dad2875a3e
fix calculating retry sleep times ( fixes #1990 )
3 years ago
e69ee41f25
implement 'page-reverse' option ( #1854 )
3 years ago
c9e6693530
allow specifying a minimum/maximum for 'sleep-*' options ( #1835 )
...
for example '"sleep-request": [5.0, 10.0]' to wait between 5 and 10
seconds between each HTTP request
3 years ago
2ff2974353
[common] update default argument handling in Extractor.request()
...
more lines of code, but slightly less execution time
3 years ago
d79bcb6236
allow extractors to register a 'finalize()' method
3 years ago
bb6a130942
automatically set required DDoS-GUARD cookies ( #1779 )
...
for kemono.party and seiso.party
3 years ago
bd08ee2859
remove most 'yield Message.Version' statements
...
only leave them in oauth.py as noop results
3 years ago
9cb5ea5eda
update default User-Agent headers
3 years ago
0179581340
add 'T' format string conversion ( #1646 )
...
to convert 'date'/datetime to timestamp
3 years ago
94faf8c85a
add type check before applying 'browser' option ( fixes #1358 )
4 years ago
6cfc9613fe
update some code in Extractor constructor
...
- combine '_init_headers' and '_emulate_browser' functionality
into new '_init_session'
- add 'headers' and 'ciphers' options
4 years ago
29ea54dc41
[patreon] use '"browser": "firefox"' by default ( #1117 )
4 years ago
cf5fa75d4c
add 'browser' option ( #1117 )
...
- change default user agent to Firefox ESR 78 on Windows 10
- remove 'ciphers' option
4 years ago
e1a12761d7
strip '/' from instance root URLs
4 years ago
d656892670
remove cloudflare.py
...
The old IUAM challenge doesn't get used anymore, i.e. code to bypass it
is pointless, and the 'is_...()' checks are simple enough to directly
include them in 'extractor.request()'.
4 years ago
88fae99811
remove 'generate_extractors()'
4 years ago
745a114c61
[common] implement BaseExtractor class
...
Should be used when the same extractor logic applies to different
instances/domains of several sites, e.g. FoolFuuka, Shopify, etc.
This will replace the functionality of 'generate_extractors()' in
a more efficient way, by condensing everything into 1 class and not
dynamically generating an extractor class for each instance.
4 years ago
0d406c8daf
[common] restrict values used in 'generate_extractors()'
4 years ago
8ca7f54750
rename '_request_…' variables
...
- remove '_' at the beginning
- _request_last -> request_timestamp
4 years ago
c57a918f4a
[e621] implement delay via '_request_interval_min'
4 years ago
1e3dd7330e
merge SharedConfigMixin functionality into Extractor
4 years ago
198c33ec36
also collect post processors from 'basecategory' entries
...
(fixes #1084 )
4 years ago
1e313d5b84
implement 'sleep-request' option
4 years ago
055c32e0f7
precompute extractor config paths
4 years ago
231dd4c800
accumulate postprocessor objects ( #994 )
...
Instead of one 'postprocessors' setting overwriting all others lower
in the hierarchy, all postprocessors along the config path will now
get collected into one big list.
For example '--mtime-from-date' will therefore no longer cause
other postprocessor settings in a config file to get ignored.
4 years ago
f6fd449b59
reduce wait time growth rate from exponential to linear
...
Waiting for 2**N seconds after each error grows too fast.
Simply waiting N seconds seems far more reasonable.
4 years ago
2c9766b29f
fix UnboundLocalError in Extractor.request()
...
introduced in d6a271d
4 years ago
d6a271d2c7
add 'response' objects to 'HttpError's
4 years ago
53cc498d9c
improve config lookup when there are multiple possible locations
...
This specifically applies to all Mastodon extractors and all
extractors with a 'basecategory', i.e. 'booru', 'foolslide', etc.
Values inside those general config locations wouldn't be recognized
when a value with the same was set on the 'extractor' level.
For example 'extractor.mastodon.directory' should be used over
'extractor.directory' when both are set, but this was impossible
with the previous implementation.
(fixes #843 )
4 years ago
1ae1df0d27
update '--write-pages' ( #737 )
...
- fix infinite recursion for responses with multiple entries in
'history'
- hide values of Set-Cookie headers
- only write the response content by default
(use '-o write-pages=all' to also include HTTP headers)
4 years ago
15c3d29062
move dump_response() into a separate function ( #737 )
4 years ago
a363da4b43
include redirects and headers in --write-pages dumps ( #737 )
4 years ago
3201fe3521
add global SENTINEL object
4 years ago
f8f95e68a7
improve '--write-pages' ( #737 )
...
- move code into its own function
- add enumeration index to filenames
- dump responses regardless of status code
4 years ago
4cc761c730
Implement --write-pages option ( #736 )
...
* Implement --write-pages option
* Fix long lines
* Fix file mode to binary
* Fix pattern for Windows compatibility
4 years ago
5d7ca76885
retry Cloudflare challenges
4 years ago
d02f7c1118
improve Extractor.wait()
...
- allow 'until' to be a datetime object
- do "time calculations" with UTC timestamps
- set a default 'reason'
5 years ago
2a4f227e08
warn about expired cookies
5 years ago
56f1c96168
implement 'parent-directory' option ( #551 )
5 years ago
2a9be48511
improve util.load/save_cookiestxt() and add tests
...
- take a file object as argument instead of an filename
- accept whitespace before comments (" # comment")
- map expiration "0" to None and not the number 0
5 years ago
c1a6862863
implement functions to load/save cookies.txt files ( closes #586 )
...
The methods of the standard libraries' MozillaCookieJar have
several shortcomings (#HttpOnly_ cookies, 0 expiration timestamps, etc.)
and require construction of an ultimately pointless CookieJar object.
5 years ago
bd5ce9855c
allow GalleryExtractors to set URL-independent extensions
5 years ago
3811fd8a25
fix time formatting for Python 3.4 and 3.5
...
'datetime.time.isoformat()' only has an optional 'timespec' argument
since Python 3.6.
5 years ago
569747a78d
implement extractor.wait()
5 years ago
ce54b8c04c
let extractors opt-out of cookie option usage
...
useful to avoid sending unnecessary cookies when all authentication
is done through OAuth tokens
5 years ago
d3e44e899d
raise NotFoundErrors for 404 responses in GalleryExtractors
5 years ago
a4dd8b3dab
improve _check_cookies()
...
Only loop over all cookies once instead of calling
cookiejar._find() for each cookie name.
5 years ago
15f9bb3d14
add option to disable pyOpenSSL usage ( #508 )
...
(pyOpenSSL is now disabled by default)
5 years ago
e17907ee2a
change default value of 'cookies-update' to 'true'
5 years ago
e2710702d4
fix Cloudflare bypss
5 years ago
Mike Fährmann