6110e3f940
[common] fix Logger names of BaseCategory extractors
...
update of d11ec0090c178846
2 months ago
eb3ef13d28
include 'zstd' in Accept-Encoding header when supported
...
… and slightly update optional dependency list
2 months ago
8aca0e6970
update default User-Agent header to Firefox 128 ESR
2 months ago
11421cf940
[skeb] fix '429 Too Many Requests' errors ( #5766 )
...
Introduce '_handle_429' method to make it easier for Extractors to react
to 429 errors regardless of 'sleep-429' settings.
3 months ago
60b4541199
improve a1bb3279, fix oauth:pixiv ( #5757 )
...
Check 'input' option only when required.
This also fixes an exception in oauth:pixiv caused by using the same
'_input' name as a method defined there.
3 months ago
a1bb32792b
do not try to read from stdin when it is non-interactive ( #5733 )
...
add '--no-input' command-line option and 'input' config file option
to allow users to manually configure this
3 months ago
5d3d03a1f1
fix 6cfbc107
...
the former condition would return True for 2.31.*
6cfbc1071f (commitcomment-142642913)
4 months ago
6cfbc1071f
workaround for requests 2.32.3 ( #5665 )
...
manually call 'load_default_certs()' for SSLContexts
in custom HTTPAdapter instances
4 months ago
28039229fe
[common] use 'create_urllib3_context' for creating SSLContexts
...
enables dumping TLS session keys by setting SSLKEYLOGFILE (#5215 )
as well as other potentially useful settings.
4 months ago
33006fe126
[common] disable 'check_hostname' for non-urllib3 SSLContexts
...
e.g. when 'browser' is set to a non-empty value and gallery-dl creates
its own SSLContext instance instead of using requests' and urllib3's
defaults.
urllib3 disables this option for its default contexts,
since it does this check on its own.
Fixes "ValueError: Cannot set verify_mode to CERT_NONE when
check_hostname is enabled" when using --no-check-certificate.
(#3614 , #4891 , #5576 )
4 months ago
d11ec00908
[common] fix _cfgpath for BaseExtractor objects
...
After the changes in 0c17884673
5 months ago
a7d8cbab0e
[common] show full URL in Extractor.request() error messages
5 months ago
a5071c9ca0
[common] fix NameError in Extractor.request()
...
… when accessing 'code' after an requests exception was raised.
Caused by the changes in 566472f080
5 months ago
566472f080
[common] add 'sleep-429' option ( #5160 )
5 months ago
923c6f3214
[common] simplify 'status_code' check in Extractor.request()
5 months ago
68f4208251
[common] update Extractor.wait() message format
5 months ago
b38a917355
[common] add Extractor.input() method
5 months ago
0d72789aa3
merge #5461 : [cookies] use tempfile when saving cookies.txt files
5 months ago
63ac06643f
compute tempfile path only once
5 months ago
0c17884673
store 'match' and 'groups' in Extractor objects
6 months ago
106dfdb4c3
cleanup sleep-request retry delay code
...
more lines but easier to read I'd say
6 months ago
89066844f4
add 'config_instance' method
...
to allow for a more streamlined access to BaseExtractor instance options
8 months ago
f36dafad06
improve 'include' handling ( #4982 )
...
- remove spaces when given as string
- warn about invalid vales
9 months ago
7dd79eee93
save cookies to tempfile, then rename
...
avoids wiping the cookies file if the disk is full
9 months ago
6a4218aa23
handle 'json' parameter in Extractor.request() manually
...
Mainly to allow passing custom classes like util.LazyPrompt,
but also to simplify and streamline how requests handles it.
10 months ago
9dd5cb8c8a
interactively prompt for passwords on login when none is provided
10 months ago
34a387b6e2
support 'metadata-*' names for '*-metadata' options
...
For example, instead of 'url-metadata' it is now also possible to use
'metadata-url' as option name.
- metadata-url
- metadata-path
- metadata-http
- metadata-version
- metadata-parent
10 months ago
61d6558322
[exhentai] try to avoid 'DH_KEY_TOO_SMALL' errors ( #1021 , #4593 )
11 months ago
eb230e4b77
[nsfwalbum] disable Referer headers by default ( #4598 )
12 months ago
3ecb512722
send Referer headers by default
1 year ago
4cdab8074e
update/fix --list-extractors
1 year ago
a453335a9f
remove test results in extractor modules
...
and add generic example URLs
1 year ago
ceb59e176f
fix default Firefox user agent string
...
note to self: do not trust some random third-party website
1 year ago
a4f7f7da17
add '_dump()' convenience method to Extractor
1 year ago
48ef062867
fix issues with 'Extractor.finalize()'
...
- prevent crash in InstagramUserExtractor (#4359 )
- call it at the end of every DownloadJob
- add it to tests
1 year ago
ed21908fda
initial support for child extractor options
...
Using "parent-category>child-category" as extractor category in a config
file allows to set options for a child extractor when it was spawned by
that parent.
For example "reddit>gfycat" to set gfycat options for when it was found
in a reddit post.
{
"extractor": {
"gfycat": {
"filename": "regular filename"
},
"reddit>gfycat": {
"filename": "reddit-specific filename"
}
}
}
Note: This does currently not work for most imgur links due to how its
extractor hierarchy is structured.
1 year ago
a383eca7f6
decouple extractor initialization
...
Introduce an 'initialize()' function that does the actual init
(session, cookies, config options) and can called separately from
the constructor __init__().
This allows, for example, to adjust config access inside a Job
before most of it already happened when calling 'extractor.find()'.
1 year ago
d97b8c2fba
consistent cookie-related names
...
- rename every cookie variable or method to 'cookies_*'
- simplify '.session.cookies' to just '.cookies'
- more consistent 'login()' structure
1 year ago
ceebacc9e1
remove 'pyopenssl' option
1 year ago
5b59a0d143
update default User-Agent header to Firefox 115 ESR
1 year ago
856f6c10cd
allow for GalleryExtractors to skip loading gallery_url
1 year ago
3ca5dac8b6
extend 'cookies-update' functionality
...
Allow writing cookies to a different file than a given cookies.txt,
making it possible to export cookies imported with --cookies-from-browser
To convert browser cookies to cookies.txt format:
gallery-dl --cookies-fr chromium \
-o cookies-update=cookies.txt \
--no-download \
http://example.org/file.jpg
1 year ago
bc6d65d203
implement 'Extractor.config_deprecated()'
...
a version of 'Extractor.config()'
that logs a warning when using a deprecated option name
1 year ago
076380e079
remove '*' indicating keyword-only arguments
...
they are kind of unnecessary and
cause a non-insignificant function call overhead (~10%)
1 year ago
9abcb2b6e5
update headers and ciphers for '"browser": "chrome"'
2 years ago
00b94946b3
[instagram] show -o cursor=… after every error ( #3440 )
2 years ago
80a2ff2d38
support setting 'write-pages' to "ALL"
...
to show authentication header, cookies, etc
2 years ago
c881548a27
add 'extractor.retry-codes' option ( #3313 )
...
do not retry 429 and 430 by default
2 years ago
9695c4e88d
emit debug logging message when loading cookies from file
...
attempt nr. 2
no idea how I managed to remove 6514828da918ce29
2 years ago
a918ce29b5
run tests on ubuntu-20.04
...
and remove Python 3.4, since that's no longer available
on this test runner
2 years ago
Mike Fährmann
