cd241bea0a
[downloader:http] add MIME type and signature for .m4v files ( #5505 )
5 months ago
bffadf35b7
[hotleak] download files with 404 status code ( #5395 )
5 months ago
a8027745e3
[downloader:http] add MIME type and signature for .mov files ( #5287 )
7 months ago
0dacb2b24c
[downloader:http] remove 'pyopenssl' import ( #5156 )
8 months ago
ea78f67860
[downloader:http] skip files not passing filesize-min/-max ( #4821 )
...
instead of failing the download
10 months ago
5448268d5c
[downloader:http] close connection when file already exists ( #3748 )
1 year ago
c182094ebf
merge #3748 : [downloader:http] add 'consume-content' option
1 year ago
6f4a843fba
[downloader:http] release connection before logging messages
...
This allows connections to be properly released when using 'actions'
feature.
1 year ago
2edcdee32f
[downloader:http] add MIME type and signature for .heic files
...
(#3915 )
https://github.com/strukturag/libheif/issues/83
1 year ago
775d2ac999
[downloader:http] improve error logging when releasing connection
1 year ago
1a977f0f62
[downloader:http] handle exceptions in 'validate'
...
This isn't strictly necessary for 'exhentai.py', but it improves
efficiency when the adapter is reused
2 years ago
fcaeaf539c
[downloader:http] handle exceptions while consuming content
2 years ago
67ec91cdbd
[downloader:http] change '_http_retry' to accept a Python function
...
and rename '_http_retry_codes' to '_http_retry'
(#3569 )
2 years ago
df77271438
[downloader:http] add 'consume-content' option
...
* fix connection not being released when the response is neither
successful nor retried
* add the ability to consume the HTTP response body instead of closing
the connection
reference:
https://docs.python-requests.org/en/latest/user/advanced/#body-content-workflow
2 years ago
d16873941c
[downloader:http] use 'time.monotonic()'
2 years ago
ec9ff7640d
merge #3535 : [downloader:http] add signature checks for .blend, .obj, and .clip files
2 years ago
b6706b373a
[downloader:http] add signature checks for some formats
...
also add the MIME type for .obj files
2 years ago
c881548a27
add 'extractor.retry-codes' option ( #3313 )
...
do not retry 429 and 430 by default
2 years ago
c0d7d2be35
[downloader:http] add 'validate' option
2 years ago
80102fa367
[downloader:http] add 'retry-codes' option ( #3313 )
2 years ago
b4253f69c9
[downloader:http] fix ZeroDivisionError ( #3328 )
...
ensure 'time_elapsed' only get used as divisor
when it is greater than zero
2 years ago
f87cfa5f66
[downloader:http] add signature check for .mp4 files
2 years ago
a4ff20cf16
[downloader:http] fix issues from inaccurate 'time.sleep()'
...
(#3143 )
Reverts part of c59b98c8
2 years ago
550f90ab56
delay enabling .part files when 'http-metadata' is set
...
otherwise 'build_path' gets called before all metadata is collected
2 years ago
8124c16a50
split 'build_path' from 'set_filename' and 'set_extension'
...
Do not automatically build a new path
when setting file metadata or updating its extension.
2 years ago
39d9c362e4
include 'http-metadata' in '-K' output
2 years ago
870e6a48a0
implement 'http-metadata' option
...
or at least attempt to.
2 years ago
bca9f965e5
[downloader:http] add 'chunk-size' option ( #3143 )
...
and double the previous default from 16384 (2**14) to 32768 (2**15)
2 years ago
0059e2bfe7
[downloader:http] add MIME type and signature for .avif files
2 years ago
f687e64513
[downloader:http] refactor file signature checks
...
use functions/lambdas instead of startswith()
2 years ago
c0c1277c5f
[downloader:http] support sending POST data ( #2433 )
...
by setting the '_http_data' metadata field for a file
needed in addition to be3492776b
3 years ago
be3492776b
[downloader:http] support using a different method than GET ( #2433 )
...
by setting the '_http_method' metadata field for a file
3 years ago
47cf05c4ab
refactor proxy handling code ( #2357 )
...
- allow gallery-dl proxy settings to overwrite environment proxies
- allow specifying different proxies for data extraction and download
- add 'downloader.proxy' option
- '-o extractor.proxy=–PROXY_URL -o downloader.proxy=null'
now has the same effect as youtube-dl's '--geo-verification-proxy'
3 years ago
ebd3d5c1cc
[bunkr] fix .mp4 downloads ( closes #2239 )
3 years ago
d0761454b1
implement a download progress indicator ( #1519 )
3 years ago
b5b1cf22b7
[downloader:http] reorder HTTP header sources
...
so that any header can be overwritten by a user, except Range
3 years ago
221015e586
[downloader:http] disable filename extension changes for ugoira
...
(#1507 )
3 years ago
cf5fa75d4c
add 'browser' option ( #1117 )
...
- change default user agent to Firefox ESR 78 on Windows 10
- remove 'ciphers' option
4 years ago
560277394e
[downloader:http] add 'headers' option ( #1322 )
4 years ago
a228bb3a5f
[downloader:http] support callbacks to validate responses
4 years ago
0594821fcd
[downloader:http] add MIME type and signature for .ico files
...
(closes #1211 )
4 years ago
476d563ec2
[downloader:http] add MIME type and signature for .swf files
4 years ago
fe0265c7a5
[downloader.http] small improvements to file signature list
...
- specify multiple entries for gif, mp3, zip
- add entries for pdf
4 years ago
1a4b61f7eb
[downloader:http] fix issues with chunked transfer encoding
...
(fixes #1144 )
4 years ago
536c088462
[downloader:http] improve 'adjust-extensions' ( #776 )
...
Check file headers against a list of file signatures before
downloading the whole file and writing it to disk.
The file signature check needs some improvements (*),
but it produces usable results for the most part.
(*)
- 'webp', 'wav', and others start with 'RFFI'
- 'svg' uses the same "signature" as all XML documents
- 'webm' has the same signature as 'mkv' files
- only 'mp3' files in an ID3v2 container get recognized
4 years ago
f6fd449b59
reduce wait time growth rate from exponential to linear
...
Waiting for 2**N seconds after each error grows too fast.
Simply waiting N seconds seems far more reasonable.
4 years ago
ac3036ef56
add 'filesize-min' and 'filesize-max' options ( closes #780 )
4 years ago
34929f673f
readd 'session' to base downloader class ( fixes #768 )
4 years ago
ece73b5b2a
make 'path' and 'keywords' available in logging messages
...
Wrap all loggers used by job, extractor, downloader, and postprocessor
objects into a (custom) LoggerAdapter that provides access to the
underlying job, extractor, pathfmt, and kwdict objects and their
properties.
__init__() signatures for all downloader and postprocessor classes have
been changed to take the current Job object as their first argument,
instead of the current extractor or pathfmt.
(#574 , #575 )
4 years ago
19a7afdd9b
[downloader:http] add MIME types for .psd files ( closes #714 )
4 years ago
Mike Fährmann