- Move the download "logic" with rate limit checks into its own
method that only gets used if a rate limit should be enforced
- Fix an issue where suspending gallery-dl during a download would
basically ignore the rate limit for the remaining download when
resuming its execution.
With pyOpenSSL installed, but disabled, the SSLError exception
would be set to the one from pyOpenSSL, which could never get raised.
This commit solves this problem by catching both, the native SSLError
exception as well as the one from pyOpenSSL (if available.1)
Downloading https://pbs.twimg.com/media/EB2cGUYX4AI2Vuu.jpg:orig (NSFW)
sometimes returns a 416 status code, even though no 'Range' header was
sent and no data was downloaded prior.
This code usually means a file has already been downloaded completely
and the download method indicates success, but in this case it causes
an exception down the pipeline since no file was created.
- change 'has_extension' from a simple flag/bool to a field that
contains the original filename extension
- rename 'keywords' to 'kwdict' and some other stuff as well
- inline 'adjust_path()'
- put enumeration index before filename extension (#306)
The RE-tries option now specifies exactly that: the maximum number a
failed HTTP request is re-tried. For example a value of 2 will now
correctly stop after 3 attempts: the initial one + 2 re-tries.
The maximum wait-time now also caps at 30min and increases exponentially
for both extractor.request() and downloader.http.download().
This change introduces 'extractor.*.retries/timeout/verify' options
as a general way to set these values for all HTTP requests.
'downloader.http.retries/timeout/verify' is a way to override these
options for file downloads only and will fall back to 'extractor.*.…*
values if they haven't been explicitly set.
Also: downloader classes now take an extractor object as first argument
instead of a requests.session.
This commit changes the general behavior from
'Retry on every exception and abort on DownloadError' to
'Only retry on DownloadRetry exceptions and abort on every other one'
The previous version would have retried on several states which
would have no chance of ever succeeding (invalid URLs, etc.)
- '--no-part' command line option to disable them
- 'downloader.http.part' and 'downloader.text.part' config options
Disabling .part files restores the behaviour of the old downloader
implementation.
There was never any "good" reason for the strict separation
between extractors and downloaders. This change allows for
reduced resource usage (probably unnoticeable) and less lines
of code at the "cost" of tighter coupling.
- remove support for old windows config paths
- catch exception if cache-database can't be opened
- fix username/password settings for unit tests
- rename variable 'max_tries' to 'retries'
Mike Fährmann
