Mike Fährmann
179d112083
[downloader] overhaul http and text modules
...
Get rid of the modular structure and simplify/specialize those modules.
5 years ago
Mike Fährmann
b17a5d6f3b
give downloader classes proper names
6 years ago
Mike Fährmann
4a348990f4
adjust value resolution for retries/timeout/verify options
...
This change introduces 'extractor.*.retries/timeout/verify' options
as a general way to set these values for all HTTP requests.
'downloader.http.retries/timeout/verify' is a way to override these
options for file downloads only and will fall back to 'extractor.*.…*
values if they haven't been explicitly set.
Also: downloader classes now take an extractor object as first argument
instead of a requests.session.
6 years ago
Mike Fährmann
707b15b586
create missing directories for 'part-directory'
...
also some code improvements regarding downloader config values
7 years ago
Mike Fährmann
caf26412dd
add option to set alternate location of .part files ( #29 )
...
Note: The path set for 'downloader.*.part-directory' needs to point to an
already existing directory.
7 years ago
Mike Fährmann
9a41002b77
fix partial downloads for 'text:' URLs
...
Using a filesize in bytes as offset into a Python string is not
a good idea if said file contains non-ASCII characters.
7 years ago
Mike Fährmann
963670d73b
add options to control usage of .part files ( #29 )
...
- '--no-part' command line option to disable them
- 'downloader.http.part' and 'downloader.text.part' config options
Disabling .part files restores the behaviour of the old downloader
implementation.
7 years ago
Mike Fährmann
b0353aa02d
rewrite download modules ( #29 )
...
- use '.part' files during file-download
- implement continuation of incomplete downloads
- check if file size matches the one reported by server
7 years ago
Mike Fährmann
d70c66c516
fix "text:" downloader
7 years ago
Mike Fährmann
107d29ad8a
improve handling of text:... URLs
...
- don't require // after the colon
- open output files in text mode
7 years ago
Mike Fährmann
4f123b8513
code adjustments according to pep8
8 years ago
Mike Fährmann
3c1daef839
don't delete downloaded files in certain edge cases
8 years ago
Mike Fährmann
29692c5784
get extension from Content-Type header if not provided
8 years ago
Mike Fährmann
4b377ccc09
use output-module during downloads
9 years ago
Mike Fährmann
28fa7c53b4
docstrings and other small fixes for downloaders
10 years ago
Mike Fährmann
5545624da1
use seperate session in http downloader
10 years ago
Mike Fährmann
deef91eddc
initial commit
10 years ago