Mike Fährmann
179bcdd349
adjust archive-ids
7 years ago
Mike Fährmann
be3ea4425d
test archive-id creation and uniqueness
7 years ago
Mike Fährmann
3cec533c28
Merge branch 'archive'
7 years ago
Mike Fährmann
20af86b2ea
add more extractor tests
...
for mangastream, reddit and imgur
7 years ago
Mike Fährmann
b73b8b4f50
add OAuth unittests
7 years ago
Mike Fährmann
4d2fadfb6f
restore skip actions with download archive
7 years ago
Mike Fährmann
65773263fc
[util] implement OAuthSession.urlencode() ( closes #75 )
...
- Python's own urllib.parse.urlencode() has no quote_via argument in
Python 3.3 and 3.4, which is necessary to follow OAuth 1.0 quoting
rules.
7 years ago
Mike Fährmann
7e0207bcf4
[imgur] strip trailing '?1' from 'ext'
7 years ago
Mike Fährmann
cf147dfee9
[hentai2read] fix manga extraction
...
- site changed its HTML structure
7 years ago
Mike Fährmann
f5f2d29f56
[nijie] fix dojin extraction
...
- correctly extract artist_id
- set extension to "jpg" if it was empty and let filetype checks do
the rest
7 years ago
Mike Fährmann
7f7c16ae37
add option to specify additional key-value pairs
7 years ago
Mike Fährmann
d38bf2f54c
[tumblr] recognize /image/... URLs
...
xyz.tumblr.com/image/123 refers to the same images
as xyz.tumblr.com/post/123.
7 years ago
Mike Fährmann
057668e17e
extend input-file format with per-URL config and comments
...
- see docstring of parse_inputfile() for details
- TODO: unittests, recursion (currently setting for example
{"extractor": {"key": "value"}} will override the whole "extractor"
branch instead of merging {"key": "value"} into the already existing
dictionary)
7 years ago
Mike Fährmann
5b3c34aa96
use generic chapter-extractor in more modules
7 years ago
Mike Fährmann
347baf7ac5
improve util.parse_range() performance
...
It is never going to actually matter, but using partition() instead
of split() is twice as fast.
7 years ago
Mike Fährmann
7b5ba69951
[hentaihere] ensure consistent extraction results
...
sometimes there is a random space before the next <a>
7 years ago
Mike Fährmann
377b78b3c9
[hentai2read] fix manga name extraction
7 years ago
Mike Fährmann
54c36a8a34
[subapics] add chapter- and manga-extractor ( #70 )
7 years ago
Mike Fährmann
2dd3aeeeae
[komikcast] add chapter- and manga-extractor ( #70 )
7 years ago
Mike Fährmann
7a412f5c32
implement generic manga-chapter extractor
7 years ago
Mike Fährmann
aa38eab2be
allow not-defined fields in format strings
...
... and replace them with "None", for now
7 years ago
Mike Fährmann
6a07e38366
implement extractor.add() and .add_module()
...
... as a public and non-hacky way to add (external) extractors to
gallery-dl's pool and make them available for extractor.find()
7 years ago
Mike Fährmann
c0dd922c13
add '--download-archive' cmdline option
...
… as well as a config file equivalent
7 years ago
Mike Fährmann
8c3b713362
rework DownloadJob.handle_url(); include archive functionality
...
todo:
"abort" and "exit" skip modes if download is skipped because of archive
7 years ago
Mike Fährmann
34873dbd90
set 'archive_fmt' values
...
These are going to be used to create an unique id for each image.
7 years ago
Mike Fährmann
a34cebc253
[luscious] jump to first image if cover does not link to it
7 years ago
Mike Fährmann
84a52a9256
add DownloadArchive class
7 years ago
Mike Fährmann
915807dd77
log HTTP errors as warnings
7 years ago
Mike Fährmann
db7f04dd97
emit log messages on download failure
...
and when retrying with fallback URLs
7 years ago
Mike Fährmann
d951f13e37
add config option for unsupported-URL file
...
for consistency's sake
7 years ago
Mike Fährmann
619387cbb1
update extractor unittest results
7 years ago
Mike Fährmann
364e335440
smaller adjustments and improvements
...
- requests and urllib3 version on 1 line
- close input file after reading from it
- use expand_path for unsupported-urls file
- remove unnecessary logging from options.py
7 years ago
Mike Fährmann
c9a9664a65
change --write-log behaviour
...
- log files now get truncated when opening them
(mode "w" instead of "a")
- log verbosity to file depends on -q/-v
(same as logging to stderr)
7 years ago
Mike Fährmann
97f4f15ec0
add option to write logging output to a file
...
- '--write-log FILE' as cmdline argument
- 'output.logfile' as config file option
7 years ago
Mike Fährmann
f94e3706a8
use logging module for error messages during downloads
7 years ago
Mike Fährmann
db91cf871c
document message identifiers
7 years ago
Mike Fährmann
0dd48d644f
update test results
...
nothing broke, but things got updated or changed
7 years ago
Mike Fährmann
1e93955170
[batoto] remove module
...
Site officially shut down on 2018.01.18
7 years ago
Mike Fährmann
27fce6f600
fix UrlJob behavior
7 years ago
Mike Fährmann
76509a6d3c
[imgur] update test results
7 years ago
Mike Fährmann
9fccd7b783
[tumblr] provide fallback URLs ( #64 )
...
Each image now produces 3 URLs:
- amazonaws.com _raw (or _1280 for older images)
- amazonaws.com _500
- media.tumblr.com (URL returned by API)
7 years ago
Mike Fährmann
b837420291
fix minor urllist issues
7 years ago
Mike Fährmann
9d69401391
initial support for multiple URLs per image
7 years ago
Mike Fährmann
6174a5c4ef
[download] adjust filename extension on filetype mismatch
...
(closes #63 )
7 years ago
Mike Fährmann
91ed147cef
[oauth] use custom key/secret values during oauth:…
7 years ago
Mike Fährmann
421a9740a3
[tumblr] add 'tumblr:' to force Tumblr extractor ( #71 )
7 years ago
Mike Fährmann
40d35c87bc
[paheal] add tag- and post-extractors ( closes #69 )
7 years ago
Mike Fährmann
cc0c2cca57
[reddit] add extractor for reddit-hosted images ( closes #68 )
7 years ago
Mike Fährmann
f10ffc0839
update extractor blacklist to also allow classes
7 years ago
Mike Fährmann
b6797032e3
release version 1.1.2
7 years ago