299ae24996
[test] add a few downloader tests
7 years ago
dd314279fb
[test] add unit tests for extractor module functions
7 years ago
f5c6a2d7f5
[nhentai] use API to get gallery info
7 years ago
8ef790de12
update .travis.yml
...
- restrict builds to master branch and release tags
- implement 'core' and 'results' test categories
7 years ago
557cb94f81
[deviantart] use proper exponential backoff on API errors
...
... and use separate API credentials for unit tests.
7 years ago
b69cc94f0e
[util] implement bencode()
7 years ago
4d74749496
[tests] rework filters for extractor tests
...
CI incompatible tests will now only be skipped if tests are run in
a CI environment.
7 years ago
32bbd12f08
update extractor tests
7 years ago
749fbbfa6c
[mangadex] add chapter- and manga-extractor
7 years ago
5008e105ee
update archive IDs
...
... to behave in a more straightforward way when dealing with
bookmarks/favourites/etc.
specific IDs are now grouped by their owner, album-id, ... to
allow for duplicates when it would be expected.
7 years ago
2fad0b1f1b
add 'U' conversion for format strings to unquote their content
...
(#74 )
7 years ago
8f338347b6
[imagehosts] cleanup
...
removed
- chronos.to - unable to resolve hostname
- coreimg.net - same
- imgmaid.net - same
- hosturimage.com - everything returns 404
- imageontime.org - redirects to some shady site
- imgupload.yt - cloudflare error 522, host down
- img4ever.net - read timeout
7 years ago
e1e0668ca8
add option to set default replacement field value
...
Missing or undefined keywords will now be replaced with the value
set for 'keywords-default'. The default is Python's 'None', which
is equivalent to setting this option to JSON's 'null'.
7 years ago
ac3da8115e
[util] don't add text: URLs to list of downloaded URLs
7 years ago
89440382ad
[tumblr] use separate API key for unit tests
7 years ago
b50bdbf3d7
change config specifiers in input file format
...
Instead of a dictionary/object, input file options are now specified
by a 'key=value' pair starting with '-' for options only applying to
the next URL or '-G' for Global options applying to all following URLs.
See the docstring of parse_inputfile() for details.
Example option specifiers:
- filename = "{id}.{extension}"
- extractor.pixiv.user.directory = ["Pixiv Users", "{user[id]}"]
-spaces="are_optional"
-G keywords = {"global": "option"}
7 years ago
be3ea4425d
test archive-id creation and uniqueness
7 years ago
b73b8b4f50
add OAuth unittests
7 years ago
f5f2d29f56
[nijie] fix dojin extraction
...
- correctly extract artist_id
- set extension to "jpg" if it was empty and let filetype checks do
the rest
7 years ago
7a412f5c32
implement generic manga-chapter extractor
7 years ago
aa38eab2be
allow not-defined fields in format strings
...
... and replace them with "None", for now
7 years ago
619387cbb1
update extractor unittest results
7 years ago
f94e3706a8
use logging module for error messages during downloads
7 years ago
0dd48d644f
update test results
...
nothing broke, but things got updated or changed
7 years ago
1e93955170
[batoto] remove module
...
Site officially shut down on 2018.01.18
7 years ago
f10ffc0839
update extractor blacklist to also allow classes
7 years ago
35e09869d1
[mangapark] fix image URLs and use HTTPS
7 years ago
4edb25346e
[slideshare] support mobile URLs ( closes #67 )
7 years ago
b33efc99a4
[idolcomplex] add support for idol.sankakucomplex.com
7 years ago
1a70857a12
update extractor-unittest capabilities
...
- "count" can now be a string defining a comparison in the form of
'<operator> <value>', for example: '> 12' or '!= 1'. If its value
is not a string, it is assumed to be a concrete integer as before.
- "keyword" can now be a dictionary defining tests for individual keys.
These tests can either be a type, a concrete value or a regex
starting with "re:". Dictionaries can be stacked inside each other.
Optional keys can be indicated with a "?" before its name.
For example:
"keyword:" {
"image_id": int,
"gallery_id", 123,
"name": "re:pattern",
"user": {
"id": 321,
},
"?optional": None,
}
7 years ago
28cd78aae0
[kissmanga] extend chapter-string regex ( closes #58 )
7 years ago
fc7d165c97
[deviantart] add support for OAuth2 authentication
...
Some user galleries [*] require you to be either logged in or
authenticated via OAuth2 to access their deviations.
[*] e.g. https://polinaegorussia.deviantart.com/gallery/
--------------
known issue:
A deviantart 'refresh_token' can only be used once and gets updated
whenever it is used to request a new 'access_token', so storing its
initial value in a config file and reusing it again and again is not
possible.
7 years ago
0a9a07a6e1
[slideshare] improve metadata; flake8
...
- added 'views' and 'published' keywords
- fixed longer titles and descriptions
7 years ago
291369eab2
various smaller changes/additions
7 years ago
300346ecdf
[mangazuki] remove extractors
...
This site has been in "rebuild"-mode for a fairly long time and the
current extractor code isn't going to work for the new version either.
7 years ago
93482a1f88
implement 'util.advance()'
7 years ago
a718c6c6cd
implement 'util.parse_bytes()'
7 years ago
214972bc9a
[gelbooru] use manual extraction
...
... to compensate for their disabled API.
(https://gelbooru.com/index.php?page=forum&s=view&id=3875 )
This also adds an extractor for image-pools.
7 years ago
b14de6ffc2
[tumblr] small improvements
...
- don't transform inline GIF URLs
- set 'type' parameter for API calls if there is only
one post type selected
7 years ago
b8cdd42cab
[senmanga] fix extraction (again)
...
this is basically a re-revert of 2ace5c7
7 years ago
6913eeaa40
[powermanga] replace manga extractor unit test
...
My Hero Academia is gone
7 years ago
f72318e593
[seiga] support more than 200 images
...
Due to API restrictions and/or missing knowledge about and
documentation of API usage, it was only possible to retrieve the
latest 200 images of a niconico seiga user with said API.
The new approach manually visits each HTML page and gets its
information from there.
7 years ago
2457b71633
skip tests on 5xx status codes
7 years ago
305da540c3
[mangahere] fix metadata extraction
7 years ago
035ef655f1
[imagefap] update unit tests
...
old gallery/image has been deleted
7 years ago
caf26412dd
add option to set alternate location of .part files ( #29 )
...
Note: The path set for 'downloader.*.part-directory' needs to point to an
already existing directory.
7 years ago
27c026543f
re-enable download unit tests
7 years ago
b0353aa02d
rewrite download modules ( #29 )
...
- use '.part' files during file-download
- implement continuation of incomplete downloads
- check if file size matches the one reported by server
7 years ago
6af921a952
[sankaku] rewrite/improve ( fixes #44 )
...
- add wait-time between HTTP requests similar to exhentai
- add 'wait-min' and 'wait-max' options
- increase retry-count for HTTP requests to 10
- implement user authentication (non-authenticated users can only view
images up to page 25)
- implement 'skip()' functionality (only works up to page 50)
- implement image-retrieval for pages >= 51
- fix issue with multiple tags
7 years ago
75d3a1f72f
[deviantart] always download original images
...
Deviation-objects returned by the DeviantArt API don't always contain
the URL and metadata of the original image ([1]). Getting this
information requires an additional API call [2], which is indicated by
the 'is_downloadable' and 'download_filesize' metadata within a
deviation-object.
[1] https://myria-moon.deviantart.com/art/Aime-Moi-part-en-vadrouille-261986576
[2] https://www.deviantart.com/developers/http/v1/20160316/deviation_download/bed6982b88949bdb08b52cd6763fcafd
7 years ago
8e6a767109
[util] restructure formatter for better exception propagation
7 years ago
0386503c80
fix (sub)category-transfer for DownloadJob instances ( #41 )
...
... and extend "parent" parameters to TestJob- and DataJob-classes
as well.
7 years ago
41adb99e9c
[pawoo] fix extraction
...
- changed access_token
- use account-search instead of general search
7 years ago
b319f4bab3
smaller code and text changes
7 years ago
c1f0afe4c6
add custom string formatter class
7 years ago
85a2b2ae59
[khinsider] fix extraction
7 years ago
8e14714c2b
[imgspice] fix extraction
7 years ago
a85f06d2d1
[foolslide] restructure; convert suitable values to int
7 years ago
9fc1d0c901
implement and use 'util.safe_int()'
...
same as Python's 'int()', except it doesn't raise any exceptions and
accepts a default value
7 years ago
a9e7145651
[hbrowse] extract hmanga metadata & general maintenance
7 years ago
84d4450410
[fallenangels] extract manga metadata
7 years ago
f32b1a0292
[imgyt] fix extraction
7 years ago
31cd5b1c1d
[luscious] detect high-load responses
7 years ago
81877bb5f6
add '-K' as shortcut for '--list-keywords'
7 years ago
9b21d3f13c
add '--filter' command-line option
...
This allows for image filtering via Python expressions by the same
metadata that is also used to build filenames (--list-keywords).
The usually shunned eval() function is used to evaluate
filter-expressions, but it seemed quite appropriate in this case and
shouldn't introduce any new security issues, as any attacker that could do
> gallery-dl --filter "delete-everything()" ...
could as well do
> python -c "delete-everything()"
7 years ago
31731cbefe
update unittests for util.py
7 years ago
f98e3e8002
[luscious] fix tag extraction
7 years ago
65997d835b
replace popular/ranking tests with older ones
...
Metadata of several year old lists shouldn't change as much as it
would for newer ones, which makes metadata-comparisons of the output
of build_testresult_db.oy easier.
7 years ago
c0755a4d5e
[exhentai] revert login-method to its old version ( #37 )
...
Additional cookies don't seem to help and have to be manually set
anyway. The older method is more likely to succeed, so I'd rather
use this one.
7 years ago
3ee39ffd93
[exhentai] update login procedure ( #37 )
...
This new version behaves pretty much exactly like a browser would and
caches all cookies sent to it and not just "ipb_member_id" and
"ipb_pass_hash".
7 years ago
07214f4007
[booru] place subcategories into base classes
7 years ago
47bcf53ec1
implement support for additional unit test result types
...
- "pattern" matches all resulting URLs against the given regex
- "count" allows to specify the amount of returned URLs
7 years ago
c7ec103e15
[batoto] fix extraction of chapter URLs
7 years ago
f7cdfd4c25
add a simplified version of 'parse_qs'
...
This version only returns a dict of plain string to string key-value
pairs and ignores multiple values for the same query variable.
7 years ago
d70c66c516
fix "text:" downloader
7 years ago
9bf9d64ad8
update unittests for util.py
7 years ago
d74a635e41
[util] update 'default' values and improve test coverage
...
for 'code_to_language()' and 'language_to_code()'
7 years ago
abd7c559cd
[yonkouprod] remove module
...
Every manga chapter on this site has been removed.
7 years ago
852e7acd31
[twitter] ignore "Promoted Tweets"
7 years ago
6950708e52
[hentaicdn] use HTTPS
7 years ago
493bd235cf
workaround for missing 'assert_called_once' method
...
this method was introduced in Python 3.6, but calling it still
works (i.e. it doesn't cause the test to fail) on Python 3.3/3.4
7 years ago
7aa9fa796a
code cleanup and fixes
7 years ago
f08af03845
Merge branch 'cookies'
7 years ago
6db93a8b9e
add cookie tests
7 years ago
d7cb3c668a
update supportedsites.rst
7 years ago
a13eb6010f
[fallenangels] fix extraction of chapter URLs
7 years ago
1cb1d2e0a3
[mangazuki] add chapter extractor
7 years ago
3460dc8950
update gallery-dl.conf
7 years ago
af9bd17b19
[deviantart] adjust default paths
...
- user.deviantart.com/(gallery|favourites|journal)/ images go into
* <user>/
* <user>/Favourites/
* <user>/Journal/
(having an extra "Gallery" folder for a user's gallery-images seems
a bit too much if these are all you want to download, which is
probably the default use-case)
- single "deviations" (user.deviantart.com/(art|journal)/name-123) go
into their owner's directory:
* <user>/
(putting them into their own directory seems weird in practice)
7 years ago
8c16cbe7ea
fix tests
7 years ago
ce55ec6490
enable extractor tests without filters
...
$ python test_extractors.py all
7 years ago
44d98e562b
[pixiv] support pixiv.me URLs ( #23 )
7 years ago
e68af4febe
[flickr] add 'width-max' option ( #16 )
...
This option allows for simple format selection by
specifying a maximum image width.
7 years ago
9759fe8c6b
allow 'only_matching' tests
7 years ago
05ed95e5b0
[flickr] add search extractor
7 years ago
c921b4f32a
code cleanup and fixing tests
7 years ago
b6fffa9e26
[directlink] update filename format and metadata
7 years ago
48ccee2505
[gfycat] add image extractor
7 years ago
691c4dd709
support direct image links
7 years ago
398506da45
update release script
7 years ago
2974d782a3
[yomanga] remove module
...
site has been shut down
7 years ago
a90c6acc9c
code cleanup + fixes
7 years ago
4c88c0d496
rework the output format for --list-keywords
7 years ago
107d29ad8a
improve handling of text:... URLs
...
- don't require // after the colon
- open output files in text mode
7 years ago
9f1c83297f
[pinterest] allow URLs with any TLD
7 years ago
ef90a2de2f
implement the "exit" option for the "skip" config-key
7 years ago
fc9223c072
add '--abort-on-skip' option and ability to control skip behavior
...
the 'skip' config option controls skipping behavior:
true - skip download if file already exist (default)
false - download and overwrite files even if it exists
"abort" - abort extractor run if a download would be skipped
(same as '--abort-on-skip')
7 years ago
d948ba1322
[readcomics] remove module
...
- site has been unavailable for two weeks
- (#12 )
7 years ago
244ab75cad
[kissmanga] update AES key retrieval
8 years ago
13dc5d72bc
update some extractors to use https
8 years ago
f4aa452bd1
update unit test results
8 years ago
fece09d326
[fallenangels] update to new domain and site-layout
8 years ago
cf79a47b59
update unit tests
8 years ago
841fd50242
move code into util.py
8 years ago
e3212dd98f
fix some smaller stuff
...
- remove support for old windows config paths
- catch exception if cache-database can't be opened
- fix username/password settings for unit tests
- rename variable 'max_tries' to 'retries'
8 years ago
e4b3077168
improve config module
...
- speed improvements, especially in the 'interpolate' function
- 'interpolate' now prioritizes base-level values if they exist
- "username" is chosen before "extractor.<category>.username"
- -u/--username & co can now override config-file values
8 years ago
f2ef49563b
fix argument order for python33 and 34
8 years ago
0257d3e7ac
[mangamint] remove extractors - site is down
8 years ago
66eb3d3488
fix tests
8 years ago
872b8aed97
[4chan] update test
8 years ago
2fa575b273
restore exception-testing to its old form
8 years ago
40be4933b8
fix exception based tests
8 years ago
6208d9dd79
implement '--images' and '--chapters' options
...
- the former '--items' has been renamed to '--chapters'
- #6
8 years ago
e87e6fbc67
change some config keys
...
directory_fmt -> directory
filename_fmt -> filename
download-original -> original
8 years ago
96baea0959
[hentaibox] remove extractor
8 years ago
9a08f8a097
improved foolslide-based extractors
...
- this includes dokireader, fallenangels, jaiminisbox, powermanga,
sensescans, worldthree, yonkouprod, gomanga, yomanga
- added 'chapter_string', 'chapter_id', 'chapter_minor' and 'count'
keywords
- changed the 'chapter' keyword to always be just a number
- changed the default directory format
8 years ago
bad623751f
[hentaihere] transition to https
8 years ago
331b413c60
update extractor test script
8 years ago
4f123b8513
code adjustments according to pep8
8 years ago
9538a873cb
move create_test_data script
8 years ago
434ef4d5a2
[mangafox] raise proper exception if chapter is not available
8 years ago
cc0b4f2661
[yomanga] add chapter extractor
8 years ago
ad4b02508f
trying to understand travis-ci unit test failures
...
- added some debug output via logging module
- unit tests work on my machine (tm)
8 years ago
72d3ca0bf9
[deviantart] use API for user-extractor
8 years ago
989820d719
workaround for python 3.6s behaviour with BEGIN EXCLUSIVE in sqlite
8 years ago
00074a71d7
several changes to make travis build work
...
- fixed html.unescape not being available on Python3.3
- removed inconsistent test result
- added username/password pairs for authenticating extractors
8 years ago
a1fb46d984
make extractor unittest discoverable
8 years ago
2c9bc50c23
support expecting exceptions in tests
8 years ago
4e75e923cd
sanatize extractor test output
8 years ago
d1f90bb424
add method to re-create existing test-data
8 years ago
57a12f5be0
fix module-names for extractor tests
8 years ago
b0c333b799
rewrite cache module
9 years ago
871f4c8a48
initialize cache-module before running tests
9 years ago
b3aab5745b
dynamically create extractor testcases
9 years ago
2b8977cb2e
script to easily create testdata
9 years ago
565ea042c9
update HashJob to generate hashes for downloaded content
9 years ago
5a8541afa5
remove 'unstable' tests
9 years ago
0961ece989
make test-parameters optional
9 years ago
5304e5beef
testing environment for extractor results
9 years ago
eeae580781
more tests
9 years ago
ca523b9f64
add helper method to text module
9 years ago
506242740f
fix capitalization issues
9 years ago
b24bb1da7d
new module for simple language code to name mapping
9 years ago
cba4b91b14
add tests
9 years ago
bea33ae9cb
implement config.setdefault
9 years ago
2026223ed1
change argument format for config-calls
9 years ago
7ac106096f
add tests for config-module
9 years ago
2962bf36f6
add tests for text-module
9 years ago
Mike Fährmann