f98e3e8002
[luscious] fix tag extraction
7 years ago
b8862ff15e
add 'downloader.http.verify' option
...
(also: change the default 'timeout' from None to 30)
7 years ago
65997d835b
replace popular/ranking tests with older ones
...
Metadata of several year old lists shouldn't change as much as it
would for newer ones, which makes metadata-comparisons of the output
of build_testresult_db.oy easier.
7 years ago
be30fb2f98
add common config category for boorus and foolslide
7 years ago
c0755a4d5e
[exhentai] revert login-method to its old version ( #37 )
...
Additional cookies don't seem to help and have to be manually set
anyway. The older method is more likely to succeed, so I'd rather
use this one.
7 years ago
3ee39ffd93
[exhentai] update login procedure ( #37 )
...
This new version behaves pretty much exactly like a browser would and
caches all cookies sent to it and not just "ipb_member_id" and
"ipb_pass_hash".
7 years ago
88a386977e
[booru] add "popular" extractors for more sites
...
- konachan.com
- behoimi.org
- e621.net
7 years ago
07214f4007
[booru] place subcategories into base classes
7 years ago
60a888a1e4
[foolfuuka] add common config category
...
All FoolFuuka based 4chan-archive extractors can now be configured using
their own config keys (extractor.<category>) as well as a common shared
one (extractor.foolfuuka).
7 years ago
47bcf53ec1
implement support for additional unit test result types
...
- "pattern" matches all resulting URLs against the given regex
- "count" allows to specify the amount of returned URLs
7 years ago
2d0dfe9d56
[exhenai] init headers before login and detect sadpanda
...
- also debug-logs html after failed login
- #37
7 years ago
c7ec103e15
[batoto] fix extraction of chapter URLs
7 years ago
18e6ed1c7e
[booru] add extractors for "Popular" images
7 years ago
f7cdfd4c25
add a simplified version of 'parse_qs'
...
This version only returns a dict of plain string to string key-value
pairs and ignores multiple values for the same query variable.
7 years ago
3b21e0703c
[deviantart] allow distinction between users and groups ( #26 )
...
This is done by prepending "group-" to an extractor's subcategory
if the URL belongs to a group ("folder" becomes "group-folder" and
so on). This changes the configuration-path being used and is also
reflected in the output of '--list-keywords'.
7 years ago
e61a3a56d1
[hentai2read] fix and update keywords
...
Added the "author" keyword and changed the name of a few others to be
consistent with other manga/chapter extractors.
7 years ago
c45770331a
use 'str.partition()'
...
The (r)partition method is always faster then split() or any other
method that has been replaced in this commit.
7 years ago
017a72f448
[pixiv] improve input validation
7 years ago
dcf42c5e89
[pixiv] add extractor for ranking lists
7 years ago
4ea82ea556
[warosu] add thread extractor
7 years ago
6078ec5908
restructure the output of --help
...
Using argument groups is a definite improvement over how things looked
previously, but general group membership of individual items might be
a thing to reconsider.
7 years ago
9aa95fba8c
[deviantart] adapt download URLs to use https
...
Even though DeviantArt is "completely switching over to HTTPS"[1],
every URL contained in an API response is still using HTTP
[1] https://danlev.deviantart.com/journal/DeviantArt-Is-Switching-To-HTTPS-697996906
7 years ago
d70c66c516
fix "text:" downloader
7 years ago
f7de048980
add additional debug output
7 years ago
9bf9d64ad8
update unittests for util.py
7 years ago
02e89700fc
[foolfuuka] ensure sorted posts
7 years ago
8bcf88bff7
[flickr] fix extraction
...
This issue was only noticeable with older Python versions, as these
don't exhibit a consistent ordering of dict keys.
7 years ago
e3bfb8325a
fix circular dependency
...
- util.py imported config.py and vice versa
- Python < 3.5 doesn't like this
7 years ago
004456d5d5
properly update the config-dictionary
...
When using 2 or more config files, the values of the second would
improperly overwrite nested dictionaries of the first one.
The new method properly combines these nested dictionaries as well.
7 years ago
ae2d61e5b3
handle format string exceptions separately
7 years ago
3c9f190757
extend output of --list-keywords
7 years ago
cfa479fab5
update error message for unspecified exceptions
...
- ask user to report unexpected errors, which usually indicate
extractor failure
- handle OSErrors separately (permissions, disk full, etc)
- revert 30eef52
7 years ago
7e936e9c06
[luscious] simplify and remove dead code
7 years ago
d74a635e41
[util] update 'default' values and improve test coverage
...
for 'code_to_language()' and 'language_to_code()'
7 years ago
0245a0ba5f
fix extraction and update test results
...
- fixes for hbrowse, imgyt, imgcandy, hosturimage
- test updates for deviantart, gfycat
7 years ago
abd7c559cd
[yonkouprod] remove module
...
Every manga chapter on this site has been removed.
7 years ago
da7219ba74
[kisscomic] remove module
...
Image links on this site are dead.
7 years ago
0b576cc131
mention '--upgrade' in README.rst
7 years ago
852e7acd31
[twitter] ignore "Promoted Tweets"
7 years ago
915a0137de
improve 'extractor.request'
...
- add 'fatal' argument
- improve internal logic and flow
- raise known exception on error
- update exception hierarchy
7 years ago
dcd573806e
chg: dev: fix error ( #32 )
...
* fix: dev: error
* fix: dev: AttributeError when getting artist
* fix: dev: typo on luscious parser
7 years ago
c4713404c8
[directlink] improve URL pattern
7 years ago
d443822fdb
[luacious] get correct image URLs ( fixes #33 )
...
Instead of using thumbnail URLs and modifying them the extractor now
goes through every single image-page and gets its download URL from
there.
7 years ago
6950708e52
[hentaicdn] use HTTPS
7 years ago
4f1e6c109f
[deviantart] remove 'invalid escape sequence' warning
...
- use r"\w" or "\\w" instead of "\w"
7 years ago
c864be479e
[directlink] update URL pattern & PEP 8
...
- combine some file extensions
- don't match '.je'
- line length < 80
7 years ago
45f9d64c23
Update directlink.py with additional file exts. ( #30 )
...
Add WebP, still not that common, but it's increasing.
Add 3rd JPEG variant (https://en.wikipedia.org/wiki/JPEG#JPEG_filename_extensions )
Never seen JFIF in the wild, would probably be overkill.
Extend Ogg formats (https://en.wikipedia.org/wiki/Ogg ; https://wiki.xiph.org/MIME_Types_and_File_Extensions )
7 years ago
4357966a70
[kissmanga] make URL pattern case-insensitive (fixes 28)
7 years ago
493bd235cf
workaround for missing 'assert_called_once' method
...
this method was introduced in Python 3.6, but calling it still
works (i.e. it doesn't cause the test to fail) on Python 3.3/3.4
7 years ago
7aa9fa796a
code cleanup and fixes
7 years ago
Mike Fährmann