Mike Fährmann
1c95a0173f
[hentaifoundry] split 'artist' into 'user'+'artist'
...
and some smaller changes ...
'user' is the name of the account an image is listed at and
'artist' is now the name of the account who created the image.
For example "https://www.hentai-foundry.com/user/Tenpura/faves/pictures "
- 'user': Tenpura
- 'artist' of the only image: LewdBrush
6 years ago
Mike Fährmann
e066f35118
update extractor tests
6 years ago
Mike Fährmann
006f75b538
[hentaifoundry] rewrite + more metadata
...
- extract width, height, artist per image
- improve pattern regex
- better extensibility for other listings
6 years ago
Mike Fährmann
eeb7424783
[hentaifoundry] add support for "scraps" ( #110 )
6 years ago
Mike Fährmann
6ea9a78588
[wallhaven] add login capabilities
...
Being logged in is required to access NSFW wallpapers.
6 years ago
Mike Fährmann
c9290d8212
[wallhaven] add wallpaper and search extractors
...
todo:
- login support to gain access to NSFW wallpapers
- extractors for tag-, similar-, latest-listings
- skip() support
6 years ago
Mike Fährmann
26cbcb3a72
[flickr] improve error handling ( #109 )
6 years ago
Mike Fährmann
2be4c9ffe3
[sankaku] small code improvements
6 years ago
Mike Fährmann
529aa21dd9
move FileAdapter definition into recursive.py
6 years ago
Mike Fährmann
22ab509a70
[bobx] rename "model" to "idol" extractor
6 years ago
Mike Fährmann
99137f1bee
[sankaku] send login info as formdata
...
Previously they were erroneously send as URL parameters.
6 years ago
Mike Fährmann
fa64c38d5b
[sankaku] fix pagination for user favorites ( #106 )
6 years ago
Mike Fährmann
69fd61ea86
[bobx] add gallery and model extractors
6 years ago
Mike Fährmann
0232d80cec
[deviantart] convert 'published_time' to int ( fixes #108 )
...
The 'published_time' field (a timestamp) changed from integer to string
and caused journal creation to fail.
6 years ago
Mike Fährmann
7742cf8601
[tumblr] change 'reblogs' option ( #103 )
...
- rename "deleted" to "same-blog"
- change test for deleted original post to test if
original post owner has the same UUID (full blog name) as the one
being downloaded from
- add 'blog[uuid]' metadata to allow comparison with
'reblogged_from_uuid'
6 years ago
Mike Fährmann
d4d95d3154
[tumblr] improve rewrite rules for video URLs
6 years ago
Mike Fährmann
542a25c389
[ngomik] fix extraction
6 years ago
Mike Fährmann
a666ddd16b
[tumblr] extend 'reblogs' functionality ( #103 )
...
Setting 'reblogs' to "deleted" will check if the parent post of a
reblog has been deleted and download its media content if that is the
case, otherwise it will be skipped.
This is a rather costly operation (1 API request per reblogged post)
and should therefore be used with care.
6 years ago
Mike Fährmann
c9b8e6aefc
[reddit] fix submission-ID parsing ( #104 )
...
Uppercase characters caused a ValueError exception
6 years ago
Mike Fährmann
488abeca0b
[hentaicafe] adjust default directory format
...
A separate folder for each chapter is rather pointless if almost all
manga have only one chapter each.
6 years ago
Mike Fährmann
b4eca2633e
[tumblr] support /archive URLs
6 years ago
Mike Fährmann
aa1de70da0
[tumblr] recognize inline videos ( #102 )
6 years ago
Mike Fährmann
3ecea4cf36
[hentaicafe] add chapter and manga extractors ( #101 )
6 years ago
Mike Fährmann
0bc8ef51c8
[smugmug] Handle albums with no explicit owner ( #100 )
6 years ago
Mike Fährmann
b47af4637a
[mangadex] update URL pattern
...
Manga URLs now begin with /title/ instead of /manga/
6 years ago
Mike Fährmann
75862715ac
[behance] add user extractor
6 years ago
Mike Fährmann
a493fed376
[deviantart] fix journal creation if no 'username' is set
6 years ago
Mike Fährmann
5b8a314de7
[tumblr] replace inline URLs with higher quality ones ( #98 )
6 years ago
Mike Fährmann
2af2bb7911
[mangadex] fix relative page URLs
6 years ago
Mike Fährmann
34b556922d
update/restore tests
6 years ago
Mike Fährmann
ab2bfaeb46
[ngomik] add replacement for 'subapics'
...
http://subapics.com/ got discontinued and replaced by http://ngomik.in/ .
ngomik.in is still displaying a link to the "old site" showing a big
"Account Suspended" sign.
6 years ago
Mike Fährmann
a2eeef1f5e
[behance] replace test
...
The "UVMW Studio" account and their galleries are gone.
6 years ago
Mike Fährmann
e9dd2eff1d
[twitter] add extractor for media-tweet timelines ( #96 )
...
For example "https://twitter.com/PicturesEarth/media ".
They are different from normal timelines in that they do not contain
any (re)tweets from other users and feature all media the user ever
posted, including responses to other tweets.
6 years ago
Mike Fährmann
f45c9f2141
[gfycat] test-updates and code-adjustments
6 years ago
Mike Fährmann
9b1c39032c
[twitter] changes and improvements
...
- rename User- to TimelineExtractor
- rename 'userid' to 'user_id' to conform to the other ..._id values
- adjust archive_fmt to deal with retweets
- emulate browser behavior for API calls
6 years ago
Mike Fährmann
10365394d7
[twitter] add support for user-timelines ( closes #96 )
...
also adds a 'retweets' option to filter retweeted content
6 years ago
Mike Fährmann
d3f1eed2a6
[pinterest] improvements
...
- add stop condition for pin-related pins
- improve URL patterns
- make Pylint happy
6 years ago
Mike Fährmann
2801a0d997
[exhentai] skip "Content Warning" page when not logged in
...
(closes #97 )
6 years ago
Mike Fährmann
63fa0b2006
[pinterest] add extractors for related pins
...
Related pins can not be accessed by adding a "#related" fragment
to the end of a Pinterest URL, for example:
- https://www.pinterest.com/pin/858146903966145189/#related
- https://www.pinterest.com/g1952849/test-/#related
There are no explicit real URLs for related pins,
using an option to enable them results in "clunky" code,
and a custom "related:<URL>" scheme doesn't feel right either.
6 years ago
Mike Fährmann
1694039de0
[komikcast] update ad-filter
6 years ago
Mike Fährmann
a74591b84b
[tumblr] remove "original image" functionality
...
Accessing higher/original quality images on
https://s3.amazonaws.com/data.tumblr.com and http://data.tumblr.com
is no longer possible and any HTTP request results in 403 Forbidden.
A few images can still be accessed through https//a.tumblr.com [1][2],
but not as "_raw", just "_1280", and that might also be "fixed" in
the near future.
[1] https://a.tumblr.com/tumblr_kzjlfiTnfe1qz4rgho1_1280.jpg
[2] https://a.tumblr.com/ee589c6345f29d2d5935cecb49b0a705/tumblr_oztu02dIHp1wgha4yo1_1280.png
6 years ago
Mike Fährmann
38d4f43cc0
[komikcast] skip ads
6 years ago
Mike Fährmann
4313c95bc9
improve error message for OAuth2 authentication
6 years ago
Mike Fährmann
b55e39d1ee
[mangadex] improve extraction
...
- cache manga API results
- add artist, author and date fields to chapter metadata
- remove Manga-/ChapterExtractor inheritance
- minor code simplifications and improvements
6 years ago
Mike Fährmann
b1c4c1e13c
[mangadex] fix extraction
6 years ago
Mike Fährmann
3c90df6635
[piczel] add user, folder and image extractors
6 years ago
Mike Fährmann
2a9f3341a2
[behance] fix title extraction
6 years ago
Mike Fährmann
3fc2f269fa
[behance] filter 'fields' list
6 years ago
Mike Fährmann
b67339155f
[rule34] update test results
...
'metadata' tag type has been removed
6 years ago
Mike Fährmann
a86f2bfc80
[pinterest] update not-found redirects
6 years ago
Mike Fährmann
b040ca0718
[rule34] small unit test fixes
6 years ago
Mike Fährmann
b164231bca
[sankaku] increase default values for 'wait-min/-max'
6 years ago
Mike Fährmann
68d6033a5d
use 'retries' and 'timeout' options for regular HTTP requests
6 years ago
Mike Fährmann
f3793660ef
update tests
6 years ago
Mike Fährmann
df082e923c
[behance] add gallery extractor ( #95 )
6 years ago
Mike Fährmann
5f27cfeff6
[deviantart] remove `prefer-public` option
...
All API requests now always use a public token and only switch to
a private token for pagination results if `refresh-token` is set
and less deviations than requested were returned.
6 years ago
Mike Fährmann
bb89a1e6d7
[mangahere] use http://
...
invalid SSL cert for quite some time now
6 years ago
Mike Fährmann
212130b048
[deviantart] improve public-private token switching
...
- rename option to `prefer-public`
- now also works for galleries with less than 24 items
6 years ago
Mike Fährmann
886d662582
[deviantart] add option to minimize refresh-token usage
...
Always trying with a public token first and repeating the API request
with a private token if deviations are missing doesn't quite work for
galleries and folders with less than 25 items, so its an option and
not the default.
6 years ago
Mike Fährmann
d98e47817d
[deviantart] reduce refresh-token usage
...
Instead of using a refresh-token-based access-token for every API
request, they are now only used for paginated results.
API requests to get a user's profile and the original download URL
now always use a public access-token.
6 years ago
Mike Fährmann
84854fcad7
[myportfolio] add user and gallery extractors ( #95 )
6 years ago
Mike Fährmann
c9f70e0a19
[paheal] use HTTPS
6 years ago
Mike Fährmann
ff436692bf
["deviantart] add 'journals' option
6 years ago
Mike Fährmann
00032b828c
[deviantart] add 'wait-min' option
6 years ago
Mike Fährmann
a6fe2bb594
[whatisthisimnotgoodwithcomputers] remove extractor
6 years ago
Mike Fährmann
0ba93650e0
[8chan] replace unit test URL
...
the other thread is no longer accessible
6 years ago
Mike Fährmann
269dc2bbd5
[sankaku] add 'tags' option ( #94 )
6 years ago
Mike Fährmann
173add6935
[nijie] fix artist_id extraction
...
view_popup.php pages for older images or dojins either have the
artist_id value at a different place or not at all.
6 years ago
Mike Fährmann
6996f5c118
[mangahere] fix and improve chapter extraction
6 years ago
Mike Fährmann
1d43cbbf52
[gelbooru] tag-splitting for non-api mode
6 years ago
Mike Fährmann
2eefaa99a3
[mangapark] support .net and .com mirrors
6 years ago
Mike Fährmann
c20c0a4820
[safebooru] add pool extractor
6 years ago
Mike Fährmann
f916279ae6
[rule34] add pool extractor
6 years ago
Mike Fährmann
3dbc7c5f8d
[gelbooru] restore pool functionality
6 years ago
Mike Fährmann
a2c74bc6f0
[gelbooru] inherit from BooruExtractor class
...
Breaks pool functionality when using API calls (for now),
but reduces code clutter and enables the `tags` option.
6 years ago
Mike Fährmann
4a57509392
generalize tag-splitting option ( #92 )
...
- extend functionality to other booru sites:
- http://behoimi.org/
- https://konachan.com/
- https://e621.net/
- https://rule34.xxx/
- https://safebooru.org/
- https://yande.re/
6 years ago
Mike Fährmann
188e956c4e
[imagefap] use HTTPS + update test results
6 years ago
Mike Fährmann
87853538b4
[yandere] add option to split tags by type ( #92 )
6 years ago
Mike Fährmann
a699787d01
[deviantart] update URL patterns to new format
...
DeviantArt changed its URL format from
https://<name>.deviantart.com/...
to
https://www.deviantart.com/ <name>/...
With this change both formats will be supported.
6 years ago
Mike Fährmann
9e3415886c
[senmanga] fix/update tests
6 years ago
Mike Fährmann
b8c97d2295
use 'extractor.request()' for more HTTP requests
6 years ago
Mike Fährmann
150a6b9064
[xvideos] fix metadata extraction
6 years ago
Mike Fährmann
7a98cc9798
[smugmug] update tests
...
My test account expired and all uploaded images got deleted.
6 years ago
Mike Fährmann
91340d9d27
[pixiv] fix ugoira test
6 years ago
Mike Fährmann
eb7a1f3b98
[pixiv] rework ugoira handling
...
Frame information now gets attached to the ZIP file's keyword dict
instead of being written to a separate text file.
6 years ago
Mike Fährmann
017188d268
improve extractor.request()
...
Replace the 'fatal' parameter with 'expect', which is a list/range
of HTTP status codes >= 400 that should also be accepted.
6 years ago
Mike Fährmann
f10bd5cdbe
[4chan] unescape filenames
6 years ago
Mike Fährmann
2d1a104739
[mangadex] unescape manga names and chapter titles
...
pretty sure I previously tested if unescaping strings from the
embedded JSON object was necessary ... maybe they changed it
6 years ago
Mike Fährmann
6ac403c5d3
add postprocessor config example
6 years ago
Mike Fährmann
a47c6136cd
[simplyhentai] avoid redirects for all-pages.json ( #89 )
6 years ago
Mike Fährmann
ad14de19c6
[imgur] support "unmuted" URLs
6 years ago
Mike Fährmann
72e66f0aac
[simplyhentai] improve URL pattern
...
[ci skip]
6 years ago
Mike Fährmann
cdcc3427a0
[simplyhentai] add video extractor ( #89 )
...
All videos hosted on their own servers seem be to dead,
but myhentai.tv embeds, which are most of the videos, work fine.
6 years ago
Mike Fährmann
f9a6a19658
[simplyhentai] add image extractor ( #89 )
6 years ago
Mike Fährmann
ebf596b399
[pawoo] restore metadata fields + smaller improvements
6 years ago
Mike Fährmann
f7e7306e5a
[komikcast] update URL pattern and unescape image URLs
6 years ago
Mike Fährmann
70f3617d88
[mangafox] fix URL extraction
6 years ago
Mike Fährmann
a62bd81e9b
[pixiv] fix filter for 'type=all'
6 years ago
Mike Fährmann
55b0913412
[simplyhentai] add gallery extractor ( #89 )
6 years ago
Mike Fährmann
15cce22d82
[mangadex] fix parsing of unusual chapter strings
6 years ago
Mike Fährmann
ecdc3475b8
[pixhost] support .to TLDs
6 years ago
Mike Fährmann
f3d770d4e2
Merge branch '1.4-dev'
6 years ago
Mike Fährmann
1ff626db97
[pixiv] improve bookmark extraction
...
- combine 'favorite' and 'bookmark' extractors
- it is now one extractor class, but its subcategory still
distinguishes between your own bookmarks ('bookmark') and other
user's bookmarks ('favorite') like before
- allow filtering by bookmark tags and public/private bookmarks
- fix pagination for bookmark results
6 years ago
Mike Fährmann
0a1863fce3
[pixiv] respect more query parameters for user URLs
...
The API endpoint responsible for user illustrations does not
provide sufficient filter capabilities* to match the actual
website, so we are spinning our own filters.
Respected parameters are
'type': illust, manga, ugoira
'tag' : any image tag (this was already supported)
'p' : the page to start on
*
- API can filter for illustrations and manga, but not for ugoira.
- 'offset' is applied before filtering
- no 'tag' filter
6 years ago
Mike Fährmann
f43d446692
[mangahere] extract chapter titles
6 years ago
Mike Fährmann
b8e53b8c6b
[pixiv] move query parsing out of constructor
...
better exception handling, among other things
6 years ago
Mike Fährmann
909d105ae6
[pixiv] add extractor for illusts from followed users
6 years ago
Mike Fährmann
7f899bd5d8
Merge branch 'master' into 1.4-dev
6 years ago
Mike Fährmann
fe69d01083
[pixiv] add extractor for search results
6 years ago
Mike Fährmann
247f785af1
[pixiv] use App API
...
Transitioning to the App API breaks favorites archive IDs (there is
no longer any bookmark ID information), but the favorites API endpoint
of the public API was gone anyways ...
6 years ago
Mike Fährmann
92fc199b07
[reddit] allow arbitrary subdomains
6 years ago
Mike Fährmann
4cea886177
[imgur] allow longer album hashes
6 years ago
Mike Fährmann
e1e23165a0
[pinterest] catch JSON decode errors
6 years ago
Mike Fährmann
789608c107
[imagebam] fix extraction for certain galleries
6 years ago
Mike Fährmann
7a58151566
fix util.parse_bytes invocations
...
(should be text.parse_bytes)
6 years ago
Mike Fährmann
1c1e086d01
use common base class for OAuth1.0 based API interfaces
6 years ago
Mike Fährmann
f3483a2b7c
[smugmug] add OAuth support
6 years ago
Mike Fährmann
6a31ada9e3
re-implement OAuth1.0 code
...
OAuth support for SmugMug needs some additional features
(auth-rebuild on redirect, query parameters in URL, ...)
and fixing this in the old code wouldn't work all that well.
6 years ago
Mike Fährmann
ec158776ed
[deviantart] add extractor for popular listings
6 years ago
Mike Fährmann
0e3883303f
[pixiv] implement AppAPI wrapper
6 years ago
Mike Fährmann
e2157f594e
[mangadex] fix manga extraction ( closes #84 )
...
Chapter listings for manga now use
https://mangadex.org/manga/ <id>/_/chapters/2/
as URL instead of
https://mangadex.org/manga/ <id>/_//2/
6 years ago
Mike Fährmann
69a5e6ddb3
Merge branch 'master' into 1.4-dev
6 years ago
Mike Fährmann
3ce5296313
[smugmug] code cleanup
...
- combine User and Node extractors
- (re)move miscellaneous helper functions
- rename "Owner" to "User"
6 years ago
Mike Fährmann
42ed7667b8
[smugmug] support user- and general album URLs
6 years ago
Mike Fährmann
2ea0d1da42
[smugmug] improve API code; use data expansions
6 years ago
Mike Fährmann
16e014baaa
[smugmug] added image and album extractor
...
just some initial code that still requires a lot of work ...
TODO:
- folders
- old-style albums (which are nearly all of them ...)
- images from users
- OAuth
It could also happen that the API credentials used will become invalid
whenever my 14 day trial period ends (7 days remaining), but that
would just require users to supply their own.
6 years ago
Mike Fährmann
d96b3474e5
[puremashiro] remove module
...
site has been unreachable for a couple of weeks
and now the DNS record is gone as well
6 years ago
Mike Fährmann
b44a296404
[gomanga] remove module
...
site has been unreachable for a couple of weeks
and the cloudflare status page shows host errors
6 years ago
Mike Fährmann
95392554ee
use text.urljoin()
6 years ago
Mike Fährmann
2395d870dd
[pinterest] unquote board and user names, better errors
6 years ago
Mike Fährmann
8b79eaafea
[tumblr] log actual time of rate limit resets
...
... instead of the amount of seconds until a reset
6 years ago
Mike Fährmann
0f1e07f627
[pinterest] scrap OAuth implementation; code improvements
...
OAuth authentication isn't needed anymore and other tools
like Postman are better suited for this job anyway.
6 years ago
Mike Fährmann
55d4d23860
[pinterest] use Pinterest's "Web" API ( #83 )
...
no access tokens, no user credentials of any kind ...
7 years ago
Mike Fährmann
2721417dd8
Merge branch 'master' into 1.4-dev
7 years ago
Mike Fährmann
c6d5154fc3
fix flake8 errors, ignore W504
...
pycodestyle 2.4.0 enforces some new style guidelines
7 years ago
Mike Fährmann
2d17a9e07f
improve extractor.request()
...
- better retry behavior
- exponential back-off
- removed 'allow_empty' argument
7 years ago
Mike Fährmann
80521ae1f6
[deviantart] improve API error handling
...
The previous implementation would retry requests with 4xx status codes
in an infinite loop, which is especially a problem when querying
non-existent users or groups. These are now properly handled with a
NotFoundError exception.
7 years ago
Mike Fährmann
e54b43be08
[mangadex] add title info for chapter extractors
7 years ago
Mike Fährmann
f471161920
Merge branch 'master' into 1.4-dev
7 years ago
Mike Fährmann
eb37fbf0e8
[hentaifoundry] improve extractor
...
- use common base class
- better pagination
- respect '.../page/<num>'
- implement skip() / --range support
- get YII_CSRF_TOKEN from cookies
7 years ago
Mike Fährmann
80bead739d
[oauth] require custom client-* values for pinterest
7 years ago
Mike Fährmann
cc36f88586
rename safe_int to parse_int; move parse_* to text module
7 years ago
Mike Fährmann
10cc59f3b5
fix extractor names
7 years ago
Mike Fährmann
b1325d4d2c
fix extractor docstrings
7 years ago
Mike Fährmann
df7e18399e
[luscious] fix image order
7 years ago
Mike Fährmann
d10579edb5
[pinterest] improve PinterestAPI code; remove OAuth mentions
...
on another note: access_tokens have been set to only allow for
10 requests per hour (from 200 yesterday)
7 years ago
Mike Fährmann
4bd182c107
[pinterest] implement `oauth:pinterest` ( #83 )
...
Pinterest access tokens are rate limited at 200 requests per
hour (or maybe per 2 or 3 hours?) so having just one access token
for all users isn't going to work in the long run.
7 years ago
Mike Fährmann
9651f3fce0
[pinterest] improve error messages ( #83 )
7 years ago
Mike Fährmann
dbe250f7e5
[pinterest] update access_token ( #83 )
7 years ago
Mike Fährmann
dd49127408
[spectrumnexus] remove module
...
Site stopped hosting manga scans (http://view.thespectrum.net/ )
7 years ago