966a9ca3a0
update test results
6 years ago
fd8ed35591
[turboimagehost] fix extraction
6 years ago
c69150f715
[imagefap] fix extraction
...
also adds tags to gallery-metadata and converts suitable values to int
6 years ago
d1f3d32eec
[fallenangels] unescape chapter titles
6 years ago
c9861ca812
adjust message for status_code based exceptions
...
from: 5xx HTTP Error: Reason
to : 5xx: Reason
The "HTTP Error" part was in there to emulate Request's error messages
from response.raise_for_status(), but it reads a lot better without.
6 years ago
eb1c24b286
[imagebam] detect nonexistent galleries
6 years ago
e1d306cc48
update unit test results
6 years ago
8faf03ed84
[pixiv] use refresh_token based authentication
...
The first login will still use username and password, but everything
afterwards will use the refresh_token obtained from that.
This will prevent pixiv from sending a "New login to pixiv" email every
time a new access_token is requested.
6 years ago
2221cf97ff
implement 'update()' for caches
6 years ago
d8492df51b
[deviantart] extend functionality of 'original' option
6 years ago
c00dce2adc
[behance] enable 'categorytransfer'
6 years ago
1532d1b690
fix 'range' tests and update a few test results
6 years ago
4a348990f4
adjust value resolution for retries/timeout/verify options
...
This change introduces 'extractor.*.retries/timeout/verify' options
as a general way to set these values for all HTTP requests.
'downloader.http.retries/timeout/verify' is a way to override these
options for file downloads only and will fall back to 'extractor.*.…*
values if they haven't been explicitly set.
Also: downloader classes now take an extractor object as first argument
instead of a requests.session.
6 years ago
f647f5d9c3
use 'verify' option for regular HTTP requests
6 years ago
ca6ac4db6a
fix 'content' tests
6 years ago
188876d814
implement youtube-dl downloader module
...
URLs starting with 'ytdl:' will now be handled by youtube-dl.
There is probably a lot to fix and improve, but the basic use case
works.
TODO:
- format selection and ytdl options in general
- better filename/path handling
- ytdl support for "unsupported URLs"
- ...
6 years ago
f4df6c2396
[pixiv] remove 'type' and 'page' query parameter handling
...
The "new and improved" /member_illust.php and /bookmark.php listings
don't quite work with how things were.
6 years ago
d70db2d555
Revert "[komikcast] fix extraction"
...
This reverts commit 5507f5ce2e
6 years ago
d69db60e2a
update unit test results
6 years ago
f8b3b00249
[twitter] add experimental 'videos' option ( #99 )
...
Enabling this option will detect videos in tweets and output them as
"unsupported" URLs, so that these can then be downloaded with youtube-dl
There are a lot of improvements to be made to the current
implementation, but it works and does what it is supposed to, even if
inefficient as can be ...
6 years ago
5507f5ce2e
[komikcast] fix extraction
6 years ago
8080071174
[flickr] improve album metadata ( closes #109 )
6 years ago
537448ba6e
[yuki] fix extraction of older threads ( closes #112 )
6 years ago
1acaed73e0
[warosu] improve extraction and metadata
...
- convert values to int
- unquote original filenames
- don't parse posts twice
6 years ago
2cf3f53839
[yuki] add thread extractor ( closes #111 )
6 years ago
c402cc4047
[hentaifoundry] add 'popular' and 'recent' extractors
...
for "Popular Pictures" and "Recent Pictures" listings
6 years ago
a5fc311dfa
[hentaifoundry] add 'favorite' extractor
6 years ago
1c95a0173f
[hentaifoundry] split 'artist' into 'user'+'artist'
...
and some smaller changes ...
'user' is the name of the account an image is listed at and
'artist' is now the name of the account who created the image.
For example "https://www.hentai-foundry.com/user/Tenpura/faves/pictures "
- 'user': Tenpura
- 'artist' of the only image: LewdBrush
6 years ago
e066f35118
update extractor tests
6 years ago
006f75b538
[hentaifoundry] rewrite + more metadata
...
- extract width, height, artist per image
- improve pattern regex
- better extensibility for other listings
6 years ago
eeb7424783
[hentaifoundry] add support for "scraps" ( #110 )
6 years ago
6ea9a78588
[wallhaven] add login capabilities
...
Being logged in is required to access NSFW wallpapers.
6 years ago
c9290d8212
[wallhaven] add wallpaper and search extractors
...
todo:
- login support to gain access to NSFW wallpapers
- extractors for tag-, similar-, latest-listings
- skip() support
6 years ago
26cbcb3a72
[flickr] improve error handling ( #109 )
6 years ago
2be4c9ffe3
[sankaku] small code improvements
6 years ago
529aa21dd9
move FileAdapter definition into recursive.py
6 years ago
22ab509a70
[bobx] rename "model" to "idol" extractor
6 years ago
99137f1bee
[sankaku] send login info as formdata
...
Previously they were erroneously send as URL parameters.
6 years ago
fa64c38d5b
[sankaku] fix pagination for user favorites ( #106 )
6 years ago
69fd61ea86
[bobx] add gallery and model extractors
6 years ago
0232d80cec
[deviantart] convert 'published_time' to int ( fixes #108 )
...
The 'published_time' field (a timestamp) changed from integer to string
and caused journal creation to fail.
6 years ago
7742cf8601
[tumblr] change 'reblogs' option ( #103 )
...
- rename "deleted" to "same-blog"
- change test for deleted original post to test if
original post owner has the same UUID (full blog name) as the one
being downloaded from
- add 'blog[uuid]' metadata to allow comparison with
'reblogged_from_uuid'
6 years ago
d4d95d3154
[tumblr] improve rewrite rules for video URLs
6 years ago
542a25c389
[ngomik] fix extraction
6 years ago
a666ddd16b
[tumblr] extend 'reblogs' functionality ( #103 )
...
Setting 'reblogs' to "deleted" will check if the parent post of a
reblog has been deleted and download its media content if that is the
case, otherwise it will be skipped.
This is a rather costly operation (1 API request per reblogged post)
and should therefore be used with care.
6 years ago
c9b8e6aefc
[reddit] fix submission-ID parsing ( #104 )
...
Uppercase characters caused a ValueError exception
6 years ago
488abeca0b
[hentaicafe] adjust default directory format
...
A separate folder for each chapter is rather pointless if almost all
manga have only one chapter each.
6 years ago
b4eca2633e
[tumblr] support /archive URLs
6 years ago
aa1de70da0
[tumblr] recognize inline videos ( #102 )
6 years ago
3ecea4cf36
[hentaicafe] add chapter and manga extractors ( #101 )
6 years ago
0bc8ef51c8
[smugmug] Handle albums with no explicit owner ( #100 )
6 years ago
b47af4637a
[mangadex] update URL pattern
...
Manga URLs now begin with /title/ instead of /manga/
6 years ago
75862715ac
[behance] add user extractor
6 years ago
a493fed376
[deviantart] fix journal creation if no 'username' is set
6 years ago
5b8a314de7
[tumblr] replace inline URLs with higher quality ones ( #98 )
6 years ago
2af2bb7911
[mangadex] fix relative page URLs
6 years ago
34b556922d
update/restore tests
6 years ago
ab2bfaeb46
[ngomik] add replacement for 'subapics'
...
http://subapics.com/ got discontinued and replaced by http://ngomik.in/ .
ngomik.in is still displaying a link to the "old site" showing a big
"Account Suspended" sign.
6 years ago
a2eeef1f5e
[behance] replace test
...
The "UVMW Studio" account and their galleries are gone.
6 years ago
e9dd2eff1d
[twitter] add extractor for media-tweet timelines ( #96 )
...
For example "https://twitter.com/PicturesEarth/media ".
They are different from normal timelines in that they do not contain
any (re)tweets from other users and feature all media the user ever
posted, including responses to other tweets.
6 years ago
f45c9f2141
[gfycat] test-updates and code-adjustments
6 years ago
9b1c39032c
[twitter] changes and improvements
...
- rename User- to TimelineExtractor
- rename 'userid' to 'user_id' to conform to the other ..._id values
- adjust archive_fmt to deal with retweets
- emulate browser behavior for API calls
6 years ago
10365394d7
[twitter] add support for user-timelines ( closes #96 )
...
also adds a 'retweets' option to filter retweeted content
6 years ago
d3f1eed2a6
[pinterest] improvements
...
- add stop condition for pin-related pins
- improve URL patterns
- make Pylint happy
6 years ago
2801a0d997
[exhentai] skip "Content Warning" page when not logged in
...
(closes #97 )
6 years ago
63fa0b2006
[pinterest] add extractors for related pins
...
Related pins can not be accessed by adding a "#related" fragment
to the end of a Pinterest URL, for example:
- https://www.pinterest.com/pin/858146903966145189/#related
- https://www.pinterest.com/g1952849/test-/#related
There are no explicit real URLs for related pins,
using an option to enable them results in "clunky" code,
and a custom "related:<URL>" scheme doesn't feel right either.
6 years ago
1694039de0
[komikcast] update ad-filter
6 years ago
a74591b84b
[tumblr] remove "original image" functionality
...
Accessing higher/original quality images on
https://s3.amazonaws.com/data.tumblr.com and http://data.tumblr.com
is no longer possible and any HTTP request results in 403 Forbidden.
A few images can still be accessed through https//a.tumblr.com [1][2],
but not as "_raw", just "_1280", and that might also be "fixed" in
the near future.
[1] https://a.tumblr.com/tumblr_kzjlfiTnfe1qz4rgho1_1280.jpg
[2] https://a.tumblr.com/ee589c6345f29d2d5935cecb49b0a705/tumblr_oztu02dIHp1wgha4yo1_1280.png
6 years ago
38d4f43cc0
[komikcast] skip ads
6 years ago
4313c95bc9
improve error message for OAuth2 authentication
6 years ago
b55e39d1ee
[mangadex] improve extraction
...
- cache manga API results
- add artist, author and date fields to chapter metadata
- remove Manga-/ChapterExtractor inheritance
- minor code simplifications and improvements
6 years ago
b1c4c1e13c
[mangadex] fix extraction
6 years ago
3c90df6635
[piczel] add user, folder and image extractors
6 years ago
2a9f3341a2
[behance] fix title extraction
6 years ago
3fc2f269fa
[behance] filter 'fields' list
6 years ago
b67339155f
[rule34] update test results
...
'metadata' tag type has been removed
6 years ago
a86f2bfc80
[pinterest] update not-found redirects
6 years ago
b040ca0718
[rule34] small unit test fixes
6 years ago
b164231bca
[sankaku] increase default values for 'wait-min/-max'
6 years ago
68d6033a5d
use 'retries' and 'timeout' options for regular HTTP requests
6 years ago
f3793660ef
update tests
6 years ago
df082e923c
[behance] add gallery extractor ( #95 )
6 years ago
5f27cfeff6
[deviantart] remove `prefer-public` option
...
All API requests now always use a public token and only switch to
a private token for pagination results if `refresh-token` is set
and less deviations than requested were returned.
6 years ago
bb89a1e6d7
[mangahere] use http://
...
invalid SSL cert for quite some time now
6 years ago
212130b048
[deviantart] improve public-private token switching
...
- rename option to `prefer-public`
- now also works for galleries with less than 24 items
6 years ago
886d662582
[deviantart] add option to minimize refresh-token usage
...
Always trying with a public token first and repeating the API request
with a private token if deviations are missing doesn't quite work for
galleries and folders with less than 25 items, so its an option and
not the default.
6 years ago
d98e47817d
[deviantart] reduce refresh-token usage
...
Instead of using a refresh-token-based access-token for every API
request, they are now only used for paginated results.
API requests to get a user's profile and the original download URL
now always use a public access-token.
6 years ago
84854fcad7
[myportfolio] add user and gallery extractors ( #95 )
6 years ago
c9f70e0a19
[paheal] use HTTPS
6 years ago
ff436692bf
["deviantart] add 'journals' option
6 years ago
00032b828c
[deviantart] add 'wait-min' option
6 years ago
a6fe2bb594
[whatisthisimnotgoodwithcomputers] remove extractor
6 years ago
0ba93650e0
[8chan] replace unit test URL
...
the other thread is no longer accessible
6 years ago
269dc2bbd5
[sankaku] add 'tags' option ( #94 )
6 years ago
173add6935
[nijie] fix artist_id extraction
...
view_popup.php pages for older images or dojins either have the
artist_id value at a different place or not at all.
6 years ago
6996f5c118
[mangahere] fix and improve chapter extraction
6 years ago
1d43cbbf52
[gelbooru] tag-splitting for non-api mode
6 years ago
2eefaa99a3
[mangapark] support .net and .com mirrors
6 years ago
c20c0a4820
[safebooru] add pool extractor
6 years ago
f916279ae6
[rule34] add pool extractor
6 years ago
3dbc7c5f8d
[gelbooru] restore pool functionality
6 years ago
a2c74bc6f0
[gelbooru] inherit from BooruExtractor class
...
Breaks pool functionality when using API calls (for now),
but reduces code clutter and enables the `tags` option.
6 years ago
4a57509392
generalize tag-splitting option ( #92 )
...
- extend functionality to other booru sites:
- http://behoimi.org/
- https://konachan.com/
- https://e621.net/
- https://rule34.xxx/
- https://safebooru.org/
- https://yande.re/
6 years ago
188e956c4e
[imagefap] use HTTPS + update test results
6 years ago
87853538b4
[yandere] add option to split tags by type ( #92 )
6 years ago
a699787d01
[deviantart] update URL patterns to new format
...
DeviantArt changed its URL format from
https://<name>.deviantart.com/...
to
https://www.deviantart.com/ <name>/...
With this change both formats will be supported.
6 years ago
9e3415886c
[senmanga] fix/update tests
6 years ago
b8c97d2295
use 'extractor.request()' for more HTTP requests
6 years ago
150a6b9064
[xvideos] fix metadata extraction
6 years ago
7a98cc9798
[smugmug] update tests
...
My test account expired and all uploaded images got deleted.
6 years ago
91340d9d27
[pixiv] fix ugoira test
6 years ago
eb7a1f3b98
[pixiv] rework ugoira handling
...
Frame information now gets attached to the ZIP file's keyword dict
instead of being written to a separate text file.
6 years ago
017188d268
improve extractor.request()
...
Replace the 'fatal' parameter with 'expect', which is a list/range
of HTTP status codes >= 400 that should also be accepted.
6 years ago
f10bd5cdbe
[4chan] unescape filenames
6 years ago
2d1a104739
[mangadex] unescape manga names and chapter titles
...
pretty sure I previously tested if unescaping strings from the
embedded JSON object was necessary ... maybe they changed it
6 years ago
6ac403c5d3
add postprocessor config example
6 years ago
a47c6136cd
[simplyhentai] avoid redirects for all-pages.json ( #89 )
6 years ago
ad14de19c6
[imgur] support "unmuted" URLs
6 years ago
72e66f0aac
[simplyhentai] improve URL pattern
...
[ci skip]
6 years ago
cdcc3427a0
[simplyhentai] add video extractor ( #89 )
...
All videos hosted on their own servers seem be to dead,
but myhentai.tv embeds, which are most of the videos, work fine.
6 years ago
f9a6a19658
[simplyhentai] add image extractor ( #89 )
6 years ago
ebf596b399
[pawoo] restore metadata fields + smaller improvements
6 years ago
f7e7306e5a
[komikcast] update URL pattern and unescape image URLs
6 years ago
70f3617d88
[mangafox] fix URL extraction
6 years ago
a62bd81e9b
[pixiv] fix filter for 'type=all'
6 years ago
55b0913412
[simplyhentai] add gallery extractor ( #89 )
6 years ago
15cce22d82
[mangadex] fix parsing of unusual chapter strings
6 years ago
ecdc3475b8
[pixhost] support .to TLDs
6 years ago
f3d770d4e2
Merge branch '1.4-dev'
6 years ago
1ff626db97
[pixiv] improve bookmark extraction
...
- combine 'favorite' and 'bookmark' extractors
- it is now one extractor class, but its subcategory still
distinguishes between your own bookmarks ('bookmark') and other
user's bookmarks ('favorite') like before
- allow filtering by bookmark tags and public/private bookmarks
- fix pagination for bookmark results
6 years ago
0a1863fce3
[pixiv] respect more query parameters for user URLs
...
The API endpoint responsible for user illustrations does not
provide sufficient filter capabilities* to match the actual
website, so we are spinning our own filters.
Respected parameters are
'type': illust, manga, ugoira
'tag' : any image tag (this was already supported)
'p' : the page to start on
*
- API can filter for illustrations and manga, but not for ugoira.
- 'offset' is applied before filtering
- no 'tag' filter
6 years ago
f43d446692
[mangahere] extract chapter titles
6 years ago
b8e53b8c6b
[pixiv] move query parsing out of constructor
...
better exception handling, among other things
6 years ago
909d105ae6
[pixiv] add extractor for illusts from followed users
6 years ago
7f899bd5d8
Merge branch 'master' into 1.4-dev
6 years ago
fe69d01083
[pixiv] add extractor for search results
6 years ago
247f785af1
[pixiv] use App API
...
Transitioning to the App API breaks favorites archive IDs (there is
no longer any bookmark ID information), but the favorites API endpoint
of the public API was gone anyways ...
6 years ago
92fc199b07
[reddit] allow arbitrary subdomains
6 years ago
4cea886177
[imgur] allow longer album hashes
6 years ago
e1e23165a0
[pinterest] catch JSON decode errors
6 years ago
789608c107
[imagebam] fix extraction for certain galleries
6 years ago
7a58151566
fix util.parse_bytes invocations
...
(should be text.parse_bytes)
6 years ago
1c1e086d01
use common base class for OAuth1.0 based API interfaces
6 years ago
f3483a2b7c
[smugmug] add OAuth support
6 years ago
6a31ada9e3
re-implement OAuth1.0 code
...
OAuth support for SmugMug needs some additional features
(auth-rebuild on redirect, query parameters in URL, ...)
and fixing this in the old code wouldn't work all that well.
6 years ago
ec158776ed
[deviantart] add extractor for popular listings
6 years ago
0e3883303f
[pixiv] implement AppAPI wrapper
6 years ago
e2157f594e
[mangadex] fix manga extraction ( closes #84 )
...
Chapter listings for manga now use
https://mangadex.org/manga/ <id>/_/chapters/2/
as URL instead of
https://mangadex.org/manga/ <id>/_//2/
6 years ago
69a5e6ddb3
Merge branch 'master' into 1.4-dev
6 years ago
3ce5296313
[smugmug] code cleanup
...
- combine User and Node extractors
- (re)move miscellaneous helper functions
- rename "Owner" to "User"
6 years ago
42ed7667b8
[smugmug] support user- and general album URLs
6 years ago
2ea0d1da42
[smugmug] improve API code; use data expansions
6 years ago
16e014baaa
[smugmug] added image and album extractor
...
just some initial code that still requires a lot of work ...
TODO:
- folders
- old-style albums (which are nearly all of them ...)
- images from users
- OAuth
It could also happen that the API credentials used will become invalid
whenever my 14 day trial period ends (7 days remaining), but that
would just require users to supply their own.
6 years ago
d96b3474e5
[puremashiro] remove module
...
site has been unreachable for a couple of weeks
and now the DNS record is gone as well
6 years ago
b44a296404
[gomanga] remove module
...
site has been unreachable for a couple of weeks
and the cloudflare status page shows host errors
6 years ago
95392554ee
use text.urljoin()
6 years ago
2395d870dd
[pinterest] unquote board and user names, better errors
6 years ago
8b79eaafea
[tumblr] log actual time of rate limit resets
...
... instead of the amount of seconds until a reset
6 years ago
0f1e07f627
[pinterest] scrap OAuth implementation; code improvements
...
OAuth authentication isn't needed anymore and other tools
like Postman are better suited for this job anyway.
6 years ago
55d4d23860
[pinterest] use Pinterest's "Web" API ( #83 )
...
no access tokens, no user credentials of any kind ...
7 years ago
2721417dd8
Merge branch 'master' into 1.4-dev
7 years ago
c6d5154fc3
fix flake8 errors, ignore W504
...
pycodestyle 2.4.0 enforces some new style guidelines
7 years ago
2d17a9e07f
improve extractor.request()
...
- better retry behavior
- exponential back-off
- removed 'allow_empty' argument
7 years ago
80521ae1f6
[deviantart] improve API error handling
...
The previous implementation would retry requests with 4xx status codes
in an infinite loop, which is especially a problem when querying
non-existent users or groups. These are now properly handled with a
NotFoundError exception.
7 years ago
e54b43be08
[mangadex] add title info for chapter extractors
7 years ago
f471161920
Merge branch 'master' into 1.4-dev
7 years ago
eb37fbf0e8
[hentaifoundry] improve extractor
...
- use common base class
- better pagination
- respect '.../page/<num>'
- implement skip() / --range support
- get YII_CSRF_TOKEN from cookies
7 years ago
80bead739d
[oauth] require custom client-* values for pinterest
7 years ago
cc36f88586
rename safe_int to parse_int; move parse_* to text module
7 years ago
10cc59f3b5
fix extractor names
7 years ago
b1325d4d2c
fix extractor docstrings
7 years ago
df7e18399e
[luscious] fix image order
7 years ago
d10579edb5
[pinterest] improve PinterestAPI code; remove OAuth mentions
...
on another note: access_tokens have been set to only allow for
10 requests per hour (from 200 yesterday)
7 years ago
4bd182c107
[pinterest] implement `oauth:pinterest` ( #83 )
...
Pinterest access tokens are rate limited at 200 requests per
hour (or maybe per 2 or 3 hours?) so having just one access token
for all users isn't going to work in the long run.
7 years ago
9651f3fce0
[pinterest] improve error messages ( #83 )
7 years ago
dbe250f7e5
[pinterest] update access_token ( #83 )
7 years ago
dd49127408
[spectrumnexus] remove module
...
Site stopped hosting manga scans (http://view.thespectrum.net/ )
7 years ago
5c487300ee
improve 'parse_query()' and add tests
...
- another irrelevant micro-optimization !
- use urllib.parse.parse_qsl directly instead of parse_qs, which
just packs the results of parse_qsl in a different data structure
- reduced memory requirements since no additional dict and lists are
created
7 years ago
728c64a3fb
[tumblr] rename 'offset' to 'num and adjust formats
...
Trying to somehow emulate Tumblr filenames is a bad idea ...
7 years ago
6bd857a319
[tumblr] handle rate limits / 429 errors
...
- wait for the hourly limit to reset
- abort upon exceeding the daily limit (it doesn't seem useful to
potentially wait for several hours)
7 years ago
7073ab7707
[komikcast] update regex to only match manga pages
...
The 'readerarea' section now includes some (shady) external
Javascript file, which got matched as well.
7 years ago
a1fa4b43b0
Revert "[tumblr] add option to sort photosets by upload order"
...
This reverts commit 4a26ae32df
7 years ago
48a83a89e9
[loveisover] remove module
...
archive.loveisover.me was shut down on 2018-03-29;
https://www.archiveteam.org/index.php?title=4chan#archive.loveisover.me
7 years ago
564e12ca8f
replace 'imgyt' with 'imxto'
...
https://img.yt/ wasn't available for a couple of days, but has now
re-emerged as https://imx.to/ with a new web-interface.
Links to older images still work (see tests).
7 years ago
1b80fa82a9
[imgur] update URL pattern and tests
7 years ago
4a26ae32df
[tumblr] add option to sort photosets by upload order
7 years ago
6b72be8ee6
[tumblr] add 'hash' keyword
...
'hash' is the middle part of the filename in a tumblr image URL.
For example an image with '.../tumblr_p6tgemp1NZ1wgha4yo1_250.png' as
its URL would have 'p6tgemp1NZ1wgha4yo1' as hash.
7 years ago
d11fcf4804
smaller changes and fixes
...
- fix the cloudflare challenge result if the last decimal places
are zero (JS`s toFixed() removes trailing zeroes)
- fix downloading of kissmanga chapter-pages hosted on blogspot
(accessing blogspot with "kissmanga.com" as referrer yields a 401)
- disable certificate validation for 'mangahere' tests
- update flickr test result
7 years ago
759ba26fb0
[luscious] proper image order for picture albums
...
... and (try) to start with the first image instead of somewhere
in the middle of an album.
7 years ago
68e9fbee16
[tumblr] check all 4 keys/secrets before using OAuth
...
it was possible to cause a crash by setting api-key or -secret to null.
(this commit also slightly improves the blog-cache implementation)
7 years ago
f8168c693e
[tumblr] avoid calls to '/blog/.../info'
...
The same information returned by the 'blog/.../info' API endpoint
is also included in the result of every 'blog/.../posts' call.
7 years ago
64d7c85b55
[exhentai] improve metadata
...
- add 'width', 'height' and 'size' (in bytes) for each image
- change the former 'size' and 'size_units' into 'gallery_size'
7 years ago
64b22e0fc1
[pawoo] update URL pattern
...
adds support for 'https://pawoo.net/@.../media '
7 years ago
7b562907c3
[nijie] add favorites extractor
...
adds support for 'https://nijie.info/user_like_illust_view.php?id= ...'
7 years ago
445db75955
[nijie] improve extraction and metadata
...
- add 'title' and 'description'
- split 'artist_id' into 'user_id' and 'artist_id'
- 'user_id' is the ID of the user from which the image entry
originates from
- 'artist_id' is the ID of the actual image artist
- improve pagination and URL patterns
7 years ago
a112e3f2a0
[nijie] add doujin extractor
...
adds support for "https://nijie.info/members_dojin.php?id= <artist_id>"
7 years ago
f39153b6e9
[nhentai] add extractor for search results
7 years ago
52d41c41e7
[exhentai] add extractor for favorited galleries
7 years ago
63cc2599c4
[exhentai] add extractor for search results
7 years ago
d1c91a1f2b
[mangadex] fix manga-page extraction
7 years ago
299ae24996
[test] add a few downloader tests
7 years ago
dd314279fb
[test] add unit tests for extractor module functions
7 years ago
e7525b1b0e
[artstation] add challenge extractor ( #80 )
7 years ago
f5c6a2d7f5
[nhentai] use API to get gallery info
7 years ago
b2ba2b821d
[hitomi] fix image URLs and improve metadata
...
- use '?a.hitomi.la' as subdomain depending in gallery-id
- add 'characters', 'tags' and 'date' information
- support multiple entires per metadata-value
- rename 'num' to 'page'
7 years ago
3905474805
[booru] call update_page() with correct dict ( closes #82 )
7 years ago
44c267e362
[artstation] add search extractor ( #80 )
7 years ago
40ca562d7b
[artstation] add album extractor ( #80 )
7 years ago
f367d5c281
[deviantart] move delay-increase after expect_error check
...
[ci skip]
7 years ago
557cb94f81
[deviantart] use proper exponential backoff on API errors
...
... and use separate API credentials for unit tests.
7 years ago
723cc66bb1
[artstation] add user-, image- and likes-extractors
7 years ago
4d74749496
[tests] rework filters for extractor tests
...
CI incompatible tests will now only be skipped if tests are run in
a CI environment.
7 years ago
d6ef52897c
[imgchili] remove module
...
All previously hosted images yield a 404
and the main page is just a logo.
7 years ago
7847ab1d5a
[imagehosts] remove even more dead sites
...
All removed sites either
- reject all incoming connections or
- display a message from their domain registrar
7 years ago
5f37d40a3e
[komikcast] bypass cloudflare challenge
7 years ago
f9884e2338
[pixiv] update URL pattern
...
add support for 'https://www.pixiv.net/user/ <id>'
7 years ago
85ed023c2e
[mangadex] remove the trailing ' - MangaDex' in a better way
...
str.rstrip() works differently than assumed.
7 years ago
32bbd12f08
update extractor tests
7 years ago
ca326bd275
[deviantart] fix folder and collection archive IDs
...
{folder[index]} and {collection[index]} are both '0' when being
delegated from Gallery- or FavoriteExtractors, as there is no
way of knowing a folder's index when getting folder-information
from the API.
7 years ago
e32fe1cdf1
[pinterest] cast IDs to int
...
... and update test results.
Image URLs changed from
https://s-media-cache-ak0.pinimg.com/ ... to
https://i.pinimg.com/ ...
7 years ago
179ecee965
[turboimagehost] fix extraction
7 years ago
1400868f53
[mangadex] general improvements
...
- support >100 chapter entries per manga
- custom archive ID format
- detect non-existing chapters
7 years ago
749fbbfa6c
[mangadex] add chapter- and manga-extractor
7 years ago
6e38cf5aab
[mangareader] use 'https://'
...
The site now redirects from http://mangareader.net/
to https://mangareader.net/
7 years ago
1d71123f91
[pixiv] update archive IDs and add metadata-fields
...
(Pixiv bookmarks actually have their own IDs, comments and tags,
independent of the bookmarked image, which makes creating an
archive ID a lot easier)
7 years ago
858fdbdb22
[tumblr] improve 'inline' extraction
...
'quote' posts store their HTML content in the 'source' field
7 years ago
5008e105ee
update archive IDs
...
... to behave in a more straightforward way when dealing with
bookmarks/favourites/etc.
specific IDs are now grouped by their owner, album-id, ... to
allow for duplicates when it would be expected.
7 years ago
829ddf4ac1
[sankaku] general improvements
...
- simplify regex
- unquote search tags
- increase default wait-time between HTTP requests
- downloading several hundreds of images always resulted
in '429 Too Many Requests' eventually
- circumvent paging restrictions for unauthenticated users by only
using the 'next' parameter
- setting 'page' to a constant, low value (or simply omitting it)
does the trick
7 years ago
49463f76bb
support multi-page URL ( #79 )
...
* support multi-page URL
* fix
* all done.
* fix, again
7 years ago
19aefdfde3
[directlink] update test results
7 years ago
74029c50bb
[directlink] unquote metadata fields
7 years ago
8f338347b6
[imagehosts] cleanup
...
removed
- chronos.to - unable to resolve hostname
- coreimg.net - same
- imgmaid.net - same
- hosturimage.com - everything returns 404
- imageontime.org - redirects to some shady site
- imgupload.yt - cloudflare error 522, host down
- img4ever.net - read timeout
7 years ago
edfd3d9fc9
[yeet] remove module
...
- archive.yeet.net returns a 500 server error
- yeet.net moved to yeet.rip, but the archive is gone
7 years ago
8704d850bf
add explicit proxy support ( #76 )
...
- '--proxy' as command-line argument
- 'extractor.*.proxy' as config option
7 years ago
367b963d37
[pixiv] fix ugoira extraction ... again ( #78 )
...
Some animations are not available for mobile devices, so we
pretend to be a desktop browser when requesting the ugoira page.
7 years ago
b79f1f2ca7
[pixiv] fix ugoira extraction ( closes #78 )
7 years ago
d122203be1
[mangastream] fix extraction
7 years ago
179bcdd349
adjust archive-ids
7 years ago
3cec533c28
Merge branch 'archive'
7 years ago
20af86b2ea
add more extractor tests
...
for mangastream, reddit and imgur
7 years ago
7e0207bcf4
[imgur] strip trailing '?1' from 'ext'
7 years ago
cf147dfee9
[hentai2read] fix manga extraction
...
- site changed its HTML structure
7 years ago
f5f2d29f56
[nijie] fix dojin extraction
...
- correctly extract artist_id
- set extension to "jpg" if it was empty and let filetype checks do
the rest
7 years ago
d38bf2f54c
[tumblr] recognize /image/... URLs
...
xyz.tumblr.com/image/123 refers to the same images
as xyz.tumblr.com/post/123.
7 years ago
5b3c34aa96
use generic chapter-extractor in more modules
7 years ago
7b5ba69951
[hentaihere] ensure consistent extraction results
...
sometimes there is a random space before the next <a>
7 years ago
377b78b3c9
[hentai2read] fix manga name extraction
7 years ago
54c36a8a34
[subapics] add chapter- and manga-extractor ( #70 )
7 years ago
2dd3aeeeae
[komikcast] add chapter- and manga-extractor ( #70 )
7 years ago
7a412f5c32
implement generic manga-chapter extractor
7 years ago
Mike Fährmann
