ecdc3475b8
[pixhost] support .to TLDs
6 years ago
f3d770d4e2
Merge branch '1.4-dev'
6 years ago
1ff626db97
[pixiv] improve bookmark extraction
...
- combine 'favorite' and 'bookmark' extractors
- it is now one extractor class, but its subcategory still
distinguishes between your own bookmarks ('bookmark') and other
user's bookmarks ('favorite') like before
- allow filtering by bookmark tags and public/private bookmarks
- fix pagination for bookmark results
6 years ago
0a1863fce3
[pixiv] respect more query parameters for user URLs
...
The API endpoint responsible for user illustrations does not
provide sufficient filter capabilities* to match the actual
website, so we are spinning our own filters.
Respected parameters are
'type': illust, manga, ugoira
'tag' : any image tag (this was already supported)
'p' : the page to start on
*
- API can filter for illustrations and manga, but not for ugoira.
- 'offset' is applied before filtering
- no 'tag' filter
6 years ago
f43d446692
[mangahere] extract chapter titles
6 years ago
b8e53b8c6b
[pixiv] move query parsing out of constructor
...
better exception handling, among other things
6 years ago
909d105ae6
[pixiv] add extractor for illusts from followed users
6 years ago
7f899bd5d8
Merge branch 'master' into 1.4-dev
6 years ago
fe69d01083
[pixiv] add extractor for search results
6 years ago
247f785af1
[pixiv] use App API
...
Transitioning to the App API breaks favorites archive IDs (there is
no longer any bookmark ID information), but the favorites API endpoint
of the public API was gone anyways ...
6 years ago
92fc199b07
[reddit] allow arbitrary subdomains
6 years ago
4cea886177
[imgur] allow longer album hashes
6 years ago
e1e23165a0
[pinterest] catch JSON decode errors
6 years ago
789608c107
[imagebam] fix extraction for certain galleries
6 years ago
7a58151566
fix util.parse_bytes invocations
...
(should be text.parse_bytes)
6 years ago
1c1e086d01
use common base class for OAuth1.0 based API interfaces
6 years ago
f3483a2b7c
[smugmug] add OAuth support
6 years ago
6a31ada9e3
re-implement OAuth1.0 code
...
OAuth support for SmugMug needs some additional features
(auth-rebuild on redirect, query parameters in URL, ...)
and fixing this in the old code wouldn't work all that well.
6 years ago
ec158776ed
[deviantart] add extractor for popular listings
6 years ago
0e3883303f
[pixiv] implement AppAPI wrapper
6 years ago
e2157f594e
[mangadex] fix manga extraction ( closes #84 )
...
Chapter listings for manga now use
https://mangadex.org/manga/ <id>/_/chapters/2/
as URL instead of
https://mangadex.org/manga/ <id>/_//2/
6 years ago
69a5e6ddb3
Merge branch 'master' into 1.4-dev
6 years ago
3ce5296313
[smugmug] code cleanup
...
- combine User and Node extractors
- (re)move miscellaneous helper functions
- rename "Owner" to "User"
6 years ago
42ed7667b8
[smugmug] support user- and general album URLs
6 years ago
2ea0d1da42
[smugmug] improve API code; use data expansions
6 years ago
16e014baaa
[smugmug] added image and album extractor
...
just some initial code that still requires a lot of work ...
TODO:
- folders
- old-style albums (which are nearly all of them ...)
- images from users
- OAuth
It could also happen that the API credentials used will become invalid
whenever my 14 day trial period ends (7 days remaining), but that
would just require users to supply their own.
6 years ago
d96b3474e5
[puremashiro] remove module
...
site has been unreachable for a couple of weeks
and now the DNS record is gone as well
6 years ago
b44a296404
[gomanga] remove module
...
site has been unreachable for a couple of weeks
and the cloudflare status page shows host errors
6 years ago
95392554ee
use text.urljoin()
6 years ago
2395d870dd
[pinterest] unquote board and user names, better errors
6 years ago
8b79eaafea
[tumblr] log actual time of rate limit resets
...
... instead of the amount of seconds until a reset
6 years ago
0f1e07f627
[pinterest] scrap OAuth implementation; code improvements
...
OAuth authentication isn't needed anymore and other tools
like Postman are better suited for this job anyway.
6 years ago
55d4d23860
[pinterest] use Pinterest's "Web" API ( #83 )
...
no access tokens, no user credentials of any kind ...
7 years ago
2721417dd8
Merge branch 'master' into 1.4-dev
7 years ago
c6d5154fc3
fix flake8 errors, ignore W504
...
pycodestyle 2.4.0 enforces some new style guidelines
7 years ago
2d17a9e07f
improve extractor.request()
...
- better retry behavior
- exponential back-off
- removed 'allow_empty' argument
7 years ago
80521ae1f6
[deviantart] improve API error handling
...
The previous implementation would retry requests with 4xx status codes
in an infinite loop, which is especially a problem when querying
non-existent users or groups. These are now properly handled with a
NotFoundError exception.
7 years ago
e54b43be08
[mangadex] add title info for chapter extractors
7 years ago
f471161920
Merge branch 'master' into 1.4-dev
7 years ago
eb37fbf0e8
[hentaifoundry] improve extractor
...
- use common base class
- better pagination
- respect '.../page/<num>'
- implement skip() / --range support
- get YII_CSRF_TOKEN from cookies
7 years ago
80bead739d
[oauth] require custom client-* values for pinterest
7 years ago
cc36f88586
rename safe_int to parse_int; move parse_* to text module
7 years ago
10cc59f3b5
fix extractor names
7 years ago
b1325d4d2c
fix extractor docstrings
7 years ago
df7e18399e
[luscious] fix image order
7 years ago
d10579edb5
[pinterest] improve PinterestAPI code; remove OAuth mentions
...
on another note: access_tokens have been set to only allow for
10 requests per hour (from 200 yesterday)
7 years ago
4bd182c107
[pinterest] implement `oauth:pinterest` ( #83 )
...
Pinterest access tokens are rate limited at 200 requests per
hour (or maybe per 2 or 3 hours?) so having just one access token
for all users isn't going to work in the long run.
7 years ago
9651f3fce0
[pinterest] improve error messages ( #83 )
7 years ago
dbe250f7e5
[pinterest] update access_token ( #83 )
7 years ago
dd49127408
[spectrumnexus] remove module
...
Site stopped hosting manga scans (http://view.thespectrum.net/ )
7 years ago
5c487300ee
improve 'parse_query()' and add tests
...
- another irrelevant micro-optimization !
- use urllib.parse.parse_qsl directly instead of parse_qs, which
just packs the results of parse_qsl in a different data structure
- reduced memory requirements since no additional dict and lists are
created
7 years ago
728c64a3fb
[tumblr] rename 'offset' to 'num and adjust formats
...
Trying to somehow emulate Tumblr filenames is a bad idea ...
7 years ago
6bd857a319
[tumblr] handle rate limits / 429 errors
...
- wait for the hourly limit to reset
- abort upon exceeding the daily limit (it doesn't seem useful to
potentially wait for several hours)
7 years ago
7073ab7707
[komikcast] update regex to only match manga pages
...
The 'readerarea' section now includes some (shady) external
Javascript file, which got matched as well.
7 years ago
a1fa4b43b0
Revert "[tumblr] add option to sort photosets by upload order"
...
This reverts commit 4a26ae32df
7 years ago
48a83a89e9
[loveisover] remove module
...
archive.loveisover.me was shut down on 2018-03-29;
https://www.archiveteam.org/index.php?title=4chan#archive.loveisover.me
7 years ago
564e12ca8f
replace 'imgyt' with 'imxto'
...
https://img.yt/ wasn't available for a couple of days, but has now
re-emerged as https://imx.to/ with a new web-interface.
Links to older images still work (see tests).
7 years ago
1b80fa82a9
[imgur] update URL pattern and tests
7 years ago
4a26ae32df
[tumblr] add option to sort photosets by upload order
7 years ago
6b72be8ee6
[tumblr] add 'hash' keyword
...
'hash' is the middle part of the filename in a tumblr image URL.
For example an image with '.../tumblr_p6tgemp1NZ1wgha4yo1_250.png' as
its URL would have 'p6tgemp1NZ1wgha4yo1' as hash.
7 years ago
d11fcf4804
smaller changes and fixes
...
- fix the cloudflare challenge result if the last decimal places
are zero (JS`s toFixed() removes trailing zeroes)
- fix downloading of kissmanga chapter-pages hosted on blogspot
(accessing blogspot with "kissmanga.com" as referrer yields a 401)
- disable certificate validation for 'mangahere' tests
- update flickr test result
7 years ago
759ba26fb0
[luscious] proper image order for picture albums
...
... and (try) to start with the first image instead of somewhere
in the middle of an album.
7 years ago
68e9fbee16
[tumblr] check all 4 keys/secrets before using OAuth
...
it was possible to cause a crash by setting api-key or -secret to null.
(this commit also slightly improves the blog-cache implementation)
7 years ago
f8168c693e
[tumblr] avoid calls to '/blog/.../info'
...
The same information returned by the 'blog/.../info' API endpoint
is also included in the result of every 'blog/.../posts' call.
7 years ago
64d7c85b55
[exhentai] improve metadata
...
- add 'width', 'height' and 'size' (in bytes) for each image
- change the former 'size' and 'size_units' into 'gallery_size'
7 years ago
64b22e0fc1
[pawoo] update URL pattern
...
adds support for 'https://pawoo.net/@.../media '
7 years ago
7b562907c3
[nijie] add favorites extractor
...
adds support for 'https://nijie.info/user_like_illust_view.php?id= ...'
7 years ago
445db75955
[nijie] improve extraction and metadata
...
- add 'title' and 'description'
- split 'artist_id' into 'user_id' and 'artist_id'
- 'user_id' is the ID of the user from which the image entry
originates from
- 'artist_id' is the ID of the actual image artist
- improve pagination and URL patterns
7 years ago
a112e3f2a0
[nijie] add doujin extractor
...
adds support for "https://nijie.info/members_dojin.php?id= <artist_id>"
7 years ago
f39153b6e9
[nhentai] add extractor for search results
7 years ago
52d41c41e7
[exhentai] add extractor for favorited galleries
7 years ago
63cc2599c4
[exhentai] add extractor for search results
7 years ago
d1c91a1f2b
[mangadex] fix manga-page extraction
7 years ago
299ae24996
[test] add a few downloader tests
7 years ago
dd314279fb
[test] add unit tests for extractor module functions
7 years ago
e7525b1b0e
[artstation] add challenge extractor ( #80 )
7 years ago
f5c6a2d7f5
[nhentai] use API to get gallery info
7 years ago
b2ba2b821d
[hitomi] fix image URLs and improve metadata
...
- use '?a.hitomi.la' as subdomain depending in gallery-id
- add 'characters', 'tags' and 'date' information
- support multiple entires per metadata-value
- rename 'num' to 'page'
7 years ago
3905474805
[booru] call update_page() with correct dict ( closes #82 )
7 years ago
44c267e362
[artstation] add search extractor ( #80 )
7 years ago
40ca562d7b
[artstation] add album extractor ( #80 )
7 years ago
f367d5c281
[deviantart] move delay-increase after expect_error check
...
[ci skip]
7 years ago
557cb94f81
[deviantart] use proper exponential backoff on API errors
...
... and use separate API credentials for unit tests.
7 years ago
723cc66bb1
[artstation] add user-, image- and likes-extractors
7 years ago
4d74749496
[tests] rework filters for extractor tests
...
CI incompatible tests will now only be skipped if tests are run in
a CI environment.
7 years ago
d6ef52897c
[imgchili] remove module
...
All previously hosted images yield a 404
and the main page is just a logo.
7 years ago
7847ab1d5a
[imagehosts] remove even more dead sites
...
All removed sites either
- reject all incoming connections or
- display a message from their domain registrar
7 years ago
5f37d40a3e
[komikcast] bypass cloudflare challenge
7 years ago
f9884e2338
[pixiv] update URL pattern
...
add support for 'https://www.pixiv.net/user/ <id>'
7 years ago
85ed023c2e
[mangadex] remove the trailing ' - MangaDex' in a better way
...
str.rstrip() works differently than assumed.
7 years ago
32bbd12f08
update extractor tests
7 years ago
ca326bd275
[deviantart] fix folder and collection archive IDs
...
{folder[index]} and {collection[index]} are both '0' when being
delegated from Gallery- or FavoriteExtractors, as there is no
way of knowing a folder's index when getting folder-information
from the API.
7 years ago
e32fe1cdf1
[pinterest] cast IDs to int
...
... and update test results.
Image URLs changed from
https://s-media-cache-ak0.pinimg.com/ ... to
https://i.pinimg.com/ ...
7 years ago
179ecee965
[turboimagehost] fix extraction
7 years ago
1400868f53
[mangadex] general improvements
...
- support >100 chapter entries per manga
- custom archive ID format
- detect non-existing chapters
7 years ago
749fbbfa6c
[mangadex] add chapter- and manga-extractor
7 years ago
6e38cf5aab
[mangareader] use 'https://'
...
The site now redirects from http://mangareader.net/
to https://mangareader.net/
7 years ago
1d71123f91
[pixiv] update archive IDs and add metadata-fields
...
(Pixiv bookmarks actually have their own IDs, comments and tags,
independent of the bookmarked image, which makes creating an
archive ID a lot easier)
7 years ago
858fdbdb22
[tumblr] improve 'inline' extraction
...
'quote' posts store their HTML content in the 'source' field
7 years ago
5008e105ee
update archive IDs
...
... to behave in a more straightforward way when dealing with
bookmarks/favourites/etc.
specific IDs are now grouped by their owner, album-id, ... to
allow for duplicates when it would be expected.
7 years ago
829ddf4ac1
[sankaku] general improvements
...
- simplify regex
- unquote search tags
- increase default wait-time between HTTP requests
- downloading several hundreds of images always resulted
in '429 Too Many Requests' eventually
- circumvent paging restrictions for unauthenticated users by only
using the 'next' parameter
- setting 'page' to a constant, low value (or simply omitting it)
does the trick
7 years ago
49463f76bb
support multi-page URL ( #79 )
...
* support multi-page URL
* fix
* all done.
* fix, again
7 years ago
19aefdfde3
[directlink] update test results
7 years ago
74029c50bb
[directlink] unquote metadata fields
7 years ago
8f338347b6
[imagehosts] cleanup
...
removed
- chronos.to - unable to resolve hostname
- coreimg.net - same
- imgmaid.net - same
- hosturimage.com - everything returns 404
- imageontime.org - redirects to some shady site
- imgupload.yt - cloudflare error 522, host down
- img4ever.net - read timeout
7 years ago
edfd3d9fc9
[yeet] remove module
...
- archive.yeet.net returns a 500 server error
- yeet.net moved to yeet.rip, but the archive is gone
7 years ago
8704d850bf
add explicit proxy support ( #76 )
...
- '--proxy' as command-line argument
- 'extractor.*.proxy' as config option
7 years ago
367b963d37
[pixiv] fix ugoira extraction ... again ( #78 )
...
Some animations are not available for mobile devices, so we
pretend to be a desktop browser when requesting the ugoira page.
7 years ago
b79f1f2ca7
[pixiv] fix ugoira extraction ( closes #78 )
7 years ago
d122203be1
[mangastream] fix extraction
7 years ago
179bcdd349
adjust archive-ids
7 years ago
3cec533c28
Merge branch 'archive'
7 years ago
20af86b2ea
add more extractor tests
...
for mangastream, reddit and imgur
7 years ago
7e0207bcf4
[imgur] strip trailing '?1' from 'ext'
7 years ago
cf147dfee9
[hentai2read] fix manga extraction
...
- site changed its HTML structure
7 years ago
f5f2d29f56
[nijie] fix dojin extraction
...
- correctly extract artist_id
- set extension to "jpg" if it was empty and let filetype checks do
the rest
7 years ago
d38bf2f54c
[tumblr] recognize /image/... URLs
...
xyz.tumblr.com/image/123 refers to the same images
as xyz.tumblr.com/post/123.
7 years ago
5b3c34aa96
use generic chapter-extractor in more modules
7 years ago
7b5ba69951
[hentaihere] ensure consistent extraction results
...
sometimes there is a random space before the next <a>
7 years ago
377b78b3c9
[hentai2read] fix manga name extraction
7 years ago
54c36a8a34
[subapics] add chapter- and manga-extractor ( #70 )
7 years ago
2dd3aeeeae
[komikcast] add chapter- and manga-extractor ( #70 )
7 years ago
7a412f5c32
implement generic manga-chapter extractor
7 years ago
6a07e38366
implement extractor.add() and .add_module()
...
... as a public and non-hacky way to add (external) extractors to
gallery-dl's pool and make them available for extractor.find()
7 years ago
34873dbd90
set 'archive_fmt' values
...
These are going to be used to create an unique id for each image.
7 years ago
a34cebc253
[luscious] jump to first image if cover does not link to it
7 years ago
84a52a9256
add DownloadArchive class
7 years ago
619387cbb1
update extractor unittest results
7 years ago
db91cf871c
document message identifiers
7 years ago
0dd48d644f
update test results
...
nothing broke, but things got updated or changed
7 years ago
1e93955170
[batoto] remove module
...
Site officially shut down on 2018.01.18
7 years ago
76509a6d3c
[imgur] update test results
7 years ago
9fccd7b783
[tumblr] provide fallback URLs ( #64 )
...
Each image now produces 3 URLs:
- amazonaws.com _raw (or _1280 for older images)
- amazonaws.com _500
- media.tumblr.com (URL returned by API)
7 years ago
9d69401391
initial support for multiple URLs per image
7 years ago
91ed147cef
[oauth] use custom key/secret values during oauth:…
7 years ago
421a9740a3
[tumblr] add 'tumblr:' to force Tumblr extractor ( #71 )
7 years ago
40d35c87bc
[paheal] add tag- and post-extractors ( closes #69 )
7 years ago
cc0c2cca57
[reddit] add extractor for reddit-hosted images ( closes #68 )
7 years ago
f10ffc0839
update extractor blacklist to also allow classes
7 years ago
35e09869d1
[mangapark] fix image URLs and use HTTPS
7 years ago
9a049bdf51
[tumblr] add 'likes' extractor ( #65 )
7 years ago
67d4462d26
[batoto] rudimentary Cloudflare bypass
7 years ago
29d75fc3fa
[tumblr] add support for OAuth authentication ( #65 )
7 years ago
4edb25346e
[slideshare] support mobile URLs ( closes #67 )
7 years ago
e420a28bbc
fix cookie tests
7 years ago
b33efc99a4
[idolcomplex] add support for idol.sankakucomplex.com
7 years ago
75b2e84b6d
[tumblr] use s3.amazonaws.com for image URLs ( #64 )
7 years ago
5b094328b5
[puremashiro] add chapter- and manga-extractor ( closes #66 )
...
Also adds support for region subtags in language codes (e.g. en-us)
7 years ago
974e73bdbb
[booru] smaller code adjustments
7 years ago
03b8a548cb
[tumblr] change `reblogs` default value to `true` ( #61 )
7 years ago
d235f68f59
[tumblr] add option to filter reblogged posts ( #61 )
...
Reblogs are ignored by default, but can be included by setting
'extractor.tumblr.reblogs' to 'true'.
7 years ago
a794fffc6d
[batoto] extend chapter-string regex ( closes #60 )
...
Non-numeric chapter indices exist after all ...
7 years ago
1219ebb7f5
[danbooru] use alternate subdomains; support safebooru
7 years ago
9e8a84ab6c
[booru] rewrite using Mixin classes ( #59 )
...
- improved code structure
- improved URL patterns
- better pagination to work around page limits on
- Danbooru
- e621
- 3dbooru
7 years ago
0876541e43
[seiga] update tests
7 years ago
88bb0798fd
delay initialization of PathFormat objects
...
This allows the DeviantArt group-check to be moved inside the
Extractor.items() method which in turn allows for better exception
handling.
As a new general rule:
Never raise exceptions during extractor initialization.
7 years ago
c24e0e70a7
[pixiv] simplify main loop
7 years ago
c1e331edbb
[mangapark] replace manga test
7 years ago
28cd78aae0
[kissmanga] extend chapter-string regex ( closes #58 )
7 years ago
a3e9b51bea
[imgbox] update test results
...
Image URLs of older galleries have been updated to the new format.
https://i.imgbox.com/qHhw7lpG.png
-->
https://images3.imgbox.com/6d/9a/qHhw7lpG_o.png
7 years ago
d0886f411e
[gelbooru] re-enable API use ( closes #56 )
...
Gelbooru's API allows access to all images and is not restricted
to the first 20000.
This also adds an option to select between API use and manual
information extraction in case their API gets disabled again.
7 years ago
8102aae311
[mangahere] support ".cc" TLD and mobile URLs
7 years ago
676602056c
[reddit] unescape output URLs
7 years ago
2eedbaaaf9
[deviantart] use cache to store new refresh_tokens
...
The 'refresh_token' set in a user's config file gets used once to
get a new 'access_token' and 'refresh_token', which is then stored
in gallery-dl's cache and gets used the next time the 'access_token'
needs to be refreshed.
This means deleting the cache file invalidates the refresh_token-
chain and requires the user to re-authenticate.
7 years ago
fc7d165c97
[deviantart] add support for OAuth2 authentication
...
Some user galleries [*] require you to be either logged in or
authenticated via OAuth2 to access their deviations.
[*] e.g. https://polinaegorussia.deviantart.com/gallery/
--------------
known issue:
A deviantart 'refresh_token' can only be used once and gets updated
whenever it is used to request a new 'access_token', so storing its
initial value in a config file and reusing it again and again is not
possible.
7 years ago
91c2aed077
[nhentai] fix JSON extraction
7 years ago
444008a14a
[khinsider] use urljoin() to complete page URLs
7 years ago
263741d243
[luscious] update URL pattern ( closes #55 )
7 years ago
0a9a07a6e1
[slideshare] improve metadata; flake8
...
- added 'views' and 'published' keywords
- fixed longer titles and descriptions
7 years ago
a8d2dde8b2
[slideshare] Add a new extractor for slideshare.net ( #54 )
7 years ago
19a6ae57b2
[sankaku] add pool extractor
7 years ago
e52f0cc1ed
[sankaku] add post extractor
7 years ago
595593a35e
[sankaku] rewrite
...
- better code structure and extensibility
- better metadata
7 years ago
a3924d2072
[sankaku] fix swf extraction ( closes #52 )
7 years ago
291369eab2
various smaller changes/additions
7 years ago
300346ecdf
[mangazuki] remove extractors
...
This site has been in "rebuild"-mode for a fairly long time and the
current extractor code isn't going to work for the new version either.
7 years ago
d275b1d9a3
[khinsider] fix extraction
...
... again
7 years ago
6b8e3003df
[hentai2read] ensure consistent extraction results
7 years ago
a1980b16f3
[gelbooru] various improvements
...
- better metadata for pools
- map ratings to s/q/e like other boorus do
- skip() support
7 years ago
93482a1f88
implement 'util.advance()'
7 years ago
038e3b3369
[kissmanga] handle "AreYouHuman" redirects ( #51 )
7 years ago
2b9a783fc7
[khinsider] fix extraction
7 years ago
214972bc9a
[gelbooru] use manual extraction
...
... to compensate for their disabled API.
(https://gelbooru.com/index.php?page=forum&s=view&id=3875 )
This also adds an extractor for image-pools.
7 years ago
55c64cad4b
[khinsider] fix filename extension and test-pattern
7 years ago
b14de6ffc2
[tumblr] small improvements
...
- don't transform inline GIF URLs
- set 'type' parameter for API calls if there is only
one post type selected
7 years ago
9296a26eae
[tumblr] add warning messages
7 years ago
65c1c53eb8
[khinsider] fix extraction
7 years ago
12de658937
[tumblr] add options to control extraction behavior ( #48 )
...
- posts : list of post-types to inspect
- inline : scan post bodies for inline images
- external: follow external links
7 years ago
077f8c12be
[tumblr] original video URLs + continuous offset
7 years ago
8eb12ebeae
[tumblr] support more post/media types ( #48 )
...
This adds support for audio and video posts (most videos are shared
from youtube/instagram which isn't supported -> youtube-dl),
as well as link posts and image-search inside of text posts.
Most of this is just WIP and will need some sort of improvement
and options to enable/disable different media types etc.
7 years ago
b8cdd42cab
[senmanga] fix extraction (again)
...
this is basically a re-revert of 2ace5c7
7 years ago
e6814aebe2
add 'extractor.*.user-agent' config option
7 years ago
6913eeaa40
[powermanga] replace manga extractor unit test
...
My Hero Academia is gone
7 years ago
7e0d9257a7
[hbrowse] fix manga extraction
7 years ago
3c576d10c0
[seiga] better metadata + 'skip()' support
7 years ago
f72318e593
[seiga] support more than 200 images
...
Due to API restrictions and/or missing knowledge about and
documentation of API usage, it was only possible to retrieve the
latest 200 images of a niconico seiga user with said API.
The new approach manually visits each HTML page and gets its
information from there.
7 years ago
baf8094868
improve Extractor.request()'s retry behavior
7 years ago
7e7b64162b
[batoto] handle error 10031
7 years ago
92027f67f9
use consistent names for URL constants
...
root := <scheme>://<host>
base_url := <root>/<common path>
7 years ago
69cbc0619f
[mangastream] fix 'next-page' URLs ( fixes #49 )
7 years ago
980fd3616d
[tumblr] use API v2 ( #48 )
7 years ago
d6bed9f36f
[tumblr] prevent premature exit to get all images ( fixes #48 )
7 years ago
305da540c3
[mangahere] fix metadata extraction
7 years ago
2d0cfb33e1
[xvideos] add user profile extractor ( #45 )
7 years ago
a393e6e538
[xvideos] add gallery extractor ( #45 )
7 years ago
3a8a0c1f35
[imgbox] rewrite / fix extraction ( closes #47 )
7 years ago
035ef655f1
[imagefap] update unit tests
...
old gallery/image has been deleted
7 years ago
239d7afea7
[hosturimage] fix extraction of larger images
7 years ago
158e60ee89
[3dbooru] enable download continuation
...
behoimi.org doesn't respect 'Range' headers and doesn't report
'Content-Length' for compressed content encodings.
7 years ago
c4fcdf2691
Revert "[senmanga] fix extraction and download"
...
This reverts commit 2ace5c7b3c
7 years ago
81a7788b40
replace space characters in unit test URLs
7 years ago
bf82181359
[jaiminisbox] fix extraction
7 years ago
16783e327f
[common] fix UnboundLocalError in Extractor.request()
7 years ago
2ace5c7b3c
[senmanga] fix extraction and download
7 years ago
4d8387f93b
[pixiv] support mobile URLs ( https://touch.pixiv.net/ )
7 years ago
ab2bf0b0dd
[deviantart] replace collection unittest
7 years ago
289d6b65d2
[danbooru] extend and improve URL regex
...
- add support for danbooru mirrors:
- hijiribe.donmai.us
- sonohara.donmai.us
- todo: actually use these domains instead of redirecting everything
to danbooru itself
- improve handling of query string parameters
7 years ago
5fa42336a2
[sankaku] add warning for unauthenticated users
...
also improve URL pattern and add missing options to default config file
7 years ago
6af921a952
[sankaku] rewrite/improve ( fixes #44 )
...
- add wait-time between HTTP requests similar to exhentai
- add 'wait-min' and 'wait-max' options
- increase retry-count for HTTP requests to 10
- implement user authentication (non-authenticated users can only view
images up to page 25)
- implement 'skip()' functionality (only works up to page 50)
- implement image-retrieval for pages >= 51
- fix issue with multiple tags
7 years ago
9aecc67841
[common] explicitly handle HTTP status code 429
7 years ago
d68a24aa70
[kissmanga] fix extraction
...
site changed '\n' to '\r\n' for newlines
7 years ago
864a63ed33
fix typo
...
[skip ci]
7 years ago
f3fbaa5c3e
[reddit] allow users to override the API User-Agent
...
Only overriding the Client-ID is not enough if you want to follow
Reddit's API access rules [1].
[1] https://github.com/reddit/reddit/wiki/API#rules
7 years ago
31ea6001e8
[dynastyscans] improve metadata and filename formats
7 years ago
2ef3c35c98
smaller textual changes
...
- swapped doc for deviantart.mature and .original
- updated gallery-dl.conf
- "transferred" -> "delegated"
7 years ago
68a0a7579c
fix/improve some regular expressions
7 years ago
393755ee94
[tumblr] update tests
7 years ago
75d3a1f72f
[deviantart] always download original images
...
Deviation-objects returned by the DeviantArt API don't always contain
the URL and metadata of the original image ([1]). Getting this
information requires an additional API call [2], which is indicated by
the 'is_downloadable' and 'download_filesize' metadata within a
deviation-object.
[1] https://myria-moon.deviantart.com/art/Aime-Moi-part-en-vadrouille-261986576
[2] https://www.deviantart.com/developers/http/v1/20160316/deviation_download/bed6982b88949bdb08b52cd6763fcafd
7 years ago
a1c8b21cfd
[senmanga] improve metadata
7 years ago
994b2fc1e7
[deviantart] replace 'author[urlname]' keyword
...
author[urlname] has always only been the lowercase version of
author[username], which can now be directly converted to lowercase
using the 'l' conversion: '{author[username]!l}'
7 years ago
633b376f35
improve/adjust default filename formats for manga sites
7 years ago
41adb99e9c
[pawoo] fix extraction
...
- changed access_token
- use account-search instead of general search
7 years ago
b319f4bab3
smaller code and text changes
7 years ago
ad4580800c
[pixiv] add support for more URL patterns
...
- https://www.pixiv.net/mypage.php#id=USERID
- https://www.pixiv.net/#id=USERID
7 years ago
82ea6c0cd3
adjust format strings with optional titles
...
... except for anything manga/comic related
7 years ago
85a2b2ae59
[khinsider] fix extraction
7 years ago
26a866e7d8
implement (sub)category-transfer between extractors ( #41 )
...
ImageFap- and all Manga-Extractors will transfer their (sub)category
values to other extractors instantiated by them, which will in turn
allow those to use options set for their parents.
Example:
ImagefapGalleryExtractors will use options set under
extractor.imagefap.user, if (and only if) they have been instantiated by
a ImagefapUserExtractor; and options from extractor.imagefap.gallery
otherwise.
7 years ago
1ab4c7986f
[mangahere] fix extraction
...
would switch to HTTPS, but there seem to be certificate issues
7 years ago
8e14714c2b
[imgspice] fix extraction
7 years ago
9c138dfc1f
[common] detect empty HTTP response bodies
7 years ago
c51616f8d8
[foolslide] fix minor chapter number
7 years ago
77bf923c56
Update imgur.py to include 'title' of single image ( #40 )
...
Add {title} keyword..
Images on Imgur don't necessarily have a title, but I think most of them do, and since this should not break anything else..
7 years ago
a85f06d2d1
[foolslide] restructure; convert suitable values to int
7 years ago
deb2e803ba
simplify MangaExtractor class
7 years ago
9fc1d0c901
implement and use 'util.safe_int()'
...
same as Python's 'int()', except it doesn't raise any exceptions and
accepts a default value
7 years ago
8963da8fd8
[spectrumnexus] extract manga metadata
7 years ago
a3e40734d1
[mangareader] extract manga metadata
7 years ago
9196005a4d
[mangazuki] extract manga metadata
7 years ago
543ba245eb
[deviantart] update test results
...
thumbnail URLs changed from //tXX.… to //t00.…
7 years ago
b7a54a51d0
[mangapark] extract manga metadata + code improvements
7 years ago
Mike Fährmann
