105f3c9666
[twitter] add extractor for direct image links ( closes #1417 )
4 years ago
ec3d5d58a8
[vk] improve extractor ( #474 )
...
- fetch all photos
- add 'metadata' option
- fix extracting photos without '?' in URL
4 years ago
ebd142e2a8
[twitter] don't use youtube-dl for cards when videos are disabled
...
(#1416 )
4 years ago
d5aad999dc
[tapas] implement login with username & password ( #692 )
4 years ago
e9ec91c811
[exhentai] improve image limits check
...
- check if current image is the '509 Bandwidth Exceeded' notification
(https://ehgt.org/g/509.gif or https://exhentai.org/img/509.gif )
- remove 'limits' option
4 years ago
387fe415d5
unescape items in text.split_html()
4 years ago
36291176bc
[pinterest] add 'search' extractor ( #1411 )
4 years ago
058cc47e9b
[bcy] improve pagination
4 years ago
ddd48ceee5
update extractor test results
4 years ago
1a540fbe00
[komikcast] fix extraction
4 years ago
5aa30c3669
[tapas] add 'series' and 'episode' extractors ( #692 )
4 years ago
ccfa5a8694
[twitter] better error message when logging in with 2FA ( #1409 )
4 years ago
214ecf62ce
[deviantart] fix arguments for search/popular results ( #1408 )
4 years ago
522d0a834c
[aryion] Unescape paths too ( #1414 )
...
Without this you'll get paths like this:
- Starcross - Ch. 2 "The Ins and Outs of Sarah"
This commit changes it to:
- Starcross - Ch. 2 "The Ins and Outs of Sarah"
4 years ago
5ad615f0db
fix PixivFavoriteExtractor regex ( #1405 )
...
* fix PixivFavoriteExtractor regex
* do not use lookbehind
4 years ago
62cfee4d28
[vk] initial support for albums ( #474 )
4 years ago
0e601de67b
[sankaku] simplify 'pool' tags ( #1388 )
...
normalize 'tags' and 'artist_tags' to a string-list
4 years ago
d085ade9d5
[sankaku] add 'tag_string' metadata field ( #1388 )
...
The 'join()'ed version of 'tags'.
Handling lists in format strings isn't properly supported yet.
4 years ago
2dffd231b7
[sankaku] add enumeration index for books ( #1388 )
4 years ago
139fb84108
[deviantart] fix username for 'watch' results ( #794 )
...
before it'd use "/" as username
4 years ago
91c2e15da9
[deviantart] add support for posts from watched users ( #794 )
4 years ago
03c20d8c8e
[deviantart] update 'watch' URL pattern ( #794 )
4 years ago
2846235669
[twitter] allow specifying a custom format for user results
...
(#1337 )
4 years ago
dc23cfd684
[deviantart] use fallback for /intermediary/ URLs
...
instead of checking availability with HEAD requests
4 years ago
b0438c8f99
Revert "[deviantart] extend 'extra' option"
...
This reverts commit
5ad2b9c82b5c32a7bf5883f465faca#1387 , #1356 )
4 years ago
58b93635ee
[architizer] add 'firm' extractor ( #1369 )
4 years ago
204523611c
[imgclick] use 'http://' for image URLs
...
The TLS certificate for main.imgclick.net is invalid.
4 years ago
0b55f5ad84
[imgur] fix/improve rate limit handling ( #1386 )
...
- also wait-and-retry on 429 status codes
- use infinite loop instead of recursive calls
- 'extractor.sleep()' -> 'extractor.wait()'
4 years ago
69ca4e29f1
[deviantart] add 'watch' extractor ( #794 )
4 years ago
fcdda6128c
[mangastream] remove module
4 years ago
c677ea19dd
[mangareader] remove module
4 years ago
71523aaab6
[architizer] add 'project' extractor ( #1369 )
4 years ago
3378b39719
[twitter] implement 'users' option ( #1337 )
4 years ago
847e9b0ed7
[philomena] support post URLs without '/images/'
...
e.g. 'derpibooru.org/1'
4 years ago
466966bf83
[hentaicafe] remove module
4 years ago
97641cd151
[hentainexus] remove module
4 years ago
c485d0a956
[philomena] add generalized extractors for philomena sites
...
(closes #1379 )
4 years ago
6be7df53da
[hentaifox] improve metadata extraction ( fixes #1378 )
4 years ago
72fe9ac0f3
[gelbooru_v01] support some more boorus by default
...
- https://drawfriends.booru.org/
- https://vidyart.booru.org/
- https://tlb.booru.org/
4 years ago
10c279f285
Weasyl: Drop the `&feature=submit` part of the favourite extractor URL ( #1374 )
...
It's optional and requiring it forces users to escape those URLs because
of the ampersand
4 years ago
4be27ff0fe
[nozomi] support '/index-N.html' URLs ( closes #1365 )
...
and '/index-Popular-N.html'
4 years ago
780bac4c8a
[gelbooru] update video server ( fixes #1368 )
...
from 'https://img2.gelbooru.com ' to 'https://img3.gelbooru.com '
and provide fallback URLs
4 years ago
f8441e851a
[hentaifox] improve image extraction ( fixes #1366 )
...
build image URLs from embedded JSON data
instead 0f rewriting thumbnail URLs
4 years ago
c7c3fef0bc
[exhentai] support '/tag/' URLs ( closes #1363 )
4 years ago
90830daf85
[exhentai] improve 'favorites' extraction ( closes #1360 )
...
add special cases for when the favorite count is 0 (Never) or 1 (Once)
4 years ago
83f465faca
[deviantart] refactor 'extra' ( #1356 )
...
- change its expected type to string
- let users specify a list of sources (stash, posts) or 'all'
4 years ago
5c32a7bf58
[deviantart] allow selecting source for 'extra' ( #1356 )
...
Setting 'extra' to "stash" or "deviations" will only download embedded
sta.sh content or deviations. 'true' still downloads both.
4 years ago
a677123abb
[instagram] recognize 'reels' as option for 'include' ( #1329 )
4 years ago
94faf8c85a
add type check before applying 'browser' option ( fixes #1358 )
4 years ago
7440d1f112
[pixiv] add 'translated-tags' option ( closes #1354 )
...
(a lot more straight forward than I thought ...)
4 years ago
106cdc37c0
[instagram] support '/user/reels/' URLs ( closes #1329 )
4 years ago
524ebb133e
[instagram] refactor reel handling
4 years ago
9785c551bc
[500px] skip unavailable photos ( #1335 )
...
instead of crashing with a KeyError exception
4 years ago
6cfc9613fe
update some code in Extractor constructor
...
- combine '_init_headers' and '_emulate_browser' functionality
into new '_init_session'
- add 'headers' and 'ciphers' options
4 years ago
f59e63669b
[hentaicafe] add 'search' and 'tag' extractors ( #1345 )
4 years ago
38e66940c1
[tumblrgallery] simplify
4 years ago
665499924d
Support naver webtoon ( #1331 )
...
* Support naver webtoon (WIP)
* Apply patch
* Change filename format
* Fill test results
* Fill test result
4 years ago
a9119da4d4
support `tumblrgallery.xyz` ( #1298 )
...
* support `tumblrgallery.xyz`
* fix format issues
* Refactor and add post and search page support
* Fix warnings
* Few improvments
* Better file names
* Fix linting errors
* move id closer to the begining of the file name
Co-authored-by: topozorra <none>
4 years ago
bff71cde80
implement 'util.unique_squence()'
4 years ago
bae874f370
replace 'wait-min/-max' with 'sleep-request'
...
on exhentai, idolcomplex, reactor
4 years ago
e165e6c265
[wallhaven] add 'collections' extractor ( #1351 )
4 years ago
faf561b6ca
[wallhaven] add 'collection' extractor ( #1351 )
4 years ago
5d3d94ba14
[wallhaven] refactor
4 years ago
8821dceb79
use __import__() to dynamically load modules
4 years ago
69ea781d32
[mangadex] improve caching of manga results
...
'manga_id' being a string or integer are treated as two different keys
4 years ago
e58039358d
[mangadex] use 'api.mangadex.org' as default API server
...
The caching issues seem to be gone.
(#1290 , #1310 )
4 years ago
fc15930266
[readcomiconline] download high quality image versions
...
(fixes #1347 )
4 years ago
f360778e60
[komikcast] fix extraction
4 years ago
3df527ee2c
update extractor test results
4 years ago
fe2ec9cf68
[patreon] reduce redirects when fetching campaign ID
4 years ago
29ea54dc41
[patreon] use '"browser": "firefox"' by default ( #1117 )
4 years ago
61fbbd2dae
[exhentai] rename metadata fields to match API results ( #1325 )
...
- gallery_id -> gid
- gallery_token -> token
- title_jp -> title_jpn
- visible -> expunged
- gallery_size -> filesize
- count -> filecount
Also changes the function of the 'metadata' option.
It is now boolean and causes extra data fields from the API to be added
instead of completely replacing the data from HTML when activated.
4 years ago
996bfe4d4b
[hentaicafe] fix manga extractor
...
was broken since 993856b8
4 years ago
5d69e437d0
[twitter] add option to download all media from a conversation
...
(fixes #1319 )
4 years ago
cf5fa75d4c
add 'browser' option ( #1117 )
...
- change default user agent to Firefox ESR 78 on Windows 10
- remove 'ciphers' option
4 years ago
e5735361ed
[exhentai] add 'metadata' option ( #1325 )
...
to select between gallery metadata from 'api' or 'html'
4 years ago
8f095a0980
[exhentai] extract more metadata from gallery pages ( #1325 )
4 years ago
ffce8d85e7
[cyberdrop] update
...
- add test and archive_fmt
- extract more metadata
4 years ago
de0656941b
[twitter] add extractor for followed users ( #1337 )
...
https://twitter.com/USER/following or
https://twitter.com/id:USERID/following
4 years ago
e39aea42cd
fix supportedsites.py for modules without docstring
...
(fixes #1332 )
4 years ago
7b5ee922b7
cyberdrop extractor ( #1328 )
...
* create cyberdrop extractor
* add cyberdrop to list of extractors
* fix formatting
* change class name from CyberdropExtractor to CyberdropAlbumExtractor
* add cyberdrop to list of supported sites
* attempt to clean up diff of supportedsites.rst
* replace regex with functions from text library
4 years ago
5ad2b9c82b
[deviantart] extend 'extra' option
...
also download from embedded DeviantArt posts
4 years ago
6b0ecbf6bc
[hentainexus] add 'orignal' option ( #1322 )
4 years ago
5542a11c46
[twitter] update GraphQL endpoints
4 years ago
e1a12761d7
strip '/' from instance root URLs
4 years ago
595bdaa4be
add extractors for gelbooru v0.1 sites
...
- support https://illusioncards.booru.org/ (closes #426 )
- support https://the-collection.booru.org/ (closes #767 )
- support https://allgirl.booru.org/
- closes #234 , closes #473 , closes #1238
To get gallery-dl to recognize other sites running Gelbooru v0.1
(most sites on booru.org), add one or more entries to the
'gelbooru_v01' block in your config file. For example:
{
"extractor": {
"gelbooru_v01": {
"rozenmaidenbooru": {"root": "http://rm.booru.org "},
"drawfriendsbooru": {"root": "http://drawfriends.booru.org "}
}
}
}
4 years ago
59fd740b47
[tbib] add support for https://tbib.org/ ( #473 , closes #1082 )
4 years ago
08d7934c6e
move extractors from booru.py into their own gelbooru_v02 module
4 years ago
d656892670
remove cloudflare.py
...
The old IUAM challenge doesn't get used anymore, i.e. code to bypass it
is pointless, and the 'is_...()' checks are simple enough to directly
include them in 'extractor.request()'.
4 years ago
65ca923b4e
fix 'whitelist' option for BaseExtractor instances
4 years ago
fbfcbcbf57
Merge branch '1.17.0'
4 years ago
ba693d8686
[patreon] skip posts without view permission ( #1316 )
4 years ago
dcbd995346
[vanillarock] fix metadata extraction
4 years ago
4b1cda4cf7
[paheal] fix metadata extraction
4 years ago
2919d78bfc
update extractor test results
4 years ago
8974f0361c
[pixiv] update ( #1304 )
...
- remove login with username & password
- require a refresh token
- add 'oauth:pixiv' functionality
See also:
- https://github.com/upbit/pixivpy/issues/158
- https://gist.github.com/ZipFile/c9ebedb224406f4f11845ab700124362
4 years ago
79c0fc249b
[mangadex] add 'api-server' option ( #1309 )
...
and change the API server back to 'https://mangadex.org/api ' for now
4 years ago
96a51ff169
[sankaku] update invalid-token detection ( fixes #1309 )
4 years ago
23be48427c
[deviantart] fix 'folders' option ( closes #1302 )
...
don't assume parent folders are listed before their children
4 years ago
ca6b0fc2ac
[imagehosts] cleanup
4 years ago
Mike Fährmann
