93ea8ca8e3
[imxto] extract additional metadata ( #3175 )
2 years ago
e3abab8629
[weibo] send 'Referer' headers ( #3188 )
2 years ago
6423f990de
[realbooru] fix 'tags' extraction ( #2530 )
2 years ago
ecad02cf3f
[realbooru] fix download URLs ( #2530 )
2 years ago
15cd114c9c
[twitter] update bookmarks pagination ( #3172 )
...
Do not stop when there aren't any tweets in a batch,
but only when the same cursor value appears twice in a row.
2 years ago
20fbba9d7c
[exhentai] add metadata to search results ( #3181 )
...
'gallery_id' and 'gallery_token'
2 years ago
6a0c5e34f4
[exhentai] fix pagination ( #3181 )
2 years ago
171262c1b6
[instagram] remove login support
...
broken feature that I cannot get to work anymore
2 years ago
93e6bd6847
[uploadir] use utf-8 filenames ( #3162 )
2 years ago
b7a83ac726
[uploadir] update ( #3162 )
...
- prevent extra HTTP request from redirects
- add 'id' metadata field
- set 'filename_fmt' and 'archive_fmt'
2 years ago
ccb80f1b8b
[uploadir] add support for 'uploadir.com' ( #3162 )
2 years ago
b0cb4a1b9c
replace 'text.extract()' with 'text.extr()' where possible
2 years ago
4fd3c893fa
[booru] adjust/match '_tags' and '_notes' code
2 years ago
88954aa2e4
[gelbooru_v02] implement 'notes' extraction
...
same code as for 'moebooru' works here as well
2 years ago
4e80d3210e
[tumblr] Fallback to `gifv` when possible ( #3095 ) ( #3159 )
2 years ago
9d3f86dbcd
[twitter] update URL for syndication API ( #3160 )
...
Twitter changed the URL format to access tweet data through their syndication API.
2 years ago
c01cad599a
[lolisafe] add support for xbunkr ( #3156 )
2 years ago
9fc142d27b
[mastodon] add "remote_instance" field ( #3119 )
...
Example Usage:
If the url is "mastodon:https://mastodon.example.org/@VoteChess@botsin.space the "remote_instance" will be "botsin.space"
...
"directory": ["mastodon", "{remote_instance|instance}", "{account[username]!l}"]
...
2 years ago
2a1cb403ee
Revert "[Deviantart] [ #1776 ] Remove the "you need session cookies to download mature scraps" warning ( #1777 )"
...
This reverts commit 1f02878351
2 years ago
86790da2d5
update Cloudflare IUAM detection
...
again
2 years ago
775895f44b
[booru] refactor 'tags' and 'notes' extraction
...
- move HTML request for post pages into its own function
- move gelbooru_v02.py notes extraction to gelbooru.py
since it only works there
- clean up some code
2 years ago
0f9dfb7e62
[instagram] Fix AttributeError on user stories extraction ( #3123 )
2 years ago
f81dd5297a
[skeb] fix extraction ( #3112 )
...
'completed_at' is no longer included in API responses
2 years ago
fb2dbb04e2
[moebooru] extract 'notes' ( #3094 )
2 years ago
4e26bf98f5
[aibooru] support 'safe' subdomain ( #3110 )
2 years ago
5c31791b3c
[mastodon] support '/web/' URLs ( #3109 )
2 years ago
9a2cfd4421
[mastodon] support cross-instance user references ( #3109 )
2 years ago
58d97188b4
[mastodon] add 'bookmark' extractor ( #3109 )
2 years ago
46b64251eb
[bcy] fix extraction ( #3103 )
...
- fix regex for non-watermarked images
- fetch data from '/item/detail' pages for all other posts,
since '/apiv3/user/selfPosts' only has incomplete data
2 years ago
77173694d5
[kemonoparty] fix 'dms' extraction ( #3106 )
2 years ago
f168ec9572
[instagram] extract 'coauthors' metadata ( #3107 )
2 years ago
7c6af27eb8
[tumblr] add 'fallback-*' options ( #2957 )
...
specifically 'fallback-delay' and 'fallback-retries'
and change default number of retries to 2 (down from 3)
2 years ago
4aa56d500b
[hentaihere] fix test results
2 years ago
75d707fd92
[hentaihere] update
...
- support minor versions in chapter URLs
- fix manga metadata extraction
- update tests
2 years ago
d2fc73f20b
[hentai2read] fix manga metadata extraction
...
and update tests
2 years ago
f4d06e5180
[manganelo] update domain to 'chapmanganato.com' ( #3097 )
2 years ago
769e6754dc
[pixiv] use 'exact_match_for_tags' as default search mode ( #3092 )
2 years ago
a90e5cb354
[instagram] support 'instagram.com/s/' highlight URLs ( #3076 )
2 years ago
fd19c4b228
[hentai2read] recognize '.' in chapter ( #3089 )
2 years ago
2ff1897421
[vichan] recognize board url w/o trailing slash ( #3087 )
2 years ago
ac6111e693
[mangasee] add support for 'mangalife' ( #3086 )
2 years ago
300bc03deb
[gelbooru] allow alternate parameter order in post URLs ( #2821 )
2 years ago
a7d23f1484
[vichan] add generic extractors for vichan imageboards
...
includes 8kun.top, smuglo.li, and wikieat.club
2 years ago
04d3ebdfb4
[redgifs] fix 'token' extraction ( #3080 , #3081 )
2 years ago
062ef238a6
add support for aibooru (using danbooru extractor) ( #3075 )
2 years ago
0163ca86f7
[smugloli] add smugloli extractors ( #3060 )
2 years ago
cf86f68864
[instagram] add 'avatar' extractor ( #929 , #1097 , #2992 )
2 years ago
ea8113ff36
[reactor] match 'best', 'new', 'all' URLs ( #3073 )
2 years ago
618c81afdf
[ngomik] remove module
...
"Access denied"
2 years ago
94a2dfe205
[kemonoparty] update pagination offset
2 years ago
52d1eb928d
[pixiv] extend 'metadata' option ( #3057 )
...
make it usable for all 'pixiv' extractors
2 years ago
0714274f1f
[instagram] remove 'channel' extractor
2 years ago
d0d4ce1a13
[danbooru] fix ugoira metadata extraction ( #3056 )
2 years ago
096b8f2cfc
[instagram] prevent request for private '/tagged' feeds ( #3045 )
2 years ago
3b369ce3d1
[nijie] add 'followed' extractor ( #3048 )
2 years ago
c4a62a48ae
[nijie] add 'feed' extractor ( #3048 )
2 years ago
d1314df6e6
[nozomi] fix extraction ( #3051 )
2 years ago
277be410a7
[2chen] update 'archive_fmt'
2 years ago
ed55bd3a5c
[redgifs] extract Bearer token ( #3037 )
2 years ago
e974c75083
[redgifs] fix extraction ( #3037 )
...
send public Bearer token as 'authorization' header
2 years ago
68466a7d61
[tumblr] support ' https://www.tumblr.com/BLOGNAME ' URLs ( #3034 )
2 years ago
b6a68f5a4b
[fanbox] extend 'content' test result ( #3020 )
2 years ago
f1f89b2436
[tumblr] add 'offset' option
2 years ago
827ab0a62d
[instagram] fix login
...
- use mobile user agent header
- update general headers
- skip /data/shared_data/ step
2 years ago
1ca6be8619
[fanbox] add 'content' metadata field ( #3020 )
2 years ago
e5d229c524
[tumblr] sleep between fallback retries ( #2957 )
2 years ago
b2b0b1c455
[hitomi] fall back to webp when format not available ( #3030 )
2 years ago
1696f68a68
[8chan] add 'thread' and 'board' extractors ( #2938 )
2 years ago
560f7b41d8
[vk] add 'tagged' extractor ( #2997 )
2 years ago
122e1a467a
[vk] unescape error messages
2 years ago
bc9d291c13
[imagefap] fix and improve folder extraction ( #3013 )
2 years ago
55fca5fe4b
[imagefap] fix and improve gallery pagination ( #3013 )
2 years ago
8b1fe0bcf1
emit debug logging messages before calling time.sleep() ( #2982 )
2 years ago
14717f3fc9
[deviantart] add 'group' option ( #3018 )
...
disabling this option allows to better download from deleted accounts
2 years ago
220a04a74a
[artstation] skip missing projects ( #3016 )
2 years ago
a12ce2bb41
[deviantart] fix 'deviation' extraction ( #2981 )
2 years ago
36afb519b3
[instagram] prevent crash on empty user profile
2 years ago
f0321f423d
[2chen] Add 2chen.moe extractor ( #2707 )
...
* [2chen] Add 2chen.moe extractor
* change "==" to is
* fix for "test_unique_pattern_matches"
* fix regex pattern and group matching
* fix regex again
* [2chen] add 'reply_no' and 'hash' metadata and change 'filename_fmt'
also made an entry in supportedsites.md
* [2chen] unescape 'title'
* [2chen] partition() -> rpartition()
* [2chen] extract 'date' and 'name' metadata
* [2chen] remove 'offset' argument
* [2chen] do some changes
* [2chen] do some more changes
* [2chen] unescape 'name' and 'filename'
2 years ago
f7ba19a1c0
[nana] add 'nana' extractors ( #2967 )
2 years ago
fce6642699
[instagram] restore warnings for private profiles ( #3004 )
2 years ago
3e65645cfa
[instagram] restore 'cursor' functionality ( #2991 )
2 years ago
b8d268f57e
allow '/' and '?' in URL queries
2 years ago
7b5dad075d
[fappic] fix extraction
2 years ago
78694a61bb
[kemonoparty] restore 'favorites' API endpoints ( #2994 )
2 years ago
5fd4374036
[sankaku] improve 429 and tag limit handling
2 years ago
b84982b2f9
[kemonoparty] send Referer headers ( #2989 , #2990 )
2 years ago
98f67ae333
[instagram] add 'count' metadata field ( #2979 )
2 years ago
4089bceddd
[sankaku] implement 'refresh' option ( #2958 )
2 years ago
779e75c6f8
[kemonoparty] fix attachment IDs overwriting post IDs ( #2984 )
...
regression from 09a5cc61
2 years ago
e1d714943b
[tumblr] catch exception when updating image token ( #2957 )
2 years ago
e3a03f335c
[instagram] fix GraphQL bugs
2 years ago
6c76b5f90c
[deviantart] fix extraction ( #2981 , #2983 )
...
send a 'csrf_token' with every Eclipse API request
2 years ago
f728b5ca06
[tumblr] add fallback for failed higher-resolution images ( #2957 )
2 years ago
6992d01e19
[artstation] support search filters ( #2970 )
2 years ago
194803f3a7
[plurk] fix extraction ( #2977 )
2 years ago
63e0924927
[pixiv] add 'series' extractor ( #2964 )
2 years ago
aafea0c4f8
[artstation] fix searches ( #2970 )
2 years ago
2c67bee5c4
[instagram] update
...
- reorder some functions and extractors
- add missing GraphQL endpoints
- fix some GraphQL bugs
2 years ago
aa49bf13d2
[instagram] add 'api' option
2 years ago
6f77193a24
[instagram] move API related code into separate classes
...
may contain bugs and is probably incomplete for the GraphQL variant
2 years ago
ac45ed2764
[skeb] implement 'filters' option ( #2945 )
2 years ago
32c30754d1
[tumblr] warn when unable to fetch higher-resolution images ( #2957 )
...
and download the smaller version
instead of failing with a 404 error
2 years ago
ff532d6c3c
[newgrounds] extract 'type' metadata
2 years ago
0393e59535
[newgrounds] add 'games' extractor ( #2955 )
2 years ago
68f11e02a9
[skeb] add 'search_tags' metadata to search results ( #2945 )
2 years ago
1378cbb8dd
[myportfolio] use fallback when no images are found ( #2959 )
2 years ago
850608551c
[sankaku] detect expired links ( #2958 )
2 years ago
09a5cc6103
[kemonoparty] add 'count' metadata field ( #2952 )
2 years ago
89610a49dc
[instagram] use REST API endpoint for user feeds ( #2666 )
...
With this change, everything is using the newer REST API endpoints
providing higher-quality photos except the now obsolete '/channel' feed.
2 years ago
6737499dbd
[instagram] use REST API endpoint for saved posts ( #2911 )
...
provides 'username' and 'fullname'
as well as higher-quality images
2 years ago
50e3179c56
[instagram] update _user_by_screen_name()
...
use REST API
2 years ago
3dacfb3c56
[instagram] update API headers
2 years ago
4b2a006871
[skeb] add 'search' extractor ( #2945 )
2 years ago
94b34f460e
[exhentai] add slash to the end of gallery URLs ( #2947 )
2 years ago
2787c8511a
[mastodon] warn about moved accounts ( #2939 )
2 years ago
d699310fdf
[blogger] add 'label' or 'query' metadata fields ( #2930 )
...
for '/search/label/…' or '/search?q=…' URLs
2 years ago
eef50c1f28
[blogger] split 'search' extractor ( #2930 )
2 years ago
d29fb94098
[bunkr] use 'media-files' servers for m4v and mov files ( #2925 )
2 years ago
bd846abba0
[hotleak] add hotleak extractor ( #2909 ) ( #2890 )
2 years ago
e99a9b2aff
[twitter] improve 'cards-blacklist' ( #2875 )
...
allow blacklisting domains and 'name:domain',
where 'domain' depends on a card's 'vanity_url' value
2 years ago
aaf6992bae
[twitter] fix new-style '/card_img/' URLs
2 years ago
40baa77630
[twitter] provide proper 'date' for syndication results ( #2920 )
2 years ago
46fe469c53
[tumblr] implement 'ratelimit' option ( #2919 )
2 years ago
d0b73fec14
[flickr] add support for secure.flickr.com ( #2910 )
2 years ago
35eddaa94e
[reddit] prevent exception with empty submission URLs ( #2913 )
2 years ago
464ea90d14
[exhentai] guess extension for original files ( #2842 )
...
makes it possible to sometimes, when guessed correctly ('.jpg'),
skip an original file download without costing image limit points
2 years ago
551fdf7ad7
[exhentai] move 509 check into its own function
2 years ago
7a799df17f
[tumblr] pre-compile regular expressions
2 years ago
73a52a95b0
update Cloudflare IUAM detection
2 years ago
673b6f1218
[poipiku] use 'img-org.poipiku.com' as image domain ( #2796 )
2 years ago
4ca1a6e5f3
[bunkr] fix extraction ( #2903 )
2 years ago
8b76149521
[exhentai] improve 509.gif detection ( #2901 )
2 years ago
2ed58029f9
{paheal[ add proper support for videos ( #2892 )
2 years ago
444dfb4aa6
[instagram] add 'highlight_title' and 'date' metadata
...
to highlight posts (#2879 )
2 years ago
7f764ebee6
[redgifs] "fix" download URLs ( #2884 )
2 years ago
3cb8327c60
[zerochan] add 'metadata' option ( #2861 )
2 years ago
9745b48830
[tumblr] attempt to fetch high-quality inline images ( #2877 )
...
* [tumblr] attempt to fetch high-quality images (again)
Fixes #1846 , and fixes #1344
* slight refactor
* update configuration.rst entry
2 years ago
daef91c925
[smugmug] update default API credentials ( #2881 )
...
The old key lacked v2 access and I'm unable to accept
the new terms of service since my old account got deleted
2 years ago
4d78ca89db
[twitter] add 'cards-blacklist' option ( #2875 )
2 years ago
4d7cb0bf56
[twitter] general support for unified cards ( #2875 )
...
just removing the 'type' check seems to work
2 years ago
7ddfff957c
[twitter] support "image_website" unified cards ( #2875 )
2 years ago
2eb0ddd083
[hitomi] fix error when number of tag results is multiple of 25
...
(#2870 )
2 years ago
3cebf787c4
[slideshare] fix metadata extraction
2 years ago
da11fb32d0
update extractor test results
2 years ago
636d03df95
[nijie] reduce cache maxage to 90 days
2 years ago
f375ec0ffa
[vsco] fix 'collection' extraction
2 years ago
8672f8a2b9
[skeb] fix archive_ids for thumbnails and article images
...
8cf5981ded (commitcomment-82316040)
2 years ago
69995d789b
Revert "[twitter] use '{author[name]' in default directory names"
...
This reverts commit 9ad3cdc5d8
2 years ago
946643c23c
[hitomi] use maxage for gg.js cache ( #2863 )
...
cached values become invalid after 1-2 hours
2 years ago
d508b2c049
[gelbooru] implement 'pool' pagination ( #2853 )
2 years ago
67a2efb885
[rule34] implement 'pool' pagination ( #2853 )
2 years ago
70dc4ce911
[skeb] ignore article images with empty URL
...
8cf5981ded (commitcomment-81980633)
2 years ago
f362d4a3c7
[e621] fix 'popular' extraction
2 years ago
7e385ed63e
[foolfuuka] update domains
...
- remove nyafuu
- add rozenarcana (https://archive.alice.al/ )
- add tokyochronos (https://www.tokyochronos.net )
2 years ago
6ba72b6bc6
[twitter] ignore invalid user entries ( #2850 )
2 years ago
e4cff67aaa
[tumblr] add count metadata field ( #2804 )
...
Fixes #2778
2 years ago
574e38a287
[kemonoparty] add 'favorites' option ( #2826 ) ( #2831 )
...
* [kemonoparty] add 'favorites' option (#2826 )
* [kemonoparty] add regex for the url parameter and fallback on the config
option
* [kemonoparty] simplify
2 years ago
a799fae2df
[catbox] add 'album' extractor ( #2410 )
...
adapted from https://github.com/mikf/gallery-dl/pull/2805
- rewrite using GalleryExtractor
- extract more metadata
- match lolisafe names
- add test
2 years ago
264f1336ad
[twitter] unescape '+' in search queries ( #2226 )
...
... and do not raise exception if searched user does not exist
2 years ago
21ff77fea0
[zerochan] extract more metadata for single posts
...
Neither HTML pages nor RSS feed entries have *all* metadata.
It might be necessary to do 1-2 extra HTTP requests to grab everything.
2 years ago
391aecf219
[instagram] provide 'date' for directories ( #2830 )
2 years ago
9ea521bec5
[oauth] add host config option ( #2806 )
2 years ago
9ad3cdc5d8
[twitter] use '{author[name]' in default directory names
...
with the changes to 'user' (749802c7
2 years ago
1c89ccb27d
[weibo] prevent errors when paginating over album entries ( #2817 )
2 years ago
a27b17481f
[tumblr] restrict condition for calling _original_image
2 years ago
df1c643dda
[tumblr] attempt to extract full-resolution photos
...
- for photos with apparent width == 2048 or height == 3072
- can be disabled with 'original' option
2 years ago
a69f198ac9
[wallhaven] implement 'metadata' option ( #2803 )
2 years ago
033faf67ec
[poipiku] update filter for static images ( #2796 )
2 years ago
b9a14b5758
[poipiku] fix posts with more than one image ( #2796 )
2 years ago
0f5826e884
[weibo] prevent exception for missing 'playback_list' ( #2792 )
2 years ago
81a37d21d3
[twitter] simplify 'user' assignment
2 years ago
8a70b94245
[twitter] implement constant 'user' for tweet URLs
2 years ago
8cf5981ded
[skeb] add option to download 'article' images ( #1031 )
2 years ago
43ec315a7f
[deviantart] use public access token for journals ( #2702 )
...
and retry with a private token if needed
2 years ago
3f08a91131
[bunkr] fix extraction ( #2788 )
...
... again
2 years ago
5038893cdd
[blogger] emit metadata for posts without files ( #2789 )
2 years ago
98af5a0409
[zerochan] implement login with username & password ( #1434 )
2 years ago
3a8addfe45
[zerochan] add 'tag' and 'image' extractors ( #1434 )
2 years ago
e660e48a60
[vk] prevent exceptions for broken/invalid photos ( #2774 )
2 years ago
f559943d77
[instagram] fix empty 'params' in '_pagination_api()'
2 years ago
1540d0e695
[twitter] use filter:links ( #2766 )
2 years ago
8d0801ad8e
[twitter] fall back to unfiltered search ( #2766 )
2 years ago
0aa8345a13
[mastodon] allow downloading without access token ( #2782 )
...
Most mastodon instances allow accessing /api/v1/accounts/XXXX/statuses and api/v1/statuses/XXXX without an API access token.
This commit allows users to download at least some links from such a mastodon instance that does not already have access tokens hard-coded into the extractor.
User extractor only works on links that include the user id such as https://mastodon.tld/@id:12345 . Status links work as-is.
2 years ago
ea5ffb19a6
fanbox: download cover images in original size ( #2784 )
2 years ago
977d53b640
[Instagram] Add support for user's saved collection ( #2769 )
...
* [Instagram] Add support for user's saved collection
* [Instagram] Run formatter
* [Instagram] Simplify collection_id retrieval and add metadata
* [Instagram] Fix bug when params is not passed to _pagination_api
2 years ago
5b63df46c0
[tumblr] attempt to get higher-quality images ( #2761 )
2 years ago
59b16b3f70
[artstation] add 'num' and 'count' metadata fields ( #2764 )
2 years ago
eb68d45544
add global 'warnings' option ( #2762 )
2 years ago
f225247670
[gelbooru] add support for `api_key` and `user_id` ( #2767 )
2 years ago
77bdd8fe0f
[twitter] implement constant 'user' for 'from:…' searches
2 years ago
a267a05a3f
[twitter] update 'quote_id' and 'quote_by'
...
- 'quote_id' is now non-null for quoted Tweets and has the ID of the
quoting Tweet, instead the other way round like before
- 'quote_by' is now the 'screen_name' of the quoting user
(was the same the new 'quote_id' is now)
2 years ago
749802c7bd
[twitter] update 'user' and 'author' fields
...
- 'author' is always the user who authored a tweet
- 'user' is always the user specified in the input URL
or equal to 'author' when the former is not given
2 years ago
a566e63cdf
[tumblr] support '/blog/view' URLs ( #2760 )
2 years ago
46f11a3118
[bunkr] fix extraction ( #2732 )
...
move bunkr.is code to its own module
2 years ago
baf3815ebd
[nozomi] small code optimizations
2 years ago
836402bf58
[twitter] unescape content ( #2756 ) ( #2757 )
...
Fixes #2756
2 years ago
62cc47755b
[nozomi] reduce memory consumption during searches ( #2754 )
...
only load and use the entire 'index.nozomi' database
if there are only negative search terms
2 years ago
467a2a4d35
[instagram] add 'pinned' metadata field ( #2752 )
...
'pinned' is a list of user IDs for which a post is pinned
and empty if not pinned anywhere.
2 years ago
fe2b3d57d4
[komikcast] update domain
2 years ago
4e11ca737e
[hentaifoundry] fix metadata extraction
2 years ago
enduser420