0b34a444e0
[pixiv:novel] only detect Pixiv embeds ( #4175 )
1 year ago
9f1aee3884
[vipergirls] limit number of requests per second ( #4166 )
1 year ago
21c75d03a3
merge #4133 : [furaffinity] extract 'favorite_id' metadata
1 year ago
5e3a1749c8
[furaffinity] simplify 'favorite_id' assignment
1 year ago
ad882291d3
[instagram] fix retrieving '/tagged' posts ( #4122 )
...
reduce number of retrieved posts per API request from 50 to 20
1 year ago
0a9aaa7a8d
[weibo] prevent fatal exception due to missing video ( #4150 )
1 year ago
ac651c604c
[senmanga] fix and update ( #4160 )
1 year ago
df106fb58b
[bunkr] fix video downloads
1 year ago
aad5e6490c
merge #4159 : [bunkr] update domain to bunkrr.su
1 year ago
e0522ffb3d
[bunkr] update
1 year ago
e04796e04b
merge #3447 : [jschan] add generic extractors for jschan imageboards
1 year ago
b9692341fe
[jschan] update
1 year ago
a7c066cbac
Update bunkr.py
1 year ago
72e697b8b5
Update bunkr.py
...
Support bunkrr.su
1 year ago
4ae925c88f
[kemonoparty] support '.su' TLD ( #4139 )
1 year ago
2d9e3093ca
merge #4134 : [postimage] add gallery support, update image extractor
1 year ago
e64b521287
merge #4136 : [acidimg] fix extractor
1 year ago
a90974178d
[jpgfish] update domain to 'jpg.pet' ( #4138 )
1 year ago
ee959052ac
merge #4138 : add jpg.pet as alias for jpgfish
1 year ago
0281cc7d08
[fanbox] skip 404ed fanbox embeds ( #4088 )
...
continuation of 4fc9675d
1 year ago
97c0d13cbb
add jpg.pet as alias for jpgfish
1 year ago
2e309a13a7
[acidimg] fix extractor
1 year ago
92178b369c
[postimage] add gallery support, update image extractor to download
...
original image instead of main image
1 year ago
952c03bc9e
Add fav_id data to FuraffinityFavoriteExtractor
...
An extra field is collected when paginating favorites, and saved to
a temporary cache variable. This field is identical for both the old
and the new page layouts for FurAffinity, but can only be collected
during pagination, hence the cache variable. Other FurAffinity
extractors should be unaffected by this change.
1 year ago
54cf1fa3e7
[twitter] use GraphQL search endpoint ( #3942 )
...
for guest users; selectable with 'search-endpoint' option.
adapted from 9c7b888ffa
1 year ago
864a654b25
[twitter] update query hashes
1 year ago
45cc7cee1a
[twitter] better error message for guest searches ( #3942 )
1 year ago
271f23d971
[twitter] extract 'conversation_id' metadata ( #3839 )
1 year ago
94b6a67666
[reddit] fix crash with empty 'crosspost_parent_lists' ( #4120 )
1 year ago
0cf7282fa0
[pixiv] add 'full-series' option for novels ( #4111 )
1 year ago
bab13402df
[redgifs] update 'search' URL pattern ( #4115 )
1 year ago
5a6fd8027d
[redgifs] support galleries ( #4021 )
1 year ago
0ad59c92b1
[blogger] download files from 'lh*.googleusercontent.com' (4070)
1 year ago
ffed7efb6f
[pixiv] use BASE_PATTERN
1 year ago
b286efefcc
[pixiv] add 'novel-bookmark' extractor ( #4111 )
1 year ago
28f6487c64
[instagram] add 'metadata' option ( #3107 )
1 year ago
8cf13f8696
merge #4104 : [lensdump] add lensdump.com extractors
1 year ago
58f7480d46
[lensdump] update
...
- update docs/supportedsites.md
- add GPL2 header
- use BASE_PATTERN
- improve LensdumpImageExtractor
1 year ago
3516fdae74
[kemonoparty] fix kemono and coomer logins using the same cache
...
(#4098 )
1 year ago
d5300cf381
[lensdump] subcategory
1 year ago
82ba6bfdc0
[lensdump] f-string fix
1 year ago
9b2326e4e1
[lensdump] add lensdump.com extractor
1 year ago
148bdc04a4
merge #2719 : [jpgfish] add 'jpgfish' extractors
1 year ago
609c4f3e07
[jpgfish] simplify and improve
1 year ago
2b1f875ef4
[jpgchurch] update to 'jpgfish'
1 year ago
3d29c42142
[mangaread] fix 'tags' extraction
1 year ago
5f86527cbe
merge #2781 : [mangaread] Add Mangaread extractor
1 year ago
cdc6549fd2
merge #3329 : [8muses] Add 'parts' to album data
...
and fix 'album[url]'
1 year ago
ad760429b1
[8muses] update
1 year ago
d0184fddcf
[twitter] optimize '_extract_twitpic()'
...
- use findall instead of finditer
- store URLs in a dict to discard duplicates
1 year ago
3dc862c7fc
merge #3796 : [twitter] extract TwitPic URLs in text ( #3792 )
1 year ago
243de697b9
merge #3976 : [reddit] support cross-posted media ( #887 , #3586 )
1 year ago
f8c4c5eef9
[reddit] simplify and add tests
1 year ago
822a77d846
[danbooru] add support for booru.borvar.art instance
1 year ago
f3cca50b9e
[mangadex] update links to API docs
1 year ago
65a9f4b124
merge #3950 : [misskey] add 'favorite' extractor
1 year ago
c76f0f3a1b
[misskey] update
...
- rename to 'MisskeyFavoriteExtractor'
- add 'access-token' option to docs
- add test URLs for other instances
- simplify 'pattern'
1 year ago
3fca455b82
[pixiv] add 'embeds' option ( #1241 )
1 year ago
d1f2ef3b7b
[imagechest] update
...
- don't load HTML page when using API
- restructure some code
- add more methods to ImagechestAPI
1 year ago
856f6c10cd
allow for GalleryExtractors to skip loading gallery_url
1 year ago
4fc9675d48
[fanbox] skip 404ed or otherwise invalid posts ( #4088 )
1 year ago
56b8b8cd36
[pixiv] support short novel URLs
...
https://www.pixiv.net/n/ <ID>
1 year ago
e6f55d1555
[imagechest] add API support and 'access-token' option ( #4065 )
1 year ago
77abcf5ab3
[gofile] automatically fetch 'website-token' by default
...
the hardcoded token changed yet again
1 year ago
e3fed9bd17
[tcbscans] update domain to 'tcbscans.com' ( #4080 )
1 year ago
a83983c651
[instagram] add 'order-posts' option ( #4017 , #3993 )
1 year ago
d680623db3
[instagram] add 'order-files' option ( #4017 , #3993 )
1 year ago
f9b7a033e0
[misskey] refactor misskey extractor
1 year ago
04dbfd994e
[misskey] add my favorites extractor
1 year ago
82a12d6126
[nsfwalbum] detect placeholder images
...
patch by an anonymous contributor
1 year ago
011e4607c3
[poipiku] extract full 'descriptions' ( #4066 )
...
don't cut it off after the first line
1 year ago
5037013e2b
[gofile] update 'website-token' ( #4056 )
1 year ago
6b6bb4be73
[weibo] require numeric IDs to have length >= 10 ( #4059 )
1 year ago
494acabd38
[danbooru] refactor pagination logic ( #4002 )
...
- only use 'b<ID>' when no other order is specified
- support 'a<ID>' when using 'order:id' as tag
1 year ago
fd0e1ffd6e
[danbooru] improve 75666cf9 ( #4002 )
...
Search for direct post IDs instead of trying to
replicate the same results as the initial request.
1 year ago
e41e45ff6b
[gofile] add basic password support ( #4056 )
1 year ago
20dc13f832
[pixiv] initial 'novel' support ( #1241 , #4044 )
...
supported URLs are
- https://www.pixiv.net/novel/show.php?id= <ID>
- https://www.pixiv.net/novel/series/ <ID>
- https://www.pixiv.net/en/users/ <ID>/novels
1 year ago
c698c3de44
[newgrounds] add default delay between requests ( #4046 )
1 year ago
708f478d15
[danbooru][e621] add 'date' metadata field ( #4047 )
1 year ago
35c23a2fd8
merge #4031 : [mangadex] add 'status' and 'tags' metadata
1 year ago
2266fc8cc5
[mangadex] update and extend test results
1 year ago
1ce5dc9e18
fix whitespaces
1 year ago
13dedae09f
add status and tags info to mangadex extractor
1 year ago
be0fa94b2e
[imagechest] load all images when a 'Load More' button is present
...
(#4028 )
1 year ago
7eadcbea70
[4chanarchives] add end condition for 'board' extractor ( #4012 )
1 year ago
1406f7125f
[4chanarchives] add 'thread' and 'board' extractors ( #4012 )
1 year ago
d12dd3813c
[imgur] fix internal image/album URLs
...
URLs from "link" attributes of newer images/albums were all returned
as 'https://imgur.com/gallery/ ...' instead of the expected format,
causing them to be ignored.
1 year ago
8520de57f0
[imgur] add 'favorite-folder' extractor ( #4016 )
1 year ago
3ca5dac8b6
extend 'cookies-update' functionality
...
Allow writing cookies to a different file than a given cookies.txt,
making it possible to export cookies imported with --cookies-from-browser
To convert browser cookies to cookies.txt format:
gallery-dl --cookies-fr chromium \
-o cookies-update=cookies.txt \
--no-download \
http://example.org/file.jpg
1 year ago
bc6d65d203
implement 'Extractor.config_deprecated()'
...
a version of 'Extractor.config()'
that logs a warning when using a deprecated option name
1 year ago
850df34c31
remove '&' from URL patterns part 2
...
follow-up on 968d3e8465
1 year ago
4d415376d1
[pinterest] fix 'pin.it' extractor
...
it really was just the single '/' at the end of the url_shortener URL
1 year ago
657b6a9100
[pinterest] update endpoint for related board pins
1 year ago
79f47f98dd
[nana] remove module
...
permanently gone since 2023-03-13
1 year ago
0e74df1de8
[420chan] remove module
...
offline since 2022-06-01
1 year ago
7499fa7075
[exhentai] remove and update sad panda check
...
there hasn't been a sad panda in several years
1 year ago
076380e079
remove '*' indicating keyword-only arguments
...
they are kind of unnecessary and
cause a non-insignificant function call overhead (~10%)
1 year ago
0c46758a93
[foolslide] remove 'sensescans.com'
...
group moved to mangadex
https://mangadex.org/group/1071e71d-cc55-4fa6-81d1-4b5913a2fde5/sense-scans
1 year ago
a08fdfac6e
[foolfuuka] add 'archive.palanq.win'
1 year ago
1870df8b23
[foolfuuka] remove 'tokyochronos.net'
1 year ago
ef4e2d8178
[foolfuuka] remove 'archive.alice.al'
1 year ago
b12dad8df5
[pixiv] fix 'pixivision' extraction
1 year ago
5fb7107f2b
[imxto] fix 'gallery' extraction
...
support both single and double quotes
1 year ago
15d7c5a199
[behance] 'items()' -> 'values()'
...
we only need 'size', 'name' is unnecessary
1 year ago
0fb580135d
[behance] fix extraction ( #3980 )
1 year ago
d4f8b2fe22
fix: linter issues
1 year ago
1b918bd937
fix(extractor): fix extraction for cross-posted reddit videos and galleries
1 year ago
215028a462
[manganelo] match more minor version separators ( #3972 )
1 year ago
9f76783ac0
[pixiv] allow sorting by popularity (requires pixiv premium)
1 year ago
7865067d19
[shimmie2] add generic extractors for Shimmie2 sites ( #3734 )
...
add support for
- loudbooru.com (#3734 )
- booru.cavemanon.xyz (#3734 )
- giantessbooru.com (#943 )
- tentaclerape.net
1 year ago
28419bf45a
[itchio] add 'game' extractor ( #3923 )
1 year ago
5297ee0cd9
[tumblr] add 'day' extractor ( #3951 )
1 year ago
de670bd7de
[tumblr] update pagination logic ( #2191 )
1 year ago
98c9fdb414
[deviantart] revert e9353c63; retry downloads with private token
1 year ago
5d7435e803
[nitter] extract user IDs from encoded banner URLs
...
still requires a banner to be present to begin with
1 year ago
7f25cab56e
[sankaku] support post URLs with MD5 hashes ( #3952 )
1 year ago
a05120412a
[oauth] catch exception from 'webbrowser.get()' ( #3947 )
...
It raises an exception instead of returning None
when no runnable browser is available.
1 year ago
3fc2223893
merge #3935 : [reddit] match 'preview.redd.it' URLs
1 year ago
1d505b39f8
[twitter] support 'profile-conversation' entries ( #3938 )
1 year ago
aaf58a1259
[imgur] document 'client-id' option ( #3937 )
1 year ago
202f5d86a7
[reddit] ignore 'id-max' value "zik0zj"/2147483647
...
(#3939 , #3862 , #3697 , #3606 , #3546 , #3521 , #3412 )
1 year ago
8586ee81be
[nana] fix 'keyword' tests
1 year ago
cd4bfb0dd1
[reddit] match 'preview.redd.it' URLs
1 year ago
faca32a850
[sankaku] sanitize 'date:…' tags ( #1790 )
1 year ago
6f1e34ec69
[vipergirls] add 'thread' and 'post' extractors
...
(#731 , #2720 , #3812 )
1 year ago
81bd2af83e
[2chen] update domain to sturdychan.help
1 year ago
f500b45b5e
[twitter] improve 480bc34e
...
only check for double user assignment where necessary
1 year ago
5b635f2317
[imxto] add 'gallery' extractor ( #1289 )
1 year ago
359e31e462
[nozomi] update file URLs ( #3925 )
...
Static images are now only available in WebP format over the 'w'
subdomain. GIFs also got their own 'g' subdomain.
1 year ago
2dfd4a3de2
[imagefap] extract 'categories' metadata and fix empty 'tags'
1 year ago
480bc34e54
[twitter] do not overwrite previously assigned users ( #3922 )
1 year ago
02ec5bb8e5
[imagefap] extract 'description' metadata ( #3905 )
1 year ago
d253a3c542
merge #3841 : [urlshortener] add support for bit.ly & t.co
1 year ago
5e63942b37
[urlshortener] update
1 year ago
c45f09d2a8
[imagechest] fix extraction ( #3914 )
1 year ago
2cd4411ff8
[nitter] extract videos from 'source' elements ( #3912 )
1 year ago
9501579279
[sexcom] fix fetching HD videos
1 year ago
a2f7274eae
[sexcom] fix pagination ( #3906 )
1 year ago
e9353c63d6
[deviantart] keep using private access tokens
...
for deviations returned from a private API call
also fixes a bug from 0a7eee3e
1 year ago
e70af6a550
[hentaifoundry] do not update filters when cookies are provided
1 year ago
9c29c904c7
[mastodon] try to get account IDs without access token
...
Try to query the public '/api/v1/accounts/lookup' endpoint
and fall back to '/v1/accounts/search' if it returns an error.
'/api/v1/accounts/lookup' is available since Mastodon v3.4.0.
The version of an instance can be found at '/api/v1/instance'.
1 year ago
1614c5c4bf
[generic] write regular expressions without 'x' flags
1 year ago
d84a617273
[hentaifoundry] fix setting content filters ( #3887 )
1 year ago
875485313f
[urlshortener] force HTTPS
1 year ago
0a7eee3ee0
[deviantart] add 'public' option
1 year ago
f5a59c4170
[twitter] add 'date_bookmarked' metadata ( #3816 )
1 year ago
1c1f6fdc80
[twitter] fix regression from 160335ad
...
Tweets from 'homeConversation' or 'conversationthread' entries do not
contain a 'sortIndex' field. Accessing it raises a KeyError and would
erroneously get them labeled as 'deleted'.
1 year ago
160335ad44
[twitter] add 'date_liked' metadata for liked Tweets ( #3816 )
1 year ago
6d850ce629
[twitter] calculate 'date' from Tweet IDs
...
20 times faster than parsing 'created_at'
1 year ago
25949bd767
merge #3871 : [hotleak] Fix downloading of creators whose name starts with a category name
1 year ago
dbe06cdba1
[twitter] warn about 'withheld' Tweets and users ( #3864 )
1 year ago
3cc1dd1572
[twitter] update query hashes
1 year ago
3846ce0de5
[twitter] update to bookmark timeline v2 ( #3859 )
1 year ago
34699fbf64
[deviantart:search] detect login redirects ( #3860 )
1 year ago
e6cb92864a
[twitter] allow setting custom features per API endpoint
1 year ago
4b141cce66
Fix indentation
1 year ago
bbc5977121
Fix line length
1 year ago
ffd30abcb3
[hotleak] Fix downloading of creators whose name starts with a category name
...
E.g. `hot4lexi` would start downloading the `hot` section by mistake
This happened because the regex had a negative lookahead for the category names, but didn't ensure that they where followed by either end-of-string or a slash.
1 year ago
5ca9d55595
merge #3870 : [blogger] update 'sub' regex to get the highest resolution url
1 year ago
fd7ce4c081
merge #3868 : [shopify] fix 'collection' extractor
1 year ago
135ac9c302
merge #3854 : [twitter] fix: graphql_timeline_v2_bookmark_timeline cannot be null
1 year ago
bbb1e34c34
[blogger] update sub regex
1 year ago
96e3dd2128
[shopify] fix 'collection' extractor
1 year ago
ac97aca99c
[realbooru] fix extraction
...
get file URLs from HTML pages
1 year ago
75666cf9c3
[danbooru] reduce API requests for fetching extended 'metadata'
...
Instead of using one additional API request per post object (N+1),
this requires only one request per 200-post batch.
1 year ago
bebbff6578
fix: graphql_timeline_v2_bookmark_timeline cannot be null
...
twitter: 400 Bad Request (The following features cannot be null: graphql_timeline_v2_bookmark_timeline)
1 year ago
71b26adb9b
[urlshortener] add tinyurl.com as an example
2 years ago
421db26aff
[bunkr] update domain to 'bunkr.la'
2 years ago
9e2a945013
[urlshortener] add support for bit.ly & t.co
2 years ago
9b5e7ce8b9
[hiperdex] fix extraction
2 years ago
89a67c45e0
[nitter] support nitter.it ( #3819 )
2 years ago
88f29a751d
[nitter] skip broadcasts
...
instead of downloading an "Unsupported feature" HTML page
2 years ago
1e013eba5a
[nitter] fix extraction for instances without user banners
2 years ago
d94aa1ee02
[gelbooru] fix --range for favorites ( #3704 )
2 years ago
1f82b00b8f
[gelbooru] fix and improve --range for pools
2 years ago
197882cf12
[twitter] add 'hashtag' extractor ( #3783 )
2 years ago
9789ebac52
[naverwebtoon] fix extraction ( #3729 )
2 years ago
72f1f16eb2
[weibo] support 'mix_media_info' entries ( #3793 )
2 years ago
d4fb4ff47f
[twitter] extract TwitPic URLs in text ( #3792 )
...
also ignore previously seen URLs
2 years ago
2bb937014f
[twitter] fall back to legacy /media endpoint when not logged in
2 years ago
b68094d326
[twitter] support 'note_tweet's
2 years ago
3dcabc97ed
[twitter] update API endpoints and parameters
2 years ago
dcb8af659a
[gelbooru] extract favorites without needing cookies ( #3704 )
...
TODO: fix --range
2 years ago
b756dc13aa
[gelbooru] warn about missing cookies for favorites ( #3704 )
...
and add docstring so it shows up in --list-extractors
2 years ago
17bd053d94
[hiperdex] fix extraction ( #3768 )
2 years ago
817fc0fbd1
[nitter] remove nitter.pussthecat.org
...
"Shutdown"
2 years ago
67ec91cdbd
[downloader:http] change '_http_retry' to accept a Python function
...
and rename '_http_retry_codes' to '_http_retry'
(#3569 )
2 years ago
175822e065
merge #3738 : [generic] add tests
2 years ago
4883420e67
[generic] revert pattern change
2 years ago
9037128315
[twitter] fix some 'original' retweets not downloading ( #3744 )
2 years ago
ea3d95e7e8
merge #3740 : [deviantart] add support for fxdeviantart.com URLs
2 years ago
9abcb2b6e5
update headers and ciphers for '"browser": "chrome"'
2 years ago
c489aecb3e
[deviantart] add support for fxdeviantart.com URLs
...
fxdeviantart.com is a service that fixes embeds on Discord, similar to
fxtwitter.com
2 years ago
34a7fab0e2
[generic] add support for IDNs
...
(internationalized domain name)
2 years ago
c9a7345228
[newgrounds] prevent archive ID overlap ( #3681 )
...
add an 'i' and 'a' prefix to image and audio files
(/art/view/, /audio/listen/)
since their numeric ID may conflict with movies and other media
2 years ago
da9840a39d
[reddit] update 'videos' option ( #3712 )
...
- add 'dash' to directly extract DASH manifest URLs
(was default behavior since a7c79531a7c79531
2 years ago
baf41d7437
[misskey] update ( #3717 )
...
- add module docstring
- add options to docs/gallery-dl.conf
2 years ago
6762d99515
merge #3717 : [misskey] add misskey extractors
2 years ago
b8a702929d
[oauth] import extractor modules on demand
2 years ago
dd88740ec7
replace remaining instances of base64 with binascii
2 years ago
Mike Fährmann
