Mike Fährmann
e99ce99284
[danbooru] remove stray 'print()'
2 years ago
Mike Fährmann
ed49e63d95
[nitter] set 'hlsPlayback' cookie
2 years ago
Mike Fährmann
e081b1fac4
[nitter] sanitize filenames ( #3294 )
2 years ago
Mike Fährmann
e31d12139c
[nitter] add 'videos' option ( #3279 )
...
with the same semantics as for twitter
2 years ago
enduser420
8c4e21b110
[itaku] remove 'Extreme' rating ( #3287 )
2 years ago
Mike Fährmann
72c5d26e85
[hotleak] fix UnboundLocalError ( #3288 , #3293 )
2 years ago
Mike Fährmann
501d9bccfe
[artstation] add 'max-posts' option ( #3270 )
2 years ago
Mike Fährmann
b1ad6f2289
[artstation] add 'pro-first' option ( #3273 )
2 years ago
Mike Fährmann
5a17e15b76
[pixiv] preserve 'tags' order ( #3266 )
...
for '"tags": "translated"'
As it turns out, set() does *not* preserve insertion order.
2 years ago
Mike Fährmann
1392b44bfe
[inkbunny] provide additional metadata ( #3274 )
...
- 'pool_id' for pools
- 'favs_user_id' for favorites
- 'search[...]' for searches
2 years ago
Mike Fährmann
42481aed59
[formatter] implement 'S' format specifier ( #3266 )
...
to Sort lists
2 years ago
Mike Fährmann
8a021e4ee4
release version 1.24.0
2 years ago
Mike Fährmann
6b97dcf2e0
[postprocessor:metadata] add 'private' option
2 years ago
Mike Fährmann
a24dcbe802
[twitter] fix login ( #3220 )
...
Using an email as 'username' seems to no longer be possible,
as Twitter will always additionally ask for username or phone number
when providing an email address as 'username'.
2 years ago
Mike Fährmann
985fd398f5
[ytdl] update 'parse_bytes' location ( #3256 )
...
https://github.com/yt-dlp/yt-dlp/commit/64c464a
2 years ago
Mike Fährmann
226d778294
do not try to fetch 'http-metadata' for ytdl URLs ( #3257 )
2 years ago
Mike Fährmann
133412bd62
remove previous 'http-metadata' entries from kwdict
2 years ago
Mike Fährmann
53a5d95b7d
[instagram] skip private check for avatars ( #3255 )
2 years ago
Mike Fährmann
08fd1ff835
[twitter] add 'avatar' and 'background' extractors ( #349 , #3023 )
2 years ago
Mike Fährmann
46d811bac0
add loaded config files to debug output
2 years ago
Mike Fährmann
4c6379e9d5
fix typo
2 years ago
Mike Fährmann
6379157543
[instagram] use REST API by default
...
regardless of logged in status
2 years ago
Mike Fährmann
f87cfa5f66
[downloader:http] add signature check for .mp4 files
2 years ago
enduser420
7897f68225
[wallhaven] update 'user' extractor ( #3226 )
...
* [wallhaven] update 'user' extractor
* [wallhaven] update 'configuration.rst'
add 'extractor.wallhaven.include' entry
* [wallhaven] add 'wallhaven.include' in gallery-dl.conf
2 years ago
enduser420
5a68b5cb3c
[wallhaven] add 'user' extractor ( #3213 )
2 years ago
enduser420
442b03f7c3
[khinsider] fix song extraction ( #3219 )
2 years ago
Mike Fährmann
eaae4d9b65
[pixiv] stop with error for invalid search/ranking parameters
...
instead of falling back to defaults
2 years ago
Mike Fährmann
368f156378
[pixiv] rankings: add support for the new daily AI and daily AI R18
...
(#3214 , #3221 )
In remembrance of @thatfuckingbird
2 years ago
Mike Fährmann
6c153750fa
[nitter] add extractors for Nitter instances ( #2696 )
2 years ago
Mike Fährmann
374f14c28c
fix repeating paths for skipped files ( #3203 )
...
fixing the fix from e3260293
2 years ago
Mike Fährmann
9f06e79868
implement '"user-agent": "browser"' ( #2636 )
2 years ago
Mike Fährmann
70c7fbe89a
[instagram] add 'guide' extractor ( #3192 )
2 years ago
enduser420
93ea8ca8e3
[imxto] extract additional metadata ( #3175 )
2 years ago
Mike Fährmann
e3abab8629
[weibo] send 'Referer' headers ( #3188 )
2 years ago
Mike Fährmann
6423f990de
[realbooru] fix 'tags' extraction ( #2530 )
2 years ago
Mike Fährmann
ecad02cf3f
[realbooru] fix download URLs ( #2530 )
2 years ago
Mike Fährmann
a4ff20cf16
[downloader:http] fix issues from inaccurate 'time.sleep()'
...
(#3143 )
Reverts part of c59b98c8
by going back to using a global timer
instead of a per-chunk one.
Reintroduces the issue of ignoring rate limits after
suspending and resuming the process.
2 years ago
Mike Fährmann
15cd114c9c
[twitter] update bookmarks pagination ( #3172 )
...
Do not stop when there aren't any tweets in a batch,
but only when the same cursor value appears twice in a row.
2 years ago
Mike Fährmann
550f90ab56
delay enabling .part files when 'http-metadata' is set
...
otherwise 'build_path' gets called before all metadata is collected
2 years ago
Mike Fährmann
20fbba9d7c
[exhentai] add metadata to search results ( #3181 )
...
'gallery_id' and 'gallery_token'
2 years ago
Mike Fährmann
6a0c5e34f4
[exhentai] fix pagination ( #3181 )
2 years ago
Mike Fährmann
05255f5be0
add 'default' argument to 'text.extr()'
2 years ago
Mike Fährmann
e326029355
build path when skipping archived files
...
fixes bug from 8124c16a
2 years ago
Mike Fährmann
171262c1b6
[instagram] remove login support
...
broken feature that I cannot get to work anymore
2 years ago
Mike Fährmann
8124c16a50
split 'build_path' from 'set_filename' and 'set_extension'
...
Do not automatically build a new path
when setting file metadata or updating its extension.
2 years ago
Mike Fährmann
39d9c362e4
include 'http-metadata' in '-K' output
2 years ago
Mike Fährmann
e2401c96ee
[postprocessor:metadata] add '"mode": "jsonl"'
2 years ago
Mike Fährmann
895f36e53b
[postprocessor:metadata] add 'open' and 'encoding' options
2 years ago
Mike Fährmann
93e6bd6847
[uploadir] use utf-8 filenames ( #3162 )
2 years ago
Mike Fährmann
870e6a48a0
implement 'http-metadata' option
...
or at least attempt to.
2 years ago
Mike Fährmann
b7a83ac726
[uploadir] update ( #3162 )
...
- prevent extra HTTP request from redirects
- add 'id' metadata field
- set 'filename_fmt' and 'archive_fmt'
2 years ago
Mike Fährmann
ccb80f1b8b
[uploadir] add support for 'uploadir.com' ( #3162 )
2 years ago
Mike Fährmann
b0cb4a1b9c
replace 'text.extract()' with 'text.extr()' where possible
2 years ago
Mike Fährmann
eb33e6cf2d
add 'text.extr()'
...
a stripped-down version of text.extract() that
- always returns a string (like 'extract_from')
- only returns a string
- does not deal with 'pos' arguments
- is ~20% faster
2 years ago
Mike Fährmann
597b63d922
move git head functionality to function in util.py
2 years ago
Mike Fährmann
4fd3c893fa
[booru] adjust/match '_tags' and '_notes' code
2 years ago
Mike Fährmann
88954aa2e4
[gelbooru_v02] implement 'notes' extraction
...
same code as for 'moebooru' works here as well
2 years ago
Mike Fährmann
942bc84962
add '--chunk-size' command-line option ( #3143 )
2 years ago
Mike Fährmann
79a9fc6e45
add '--user-agent' command-line option
2 years ago
ClosedPort22
4e80d3210e
[tumblr] Fallback to `gifv` when possible ( #3095 ) ( #3159 )
2 years ago
thatfuckingbird
9d3f86dbcd
[twitter] update URL for syndication API ( #3160 )
...
Twitter changed the URL format to access tweet data through their syndication API.
2 years ago
enduser420
c01cad599a
[lolisafe] add support for xbunkr ( #3156 )
2 years ago
Allen
9fc142d27b
[mastodon] add "remote_instance" field ( #3119 )
...
Example Usage:
If the url is "mastodon:https://mastodon.example.org/@VoteChess@botsin.space the "remote_instance" will be "botsin.space"
...
"directory": ["mastodon", "{remote_instance|instance}", "{account[username]!l}"]
...
2 years ago
Mike Fährmann
bca9f965e5
[downloader:http] add 'chunk-size' option ( #3143 )
...
and double the previous default from 16384 (2**14) to 32768 (2**15)
2 years ago
Mike Fährmann
2a1cb403ee
Revert "[Deviantart] [ #1776 ] Remove the "you need session cookies to download mature scraps" warning ( #1777 )"
...
This reverts commit 1f02878351
.
Mature scraps do yet again require cookies.
2 years ago
Mike Fährmann
0059e2bfe7
[downloader:http] add MIME type and signature for .avif files
2 years ago
Mike Fährmann
f687e64513
[downloader:http] refactor file signature checks
...
use functions/lambdas instead of startswith()
2 years ago
Mike Fährmann
86790da2d5
update Cloudflare IUAM detection
...
again
2 years ago
Mike Fährmann
c12a97bcde
[postprocessor] add 'post-after' event ( #3117 )
2 years ago
Mike Fährmann
775895f44b
[booru] refactor 'tags' and 'notes' extraction
...
- move HTML request for post pages into its own function
- move gelbooru_v02.py notes extraction to gelbooru.py
since it only works there
- clean up some code
2 years ago
Luc Ritchie
0f9dfb7e62
[instagram] Fix AttributeError on user stories extraction ( #3123 )
2 years ago
Mike Fährmann
f81dd5297a
[skeb] fix extraction ( #3112 )
...
'completed_at' is no longer included in API responses
2 years ago
Mike Fährmann
b337e51e91
run flake8 on all .py files
2 years ago
enduser420
fb2dbb04e2
[moebooru] extract 'notes' ( #3094 )
2 years ago
Mike Fährmann
4e26bf98f5
[aibooru] support 'safe' subdomain ( #3110 )
2 years ago
Mike Fährmann
f037429fa4
attempt to improve '-K' output for lists
...
- use [N] instead if [] to indicate a Number needs to be placed there
- enumerate list items
2 years ago
Mike Fährmann
e140b85342
reword error text for unsupported URLs
2 years ago
Mike Fährmann
5c31791b3c
[mastodon] support '/web/' URLs ( #3109 )
2 years ago
Mike Fährmann
9a2cfd4421
[mastodon] support cross-instance user references ( #3109 )
2 years ago
Mike Fährmann
58d97188b4
[mastodon] add 'bookmark' extractor ( #3109 )
2 years ago
Mike Fährmann
46b64251eb
[bcy] fix extraction ( #3103 )
...
- fix regex for non-watermarked images
- fetch data from '/item/detail' pages for all other posts,
since '/apiv3/user/selfPosts' only has incomplete data
2 years ago
Mike Fährmann
77173694d5
[kemonoparty] fix 'dms' extraction ( #3106 )
2 years ago
Mike Fährmann
f168ec9572
[instagram] extract 'coauthors' metadata ( #3107 )
2 years ago
Mike Fährmann
7c6af27eb8
[tumblr] add 'fallback-*' options ( #2957 )
...
specifically 'fallback-delay' and 'fallback-retries'
and change default number of retries to 2 (down from 3)
2 years ago
Mike Fährmann
4aa56d500b
[hentaihere] fix test results
2 years ago
Mike Fährmann
75d707fd92
[hentaihere] update
...
- support minor versions in chapter URLs
- fix manga metadata extraction
- update tests
2 years ago
Mike Fährmann
d2fc73f20b
[hentai2read] fix manga metadata extraction
...
and update tests
2 years ago
Mike Fährmann
f4d06e5180
[manganelo] update domain to 'chapmanganato.com' ( #3097 )
2 years ago
Mike Fährmann
769e6754dc
[pixiv] use 'exact_match_for_tags' as default search mode ( #3092 )
2 years ago
Mike Fährmann
a90e5cb354
[instagram] support 'instagram.com/s/' highlight URLs ( #3076 )
2 years ago
enduser420
fd19c4b228
[hentai2read] recognize '.' in chapter ( #3089 )
2 years ago
enduser420
2ff1897421
[vichan] recognize board url w/o trailing slash ( #3087 )
2 years ago
enduser420
ac6111e693
[mangasee] add support for 'mangalife' ( #3086 )
2 years ago
ClosedPort22
455e34113e
Improve compatibility of DownloadArchive ( #3078 )
...
Other programs can add additional columns to the table without affecting
gallery-dl
2 years ago
KJ16609
300bc03deb
[gelbooru] allow alternate parameter order in post URLs ( #2821 )
2 years ago
Mike Fährmann
a7d23f1484
[vichan] add generic extractors for vichan imageboards
...
includes 8kun.top, smuglo.li, and wikieat.club
2 years ago
Mike Fährmann
04d3ebdfb4
[redgifs] fix 'token' extraction ( #3080 , #3081 )
2 years ago
thatfuckingbird
062ef238a6
add support for aibooru (using danbooru extractor) ( #3075 )
2 years ago
enduser420
0163ca86f7
[smugloli] add smugloli extractors ( #3060 )
2 years ago
Mike Fährmann
cf86f68864
[instagram] add 'avatar' extractor ( #929 , #1097 , #2992 )
2 years ago
Mike Fährmann
ea8113ff36
[reactor] match 'best', 'new', 'all' URLs ( #3073 )
2 years ago
Mike Fährmann
618c81afdf
[ngomik] remove module
...
"Access denied"
2 years ago
Mike Fährmann
94a2dfe205
[kemonoparty] update pagination offset
2 years ago
Mike Fährmann
52d1eb928d
[pixiv] extend 'metadata' option ( #3057 )
...
make it usable for all 'pixiv' extractors
2 years ago
Mike Fährmann
0714274f1f
[instagram] remove 'channel' extractor
2 years ago
Mike Fährmann
51e3b380ac
update 'virtualenv' call in release.sh
2 years ago
Mike Fährmann
b6682f3a2e
release version 1.23.3
2 years ago
Mike Fährmann
d0d4ce1a13
[danbooru] fix ugoira metadata extraction ( #3056 )
2 years ago
Mike Fährmann
096b8f2cfc
[instagram] prevent request for private '/tagged' feeds ( #3045 )
2 years ago
Mike Fährmann
3b369ce3d1
[nijie] add 'followed' extractor ( #3048 )
2 years ago
Mike Fährmann
c4a62a48ae
[nijie] add 'feed' extractor ( #3048 )
2 years ago
Mike Fährmann
d1314df6e6
[nozomi] fix extraction ( #3051 )
2 years ago
Mike Fährmann
277be410a7
[2chen] update 'archive_fmt'
2 years ago
pink-red
88f8975ab9
Fix duplicated metadata bug ( #3033 )
2 years ago
Mike Fährmann
ed55bd3a5c
[redgifs] extract Bearer token ( #3037 )
2 years ago
Mike Fährmann
e974c75083
[redgifs] fix extraction ( #3037 )
...
send public Bearer token as 'authorization' header
2 years ago
Mike Fährmann
68466a7d61
[tumblr] support ' https://www.tumblr.com/BLOGNAME ' URLs ( #3034 )
2 years ago
Mike Fährmann
b6a68f5a4b
[fanbox] extend 'content' test result ( #3020 )
2 years ago
Mike Fährmann
f1f89b2436
[tumblr] add 'offset' option
2 years ago
Mike Fährmann
827ab0a62d
[instagram] fix login
...
- use mobile user agent header
- update general headers
- skip /data/shared_data/ step
2 years ago
Mike Fährmann
1ca6be8619
[fanbox] add 'content' metadata field ( #3020 )
2 years ago
Mike Fährmann
e5d229c524
[tumblr] sleep between fallback retries ( #2957 )
2 years ago
Mike Fährmann
b2b0b1c455
[hitomi] fall back to webp when format not available ( #3030 )
2 years ago
Mike Fährmann
1696f68a68
[8chan] add 'thread' and 'board' extractors ( #2938 )
2 years ago
Mike Fährmann
560f7b41d8
[vk] add 'tagged' extractor ( #2997 )
2 years ago
Mike Fährmann
122e1a467a
[vk] unescape error messages
2 years ago
Mike Fährmann
7f30a0d7a7
add 'path-extended' option ( #3021 )
2 years ago
Mike Fährmann
bc9d291c13
[imagefap] fix and improve folder extraction ( #3013 )
2 years ago
Mike Fährmann
55fca5fe4b
[imagefap] fix and improve gallery pagination ( #3013 )
2 years ago
Mike Fährmann
8b1fe0bcf1
emit debug logging messages before calling time.sleep() ( #2982 )
2 years ago
Mike Fährmann
a6e2d96dde
fix bug when processing input file comments ( #2808 )
...
and move 'parse_inputfile()' to util.py
2 years ago
Mike Fährmann
14717f3fc9
[deviantart] add 'group' option ( #3018 )
...
disabling this option allows to better download from deleted accounts
2 years ago
Mike Fährmann
5a0b4cfab4
[postprocessor:metadata] don't ignore mode:json
...
fixes a bug introduced in b57015cf
where {"mode": "json", "format": "foobar"}
would behave like mode:custom
2 years ago
Mike Fährmann
220a04a74a
[artstation] skip missing projects ( #3016 )
2 years ago
Mike Fährmann
a12ce2bb41
[deviantart] fix 'deviation' extraction ( #2981 )
2 years ago
Mike Fährmann
36afb519b3
[instagram] prevent crash on empty user profile
2 years ago
Mike Fährmann
b57015cf0a
[postprocessor:metadata] assume 'mode: custom' when format is set
...
{"name": "metadata", "format": "foobar"}
will now implicitly use mode:custom and no longer mode:json like before
2 years ago
enduser420
f0321f423d
[2chen] Add 2chen.moe extractor ( #2707 )
...
* [2chen] Add 2chen.moe extractor
* change "==" to is
* fix for "test_unique_pattern_matches"
* fix regex pattern and group matching
* fix regex again
* [2chen] add 'reply_no' and 'hash' metadata and change 'filename_fmt'
also made an entry in supportedsites.md
* [2chen] unescape 'title'
* [2chen] partition() -> rpartition()
* [2chen] extract 'date' and 'name' metadata
* [2chen] remove 'offset' argument
* [2chen] do some changes
* [2chen] do some more changes
* [2chen] unescape 'name' and 'filename'
2 years ago
enduser420
f7ba19a1c0
[nana] add 'nana' extractors ( #2967 )
2 years ago
Mike Fährmann
fce6642699
[instagram] restore warnings for private profiles ( #3004 )
2 years ago
Mike Fährmann
3e65645cfa
[instagram] restore 'cursor' functionality ( #2991 )
2 years ago
Mike Fährmann
b8d268f57e
allow '/' and '?' in URL queries
2 years ago
Mike Fährmann
60cce7462c
release version 1.23.2
2 years ago
Mike Fährmann
7b5dad075d
[fappic] fix extraction
2 years ago
Mike Fährmann
78694a61bb
[kemonoparty] restore 'favorites' API endpoints ( #2994 )
2 years ago
Mike Fährmann
5fd4374036
[sankaku] improve 429 and tag limit handling
2 years ago
Mike Fährmann
b84982b2f9
[kemonoparty] send Referer headers ( #2989 , #2990 )
2 years ago
blankie
98f67ae333
[instagram] add 'count' metadata field ( #2979 )
2 years ago
Mike Fährmann
4089bceddd
[sankaku] implement 'refresh' option ( #2958 )
2 years ago
Mike Fährmann
779e75c6f8
[kemonoparty] fix attachment IDs overwriting post IDs ( #2984 )
...
regression from 09a5cc61
2 years ago