ce8dbb1ccc
[twitter] fix crash when using 'expand' and 'syndication'
...
caused by KeyError: 'conversation_id_str'
2 years ago
d651d45239
implement specifying ranges in slice notation ( #918 , #2865 )
...
e.g.
- '1:101' or ':101' or ':101:' for files 1 to 100
- '1::2' or '::2' for every second file
- '1:101:5' or ':101:5' for files 1, 6, 11, ..., 91, 96
(the second argument specifies the first index NOT included)
2 years ago
527bb2c4ab
[lynxchan/bbw-chan] add 'thread' and 'board' extractors
2 years ago
64902f518e
[docs] Update links and fix field typo
2 years ago
5d5ea58887
[twitter] update docs for 'quoted' & 'unique'
2 years ago
f82ee93676
[danbooru] extract uploader metadata (if metadata is set)
2 years ago
1800bd7d14
allow '*-filter' options to be a list of expressions
2 years ago
895b41f1ac
[jschan] add generic jschan extractor
2 years ago
73ab5d84c0
update docs/configuration.rst
2 years ago
bee354c264
Merge pull request #3415 from enduser420/extractor/fapello
...
[fapello] add 'post', 'user' and 'path' extractors
2 years ago
dd4a4a3fa6
[e621] softcode the pagination threshold
2 years ago
e5076ba056
[fapello] add 'post', 'user' and 'path' extractors
2 years ago
d0ad6d0e67
[e621] implement manual pagination mode
2 years ago
1317625ec4
[webmshare] add 'video' extractor ( #2410 )
2 years ago
41bf236d36
[lynxchan] add generic extractors for lynxchan imageboards ( #3394 )
...
* [lynxchan] add generic extractors for lynxchan imageboards
includes kohlchan.net, endchan.org:wq
* [lynxchan] set pop default to empty tuple
* Apply suggestions from code review
Co-authored-by: Mike Fährmann <mike_faehrmann@web.de>
2 years ago
3c75c3bbc4
[soundgasm] add 'user' extractor ( #3384 )
...
based on code from PR #3388 by @enduser420
2 years ago
2952add4a8
[reddit] increase 'id-max' default value ( #3397 )
...
to float("inf")
2 years ago
6b6f886dcf
[bunkr] update domain ( #3391 )
...
and improve bunkr/app.bunkr handling
2 years ago
eb94568e1f
[soundgasm] add 'audio' extractor ( #3384 )
2 years ago
dfe7b23579
support Firefox containers for --cookies-from-browser ( #3346 )
2 years ago
4a3a1f4c87
[komikcast] update domain and fix extraction
2 years ago
a42ba25ca1
[foolslide] remove 'kireicake'
...
site redirects to (unclaimed) mangadex group
2 years ago
86f0597c95
[kissgoddess] remove module
...
site does not host albums anymore
2 years ago
fceaee3c4f
[lolisafe] remove zz.ht
2 years ago
54844944ab
[pixhost] add 'gallery' support ( #3353 )
2 years ago
213676c785
[fapachi] add 'post' and 'user' extractors ( #3347 )
...
* [fapachi] add 'post' and 'user' extractors
* [fapachi] add 'keyword' to test
* [fapachi] remove whitespaces
2 years ago
80102fa367
[downloader:http] add 'retry-codes' option ( #3313 )
2 years ago
b14b33f19e
Implement `version-metadata` option ( #3201 )
2 years ago
6cb12f513b
[nitter] support quoted Tweets
...
- distinguish between regular and quoted Tweets and media
- add 'quoted' option and metadata field
2 years ago
a41d093bb1
[nitter] add 'retweets' option ( #3278 )
2 years ago
e31d12139c
[nitter] add 'videos' option ( #3279 )
...
with the same semantics as for twitter
2 years ago
501d9bccfe
[artstation] add 'max-posts' option ( #3270 )
2 years ago
b1ad6f2289
[artstation] add 'pro-first' option ( #3273 )
2 years ago
42481aed59
[formatter] implement 'S' format specifier ( #3266 )
...
to Sort lists
2 years ago
6b97dcf2e0
[postprocessor:metadata] add 'private' option
2 years ago
08fd1ff835
[twitter] add 'avatar' and 'background' extractors ( #349 , #3023 )
2 years ago
6379157543
[instagram] use REST API by default
...
regardless of logged in status
2 years ago
7897f68225
[wallhaven] update 'user' extractor ( #3226 )
...
* [wallhaven] update 'user' extractor
* [wallhaven] update 'configuration.rst'
add 'extractor.wallhaven.include' entry
* [wallhaven] add 'wallhaven.include' in gallery-dl.conf
2 years ago
5a68b5cb3c
[wallhaven] add 'user' extractor ( #3213 )
2 years ago
6c153750fa
[nitter] add extractors for Nitter instances ( #2696 )
2 years ago
9f06e79868
implement '"user-agent": "browser"' ( #2636 )
2 years ago
039d06c8f6
[mangaread] add 'chapter' and 'manga' extractors
2 years ago
70c7fbe89a
[instagram] add 'guide' extractor ( #3192 )
2 years ago
8d543b040d
update docs
...
- document '"mode": "jsonl"' for metadata pp
- remove 'username' and 'password' from 'instagram'
- add 'ytdl' example (#3180 )
2 years ago
171262c1b6
[instagram] remove login support
...
broken feature that I cannot get to work anymore
2 years ago
895f36e53b
[postprocessor:metadata] add 'open' and 'encoding' options
2 years ago
870e6a48a0
implement 'http-metadata' option
...
or at least attempt to.
2 years ago
ccb80f1b8b
[uploadir] add support for 'uploadir.com' ( #3162 )
2 years ago
c01cad599a
[lolisafe] add support for xbunkr ( #3156 )
2 years ago
bca9f965e5
[downloader:http] add 'chunk-size' option ( #3143 )
...
and double the previous default from 16384 (2**14) to 32768 (2**15)
2 years ago
c12a97bcde
[postprocessor] add 'post-after' event ( #3117 )
2 years ago
58d97188b4
[mastodon] add 'bookmark' extractor ( #3109 )
2 years ago
7c6af27eb8
[tumblr] add 'fallback-*' options ( #2957 )
...
specifically 'fallback-delay' and 'fallback-retries'
and change default number of retries to 2 (down from 3)
2 years ago
ac6111e693
[mangasee] add support for 'mangalife' ( #3086 )
2 years ago
a7d23f1484
[vichan] add generic extractors for vichan imageboards
...
includes 8kun.top, smuglo.li, and wikieat.club
2 years ago
062ef238a6
add support for aibooru (using danbooru extractor) ( #3075 )
2 years ago
0163ca86f7
[smugloli] add smugloli extractors ( #3060 )
2 years ago
cf86f68864
[instagram] add 'avatar' extractor ( #929 , #1097 , #2992 )
2 years ago
618c81afdf
[ngomik] remove module
...
"Access denied"
2 years ago
52d1eb928d
[pixiv] extend 'metadata' option ( #3057 )
...
make it usable for all 'pixiv' extractors
2 years ago
0714274f1f
[instagram] remove 'channel' extractor
2 years ago
3b369ce3d1
[nijie] add 'followed' extractor ( #3048 )
2 years ago
c4a62a48ae
[nijie] add 'feed' extractor ( #3048 )
2 years ago
f1f89b2436
[tumblr] add 'offset' option
2 years ago
15a4d7696f
[docs] update 'adjust-extensions' explanation ( #3031 )
2 years ago
1696f68a68
[8chan] add 'thread' and 'board' extractors ( #2938 )
2 years ago
560f7b41d8
[vk] add 'tagged' extractor ( #2997 )
2 years ago
7f30a0d7a7
add 'path-extended' option ( #3021 )
2 years ago
14717f3fc9
[deviantart] add 'group' option ( #3018 )
...
disabling this option allows to better download from deleted accounts
2 years ago
f0321f423d
[2chen] Add 2chen.moe extractor ( #2707 )
...
* [2chen] Add 2chen.moe extractor
* change "==" to is
* fix for "test_unique_pattern_matches"
* fix regex pattern and group matching
* fix regex again
* [2chen] add 'reply_no' and 'hash' metadata and change 'filename_fmt'
also made an entry in supportedsites.md
* [2chen] unescape 'title'
* [2chen] partition() -> rpartition()
* [2chen] extract 'date' and 'name' metadata
* [2chen] remove 'offset' argument
* [2chen] do some changes
* [2chen] do some more changes
* [2chen] unescape 'name' and 'filename'
2 years ago
f7ba19a1c0
[nana] add 'nana' extractors ( #2967 )
2 years ago
4089bceddd
[sankaku] implement 'refresh' option ( #2958 )
2 years ago
63e0924927
[pixiv] add 'series' extractor ( #2964 )
2 years ago
aa49bf13d2
[instagram] add 'api' option
2 years ago
ac45ed2764
[skeb] implement 'filters' option ( #2945 )
2 years ago
08b0c4c0b1
[docs] add kemono text content example ( #2963 )
2 years ago
0393e59535
[newgrounds] add 'games' extractor ( #2955 )
2 years ago
4b2a006871
[skeb] add 'search' extractor ( #2945 )
2 years ago
eef50c1f28
[blogger] split 'search' extractor ( #2930 )
2 years ago
bd846abba0
[hotleak] add hotleak extractor ( #2909 ) ( #2890 )
2 years ago
e99a9b2aff
[twitter] improve 'cards-blacklist' ( #2875 )
...
allow blacklisting domains and 'name:domain',
where 'domain' depends on a card's 'vanity_url' value
2 years ago
46fe469c53
[tumblr] implement 'ratelimit' option ( #2919 )
2 years ago
b36125333f
[postprocessor:zip] implement 'files' option ( #2872 )
2 years ago
3cb8327c60
[zerochan] add 'metadata' option ( #2861 )
2 years ago
9745b48830
[tumblr] attempt to fetch high-quality inline images ( #2877 )
...
* [tumblr] attempt to fetch high-quality images (again)
Fixes #1846 , and fixes #1344
* slight refactor
* update configuration.rst entry
2 years ago
4d78ca89db
[twitter] add 'cards-blacklist' option ( #2875 )
2 years ago
8839b0d2ee
add section about global replacement fields to formatting.md
...
(#2862 )
2 years ago
f16fbe9f93
document 'extractor.twitter.expand' ( #2848 )
2 years ago
7e385ed63e
[foolfuuka] update domains
...
- remove nyafuu
- add rozenarcana (https://archive.alice.al/ )
- add tokyochronos (https://www.tokyochronos.net )
2 years ago
67bad04dda
[formatter] add 'g' conversion to sluGify a string ( #2410 )
2 years ago
3d87cedc58
[jpgchurch] rework the image extractor
...
now the image extractor can recognize if an image if from an album
also removed some unnecessary methods
2 years ago
574e38a287
[kemonoparty] add 'favorites' option ( #2826 ) ( #2831 )
...
* [kemonoparty] add 'favorites' option (#2826 )
* [kemonoparty] add regex for the url parameter and fallback on the config
option
* [kemonoparty] simplify
2 years ago
a799fae2df
[catbox] add 'album' extractor ( #2410 )
...
adapted from https://github.com/mikf/gallery-dl/pull/2805
- rewrite using GalleryExtractor
- extract more metadata
- match lolisafe names
- add test
2 years ago
9ea521bec5
[oauth] add host config option ( #2806 )
2 years ago
df1c643dda
[tumblr] attempt to extract full-resolution photos
...
- for photos with apparent width == 2048 or height == 3072
- can be disabled with 'original' option
2 years ago
a69f198ac9
[wallhaven] implement 'metadata' option ( #2803 )
2 years ago
7d1a95ada6
implement 'path-metadata' option ( #2734 )
2 years ago
8cf5981ded
[skeb] add option to download 'article' images ( #1031 )
2 years ago
98af5a0409
[zerochan] implement login with username & password ( #1434 )
2 years ago
3a8addfe45
[zerochan] add 'tag' and 'image' extractors ( #1434 )
2 years ago
977d53b640
[Instagram] Add support for user's saved collection ( #2769 )
...
* [Instagram] Add support for user's saved collection
* [Instagram] Run formatter
* [Instagram] Simplify collection_id retrieval and add metadata
* [Instagram] Fix bug when params is not passed to _pagination_api
2 years ago
0c73914848
[postprocessor:metadata] implement 'mode: modify' ( #2640 )
2 years ago
f3de6b7a87
[postprocessor:metadata] implement 'mode: delete' ( #2640 )
2 years ago
eb68d45544
add global 'warnings' option ( #2762 )
2 years ago
f225247670
[gelbooru] add support for `api_key` and `user_id` ( #2767 )
2 years ago
46f11a3118
[bunkr] fix extraction ( #2732 )
...
move bunkr.is code to its own module
2 years ago
fe2b3d57d4
[komikcast] update domain
2 years ago
2dc57637cf
[foolfuuka] remove archive.wakarimasen.moe
2 years ago
74865adae5
implement 'format-separator' option ( #2737 )
...
a global option, that servers as a workaround for shortcomings due to
lack of a proper format string parser
2 years ago
117eeefda0
[postprocessor:mtime] add 'value' option ( #2739 )
2 years ago
90ae48c40c
[formatter] implement 'O' format specifier ( #2736 )
...
to apply a UTC offset to 'date' values and other datetime objects
2 years ago
1d14928bd9
[twitter] ignore previously seen Tweets ( #2712 )
...
occurs primarily for /with_replies results when logged in
2 years ago
4b2a0a0eda
[twitter] implement 'strategy' option ( #2712 )
...
to be able to better control what Tweets get used an returned
for twitter.com/USER URLs.
2 years ago
124b5c6410
[postprocessor:ugoira] enable 'mtime' by default ( #2714 )
2 years ago
3a5d5c3a91
update default User-Agent header to Firefox 102 ESR
...
snd update headers and ciphers for "browser": "firefox"
2 years ago
54525d2e21
[formatter] implement slice operator as format specifier
...
this allows using a slice operator alongside other (special) format
specifiers like J, to first join list elements to a string and then
trimming that with a slice.
{tags:J, /[:50]}
2 years ago
241e82e18d
[horne] add support for horne.red ( #2700 )
2 years ago
9d8e99af80
[itaku] support videos ( #1842 )
2 years ago
27e8078fb7
[poipiku] add 'user' and 'post' extractors ( #1602 )
2 years ago
fa902cd54d
[itaku] add 'gallery' and 'image' extractors ( #1842 )
2 years ago
9c8d895d19
[twitter] implement 'csrf' option ( #2676 )
2 years ago
6db77d4656
[weibo] support '?tabtype=video' listings ( #2601 )
2 years ago
4b78bd423f
[paheal] add 'metadata' option ( #2641 )
2 years ago
57508d3bb7
[weibo] support all different 'tabtype' listings ( #686 , #2601 )
2 years ago
6d0aa8c0f3
[docs/configuration.rst] Fix small typo ( #2649 )
2 years ago
e0ac358aa5
[gofile] fix 401 Unauthorized errors ( #2632 )
2 years ago
8a42d859bf
[bunkr] change domain to 'app.bunkr.is' ( #2634 )
2 years ago
4bf5bc2403
[weibo] support 'livephoto' entries ( #2146 )
2 years ago
5b43faffed
[postprocessor:metadata] write to stdout by setting filename to "-"
...
(#2624 )
2 years ago
603af48265
implement 'output.ansi' option ( #2628 )
2 years ago
ff5e10a86d
[hypnohub] move to gelbooru_v02 instances ( #2631 )
2 years ago
d6e744bf0f
[gfycat] add 'collection' extractor ( #2629 )
2 years ago
d4e9d51760
[reddit] add 'home' extractor ( #2614 )
2 years ago
915dba8345
[twitter] improve results for regular user URLs
...
- continuation of 3346f58ahttps://twitter.com/USER/tweets )
although there should be no need to ever do that
2 years ago
9df4e0f65b
[twitter] disable 'cards' by default
2 years ago
79dce8ae68
[weasyl] implement 'metadata' option ( #2610 )
2 years ago
feb470d19a
[shopifx] natively support a few more sites ( closes #2089 )
...
- chelseacrew.com
- michaels.com.au
- modcloth.com
- pinupgirlclothing.com
- raidlondon.com (loveraid.com)
- unique-vintage.com
2 years ago
60f4d59b1e
[gelbooru_v01] remove 'tlb.booru.org' from supported domains
...
403 Forbidden
nginx
it is also no longer listed on https://booru.org/top
2 years ago
6b6eb0b8f6
[lolisafe] implement 'domain' option ( #2575 )
2 years ago
6742f3bc1e
implement --cookies-from-browser ( #1606 )
...
most of the code is adapted from yt-dlp's implementation
and *should* work the same.
2 years ago
61887c895b
implement 'output.colors' options ( #2532 )
2 years ago
52b47c3cf9
[gelbooru_v01] add 'favorite' extractor ( #2546 )
2 years ago
84756982e9
[pixiv] implement 'include' option
...
- split 'user' extractor and its 'avatar' and 'background' options into
separate extractors ('artworks', 'avatar', 'background')
- avatars can now be downloaded with
https://www.pixiv.net/en/users/ID/avatar
as URL and will use a proper archive key; similar for backgrounds
- options for the 'user' subcategory must be moved to 'artworks' to have
the same effect as before
2 years ago
d11e2191ae
[nijie] support /history_nuita.php listings ( closes #2541 )
2 years ago
61afa973b7
use extended format strings for archive keys
2 years ago
0381752575
[photovogue] switch to .com, update api endpoint ( #2494 )
2 years ago
9e6ff42a9d
[pixiv] implement 'background' option ( #623 , #1124 , #2495 )
2 years ago
97e8a15295
[deviantart] implement 'pagination' option ( #2488 )
2 years ago
12bd9ba33a
[readcomiconline] add 'quality' option ( #2467 )
2 years ago
24ea1398bc
[postprocessor:ugoira] do not auto-select image2 demuxer ( #2492 )
2 years ago
ffa1dd1d91
add 'coomer.party' to supported sites ( closes #2462 )
2 years ago
9c5d2d7af3
[pinterest] add extractor for created pins ( #2452 )
3 years ago
1171911dc3
[twitter] add 'syndication' option ( #2354 )
...
to fetch age-restricted content using Twitter's syndication API
3 years ago
7e545a3ae9
[gofile] add gofile.io extractor ( #2364 )
...
* Add gofile extractor
* add gofile extractor to module list
* add support for tiny monitors and ancient python versions
* seriously, f-strings are not *that* new...
* i love flake8 :)
* add 'api-token' and 'recursive' options
* add tests
3 years ago
625f4d4cc4
[telegraph] Add telegra.ph extractor ( #2312 )
3 years ago
5c2d953766
[postprocessor:ugoira] auto-select demuxer by default
3 years ago
37d584a9b2
[hitomi] update metadata extraction ( fixes #2444 )
...
remove 'hitomi.metadata' option, as it is no longer necessary
to make additional HTTP requests to fetch all metadata.
3 years ago
e718dd7b32
[postprocessor:ugoira] support setting timecodes with mkvmerge
...
by selecting the "mkvmerge" demuxer
(#1550 )
3 years ago
71bba774da
respect 'output.private' in '-K/--list-keywords' output
3 years ago
ba69fb669d
[kemonoparty] add 'duplicates' option ( closes #2440 )
3 years ago
40ce505420
[postprocessor:ugoira] add 'mtime' option ( #2307 )
3 years ago
9bd27b1b8d
[postprocessor:metadata] implement archive options ( #2421 )
...
'archive', 'archive-format', and 'archive-prefix'
3 years ago
f8230dde43
[instagram] add 'previews' option ( #2135 )
3 years ago
cf44aba333
[formatter] allow evaluating f-string literals
...
by starting a format string with '\fF'.
This was technically already possible with '\fE',
but this makes it a bit more convenient.
3 years ago
58e0b17211
add note about using '\f' in shells to docs/formatting.md
...
(closes #2398 , closes #2406 )
3 years ago
b3d8b40690
Rename meme.museum ( #2393 )
3 years ago
fc277fa45f
[seiga] require authentication with 'user_session' cookie ( #2372 )
...
Login with username & password would now require entering a 2FA token.
see also 7b009cc893
3 years ago
47cf05c4ab
refactor proxy handling code ( #2357 )
...
- allow gallery-dl proxy settings to overwrite environment proxies
- allow specifying different proxies for data extraction and download
- add 'downloader.proxy' option
- '-o extractor.proxy=–PROXY_URL -o downloader.proxy=null'
now has the same effect as youtube-dl's '--geo-verification-proxy'
3 years ago
acbb0f5a57
clarify 'metadata.mtime' usage and shortcomings ( #2307 )
3 years ago
bfa5e61900
[patreon] add explicit 'image_large' file type ( #2257 )
...
to allow more control over when and if to download 'large_url' images
4fee3a0e52
3 years ago
f1cab23724
[skeb] add 'sent-requests' option ( #2322 ) ( #2330 )
...
* Update skeb.py
* Update configuration.rst
* flake8
3 years ago
4385a34e05
[twitter] fix handling of 429 responses ( fixes #2339 )
...
Twitter doesn't return a valid JSON response for 429 errors anymore.
3 years ago
5a50569360
[toyhouse] support 'art' listings ( #1546 , #2331 )
3 years ago
59749555b1
[postprocessor:mtime] add 'event' option ( #2307 )
3 years ago
7958995398
[postprocessor:metadata] implement 'mtime' option ( #2307 )
3 years ago
fdfdc1b614
[kissgoddess] add 'gallery' and 'model' extractors
...
(closes #1052 , #2304 )
3 years ago
79a461a2c1
[mememuseum] add 'tag' and 'post' extractors ( closes #2264 )
3 years ago
254a5b26e0
[twibooru] add extractors for searches, galleries, and posts
...
(#2219 )
3 years ago
f5b2a9fcc6
add 'signals-ignore' option ( #2296 )
3 years ago
b17e2dcf93
[wallpapercave] add extractor for images ( #2205 )
3 years ago
a7de819aca
[lightroom] add Lightroom gallery extractor ( #2263 )
3 years ago
563bd0ecf4
[danbooru] inherit from BaseExtractor
...
- merge danbooru and e621 code
- support booru.allthefallen.moe (closes #2283 )
- remove support for old e621 tag search URLs
3 years ago
f1c853c6ef
[furaffinity] add 'layout' option ( #2277 )
...
to be able to force gallery-dl to parse according to a specific layout
in case its auto-detect fails
3 years ago
4e4cb40663
do not sort BaseExtractor instances by name
3 years ago
f351746483
release version 1.20.4
3 years ago
f4a7c6eca2
Add two examples for custom instances of gelbooru ( #2262 )
...
* Add two examples for custom instances of gelbooru
- First example uses some custom options at the "gelbooru_v01" basecategory
These apply to all instances used herein
- Second example adds all options to the specific "gelbooru_v02" basecategory of TBIB
These settings are only used by TBIB
* move tbib settings into their own block
3 years ago
cc7dce5755
[sexcom] add 'pins' extractor ( closes #2265 )
3 years ago
02e18f56be
[e621] add 'favorite' extractor ( closes #2250 )
3 years ago
86fa412b47
[hitomi] add 'format' option ( #2260 )
...
default is 'webp' since downloading original files is no longer allowed
3 years ago
492436f936
[twitter] add 'warnings' option ( #2258 )
...
disable reporting any non-fatal errors by default
3 years ago
9a221494c3
[twitter] add 'event' extractor ( closes #2109 )
3 years ago
de754590e0
add --source-address command-line option ( closes #2206 )
3 years ago
f2e8aedd74
[twitter] changes to 'cards' option
...
- change default value to 'true'
- only invoke youtube-dl for cards unsupported by gallery
when 'cards' is set to "ytdl"
"cards": true --> only download card images
"cards": "ytdl" --> download card images and
use youtube_dl on otherwise unsupported cards
3 years ago
c559c49781
formatting.md: add environment variable syntax ( #2065 )
3 years ago
3c79c9b271
document extended blacklist/whitelist syntax ( #2025 )
...
and not just in the commit message of 010d65dc
3 years ago
37beb1298e
[newgrounds] add 'search' extractor ( closes #2161 )
3 years ago
fa7d92f7a9
add docs for 'extractor.generic.enabled'
3 years ago
89bebe1bef
[500px] add 'favorite' extractor ( closes #1927 )
3 years ago
882c614281
add album extractor for lolisafe/chibisafe instances
...
- support bunkr.is (closes #2038 )
- support zz.ht (closes #2105 )
3 years ago
64cf26eaf4
allow specifying sleep-* options as string
...
either as single value or as range: "3.5", "2.1 - 5.0"
3 years ago
62692c6842
[exhentai] add 'source' option
...
setting it to "hitomi" downloads the corresponding gallery from
hitomi.la; might be extended to other sources in the future
3 years ago
099ed72de7
[hitomi] disable extra 'metadata' by default
...
safes one HTTP request that not needed with default filename settings
3 years ago
f587458a3c
[twitter] include '4096x4096' as a default image fallback
...
(closes #2107 , closes #1881 )
3 years ago
299bd2f1f5
[rule34us] add 'tag' and 'post' extractors ( #1527 )
3 years ago
3cf1075d86
[inkbunny] add 'search' extractor ( closes #2094 )
3 years ago
f4e3cee6ac
use yt-dlp by default ( #1850 , #2028 )
3 years ago
f1b142e993
{kemonoparty[ change default 'files' order to attachments,file,inline
...
(#1991 )
3 years ago
addb72e1bb
[reactor] support thatpervert.com ( closes #2029 )
3 years ago
d8d9502e1e
[reactor] inherit from BaseExtractor
3 years ago
f4ea216c95
[shopify] support loungeunderwear.com ( closes #2053 )
3 years ago
612850438e
[skeb] add 'thumbnails' option ( #2047 ) ( #2051 )
3 years ago
11a3d96d13
[mangadex] load additional metadata using includes[] directives
...
- always provide 'artist', 'author', and 'group' metadata fields (#2049 )
- remove 'metadata' option
3 years ago
c67756e187
[kemonoparty] add 'dms' option ( #2008 )
3 years ago
3842cdcd8f
[formatter] implement 'D' format specifier
...
To be able to parse any string into a 'datetime' object
and format it as necessary.
Example:
{created_at:D%Y-%m-%dT%H:%M:%S%z}
->
"2010-01-01 00:00:00"
{created_at:D%Y-%m-%dT%H:%M:%S%z/%b %d %Y %I:%M %p}
->
"Jan 01 2010 12:00 AM"
with 'created_at' == "2010-01-01T01:00:00+0100"
3 years ago
3a7a19c7b9
[dynastyscans] add 'manga' extractor ( closes #2035 )
3 years ago
9bc83af3a6
[kemonoparty] 'postfile' -> 'file' ( #1991 )
...
to stay consistent with the existing file types for kemono
3 years ago
d433735750
[kemonoparty] skip duplicate files ( #2032 , #1991 , #1899 )
...
Extract the SHA-256 file hash from URLs
and skip files with the same hash in the same post.
- provide a 'hash' metadata field (empty string if not available)
- remove 'patreon-skip-file' option
3 years ago
d4ec245554
[kemonoparty] implement a 'files' option ( #1991 )
...
similar to 8d676151
3 years ago
cfa4876848
[philomena] support furbooru.org ( closes #1995 )
3 years ago
7c50f473e4
[ytdl] document 'cmdline-args' and 'config-file' options ( #1680 )
3 years ago
37c9dedee1
[seisoparty] remove module
3 years ago
b315a0ecef
[redgifs] update to API v2 ( #1984 )
3 years ago
f0fc3b0ba1
[kemonoparty] add 'comments' option ( #1980 )
3 years ago
4c49174579
[mangakakalot] update domain and fix extraction
3 years ago
69cd573809
add special type format strings to docs ( #1987 )
3 years ago
9156e90f1f
[twitter] add 'pinned' option
3 years ago
06b414c9a3
[redgifs] 'gfyId' -> 'id' ( #1984 )
3 years ago
bfd7401b1e
[skeb] add 'user' and 'post' extractors ( #1031 ) ( #1971 )
...
* Create skeb.py
* Update __init__.py
* Update supportedsites.py
* Update supportedsites.md
* Update supportedsites.py
* Update skeb.py
3 years ago
02a247f4e5
[deviantart] full resolution for non-downloadable images ( #293 )
...
Many thanks to @Ironchest337 for discovering this method
and providing a well-documented implementation.
3 years ago
c19e762fdf
[vk] add 'album' extractor ( #474 , fixes #1952 )
...
todo: better metadata for albums
3 years ago
8bb442f20d
[redgifs][gfycat] provide fallback URLs ( fixes #1962 )
...
and extend the 'format' option
3 years ago
8d676151b7
[patreon] implement 'files' option ( #1935 )
3 years ago
80314e981b
small corrections to gallery-dl-example.conf
...
use "'_reddit' in locals()"
instead of ""locals().get('_reddit')"
since the reddit metadata dict could be empty. The latter expression
would evaluate to False in that case, even though it has a reddit parent.
3 years ago
3ecf4a9729
update gallery-dl-example.conf
...
- add comments
- remove/replace some older options
- display more post processor settings
3 years ago
50098762e3
[nhentai] add 'tag' extractor ( closes #1950 )
3 years ago
fe6ce5495a
[kemonoparty] add 'discord' extractor ( #1827 , #1940 )
3 years ago
918fc9974d
[picarto] add 'gallery' extractor ( closes #1931 )
3 years ago
e33125ad39
[pixiv] add 'sketch' extractor ( #1497 )
3 years ago
b93915c113
[inkbunny] add 'pool' extractor ( #1937 )
3 years ago
373d3e1c57
[seisoparty] implement login with username & password ( #1906 )
3 years ago
7c5f62b453
[seisoparty] add 'favorite' extractor ( #1906 )
3 years ago
d93b5474c3
[mangadex] update parameter handling for API requests
...
- move common parameters into '_pagination()'
- add 'ratings' (#1908 ) and 'api-parameters' options
3 years ago
cd66c3c415
[twitter] add 'size' option ( #1881 )
3 years ago
df8050b81d
[postprocessor:compare] add 'equal' option ( #1592 )
...
Move functionality from cdd72e14
3 years ago
cdd72e1413
[postprocessor:compare] extend 'action' option ( #1592 )
...
allow setting it to "abort", "terminate", or "exit" as with 'skip'
3 years ago
e0bdacd932
[fappic] add 'image' extractor ( closes #1898 )
3 years ago
8c29a6e491
add example for 'url-metadata' option ( closes #1884 )
3 years ago
d0761454b1
implement a download progress indicator ( #1519 )
3 years ago
140a7bfdcd
update --clear-cache command in deviantart client-id instructions
3 years ago
e69ee41f25
implement 'page-reverse' option ( #1854 )
3 years ago
e4684c5cb9
[desktopography] simplify ( #1740 )
3 years ago
7b05600056
[pinterest] only mention cookies for authentication ( #1856 )
...
regular login is broken due to CAPTCHA
3 years ago
266ed9b62e
[aryion] add 'tag' extractor ( closes #1849 )
3 years ago
c9e6693530
allow specifying a minimum/maximum for 'sleep-*' options ( #1835 )
...
for example '"sleep-request": [5.0, 10.0]' to wait between 5 and 10
seconds between each HTTP request
3 years ago
bd845303ad
implement a way to shorten filenames with east-asian characters
...
(#1377 )
Setting 'output.shorten' to "eaw" (East-Asian Width) uses a slower
algorithm that also considers characters with a width > 1.
3 years ago
7bbb1f92d7
[gelbooru_v02] add 'favorite' extractor ( closes #1834 )
3 years ago
4ec11af6a4
[kemonoparty] implement login with username & password ( #1824 )
3 years ago
83bbb628d8
[kemonoparty] add 'favorite' extractor ( #1824 )
3 years ago
44f572c27f
[deviantart] implement a 'auto-unwatch' option ( #1466 , #1757 )
3 years ago
268c9ca034
update default gallery-dl.conf
3 years ago
3e36543c98
[nhentai] add 'favorite' extractor ( #1814 )
3 years ago
96fec14ef7
[deviantart] rename 'watch' option to 'auto-watch'
...
(#1466 , #1757 )
Similar reason as in e05a96db
3 years ago
e75f2de9da
[deviantart] add 'comments' option ( #1800 )
3 years ago
6ce16c6d31
[deviantart] add 'tag' extractor ( closes #1803 )
3 years ago
4e9f8fe395
[shopify] support windsorstore.com ( #1793 )
3 years ago
292fffc83c
add 'j' format string conversion
...
to convert to a JSON formatted string
3 years ago
ecc8da4704
[deviantart] implement a 'watch' option ( #1466 , #1757 )
3 years ago
d3eab417ed
implement a 'path-strip' option
3 years ago
c04f7ab139
[foolfuuka] add 'gallery' extractor ( #1785 )
3 years ago
20ee091289
[429chan] add 'thread' and 'board' extractors ( closes #1773 )
3 years ago
51eb50749f
[foolslide] remove entry for kobato.hologfx.com
3 years ago
d320ee6251
implement a 'fallback' option ( closes #1770 )
3 years ago
c866fcba48
[twitter] fix 'logout' ( #1719 )
...
delete 'auth_token' cookie and cookies.txt path
3 years ago
ed4b3c48cb
fix flake8 and other tests
3 years ago
fa6af46756
Added ability to download GIFs instead of mp4 from Luscious and Reactor ( #1701 )
3 years ago
e5a93e113f
[twitter] extend 'replies' option ( #1254 )
...
Allow setting 'replies to '"self"' to only download from self-replies.
3 years ago
cadfad4eea
[danbooru] add 'external' option ( closes #1747 )
3 years ago
5b1c62bfa9
[furaffinity] add 'external' option ( closes #1492 )
3 years ago
a316e44f8e
[bbc] add 'width' option ( #1706 )
3 years ago
7a0da4f93f
[newgrounds] add 'format' option ( closes #1729 )
3 years ago
486474800f
[kemonoparty] skip duplicated patreon files ( closes #1689 )
...
this behavior can be disabled with the 'patreon-skip-file' option
3 years ago
da7297c0b9
[comicvine] add extractor ( closes #1712 )
3 years ago
e4788fa663
[bbc] add 'gallery' and 'programme' extractors ( closes #1706 )
3 years ago
13d4045a8a
add 'archive-prefix' option ( #1711 )
3 years ago
74b457b2b5
remove unused entries
3 years ago
3868ec02d1
[pururin] update domain and fix extraction
3 years ago
b9783403d9
add 'url-metadata' option ( #1659 , #1073 )
3 years ago
e95f99882f
extend 'parent-metadata' functionality ( #1687 , #1651 , #1364 )
3 years ago
dff0da60f9
[ytdl] add 'generic' option ( #1680 )
3 years ago
36ac2197db
[ytdl] add extractor for sites supported by youtube-dl
...
(#1680 , #878 )
Can be used by prefixing any URL with 'ytdl:',
or by setting 'extractor,ytdl.enabled' to 'true'.
3 years ago
8b036778e3
[kemonoparty] add 'max-posts' option ( #1674 )
3 years ago
8ecca3af58
[pixiv] add extractor for 'pixivision' articles ( #1672 )
3 years ago
312a28e78a
[mastodon] add 'replies' option ( #1669 )
3 years ago
a1f5b78039
[mastodon] add 'reblogs' option ( #1669 )
3 years ago
317ecc8180
use HTML tables in docs/formatting.md
3 years ago
7f591c78cb
[mangafox] cleanup
3 years ago
93d356712c
[mastodon] implement 'text-posts' option ( #1569 )
...
similar to Twitter's 'text-tweets'
3 years ago
963d177a68
document format string syntax
...
or at least attempt to
3 years ago
267bbf5996
[mangasee] add 'chapter' and 'manga' extractors
3 years ago
f74cf52e2b
[seisoparty] add 'user' and 'post' extractors ( #1635 )
3 years ago
e661607e8b
[mangadex] document 'metadata' and 'lang' options ( #1535 )
3 years ago
fd00d47116
implement conditional directories ( #1394 )
...
They work the same way as conditional filenames (84d2e640
3 years ago
a751afdfb3
[twitter] change some defaults
...
- 'retweets' option: true -> false
- 'quoted' option : true -> false
i.e. disable downloading tweets from other user's timelines by default
- search directory:
'["{category}", "Search", "{search}"]' ->
'["{category}", "{user[name]}"]'
i.e. change it to the same as other twitter extractors (#1308 )
3 years ago
84d2e64024
combine conditional filenames into `filename` option ( #1394 )
3 years ago
1197ee2c20
[mangadex] add extractor for a user's followed feed ( #1535 )
3 years ago
07c8adbd8b
[mangadex] implement login with username & password ( #1535 )
3 years ago
3cbbefd4ed
support 'filter' option for post processors ( #1460 )
3 years ago
4cf40434d7
initial support for conditional filenames ( #1394 )
3 years ago
f3d692a378
improve docs for danbooru and e621 passwords ( #1593 , #1596 )
3 years ago
407627ec86
[foolfuuka] support 'archive.wakarimasen.moe' ( closes #1595 )
3 years ago
52052a0e1a
[manganelo] update domain to 'manganato.com'
3 years ago
c80b18a477
[weibo] extend 'retweets' option ( closes #1542 )
...
Setting 'retweets' to "original" will use metadata from the
original posts, and not from the retweeted ones.
3 years ago
74d97e6c6b
[postprocessor:ugoira] add 'repeat-last-frame' option ( #1550 )
3 years ago
be9547a5f3
[postprocessor:ugoira] use 'image2' demuxer ( #1550 )
...
Ensures exact frame timecodes with no duplicate frames.
Possible issues are the duration the last frame in an Ugoira with variable
frame durations is shown and insufficient timestamp precision of the
underlying file system (e.g. FAT32, ext3; works on ext4, tmpfs, NTFS).
3 years ago
b8de0671e5
update description for 'instagram.include'
...
- add double quotes for "reels"
- add "tagged"
3 years ago
e6811c7450
[pixiv] implement 'max-posts' option ( #1558 )
...
* implement max-rank for pixiv
* rename to max-posts and make more generic
3 years ago
b5affc62aa
[twitter] rename 'text-only' to 'text-tweets' ( #570 )
3 years ago
724ca61f36
[twitter] add 'text-only' option ( #570 )
3 years ago
1eabfa5c7a
[pillowfort] implement login with username & password ( #846 )
3 years ago
a7e4917ee1
[pillowfort] add 'inline' option ( #846 )
...
to support images present in a post's 'content',
but not listed in 'media'.
also separates the file hash present at the beginning
of each 'filename' into its own field.
3 years ago
efa6cc8ec3
[pillowfort] add 'external' option ( #846 )
...
for links to external Twitter posts etc.
3 years ago
5eeaaee01d
[pixiv] add 'metadata' option ( #1551 )
3 years ago
0717456b4e
[kemonoparty] add 'metadata' option ( closes #1548 )
...
to fetch creator names with an additional HTTP request
3 years ago
7ab8374385
add 'parent-skip' option ( #1399 )
3 years ago
c693db5b1a
add '"skip": "terminate"' option
...
Stops not only the current extractor/job,
but all parent extractors/jobs as well.
3 years ago
36ed1efcfb
[pixiv] rename "noop" value for 'tags' option to "original"
...
(#1507 )
3 years ago
4e1d33cea3
update default gallery-dl.conf
...
add (empty) entries for
- deviantart.client-id and deviantart.client-secret
- pixiv.refresh-token
3 years ago
3e4ffb0821
[gelbooru] add extractor for '/redirect.php' URLs ( #1530 )
3 years ago
15b0241bbc
[imagebam] fix extraction
3 years ago
38ae61edd4
[inkbunny] add 'favorite' extractor ( #1521 )
3 years ago
e300da1424
add 'output.skip' option
3 years ago
2133f1d77f
[readcomiconline] change domain to 'readcomiconline.li'
...
(closes #1517 )
3 years ago
fa519f9202
[pixiv] change 'translated-tags' option ( #1507 )
...
- rename to 'tags'
- use string-values: "japanese", "translated", "noop"
- remove duplicate entries for "translated" tags
3 years ago
c6c4a73f87
update fanbox entry in supportedsites.md
3 years ago
e47952ac14
add extractors for fantia and fanbox ( #1459 )
...
* add extractors for fantia and fanbox
* appease linter
* make docstrings unique
* [fantia] refactor post extraction
* [fantia] capitalize
* [fantia] improve regex pattern
* code style
* capitalize
* [fanbox] use BASE_PATTERN for url regexes
* [fanbox] refactor metadata and post extraction
* [fanbox] improve url base pattern
* [fanbox] accept creator page links ending with /posts
* [fanbox] more tests
* [fantia] improved pagination
* [fanbox] misc. code logic improvements
* [fantia] finish restructuring pagination code
* [fanbox] avoid making a request for each individual post when processing a creator page
* [fanbox] support embedded videos
* [fanbox] fix errors
* [fanbox] document extractor.fanbox.videos
* [fanbox] handle "article" and "entry" post types, all embeds
* [fanbox] fix downloading of embedded fanbox posts
3 years ago
77a9cc6fd6
update supportedsites.md entry for Instagram
3 years ago
9514cb8c12
[exhentai] update 'limits' check ( #1487 )
...
Only use 'limits' to set a custom upper bound.
Checking if the actual maximum gets exceeded is not necessary.
3 years ago
9322c5e43b
[exhentai] restore limit config ( #1487 )
...
This partially reverts commit e9ec91c8
3 years ago
334d690687
[hentaicosplays] Add extractor ( #1473 )
3 years ago
086925e685
[shopify] support omgmiamiswimwear.com ( closes #1280 )
3 years ago
224b883ff4
[danbooru] add option for extended metadata extraction ( #1458 )
...
* [danbooru] add option for extended metadata extraction
* appease linter
* [danbooru] update docs/configuration.rst
* [danbooru] rename extended-metadata -> metadata
3 years ago
dff03a6605
[booru] add an option to extract notes (only gelbooru for now) ( #1457 )
...
* [booru] add an option to extract notes (currently implemented only for gelbooru)
* appease linter
* [gelbooru] rename "text" to "body" in note extraction
* add a code comment about reusing return value of _extended_tags
3 years ago
78d7ee3ef4
[yuki] remove module for yuki.la
3 years ago
a86ffb04bb
add 'output.fallback' option
...
to enable/disable fallback URLs for -g/--get-urls
3 years ago
e3b9f88540
Add manganelo extractor ( #1415 )
4 years ago
105f3c9666
[twitter] add extractor for direct image links ( closes #1417 )
4 years ago
ec3d5d58a8
[vk] improve extractor ( #474 )
...
- fetch all photos
- add 'metadata' option
- fix extracting photos without '?' in URL
4 years ago
d5aad999dc
[tapas] implement login with username & password ( #692 )
4 years ago
e9ec91c811
[exhentai] improve image limits check
...
- check if current image is the '509 Bandwidth Exceeded' notification
(https://ehgt.org/g/509.gif or https://exhentai.org/img/509.gif )
- remove 'limits' option
4 years ago
36291176bc
[pinterest] add 'search' extractor ( #1411 )
4 years ago
ClosedPort22
