61afa973b7
use extended format strings for archive keys
2 years ago
9e6ff42a9d
[pixiv] implement 'background' option ( #623 , #1124 , #2495 )
2 years ago
97e8a15295
[deviantart] implement 'pagination' option ( #2488 )
2 years ago
12bd9ba33a
[readcomiconline] add 'quality' option ( #2467 )
2 years ago
24ea1398bc
[postprocessor:ugoira] do not auto-select image2 demuxer ( #2492 )
2 years ago
1171911dc3
[twitter] add 'syndication' option ( #2354 )
...
to fetch age-restricted content using Twitter's syndication API
3 years ago
7e545a3ae9
[gofile] add gofile.io extractor ( #2364 )
...
* Add gofile extractor
* add gofile extractor to module list
* add support for tiny monitors and ancient python versions
* seriously, f-strings are not *that* new...
* i love flake8 :)
* add 'api-token' and 'recursive' options
* add tests
3 years ago
5c2d953766
[postprocessor:ugoira] auto-select demuxer by default
3 years ago
37d584a9b2
[hitomi] update metadata extraction ( fixes #2444 )
...
remove 'hitomi.metadata' option, as it is no longer necessary
to make additional HTTP requests to fetch all metadata.
3 years ago
e718dd7b32
[postprocessor:ugoira] support setting timecodes with mkvmerge
...
by selecting the "mkvmerge" demuxer
(#1550 )
3 years ago
71bba774da
respect 'output.private' in '-K/--list-keywords' output
3 years ago
ba69fb669d
[kemonoparty] add 'duplicates' option ( closes #2440 )
3 years ago
40ce505420
[postprocessor:ugoira] add 'mtime' option ( #2307 )
3 years ago
9bd27b1b8d
[postprocessor:metadata] implement archive options ( #2421 )
...
'archive', 'archive-format', and 'archive-prefix'
3 years ago
f8230dde43
[instagram] add 'previews' option ( #2135 )
3 years ago
fc277fa45f
[seiga] require authentication with 'user_session' cookie ( #2372 )
...
Login with username & password would now require entering a 2FA token.
see also 7b009cc893
3 years ago
47cf05c4ab
refactor proxy handling code ( #2357 )
...
- allow gallery-dl proxy settings to overwrite environment proxies
- allow specifying different proxies for data extraction and download
- add 'downloader.proxy' option
- '-o extractor.proxy=–PROXY_URL -o downloader.proxy=null'
now has the same effect as youtube-dl's '--geo-verification-proxy'
3 years ago
acbb0f5a57
clarify 'metadata.mtime' usage and shortcomings ( #2307 )
3 years ago
bfa5e61900
[patreon] add explicit 'image_large' file type ( #2257 )
...
to allow more control over when and if to download 'large_url' images
4fee3a0e52
3 years ago
f1cab23724
[skeb] add 'sent-requests' option ( #2322 ) ( #2330 )
...
* Update skeb.py
* Update configuration.rst
* flake8
3 years ago
4385a34e05
[twitter] fix handling of 429 responses ( fixes #2339 )
...
Twitter doesn't return a valid JSON response for 429 errors anymore.
3 years ago
59749555b1
[postprocessor:mtime] add 'event' option ( #2307 )
3 years ago
7958995398
[postprocessor:metadata] implement 'mtime' option ( #2307 )
3 years ago
254a5b26e0
[twibooru] add extractors for searches, galleries, and posts
...
(#2219 )
3 years ago
f5b2a9fcc6
add 'signals-ignore' option ( #2296 )
3 years ago
f1c853c6ef
[furaffinity] add 'layout' option ( #2277 )
...
to be able to force gallery-dl to parse according to a specific layout
in case its auto-detect fails
3 years ago
86fa412b47
[hitomi] add 'format' option ( #2260 )
...
default is 'webp' since downloading original files is no longer allowed
3 years ago
492436f936
[twitter] add 'warnings' option ( #2258 )
...
disable reporting any non-fatal errors by default
3 years ago
de754590e0
add --source-address command-line option ( closes #2206 )
3 years ago
f2e8aedd74
[twitter] changes to 'cards' option
...
- change default value to 'true'
- only invoke youtube-dl for cards unsupported by gallery
when 'cards' is set to "ytdl"
"cards": true --> only download card images
"cards": "ytdl" --> download card images and
use youtube_dl on otherwise unsupported cards
3 years ago
3c79c9b271
document extended blacklist/whitelist syntax ( #2025 )
...
and not just in the commit message of 010d65dc
3 years ago
fa7d92f7a9
add docs for 'extractor.generic.enabled'
3 years ago
64cf26eaf4
allow specifying sleep-* options as string
...
either as single value or as range: "3.5", "2.1 - 5.0"
3 years ago
62692c6842
[exhentai] add 'source' option
...
setting it to "hitomi" downloads the corresponding gallery from
hitomi.la; might be extended to other sources in the future
3 years ago
099ed72de7
[hitomi] disable extra 'metadata' by default
...
safes one HTTP request that not needed with default filename settings
3 years ago
f587458a3c
[twitter] include '4096x4096' as a default image fallback
...
(closes #2107 , closes #1881 )
3 years ago
f4e3cee6ac
use yt-dlp by default ( #1850 , #2028 )
3 years ago
f1b142e993
{kemonoparty[ change default 'files' order to attachments,file,inline
...
(#1991 )
3 years ago
612850438e
[skeb] add 'thumbnails' option ( #2047 ) ( #2051 )
3 years ago
11a3d96d13
[mangadex] load additional metadata using includes[] directives
...
- always provide 'artist', 'author', and 'group' metadata fields (#2049 )
- remove 'metadata' option
3 years ago
c67756e187
[kemonoparty] add 'dms' option ( #2008 )
3 years ago
9bc83af3a6
[kemonoparty] 'postfile' -> 'file' ( #1991 )
...
to stay consistent with the existing file types for kemono
3 years ago
d433735750
[kemonoparty] skip duplicate files ( #2032 , #1991 , #1899 )
...
Extract the SHA-256 file hash from URLs
and skip files with the same hash in the same post.
- provide a 'hash' metadata field (empty string if not available)
- remove 'patreon-skip-file' option
3 years ago
d4ec245554
[kemonoparty] implement a 'files' option ( #1991 )
...
similar to 8d676151
3 years ago
7c50f473e4
[ytdl] document 'cmdline-args' and 'config-file' options ( #1680 )
3 years ago
b315a0ecef
[redgifs] update to API v2 ( #1984 )
3 years ago
f0fc3b0ba1
[kemonoparty] add 'comments' option ( #1980 )
3 years ago
9156e90f1f
[twitter] add 'pinned' option
3 years ago
06b414c9a3
[redgifs] 'gfyId' -> 'id' ( #1984 )
3 years ago
02a247f4e5
[deviantart] full resolution for non-downloadable images ( #293 )
...
Many thanks to @Ironchest337 for discovering this method
and providing a well-documented implementation.
3 years ago
8bb442f20d
[redgifs][gfycat] provide fallback URLs ( fixes #1962 )
...
and extend the 'format' option
3 years ago
8d676151b7
[patreon] implement 'files' option ( #1935 )
3 years ago
373d3e1c57
[seisoparty] implement login with username & password ( #1906 )
3 years ago
d93b5474c3
[mangadex] update parameter handling for API requests
...
- move common parameters into '_pagination()'
- add 'ratings' (#1908 ) and 'api-parameters' options
3 years ago
cd66c3c415
[twitter] add 'size' option ( #1881 )
3 years ago
df8050b81d
[postprocessor:compare] add 'equal' option ( #1592 )
...
Move functionality from cdd72e14
3 years ago
cdd72e1413
[postprocessor:compare] extend 'action' option ( #1592 )
...
allow setting it to "abort", "terminate", or "exit" as with 'skip'
3 years ago
8c29a6e491
add example for 'url-metadata' option ( closes #1884 )
3 years ago
d0761454b1
implement a download progress indicator ( #1519 )
3 years ago
140a7bfdcd
update --clear-cache command in deviantart client-id instructions
3 years ago
e69ee41f25
implement 'page-reverse' option ( #1854 )
3 years ago
7b05600056
[pinterest] only mention cookies for authentication ( #1856 )
...
regular login is broken due to CAPTCHA
3 years ago
c9e6693530
allow specifying a minimum/maximum for 'sleep-*' options ( #1835 )
...
for example '"sleep-request": [5.0, 10.0]' to wait between 5 and 10
seconds between each HTTP request
3 years ago
bd845303ad
implement a way to shorten filenames with east-asian characters
...
(#1377 )
Setting 'output.shorten' to "eaw" (East-Asian Width) uses a slower
algorithm that also considers characters with a width > 1.
3 years ago
4ec11af6a4
[kemonoparty] implement login with username & password ( #1824 )
3 years ago
44f572c27f
[deviantart] implement a 'auto-unwatch' option ( #1466 , #1757 )
3 years ago
96fec14ef7
[deviantart] rename 'watch' option to 'auto-watch'
...
(#1466 , #1757 )
Similar reason as in e05a96db
3 years ago
e75f2de9da
[deviantart] add 'comments' option ( #1800 )
3 years ago
ecc8da4704
[deviantart] implement a 'watch' option ( #1466 , #1757 )
3 years ago
d3eab417ed
implement a 'path-strip' option
3 years ago
d320ee6251
implement a 'fallback' option ( closes #1770 )
3 years ago
c866fcba48
[twitter] fix 'logout' ( #1719 )
...
delete 'auth_token' cookie and cookies.txt path
3 years ago
ed4b3c48cb
fix flake8 and other tests
3 years ago
fa6af46756
Added ability to download GIFs instead of mp4 from Luscious and Reactor ( #1701 )
3 years ago
e5a93e113f
[twitter] extend 'replies' option ( #1254 )
...
Allow setting 'replies to '"self"' to only download from self-replies.
3 years ago
cadfad4eea
[danbooru] add 'external' option ( closes #1747 )
3 years ago
5b1c62bfa9
[furaffinity] add 'external' option ( closes #1492 )
3 years ago
a316e44f8e
[bbc] add 'width' option ( #1706 )
3 years ago
7a0da4f93f
[newgrounds] add 'format' option ( closes #1729 )
3 years ago
486474800f
[kemonoparty] skip duplicated patreon files ( closes #1689 )
...
this behavior can be disabled with the 'patreon-skip-file' option
3 years ago
13d4045a8a
add 'archive-prefix' option ( #1711 )
3 years ago
b9783403d9
add 'url-metadata' option ( #1659 , #1073 )
3 years ago
e95f99882f
extend 'parent-metadata' functionality ( #1687 , #1651 , #1364 )
3 years ago
dff0da60f9
[ytdl] add 'generic' option ( #1680 )
3 years ago
36ac2197db
[ytdl] add extractor for sites supported by youtube-dl
...
(#1680 , #878 )
Can be used by prefixing any URL with 'ytdl:',
or by setting 'extractor,ytdl.enabled' to 'true'.
3 years ago
8b036778e3
[kemonoparty] add 'max-posts' option ( #1674 )
3 years ago
312a28e78a
[mastodon] add 'replies' option ( #1669 )
3 years ago
a1f5b78039
[mastodon] add 'reblogs' option ( #1669 )
3 years ago
93d356712c
[mastodon] implement 'text-posts' option ( #1569 )
...
similar to Twitter's 'text-tweets'
3 years ago
963d177a68
document format string syntax
...
or at least attempt to
3 years ago
e661607e8b
[mangadex] document 'metadata' and 'lang' options ( #1535 )
3 years ago
fd00d47116
implement conditional directories ( #1394 )
...
They work the same way as conditional filenames (84d2e640
3 years ago
a751afdfb3
[twitter] change some defaults
...
- 'retweets' option: true -> false
- 'quoted' option : true -> false
i.e. disable downloading tweets from other user's timelines by default
- search directory:
'["{category}", "Search", "{search}"]' ->
'["{category}", "{user[name]}"]'
i.e. change it to the same as other twitter extractors (#1308 )
3 years ago
84d2e64024
combine conditional filenames into `filename` option ( #1394 )
3 years ago
07c8adbd8b
[mangadex] implement login with username & password ( #1535 )
3 years ago
3cbbefd4ed
support 'filter' option for post processors ( #1460 )
3 years ago
4cf40434d7
initial support for conditional filenames ( #1394 )
3 years ago
f3d692a378
improve docs for danbooru and e621 passwords ( #1593 , #1596 )
3 years ago
c80b18a477
[weibo] extend 'retweets' option ( closes #1542 )
...
Setting 'retweets' to "original" will use metadata from the
original posts, and not from the retweeted ones.
3 years ago
74d97e6c6b
[postprocessor:ugoira] add 'repeat-last-frame' option ( #1550 )
3 years ago
be9547a5f3
[postprocessor:ugoira] use 'image2' demuxer ( #1550 )
...
Ensures exact frame timecodes with no duplicate frames.
Possible issues are the duration the last frame in an Ugoira with variable
frame durations is shown and insufficient timestamp precision of the
underlying file system (e.g. FAT32, ext3; works on ext4, tmpfs, NTFS).
3 years ago
b8de0671e5
update description for 'instagram.include'
...
- add double quotes for "reels"
- add "tagged"
3 years ago
e6811c7450
[pixiv] implement 'max-posts' option ( #1558 )
...
* implement max-rank for pixiv
* rename to max-posts and make more generic
3 years ago
b5affc62aa
[twitter] rename 'text-only' to 'text-tweets' ( #570 )
3 years ago
724ca61f36
[twitter] add 'text-only' option ( #570 )
3 years ago
1eabfa5c7a
[pillowfort] implement login with username & password ( #846 )
3 years ago
a7e4917ee1
[pillowfort] add 'inline' option ( #846 )
...
to support images present in a post's 'content',
but not listed in 'media'.
also separates the file hash present at the beginning
of each 'filename' into its own field.
3 years ago
efa6cc8ec3
[pillowfort] add 'external' option ( #846 )
...
for links to external Twitter posts etc.
3 years ago
5eeaaee01d
[pixiv] add 'metadata' option ( #1551 )
3 years ago
0717456b4e
[kemonoparty] add 'metadata' option ( closes #1548 )
...
to fetch creator names with an additional HTTP request
3 years ago
7ab8374385
add 'parent-skip' option ( #1399 )
3 years ago
c693db5b1a
add '"skip": "terminate"' option
...
Stops not only the current extractor/job,
but all parent extractors/jobs as well.
3 years ago
36ed1efcfb
[pixiv] rename "noop" value for 'tags' option to "original"
...
(#1507 )
3 years ago
e300da1424
add 'output.skip' option
3 years ago
fa519f9202
[pixiv] change 'translated-tags' option ( #1507 )
...
- rename to 'tags'
- use string-values: "japanese", "translated", "noop"
- remove duplicate entries for "translated" tags
3 years ago
e47952ac14
add extractors for fantia and fanbox ( #1459 )
...
* add extractors for fantia and fanbox
* appease linter
* make docstrings unique
* [fantia] refactor post extraction
* [fantia] capitalize
* [fantia] improve regex pattern
* code style
* capitalize
* [fanbox] use BASE_PATTERN for url regexes
* [fanbox] refactor metadata and post extraction
* [fanbox] improve url base pattern
* [fanbox] accept creator page links ending with /posts
* [fanbox] more tests
* [fantia] improved pagination
* [fanbox] misc. code logic improvements
* [fantia] finish restructuring pagination code
* [fanbox] avoid making a request for each individual post when processing a creator page
* [fanbox] support embedded videos
* [fanbox] fix errors
* [fanbox] document extractor.fanbox.videos
* [fanbox] handle "article" and "entry" post types, all embeds
* [fanbox] fix downloading of embedded fanbox posts
3 years ago
9514cb8c12
[exhentai] update 'limits' check ( #1487 )
...
Only use 'limits' to set a custom upper bound.
Checking if the actual maximum gets exceeded is not necessary.
3 years ago
9322c5e43b
[exhentai] restore limit config ( #1487 )
...
This partially reverts commit e9ec91c8
3 years ago
224b883ff4
[danbooru] add option for extended metadata extraction ( #1458 )
...
* [danbooru] add option for extended metadata extraction
* appease linter
* [danbooru] update docs/configuration.rst
* [danbooru] rename extended-metadata -> metadata
3 years ago
dff03a6605
[booru] add an option to extract notes (only gelbooru for now) ( #1457 )
...
* [booru] add an option to extract notes (currently implemented only for gelbooru)
* appease linter
* [gelbooru] rename "text" to "body" in note extraction
* add a code comment about reusing return value of _extended_tags
3 years ago
a86ffb04bb
add 'output.fallback' option
...
to enable/disable fallback URLs for -g/--get-urls
3 years ago
d5aad999dc
[tapas] implement login with username & password ( #692 )
4 years ago
e9ec91c811
[exhentai] improve image limits check
...
- check if current image is the '509 Bandwidth Exceeded' notification
(https://ehgt.org/g/509.gif or https://exhentai.org/img/509.gif )
- remove 'limits' option
4 years ago
2846235669
[twitter] allow specifying a custom format for user results
...
(#1337 )
4 years ago
b0438c8f99
Revert "[deviantart] extend 'extra' option"
...
This reverts commit
5ad2b9c82b5c32a7bf5883f465faca#1387 , #1356 )
4 years ago
3378b39719
[twitter] implement 'users' option ( #1337 )
4 years ago
df94182e11
implement 'parent-metadata' option ( #1364 )
...
experimental, might not work as expected, etc.
4 years ago
83f465faca
[deviantart] refactor 'extra' ( #1356 )
...
- change its expected type to string
- let users specify a list of sources (stash, posts) or 'all'
4 years ago
5c32a7bf58
[deviantart] allow selecting source for 'extra' ( #1356 )
...
Setting 'extra' to "stash" or "deviations" will only download embedded
sta.sh content or deviations. 'true' still downloads both.
4 years ago
a677123abb
[instagram] recognize 'reels' as option for 'include' ( #1329 )
4 years ago
7440d1f112
[pixiv] add 'translated-tags' option ( closes #1354 )
...
(a lot more straight forward than I thought ...)
4 years ago
6a28b7045c
update 'docs/gallery-dl.conf'
4 years ago
bae874f370
replace 'wait-min/-max' with 'sleep-request'
...
on exhentai, idolcomplex, reactor
4 years ago
1a38fae785
add option to use different youtube-dl modules ( fixes #1330 )
...
by setting the 'downloader.ytdl.module' value. For example
{
"downloader": {
"ytdl": {
"module": "yt_dlp"
}
}
}
or '-o module=yt_dlp'
4 years ago
e58039358d
[mangadex] use 'api.mangadex.org' as default API server
...
The caching issues seem to be gone.
(#1290 , #1310 )
4 years ago
29ea54dc41
[patreon] use '"browser": "firefox"' by default ( #1117 )
4 years ago
61fbbd2dae
[exhentai] rename metadata fields to match API results ( #1325 )
...
- gallery_id -> gid
- gallery_token -> token
- title_jp -> title_jpn
- visible -> expunged
- gallery_size -> filesize
- count -> filecount
Also changes the function of the 'metadata' option.
It is now boolean and causes extra data fields from the API to be added
instead of completely replacing the data from HTML when activated.
4 years ago
5d69e437d0
[twitter] add option to download all media from a conversation
...
(fixes #1319 )
4 years ago
cf5fa75d4c
add 'browser' option ( #1117 )
...
- change default user agent to Firefox ESR 78 on Windows 10
- remove 'ciphers' option
4 years ago
e5735361ed
[exhentai] add 'metadata' option ( #1325 )
...
to select between gallery metadata from 'api' or 'html'
4 years ago
5ad2b9c82b
[deviantart] extend 'extra' option
...
also download from embedded DeviantArt posts
4 years ago
560277394e
[downloader:http] add 'headers' option ( #1322 )
4 years ago
6b0ecbf6bc
[hentainexus] add 'orignal' option ( #1322 )
4 years ago
8974f0361c
[pixiv] update ( #1304 )
...
- remove login with username & password
- require a refresh token
- add 'oauth:pixiv' functionality
See also:
- https://github.com/upbit/pixivpy/issues/158
- https://gist.github.com/ZipFile/c9ebedb224406f4f11845ab700124362
4 years ago
cdb0b02e30
[pillowfort] add 'reblogs' option ( #846 )
4 years ago
7a096c443f
[unsplash] add 'format' option ( #1197 )
4 years ago
89a2bcbb2d
[furaffinity] add 'descriptions' option ( #1231 )
4 years ago
36f281330a
[newgrounds] fix flash file extraction ( closes #1257 )
...
… and add a 'flash' option to choose between flash and video formats.
4 years ago
c008cb5100
[pixiv] add 'related' option ( #1237 )
4 years ago
01b9ccd4de
[derpibooru] use the "Everything" filter by default ( #1243 )
...
when neither 'api-key' nor 'filter' are set
4 years ago
Mike Fährmann
