d433735750
[kemonoparty] skip duplicate files ( #2032 , #1991 , #1899 )
...
Extract the SHA-256 file hash from URLs
and skip files with the same hash in the same post.
- provide a 'hash' metadata field (empty string if not available)
- remove 'patreon-skip-file' option
3 years ago
d4ec245554
[kemonoparty] implement a 'files' option ( #1991 )
...
similar to 8d676151
3 years ago
7c50f473e4
[ytdl] document 'cmdline-args' and 'config-file' options ( #1680 )
3 years ago
b315a0ecef
[redgifs] update to API v2 ( #1984 )
3 years ago
f0fc3b0ba1
[kemonoparty] add 'comments' option ( #1980 )
3 years ago
9156e90f1f
[twitter] add 'pinned' option
3 years ago
06b414c9a3
[redgifs] 'gfyId' -> 'id' ( #1984 )
3 years ago
02a247f4e5
[deviantart] full resolution for non-downloadable images ( #293 )
...
Many thanks to @Ironchest337 for discovering this method
and providing a well-documented implementation.
3 years ago
8bb442f20d
[redgifs][gfycat] provide fallback URLs ( fixes #1962 )
...
and extend the 'format' option
3 years ago
8d676151b7
[patreon] implement 'files' option ( #1935 )
3 years ago
373d3e1c57
[seisoparty] implement login with username & password ( #1906 )
3 years ago
d93b5474c3
[mangadex] update parameter handling for API requests
...
- move common parameters into '_pagination()'
- add 'ratings' (#1908 ) and 'api-parameters' options
3 years ago
cd66c3c415
[twitter] add 'size' option ( #1881 )
3 years ago
df8050b81d
[postprocessor:compare] add 'equal' option ( #1592 )
...
Move functionality from cdd72e14
3 years ago
cdd72e1413
[postprocessor:compare] extend 'action' option ( #1592 )
...
allow setting it to "abort", "terminate", or "exit" as with 'skip'
3 years ago
8c29a6e491
add example for 'url-metadata' option ( closes #1884 )
3 years ago
d0761454b1
implement a download progress indicator ( #1519 )
3 years ago
140a7bfdcd
update --clear-cache command in deviantart client-id instructions
3 years ago
e69ee41f25
implement 'page-reverse' option ( #1854 )
3 years ago
7b05600056
[pinterest] only mention cookies for authentication ( #1856 )
...
regular login is broken due to CAPTCHA
3 years ago
c9e6693530
allow specifying a minimum/maximum for 'sleep-*' options ( #1835 )
...
for example '"sleep-request": [5.0, 10.0]' to wait between 5 and 10
seconds between each HTTP request
3 years ago
bd845303ad
implement a way to shorten filenames with east-asian characters
...
(#1377 )
Setting 'output.shorten' to "eaw" (East-Asian Width) uses a slower
algorithm that also considers characters with a width > 1.
3 years ago
4ec11af6a4
[kemonoparty] implement login with username & password ( #1824 )
3 years ago
44f572c27f
[deviantart] implement a 'auto-unwatch' option ( #1466 , #1757 )
3 years ago
96fec14ef7
[deviantart] rename 'watch' option to 'auto-watch'
...
(#1466 , #1757 )
Similar reason as in e05a96db
3 years ago
e75f2de9da
[deviantart] add 'comments' option ( #1800 )
3 years ago
ecc8da4704
[deviantart] implement a 'watch' option ( #1466 , #1757 )
3 years ago
d3eab417ed
implement a 'path-strip' option
3 years ago
d320ee6251
implement a 'fallback' option ( closes #1770 )
3 years ago
c866fcba48
[twitter] fix 'logout' ( #1719 )
...
delete 'auth_token' cookie and cookies.txt path
3 years ago
ed4b3c48cb
fix flake8 and other tests
3 years ago
fa6af46756
Added ability to download GIFs instead of mp4 from Luscious and Reactor ( #1701 )
3 years ago
e5a93e113f
[twitter] extend 'replies' option ( #1254 )
...
Allow setting 'replies to '"self"' to only download from self-replies.
3 years ago
cadfad4eea
[danbooru] add 'external' option ( closes #1747 )
3 years ago
5b1c62bfa9
[furaffinity] add 'external' option ( closes #1492 )
3 years ago
a316e44f8e
[bbc] add 'width' option ( #1706 )
3 years ago
7a0da4f93f
[newgrounds] add 'format' option ( closes #1729 )
3 years ago
486474800f
[kemonoparty] skip duplicated patreon files ( closes #1689 )
...
this behavior can be disabled with the 'patreon-skip-file' option
3 years ago
13d4045a8a
add 'archive-prefix' option ( #1711 )
3 years ago
b9783403d9
add 'url-metadata' option ( #1659 , #1073 )
3 years ago
e95f99882f
extend 'parent-metadata' functionality ( #1687 , #1651 , #1364 )
3 years ago
dff0da60f9
[ytdl] add 'generic' option ( #1680 )
3 years ago
36ac2197db
[ytdl] add extractor for sites supported by youtube-dl
...
(#1680 , #878 )
Can be used by prefixing any URL with 'ytdl:',
or by setting 'extractor,ytdl.enabled' to 'true'.
3 years ago
8b036778e3
[kemonoparty] add 'max-posts' option ( #1674 )
3 years ago
312a28e78a
[mastodon] add 'replies' option ( #1669 )
3 years ago
a1f5b78039
[mastodon] add 'reblogs' option ( #1669 )
3 years ago
93d356712c
[mastodon] implement 'text-posts' option ( #1569 )
...
similar to Twitter's 'text-tweets'
3 years ago
963d177a68
document format string syntax
...
or at least attempt to
3 years ago
e661607e8b
[mangadex] document 'metadata' and 'lang' options ( #1535 )
3 years ago
fd00d47116
implement conditional directories ( #1394 )
...
They work the same way as conditional filenames (84d2e640
3 years ago
a751afdfb3
[twitter] change some defaults
...
- 'retweets' option: true -> false
- 'quoted' option : true -> false
i.e. disable downloading tweets from other user's timelines by default
- search directory:
'["{category}", "Search", "{search}"]' ->
'["{category}", "{user[name]}"]'
i.e. change it to the same as other twitter extractors (#1308 )
3 years ago
84d2e64024
combine conditional filenames into `filename` option ( #1394 )
3 years ago
07c8adbd8b
[mangadex] implement login with username & password ( #1535 )
3 years ago
3cbbefd4ed
support 'filter' option for post processors ( #1460 )
3 years ago
4cf40434d7
initial support for conditional filenames ( #1394 )
3 years ago
f3d692a378
improve docs for danbooru and e621 passwords ( #1593 , #1596 )
3 years ago
c80b18a477
[weibo] extend 'retweets' option ( closes #1542 )
...
Setting 'retweets' to "original" will use metadata from the
original posts, and not from the retweeted ones.
3 years ago
74d97e6c6b
[postprocessor:ugoira] add 'repeat-last-frame' option ( #1550 )
3 years ago
be9547a5f3
[postprocessor:ugoira] use 'image2' demuxer ( #1550 )
...
Ensures exact frame timecodes with no duplicate frames.
Possible issues are the duration the last frame in an Ugoira with variable
frame durations is shown and insufficient timestamp precision of the
underlying file system (e.g. FAT32, ext3; works on ext4, tmpfs, NTFS).
3 years ago
b8de0671e5
update description for 'instagram.include'
...
- add double quotes for "reels"
- add "tagged"
3 years ago
e6811c7450
[pixiv] implement 'max-posts' option ( #1558 )
...
* implement max-rank for pixiv
* rename to max-posts and make more generic
3 years ago
b5affc62aa
[twitter] rename 'text-only' to 'text-tweets' ( #570 )
3 years ago
724ca61f36
[twitter] add 'text-only' option ( #570 )
3 years ago
1eabfa5c7a
[pillowfort] implement login with username & password ( #846 )
3 years ago
a7e4917ee1
[pillowfort] add 'inline' option ( #846 )
...
to support images present in a post's 'content',
but not listed in 'media'.
also separates the file hash present at the beginning
of each 'filename' into its own field.
3 years ago
efa6cc8ec3
[pillowfort] add 'external' option ( #846 )
...
for links to external Twitter posts etc.
3 years ago
5eeaaee01d
[pixiv] add 'metadata' option ( #1551 )
3 years ago
0717456b4e
[kemonoparty] add 'metadata' option ( closes #1548 )
...
to fetch creator names with an additional HTTP request
3 years ago
7ab8374385
add 'parent-skip' option ( #1399 )
3 years ago
c693db5b1a
add '"skip": "terminate"' option
...
Stops not only the current extractor/job,
but all parent extractors/jobs as well.
3 years ago
36ed1efcfb
[pixiv] rename "noop" value for 'tags' option to "original"
...
(#1507 )
3 years ago
e300da1424
add 'output.skip' option
3 years ago
fa519f9202
[pixiv] change 'translated-tags' option ( #1507 )
...
- rename to 'tags'
- use string-values: "japanese", "translated", "noop"
- remove duplicate entries for "translated" tags
3 years ago
e47952ac14
add extractors for fantia and fanbox ( #1459 )
...
* add extractors for fantia and fanbox
* appease linter
* make docstrings unique
* [fantia] refactor post extraction
* [fantia] capitalize
* [fantia] improve regex pattern
* code style
* capitalize
* [fanbox] use BASE_PATTERN for url regexes
* [fanbox] refactor metadata and post extraction
* [fanbox] improve url base pattern
* [fanbox] accept creator page links ending with /posts
* [fanbox] more tests
* [fantia] improved pagination
* [fanbox] misc. code logic improvements
* [fantia] finish restructuring pagination code
* [fanbox] avoid making a request for each individual post when processing a creator page
* [fanbox] support embedded videos
* [fanbox] fix errors
* [fanbox] document extractor.fanbox.videos
* [fanbox] handle "article" and "entry" post types, all embeds
* [fanbox] fix downloading of embedded fanbox posts
3 years ago
9514cb8c12
[exhentai] update 'limits' check ( #1487 )
...
Only use 'limits' to set a custom upper bound.
Checking if the actual maximum gets exceeded is not necessary.
3 years ago
9322c5e43b
[exhentai] restore limit config ( #1487 )
...
This partially reverts commit e9ec91c8
3 years ago
224b883ff4
[danbooru] add option for extended metadata extraction ( #1458 )
...
* [danbooru] add option for extended metadata extraction
* appease linter
* [danbooru] update docs/configuration.rst
* [danbooru] rename extended-metadata -> metadata
3 years ago
dff03a6605
[booru] add an option to extract notes (only gelbooru for now) ( #1457 )
...
* [booru] add an option to extract notes (currently implemented only for gelbooru)
* appease linter
* [gelbooru] rename "text" to "body" in note extraction
* add a code comment about reusing return value of _extended_tags
3 years ago
a86ffb04bb
add 'output.fallback' option
...
to enable/disable fallback URLs for -g/--get-urls
3 years ago
d5aad999dc
[tapas] implement login with username & password ( #692 )
4 years ago
e9ec91c811
[exhentai] improve image limits check
...
- check if current image is the '509 Bandwidth Exceeded' notification
(https://ehgt.org/g/509.gif or https://exhentai.org/img/509.gif )
- remove 'limits' option
4 years ago
2846235669
[twitter] allow specifying a custom format for user results
...
(#1337 )
4 years ago
b0438c8f99
Revert "[deviantart] extend 'extra' option"
...
This reverts commit
5ad2b9c82b5c32a7bf5883f465faca#1387 , #1356 )
4 years ago
3378b39719
[twitter] implement 'users' option ( #1337 )
4 years ago
df94182e11
implement 'parent-metadata' option ( #1364 )
...
experimental, might not work as expected, etc.
4 years ago
83f465faca
[deviantart] refactor 'extra' ( #1356 )
...
- change its expected type to string
- let users specify a list of sources (stash, posts) or 'all'
4 years ago
5c32a7bf58
[deviantart] allow selecting source for 'extra' ( #1356 )
...
Setting 'extra' to "stash" or "deviations" will only download embedded
sta.sh content or deviations. 'true' still downloads both.
4 years ago
a677123abb
[instagram] recognize 'reels' as option for 'include' ( #1329 )
4 years ago
7440d1f112
[pixiv] add 'translated-tags' option ( closes #1354 )
...
(a lot more straight forward than I thought ...)
4 years ago
6a28b7045c
update 'docs/gallery-dl.conf'
4 years ago
bae874f370
replace 'wait-min/-max' with 'sleep-request'
...
on exhentai, idolcomplex, reactor
4 years ago
1a38fae785
add option to use different youtube-dl modules ( fixes #1330 )
...
by setting the 'downloader.ytdl.module' value. For example
{
"downloader": {
"ytdl": {
"module": "yt_dlp"
}
}
}
or '-o module=yt_dlp'
4 years ago
e58039358d
[mangadex] use 'api.mangadex.org' as default API server
...
The caching issues seem to be gone.
(#1290 , #1310 )
4 years ago
29ea54dc41
[patreon] use '"browser": "firefox"' by default ( #1117 )
4 years ago
61fbbd2dae
[exhentai] rename metadata fields to match API results ( #1325 )
...
- gallery_id -> gid
- gallery_token -> token
- title_jp -> title_jpn
- visible -> expunged
- gallery_size -> filesize
- count -> filecount
Also changes the function of the 'metadata' option.
It is now boolean and causes extra data fields from the API to be added
instead of completely replacing the data from HTML when activated.
4 years ago
5d69e437d0
[twitter] add option to download all media from a conversation
...
(fixes #1319 )
4 years ago
cf5fa75d4c
add 'browser' option ( #1117 )
...
- change default user agent to Firefox ESR 78 on Windows 10
- remove 'ciphers' option
4 years ago
e5735361ed
[exhentai] add 'metadata' option ( #1325 )
...
to select between gallery metadata from 'api' or 'html'
4 years ago
5ad2b9c82b
[deviantart] extend 'extra' option
...
also download from embedded DeviantArt posts
4 years ago
560277394e
[downloader:http] add 'headers' option ( #1322 )
4 years ago
6b0ecbf6bc
[hentainexus] add 'orignal' option ( #1322 )
4 years ago
8974f0361c
[pixiv] update ( #1304 )
...
- remove login with username & password
- require a refresh token
- add 'oauth:pixiv' functionality
See also:
- https://github.com/upbit/pixivpy/issues/158
- https://gist.github.com/ZipFile/c9ebedb224406f4f11845ab700124362
4 years ago
cdb0b02e30
[pillowfort] add 'reblogs' option ( #846 )
4 years ago
7a096c443f
[unsplash] add 'format' option ( #1197 )
4 years ago
89a2bcbb2d
[furaffinity] add 'descriptions' option ( #1231 )
4 years ago
36f281330a
[newgrounds] fix flash file extraction ( closes #1257 )
...
… and add a 'flash' option to choose between flash and video formats.
4 years ago
c008cb5100
[pixiv] add 'related' option ( #1237 )
4 years ago
01b9ccd4de
[derpibooru] use the "Everything" filter by default ( #1243 )
...
when neither 'api-key' nor 'filter' are set
4 years ago
5d4494b15f
add "ascii" as a special 'path-restrict' value
4 years ago
fa8ee6eac4
[derpibooru] add search and gallery extractors ( #862 )
4 years ago
371e9ca6df
[pinterest] implement video support ( closes #1189 )
4 years ago
bf629a2818
[instagram] add 'include' option ( closes #1180 )
...
Split the functionality of the old 'user' extractor into separate
'posts' and 'highlights' extractors, which respond to virtual URLs
('/<user>/posts' and '/<user>/highlights')
4 years ago
465015f75a
[sankaku] reimplement login support ( #1176 , #1182 )
4 years ago
b2c55f0a72
[sankaku] remove login support
...
The old login method for 'https://chan.sankakucomplex.com/user/login '
and the cookies it produces have no effect on the results from
'beta.sankakucomplex.com'.
4 years ago
0bc492c0fa
add docs for 'event' and 'filename' options
...
from 9c3568c3ca59bd69
4 years ago
9b1bd09454
change 'extension-map' default
...
Replace all JPEG filename extensions with 'jpg'.
4 years ago
bc7b1d91bc
fix rST markup in configuration.rst
...
[ci skip]
4 years ago
e3480bc8de
implement 'extension-map' option ( #318 )
4 years ago
98a4d86a01
[sankakucomplex] extract videos and embeds ( closes #308 )
4 years ago
d83b95fd28
[postprocessor:metadata] accept a string-list for 'content-format'
...
(closes #1080 )
4 years ago
5bcf28de93
add a 'extractor.modules' option
4 years ago
1686dc1757
[twitter] support media from Cards ( #1005 , #937 )
...
Can be enabled with 'extractor.twitter.cards', but for now disabled by
default because cards can redirect to rather large videos from YouTube
or Twitch.
4 years ago
bac8af69e8
update configuration.rst
...
- add some lines to better explain post processor usage
- syntax highlighting for JSON blocks and other smaller stuff
4 years ago
05d7009cc6
rename 'Authentication' entries in supportedsites.rst
...
- change 'Optional' to 'Supported'
- use 'OAuth' and 'Cookies' in their own
- add link to weasyl API key option
4 years ago
d1c6d78477
fix rST markup in configuration.rst
4 years ago
b8daabc3ca
[pinterest] implement login support ( closes #1055 )
...
being logged allows access to secret/protected boards
4 years ago
3ec60e894a
[weasyl] api-key authentication ( #1057 )
...
* [weasyl] support api keys
* [weasyl] document api-key authentication
* [weasyl] usernames can contain ~
4 years ago
35056a07d1
release version 1.15.1
4 years ago
c874071f5a
[kissmanga] remove module
4 years ago
430b6d6e2e
[twitter] extend 'retweets' option ( closes #1026 )
...
Setting 'retweets' to '"original"' will use metadata from the
original retweeted Tweets, and not from the Retweet entry.
4 years ago
b9bdd2c564
[hentaifoundry] add support for stories ( closes #734 )
4 years ago
117582d798
use definition lists instead of tables in docs/configuration.rst
...
and strip some clutter from generated man pages
4 years ago
6514312126
[nijie] add 'include' option ( closes #1018 )
4 years ago
0d43456323
[hentaifoundry] add 'include' option
4 years ago
1e313d5b84
implement 'sleep-request' option
4 years ago
6a5f6396d4
add clearing cache to dA client-id instructions ( #996 )
4 years ago
3afd362e2e
add 'sleep-extractor' option ( closes #964 )
...
(would have been nice if this were possible without code duplication)
4 years ago
3918b69677
remove 'extractor.blacklist' context manager
4 years ago
c78aa17506
add general 'blacklist' and 'whitelist' options ( #492 , #844 )
4 years ago
5912727b88
support format string replacement fields in archive paths
...
(closes #985 )
4 years ago
f6fd449b59
reduce wait time growth rate from exponential to linear
...
Waiting for 2**N seconds after each error grows too fast.
Simply waiting N seconds seems far more reasonable.
4 years ago
ac3036ef56
add 'filesize-min' and 'filesize-max' options ( closes #780 )
4 years ago
0e941553ec
[aryion] Add username/password support ( #960 )
...
* Add username/password support to aryion extractor
* Update docs to match
* Fix code style
4 years ago
dbbbb21180
[exhentai] add ability to specify custom image limit ( #940 )
4 years ago
901a897e6f
[inkbunny] mention login support
4 years ago
b921fee24d
[inkbunny] fix submission order ( #283 )
...
Getting detailed submission info via /api_submissions.php reordered the
input submissions and sorted them by ID. InkbunnyAPI.detail() now sorts
them back and ensures they are returned in their original order.
This commit also removes the 'metadata' option and always requests
submission descriptions.
4 years ago
c4ed9f4faa
[inkbunny] add 'metadata' option ( #283 )
4 years ago
493cadb1e7
[inkbunny] add 'orderby' option ( #283 )
4 years ago
d5fcffcced
[subscribestar] add login capabilities ( #852 )
4 years ago
ecaecc4064
[exhentai] add 'domain' option ( #897 )
4 years ago
cb0132e441
[khinsider] add 'format' option ( closes #840 )
4 years ago
f1ddbff0b5
[aryion] add 'recursive' option ( fixes #832 )
...
This is enabled by default and will recursively go through all
(sub)folders in an artist's gallery.
The old method of using "Latest Updates" lists can be restored by
disabling this option.
4 years ago
9da2bc67f8
[twitter] add option to filter media from quoted tweets ( #854 )
4 years ago
5ba90f72ca
[pinterest] add support for sections ( closes #835 )
4 years ago
c184cce876
update configuration.rst
...
- fix anonymous links
- update description of 'extractor.twitter.videos'
- document 'extractor.redgifs.format' (#724 )
4 years ago
41d03160ff
[deviantart] also search journals for sta.sh links ( #712 )
...
when 'extra' is enabled
4 years ago
0138e9c1d8
[twitter] remove 'content' option
...
The text content of each tweet is always available as 'full_text'
4 years ago
0f459f340b
[instagram] fix and re-enable login with username&password
...
This reverts commit 3e0848a482#756 , #771 , #797 , #803 )
https://github.com/althonos/InsaLooter/issues/287#issuecomment-630456522
4 years ago
3e0848a482
[instagram] disable login with username&password ( #756 )
4 years ago
da22ea8ced
use %APPDATA%\gallery-dl for config/cache on Windows
4 years ago
b489f4de12
fix internal links in configuration.rst
4 years ago
dfcf2a2c91
write OAuth token to cache by default ( #616 )
4 years ago
ddc253cf9a
implement a 'path-replace' option ( #662 , #755 )
4 years ago
bc53302ad6
extend 'path-restrict' option
...
Allow its value to be a JSON object / Python dict that specifies
a mapping from invalid/unwanted input characters to specific
output characters.
For example {"/": "-", "*": "+"} will transform
"foo / ***bar***" into "foo - +++bar+++"
(closes #662 , #755 )
4 years ago
e19f665a44
[danbooru] change default for 'ugoira' to 'false'
...
Downloading the pre-rendered versions should be a better default
than .zip files with individual frames.
4 years ago
dba87ca99e
[downloader:ytdl] change 'forward-cookies' default to 'false'
...
There are currently no situations where forwarding gallery-dl's cookies
to youtube-dl is necessary, and it only causes problems when forcing
youtube-dl for Twitter video downloads while logged in.
4 years ago
be16d77073
simplify and improve documentation markup ( #742 )
...
- use | to get a more "compact" representation
- link directly to example config files
4 years ago
fe224416bf
Some documentation improvements ( #742 )
...
* Link configuration examples in the intro, see #712
* Clarify overriding conf files by options, see #678
4 years ago
5e2974d699
[weibo] add 'videos' option
4 years ago
9f638c2e01
[twitter] add 'replies' option ( closes #705 )
4 years ago
5b676ea59d
[e621] document username & password support ( #640 )
5 years ago
59edcdc822
[hitomi] restore metadata fields from before f33b13a
...
... and add a 'metadata' option to disable
visiting the gallery page and extracting data from it
if this is not needed.
5 years ago
a45fbc38ea
[pixiv] implement 'avatar' option ( #595 , #623 )
5 years ago
6e759fe2f5
reword note for 'extractor.deviantart.refresh-token' ( #612 )
5 years ago
67ea80222c
link to post processor example in 'pixiv.ugoira' docs
5 years ago
74e684e828
[twitter] change default value for 'videos' to 'true'
...
Every other 'videos' option defaulted to 'true', except Twitter.
5 years ago
ab13f81fd7
add example to 'cookies' docs
5 years ago
c7cf9dd111
[furaffinity] support classic layout ( #284 )
5 years ago
da0d5f6092
[oauth] add 'port' option ( #604 )
5 years ago
091f9a107d
[docs] add examples of how to filter manga chapters by language
...
(closes #599 )
5 years ago
dff33b260c
[reddit] add 'videos' option
5 years ago
56f1c96168
implement 'parent-directory' option ( #551 )
5 years ago
6703b8a86b
[blogger] implement video extraction ( closes #587 )
5 years ago
37247dbaff
miscellaneous fixes
5 years ago
25d5ec4ff3
[twitter] add option to extract TwitPic embeds ( #579 )
5 years ago
254f7c3999
implement a post-processor module to compare file versions
...
(#530 )
5 years ago
87c8b89ddd
[postprocessor:metadata] add 'directory' option ( #520 )
5 years ago
48e42e73fb
[reddit] change default value for 'comments' to '0'
5 years ago
bf658fd84b
[vsco] implement 'videos' option
5 years ago
95c90722ee
[instagram] implement 'videos' option ( closes #521 )
5 years ago
15f9bb3d14
add option to disable pyOpenSSL usage ( #508 )
...
(pyOpenSSL is now disabled by default)
5 years ago
93b43f74f4
update docs/configuration.rst
...
- fix description of "skip": "enumerate"
- use line breaks for lists of multiple examples
- improve formatting of auto-generated man pages
5 years ago
e17907ee2a
change default value of 'cookies-update' to 'true'
5 years ago
26d2334550
[postprocessor:metadata] rename 'format' to 'content-format'
...
Just to be consistent with the other 'extension-format' option name,
and only 'format' is also still accepted.
5 years ago
a412531451
[postprocessor:metadata] implement 'extension-format' option
...
closes #477
5 years ago
d45fabb79d
match user profile handling on deviantart and newgrounds
5 years ago
dd5d2b2eac
[deviantart] add user profile extractor ( #377 , #419 )
5 years ago
9e88e7a344
[postprocessor:exec] improve ( #421 , #413 )
...
- add 'final' option
- include job status in pp finalization
- improve and extend documentation
5 years ago
c01ff78467
[twitter] extend 'videos' option to force extraction with ytdl
...
(closes #459 )
5 years ago
a5e6252137
update deviantart client-id/-secret instructions ( #451 , #424 )
5 years ago
389d2d7e38
implement 'cookies-update' option ( #445 )
5 years ago
1adafdd3d0
document cache file requirement for DeviantArt refresh tokens
5 years ago
8af59a4bba
fix & update docs
...
- update Requests links
- add example for --exec
- set '-dev' version
5 years ago
46ba173ded
[reddit] fix documentation inconsistencies ( closes #429 )
...
- Require 'reddit.comments' to be a number and convert it to an
integer to be extra sure
- Link to the README's OAuth section were appropriate
5 years ago
ee8b654464
[instagram] implement 'highlights' option ( closes #329 )
5 years ago
f02a768b5c
[danbooru] add 'ugoira' option ( #406 )
...
to choose between ZIP archives or converted video files
for Ugoira posts
5 years ago
c6c5cb1898
improve 'deviantart.quality' description
5 years ago
7c09545f70
[downloader:ytdl] add 'outtmpl' option ( #395 )
5 years ago
5a210991b6
Remove control characters from filesystem paths
...
- add 'path-remove' option to specify the set of characters that
should be removed
- rename 'restrict-filenames' to 'path-restrict'
- #348 , #380
5 years ago
e661a1ce19
update links to Request's documentation
5 years ago
8dc42bb178
implement 'enumerate' for 'extractor.skip' ( #306 )
...
[ci skip]
5 years ago
b7fb93e2b2
[downloader:http] add 'adjust-extensions' option
5 years ago
3957d27d79
[deviantart] add 'quality' option ( #369 )
5 years ago
0609afd1e4
update default cache directory ... again
...
Use a 'gallery-dl' subdirectory in ~/.cache to adhere to how other
programs store their cached data, and call os.makedirs() so it also
works without an existing ~/.cache directory.
5 years ago
a8b60b2bd9
change default cache directory for unix systems
...
Use either $XDG_CACHE_HOME or ~/.cache (if the former isn't set)
and store potentially sensitive cookies and tokens in a user's
home directory and not in the world-readable /tmp.
5 years ago
923e1bb714
[docs] Fix inconsistency about which sites have optional authentication ( #359 )
...
* [docs] Fix inconsistency about which sites have optional authentication
* update authentication docs
5 years ago
a90280f4e7
[postprocessor:zip] add 'mode' option ( #355 )
5 years ago
547ea71463
[downloader.ytdl] add 'forward-cookies' option ( #352 )
...
The "long" name is necessary because just calling it 'cookies' would
clash with how the lookup for '--cookies' is implemented.
5 years ago
b1bea8aaeb
add 'restrict-filenames' option ( #348 )
5 years ago
21991acc49
add 'ciphers' option; update default User-Agent
5 years ago
feb98cf196
[twitter] improve 'content' formatting; add option ( #338 )
...
- include emoticons
- leave newlines intact
- remove pic.twitter.com/ links at the end
5 years ago
8d1ae9b715
[tumblr] enable date-min/-max/-format options ( #337 )
5 years ago
16c582aaf9
implement 'mtime' post-processor ( #332 )
...
This can set a file's modification time according to a UNIX timestamp
or a datetime object from its metadata.
5 years ago
62097284fe
add 'download' option ( #220 )
5 years ago
69205df68d
allow '-1' for infinite retries ( #300 )
5 years ago
f7b5c4c3e7
use values of 'retries' options correctly
...
The RE-tries option now specifies exactly that: the maximum number a
failed HTTP request is re-tried. For example a value of 2 will now
correctly stop after 3 attempts: the initial one + 2 re-tries.
The maximum wait-time now also caps at 30min and increases exponentially
for both extractor.request() and downloader.http.download().
5 years ago
f2000a69aa
implement 'image-unique' and 'chapter-unique' options ( #303 )
...
The default value for both is 'false', i.e. duplicate URLs are NOT
ignored.
The previous behavior was to always ignore duplicate URLs to make
'--abort-on-skip' work properly when new images where added to the
beginning of a collection while gallery-dl is running.
5 years ago
15e4ddf46d
implement custom logging formatter
...
supports custom log message formats for each loglevel and, by
extension, custom ANSI codes and colors for errors and warnings
(#304 )
5 years ago
db3f52881a
add 'mtime' option
5 years ago
e05a96db5e
[deviantart] rename 'stash' to 'extra' ( #302 )
...
'stash' is already used as a name for the StashExtractor and therefore
expected to be a dictionary.
5 years ago
c23bf263fe
[deviantart] rename 'external' to 'stash' ( #302 )
...
restrict extracted URLs to ones from https://sta.sh/ ...
5 years ago
2fb85178da
[deviantart] add 'external' option ( #302 )
...
If a description is available, this will extract URLs from the
description text and try to find Extractors for them.
5 years ago
d33f5a7423
[wallhaven] rewrite
...
- use API
- remove login support, add 'api-key' option
- remove support for "alpha" subdomain - alpha.wallhaven.cc used numeric
IDs that can't be translated to the new ID system
- support direct links to wallpapers
5 years ago
f1893b2b5b
[deviantart] add 'folders' option ( #276 )
5 years ago
4465a3ea68
[kissmanga][readcomiconline] add 'captcha' option ( #279 )
...
to configure how to handle CAPTCHA page redirects:
- either interactively wait for the user to solve the CAPTCHA
- or raise StopExtraction like before
5 years ago
9890bfdf23
[flickr] improve code and metadata
...
- simplify pagination
- add more metadata and slightly change its structure
- convert suitable values to int or list
- move keys from ["photo"] to the base level
- proper video support (#246 )
- rename method and variable names to better fit with other extractors
5 years ago
70be494161
[plurk] add a 'comments' options ( #212 )
6 years ago
369eb66125
consistently use '*' for rst lists
6 years ago
c7a6b0ed90
[deviantart] add 'metadata' option ( #189 )
6 years ago
12482553bd
update links to youtube-dl
6 years ago
25aaf55514
[smugmug] improve format selection ( closes #183 )
...
- use original image if available
- support video formats
- remove user info for ImageExtractor (it is no longer possible to get
image owner information for a single image)
6 years ago
ae353ed3b0
provide "extractor" and "job" keys for logging output
...
This allows for stuff like "{extractor.url}" and "{extractor.category}"
in logging format strings.
Accessing 'extractor' and 'job' in any way will return "None" if those
fields aren't defined, i.e. in general logging messages.
6 years ago
134487ffb0
[exhentai] stop extraction if image limit is exceeded ( #141 )
...
can be turned off with the `exhentai.limits' option
6 years ago
b50b30f1c9
[photobucket] download subalbums ( #117 )
6 years ago
277b52101a
add 'category-transfer' option
...
[ci skip]
6 years ago
8aba2bdebf
[postprocessor:metadata] add 'tags' and 'custom' modes ( #135 )
6 years ago
1e4d351ad3
[danbooru] add authentication support ( closes #151 )
...
... via HTTP Basic Auth with username and "password".
The password value in this case is not the account password itself,
but the"api_key" found in your user profile.
6 years ago
06cbf5f9c4
implement 'chapter-reverse' option ( #149 )
...
Setting it to `true` will start with the latest chapter instead of the
first one.
6 years ago
e95b24f056
[reactor] add wait-min & -max options ( #148 )
6 years ago
28f9539551
[tumblr] change default values for post types and inline media
6 years ago
Mike Fährmann
