07c8adbd8b
[mangadex] implement login with username & password ( #1535 )
3 years ago
b93cbe6720
[postprocessor:zip] run 'finalize()' regardless of 'filter' ( #1460 )
3 years ago
3e332eaf53
[mangadex] update to API v5 ( #1535 )
3 years ago
ff8d0d7280
[postprocessor:metadata] ensure reliable tags order ( fixes #1598 )
3 years ago
04f4f9badb
[oauth] prevent exceptions when reporting errors ( #1603 )
3 years ago
dfe1e09df0
add '-T/--terminate' command-line option ( #1399 )
3 years ago
6ecb710a7f
add '-P/--postprocessor' command-line option
3 years ago
83fc4c1098
update post processor config capabilities
...
This change makes it possible to specify just the name of a post processor
in the "postprocessors" list instead of a dict with all of its options.
The options for it will then be taken from inside the "postprocessor"
block similar to "extractor", "downloader", or "output" blocks.
This makes it possible to for example override the default settings for
--write-metadata by specifying a custom "metadata" block, or to set a
custom post processor block ("cbz") and then use it by referencing just
its name in "postprocessors" lists.
{
"postprocessor":
{
"metadata": {
"name": "metadata",
"event": "post",
"filename": "{tweet_id|post_id|id}.json"
},
"cbz": {
"name" : "zip",
"compression": "store",
"extension" : "cbz"
}
}
}
3 years ago
4a747a31a3
[postprocessor:metadata] handle dicts in mode;tags ( fixes #1598 )
3 years ago
a3bf878329
[idolcomplex] improve and fix pagination ( #1601 )
...
always rely on the 'next-page-url' value and its query parameters
3 years ago
3cbbefd4ed
support 'filter' option for post processors ( #1460 )
3 years ago
4cf40434d7
initial support for conditional filenames ( #1394 )
3 years ago
0abad8bc12
implement 'compile_expression()'
3 years ago
e39c4633ba
[cyberdrop] b64decode -> a2b_base64
3 years ago
407627ec86
[foolfuuka] support 'archive.wakarimasen.moe' ( closes #1595 )
3 years ago
78f89d2e61
[idolcomplex] fix pagination ( closes #1594 )
3 years ago
f14a36cfc9
release version 1.17.5
3 years ago
52052a0e1a
[manganelo] update domain to 'manganato.com'
3 years ago
c80b18a477
[weibo] extend 'retweets' option ( closes #1542 )
...
Setting 'retweets' to "original" will use metadata from the
original posts, and not from the retweeted ones.
3 years ago
c0fa5058da
[kemonoparty] actually add a 'type' metadata field ( #1556 )
3 years ago
bb50e85795
[postprocessor:ugoira] optimize writing ffconcat files
...
collect all content in-memory first and
write everything with a single 'write()'
3 years ago
5c18db5fab
[postprocessor:ugoira] use 'concat' demuxer by default ( #1550 )
...
'image2' with nanasecond mtime timestamps doesn't work on Windows
3 years ago
19a11fa11d
[postprocessor:ugoira] use frame extension from filenames ( #1550 )
3 years ago
74d97e6c6b
[postprocessor:ugoira] add 'repeat-last-frame' option ( #1550 )
3 years ago
be9547a5f3
[postprocessor:ugoira] use 'image2' demuxer ( #1550 )
...
Ensures exact frame timecodes with no duplicate frames.
Possible issues are the duration the last frame in an Ugoira with variable
frame durations is shown and insufficient timestamp precision of the
underlying file system (e.g. FAT32, ext3; works on ext4, tmpfs, NTFS).
3 years ago
264beb8556
recognize v2.mangapark URLs ( #1578 )
...
* recognize v2.mangapark URLs
* update mangapark root url to use the v2 subdomain
3 years ago
e6811c7450
[pixiv] implement 'max-posts' option ( #1558 )
...
* implement max-rank for pixiv
* rename to max-posts and make more generic
3 years ago
8a909e478d
[imagebam] fix extraction of NSFW images ( #1534 )
3 years ago
b5affc62aa
[twitter] rename 'text-only' to 'text-tweets' ( #570 )
3 years ago
724ca61f36
[twitter] add 'text-only' option ( #570 )
3 years ago
8fd8126117
fix ISO 639-1 code for Japanese
...
"jp" -> "ja"
3 years ago
2c60c7d798
[reactor] skip deleted/empty posts
3 years ago
532ac79fb0
update extractor test results
3 years ago
d7bc4a2b8b
[500px] update query hashes
3 years ago
0f35aca728
[aryion] minor code updates
3 years ago
2eb46452ad
[aryion] update 'needle' to not skip text posts ( fixes #1568 )
...
on "Latest Updates" pages
"class='thumb scrollthumb' href='/g4/view/" and
"class='thumb' href='/g4/view/" both end with
"thumb' href='/g4/view/"
3 years ago
adf4d661b3
use '_extractor' info in UrlJobs
3 years ago
4fc9668922
[imgur] update URL patterns ( #1561 )
3 years ago
1eabfa5c7a
[pillowfort] implement login with username & password ( #846 )
3 years ago
24dd10ac3c
[patreon] extract user defined 'tags' ( #1539 , closes #1540 )
3 years ago
a7e4917ee1
[pillowfort] add 'inline' option ( #846 )
...
to support images present in a post's 'content',
but not listed in 'media'.
also separates the file hash present at the beginning
of each 'filename' into its own field.
3 years ago
efa6cc8ec3
[pillowfort] add 'external' option ( #846 )
...
for links to external Twitter posts etc.
3 years ago
394fbb5f56
[twitter] strip useless t.co links ( #1532 )
...
The 'full_text' of Tweets with media content usually ends with a t.co
link to itself. This commit removes those.
3 years ago
3a7c3ff138
support XDG_CONFIG_HOME ( closes #1545 )
...
This will only load either
${XDG_CONFIG_HOME}/gallery-dl/config.json or
${HOME}/.config/gallery-dl/config.json
if XDG_CONFIG_HOME is empty, never both.
3 years ago
41457dbb1b
[twitter] resolve t.co URLs in 'content' ( #1532 )
3 years ago
2b5d80862e
[kemonoparty] add 'type' metadata field ( #1556 )
...
'file', 'attachment', or 'inline'
3 years ago
17b0ccb071
[twitter] add missing retweet media entities ( fixes #1555 )
...
from the original tweets
3 years ago
5eeaaee01d
[pixiv] add 'metadata' option ( #1551 )
3 years ago
0717456b4e
[kemonoparty] add 'metadata' option ( closes #1548 )
...
to fetch creator names with an additional HTTP request
3 years ago
b50b8e6cf4
refactor applying 'parent-…' options
3 years ago
7ab8374385
add 'parent-skip' option ( #1399 )
3 years ago
c693db5b1a
add '"skip": "terminate"' option
...
Stops not only the current extractor/job,
but all parent extractors/jobs as well.
3 years ago
4835888acc
release version 1.17.4
3 years ago
36ed1efcfb
[pixiv] rename "noop" value for 'tags' option to "original"
...
(#1507 )
3 years ago
14f983eab6
[deviantart] use default ID when 'client-id' is None
3 years ago
3e4ffb0821
[gelbooru] add extractor for '/redirect.php' URLs ( #1530 )
3 years ago
5e54105ae4
[instagram] update query hashes
3 years ago
b3ee10a7fb
[500px] update query hashes
3 years ago
15b0241bbc
[imagebam] fix extraction
3 years ago
38ae61edd4
[inkbunny] add 'favorite' extractor ( #1521 )
3 years ago
577fffad5f
[nozomi] update 'archive_fmt' values for tag and search extractors
...
… so they actually work for posts with more than 1 file.
(fixes #1523 )
3 years ago
e300da1424
add 'output.skip' option
3 years ago
c5ca7905ce
add 'noop()' and 'identity()' functions
3 years ago
755164b36a
improve --clear-cache ( #1230 )
...
Allow for an optional argument to only delete cached entries from
a specific module.
delete all cache entries
$ gallery-dl --clear-cache
or
$ gallery-dl --clear-cache all
only delete entries for instagram
$ gallery-dl --clear-cache instagram
3 years ago
e13cae182b
[nozomi] Extend default archive-fmt for Tag and Search Extractor ( #1529 )
...
Closes #1523
3 years ago
bc868e7bb8
consider apparently long extensions as part of the filename
...
(#1516 )
3 years ago
2133f1d77f
[readcomiconline] change domain to 'readcomiconline.li'
...
(closes #1517 )
3 years ago
66f28e471c
[kemonoparty] update file URLs directly linking to kemono.party
...
(#1514 )
3 years ago
6fa20d456b
[sankaku] update invalid-token detection ( fixes #1515 )
3 years ago
4b65ebf652
[kemonoparty] fix file URLs ( #1514 )
...
files are now hosted on https://data.kemono.party/
3 years ago
fa519f9202
[pixiv] change 'translated-tags' option ( #1507 )
...
- rename to 'tags'
- use string-values: "japanese", "translated", "noop"
- remove duplicate entries for "translated" tags
3 years ago
5b4da4b4bf
reorder config access in Job constructor
...
(#1111 )
3 years ago
221015e586
[downloader:http] disable filename extension changes for ugoira
...
(#1507 )
3 years ago
e5123f56c9
fix crash when using --no-download with --ugoira-conv ( #1507 )
3 years ago
07b6661a87
release version 1.17.3
3 years ago
e47952ac14
add extractors for fantia and fanbox ( #1459 )
...
* add extractors for fantia and fanbox
* appease linter
* make docstrings unique
* [fantia] refactor post extraction
* [fantia] capitalize
* [fantia] improve regex pattern
* code style
* capitalize
* [fanbox] use BASE_PATTERN for url regexes
* [fanbox] refactor metadata and post extraction
* [fanbox] improve url base pattern
* [fanbox] accept creator page links ending with /posts
* [fanbox] more tests
* [fantia] improved pagination
* [fanbox] misc. code logic improvements
* [fantia] finish restructuring pagination code
* [fanbox] avoid making a request for each individual post when processing a creator page
* [fanbox] support embedded videos
* [fanbox] fix errors
* [fanbox] document extractor.fanbox.videos
* [fanbox] handle "article" and "entry" post types, all embeds
* [fanbox] fix downloading of embedded fanbox posts
3 years ago
d900edfcfb
[simplyhentai] fix extraction
3 years ago
ba8180b5e6
[bcy] don't crash with deleted posts
3 years ago
d108421461
[myportfolio] fix extraction
3 years ago
8b22d4e667
[mangapark] use '"browser": "firefox"' by default
...
to get rid of Cloudflare CAPTCHA resonses
3 years ago
9514cb8c12
[exhentai] update 'limits' check ( #1487 )
...
Only use 'limits' to set a custom upper bound.
Checking if the actual maximum gets exceeded is not necessary.
3 years ago
141ca4ac0a
[pixiv] also save untranslated tags when translated-tags is enabled ( #1501 )
3 years ago
9322c5e43b
[exhentai] restore limit config ( #1487 )
...
This partially reverts commit e9ec91c8
3 years ago
cb86bb9cc9
[hentaicosplays] add 'slug' metadata field ( closes #1483 )
3 years ago
b4ed7cb961
fix 'category-transfer' ( #1111 )
...
broken since commit 055c32e0
3 years ago
dddda7d0e7
[hentaicosplays] use GalleryExtractor ( #1473 )
3 years ago
d88e34f17e
[webtoons] use GalleryExtractor
3 years ago
c4210b5371
[webtoons] update agegate/GDPR cookies
3 years ago
d89eb7536b
[naverwebtoon] use GalleryExtractor
3 years ago
9b52eb9bf1
[naverwebtoon] ignore non-comic images
3 years ago
bdfcc9c4b1
update extractor test results
3 years ago
334d690687
[hentaicosplays] Add extractor ( #1473 )
3 years ago
82c32d25af
[500px] update query hashes
3 years ago
de14b7ad7a
[slideshare] fix extraction
3 years ago
bef3105121
[komikcast] fix extraction
3 years ago
086925e685
[shopify] support omgmiamiswimwear.com ( closes #1280 )
3 years ago
224b883ff4
[danbooru] add option for extended metadata extraction ( #1458 )
...
* [danbooru] add option for extended metadata extraction
* appease linter
* [danbooru] update docs/configuration.rst
* [danbooru] rename extended-metadata -> metadata
3 years ago
dff03a6605
[booru] add an option to extract notes (only gelbooru for now) ( #1457 )
...
* [booru] add an option to extract notes (currently implemented only for gelbooru)
* appease linter
* [gelbooru] rename "text" to "body" in note extraction
* add a code comment about reusing return value of _extended_tags
3 years ago
78d7ee3ef4
[yuki] remove module for yuki.la
3 years ago
a86ffb04bb
add 'output.fallback' option
...
to enable/disable fallback URLs for -g/--get-urls
3 years ago
Mike Fährmann