2eb46452ad
[aryion] update 'needle' to not skip text posts ( fixes #1568 )
...
on "Latest Updates" pages
"class='thumb scrollthumb' href='/g4/view/" and
"class='thumb' href='/g4/view/" both end with
"thumb' href='/g4/view/"
3 years ago
adf4d661b3
use '_extractor' info in UrlJobs
3 years ago
4fc9668922
[imgur] update URL patterns ( #1561 )
3 years ago
1eabfa5c7a
[pillowfort] implement login with username & password ( #846 )
3 years ago
24dd10ac3c
[patreon] extract user defined 'tags' ( #1539 , closes #1540 )
3 years ago
a7e4917ee1
[pillowfort] add 'inline' option ( #846 )
...
to support images present in a post's 'content',
but not listed in 'media'.
also separates the file hash present at the beginning
of each 'filename' into its own field.
3 years ago
efa6cc8ec3
[pillowfort] add 'external' option ( #846 )
...
for links to external Twitter posts etc.
3 years ago
394fbb5f56
[twitter] strip useless t.co links ( #1532 )
...
The 'full_text' of Tweets with media content usually ends with a t.co
link to itself. This commit removes those.
3 years ago
3a7c3ff138
support XDG_CONFIG_HOME ( closes #1545 )
...
This will only load either
${XDG_CONFIG_HOME}/gallery-dl/config.json or
${HOME}/.config/gallery-dl/config.json
if XDG_CONFIG_HOME is empty, never both.
3 years ago
41457dbb1b
[twitter] resolve t.co URLs in 'content' ( #1532 )
3 years ago
2b5d80862e
[kemonoparty] add 'type' metadata field ( #1556 )
...
'file', 'attachment', or 'inline'
3 years ago
17b0ccb071
[twitter] add missing retweet media entities ( fixes #1555 )
...
from the original tweets
3 years ago
5eeaaee01d
[pixiv] add 'metadata' option ( #1551 )
3 years ago
0717456b4e
[kemonoparty] add 'metadata' option ( closes #1548 )
...
to fetch creator names with an additional HTTP request
3 years ago
b50b8e6cf4
refactor applying 'parent-…' options
3 years ago
7ab8374385
add 'parent-skip' option ( #1399 )
3 years ago
c693db5b1a
add '"skip": "terminate"' option
...
Stops not only the current extractor/job,
but all parent extractors/jobs as well.
3 years ago
4835888acc
release version 1.17.4
3 years ago
36ed1efcfb
[pixiv] rename "noop" value for 'tags' option to "original"
...
(#1507 )
3 years ago
14f983eab6
[deviantart] use default ID when 'client-id' is None
3 years ago
3e4ffb0821
[gelbooru] add extractor for '/redirect.php' URLs ( #1530 )
3 years ago
5e54105ae4
[instagram] update query hashes
3 years ago
b3ee10a7fb
[500px] update query hashes
3 years ago
15b0241bbc
[imagebam] fix extraction
3 years ago
38ae61edd4
[inkbunny] add 'favorite' extractor ( #1521 )
3 years ago
577fffad5f
[nozomi] update 'archive_fmt' values for tag and search extractors
...
… so they actually work for posts with more than 1 file.
(fixes #1523 )
3 years ago
e300da1424
add 'output.skip' option
3 years ago
c5ca7905ce
add 'noop()' and 'identity()' functions
3 years ago
755164b36a
improve --clear-cache ( #1230 )
...
Allow for an optional argument to only delete cached entries from
a specific module.
delete all cache entries
$ gallery-dl --clear-cache
or
$ gallery-dl --clear-cache all
only delete entries for instagram
$ gallery-dl --clear-cache instagram
3 years ago
e13cae182b
[nozomi] Extend default archive-fmt for Tag and Search Extractor ( #1529 )
...
Closes #1523
3 years ago
bc868e7bb8
consider apparently long extensions as part of the filename
...
(#1516 )
3 years ago
2133f1d77f
[readcomiconline] change domain to 'readcomiconline.li'
...
(closes #1517 )
3 years ago
66f28e471c
[kemonoparty] update file URLs directly linking to kemono.party
...
(#1514 )
3 years ago
6fa20d456b
[sankaku] update invalid-token detection ( fixes #1515 )
3 years ago
4b65ebf652
[kemonoparty] fix file URLs ( #1514 )
...
files are now hosted on https://data.kemono.party/
3 years ago
fa519f9202
[pixiv] change 'translated-tags' option ( #1507 )
...
- rename to 'tags'
- use string-values: "japanese", "translated", "noop"
- remove duplicate entries for "translated" tags
3 years ago
5b4da4b4bf
reorder config access in Job constructor
...
(#1111 )
3 years ago
221015e586
[downloader:http] disable filename extension changes for ugoira
...
(#1507 )
3 years ago
e5123f56c9
fix crash when using --no-download with --ugoira-conv ( #1507 )
3 years ago
07b6661a87
release version 1.17.3
3 years ago
e47952ac14
add extractors for fantia and fanbox ( #1459 )
...
* add extractors for fantia and fanbox
* appease linter
* make docstrings unique
* [fantia] refactor post extraction
* [fantia] capitalize
* [fantia] improve regex pattern
* code style
* capitalize
* [fanbox] use BASE_PATTERN for url regexes
* [fanbox] refactor metadata and post extraction
* [fanbox] improve url base pattern
* [fanbox] accept creator page links ending with /posts
* [fanbox] more tests
* [fantia] improved pagination
* [fanbox] misc. code logic improvements
* [fantia] finish restructuring pagination code
* [fanbox] avoid making a request for each individual post when processing a creator page
* [fanbox] support embedded videos
* [fanbox] fix errors
* [fanbox] document extractor.fanbox.videos
* [fanbox] handle "article" and "entry" post types, all embeds
* [fanbox] fix downloading of embedded fanbox posts
3 years ago
d900edfcfb
[simplyhentai] fix extraction
3 years ago
ba8180b5e6
[bcy] don't crash with deleted posts
3 years ago
d108421461
[myportfolio] fix extraction
3 years ago
8b22d4e667
[mangapark] use '"browser": "firefox"' by default
...
to get rid of Cloudflare CAPTCHA resonses
3 years ago
9514cb8c12
[exhentai] update 'limits' check ( #1487 )
...
Only use 'limits' to set a custom upper bound.
Checking if the actual maximum gets exceeded is not necessary.
3 years ago
141ca4ac0a
[pixiv] also save untranslated tags when translated-tags is enabled ( #1501 )
3 years ago
9322c5e43b
[exhentai] restore limit config ( #1487 )
...
This partially reverts commit e9ec91c8
3 years ago
cb86bb9cc9
[hentaicosplays] add 'slug' metadata field ( closes #1483 )
3 years ago
b4ed7cb961
fix 'category-transfer' ( #1111 )
...
broken since commit 055c32e0
3 years ago
dddda7d0e7
[hentaicosplays] use GalleryExtractor ( #1473 )
3 years ago
d88e34f17e
[webtoons] use GalleryExtractor
3 years ago
c4210b5371
[webtoons] update agegate/GDPR cookies
3 years ago
d89eb7536b
[naverwebtoon] use GalleryExtractor
3 years ago
9b52eb9bf1
[naverwebtoon] ignore non-comic images
3 years ago
bdfcc9c4b1
update extractor test results
3 years ago
334d690687
[hentaicosplays] Add extractor ( #1473 )
3 years ago
82c32d25af
[500px] update query hashes
3 years ago
de14b7ad7a
[slideshare] fix extraction
3 years ago
bef3105121
[komikcast] fix extraction
3 years ago
086925e685
[shopify] support omgmiamiswimwear.com ( closes #1280 )
3 years ago
224b883ff4
[danbooru] add option for extended metadata extraction ( #1458 )
...
* [danbooru] add option for extended metadata extraction
* appease linter
* [danbooru] update docs/configuration.rst
* [danbooru] rename extended-metadata -> metadata
3 years ago
dff03a6605
[booru] add an option to extract notes (only gelbooru for now) ( #1457 )
...
* [booru] add an option to extract notes (currently implemented only for gelbooru)
* appease linter
* [gelbooru] rename "text" to "body" in note extraction
* add a code comment about reusing return value of _extended_tags
3 years ago
78d7ee3ef4
[yuki] remove module for yuki.la
3 years ago
a86ffb04bb
add 'output.fallback' option
...
to enable/disable fallback URLs for -g/--get-urls
3 years ago
5a98bcec3a
[deviantart] improve folder name matching ( fixes #1451 )
3 years ago
918b0441fb
[gelbooru] fix tag category extraction ( #1455 )
3 years ago
fe6ce5b8f8
[erome] skip deleted albums ( fixes #1447 )
3 years ago
457abf0e71
[deviantart] fix pagination for Eclipse results ( fixes #1444 )
...
- don't crash on missing keys
- use fallback for invalid 'nextOffset' values
4 years ago
dee540050f
[8muses] fix JSON unobfuscation
...
limit the characters that get modified,
leave non-ASCII characters alone
4 years ago
b869b3a9eb
[instagram] fetch media for incomplete GraphSidecar posts
...
GraphSidecar results from /tagged pages don't contain
all media elements, only the first one.
(#1439 )
4 years ago
b0686d2174
[instagram] update query hashes
4 years ago
e8e3717b71
[instagram] add extractor for /tagged posts ( #1439 )
4 years ago
abafe71e04
[exhentai] fix image limit detection ( closes #1437 )
...
check for image limit message when downloading original files
4 years ago
a75e485461
add archive format to InfoJob output ( #875 )
4 years ago
52a7913abe
[artstation] download /4k/ images ( #1422 )
4 years ago
37940193a6
build executables with SOCKS proxy support ( closes #1424 )
4 years ago
41fbc20020
[webtoons]: Add cookie rstagGDPR_DE=true ( #1431 )
4 years ago
583bee7725
release version 1.17.2
4 years ago
e3b9f88540
Add manganelo extractor ( #1415 )
4 years ago
fd858eed7b
[twitter] add 'user_likes' metadata field for liked tweets
...
i.e. the 'screen_name' of the user whose liked tweets get extracted.
Ideally this would replace 'user' or at least be in the same format,
but that would break backwards compatibility or be impossible/too
complicated thanks to API result differences.
(#1421 )
4 years ago
8d124a3766
[twitter] rename variables
4 years ago
105f3c9666
[twitter] add extractor for direct image links ( closes #1417 )
4 years ago
ec3d5d58a8
[vk] improve extractor ( #474 )
...
- fetch all photos
- add 'metadata' option
- fix extracting photos without '?' in URL
4 years ago
ebd142e2a8
[twitter] don't use youtube-dl for cards when videos are disabled
...
(#1416 )
4 years ago
d5aad999dc
[tapas] implement login with username & password ( #692 )
4 years ago
e9ec91c811
[exhentai] improve image limits check
...
- check if current image is the '509 Bandwidth Exceeded' notification
(https://ehgt.org/g/509.gif or https://exhentai.org/img/509.gif )
- remove 'limits' option
4 years ago
387fe415d5
unescape items in text.split_html()
4 years ago
36291176bc
[pinterest] add 'search' extractor ( #1411 )
4 years ago
058cc47e9b
[bcy] improve pagination
4 years ago
ddd48ceee5
update extractor test results
4 years ago
1a540fbe00
[komikcast] fix extraction
4 years ago
78fd63b8f0
remove 'text.clean_xml()'
...
was not used anywhere
4 years ago
8553b218d9
replace calls to 'os.path.splitext()' with 'str.rpartition()'
...
Makes functions who used it more than twice as fast
and we can get rid of an import as well.
4 years ago
5aa30c3669
[tapas] add 'series' and 'episode' extractors ( #692 )
4 years ago
ccfa5a8694
[twitter] better error message when logging in with 2FA ( #1409 )
4 years ago
214ecf62ce
[deviantart] fix arguments for search/popular results ( #1408 )
4 years ago
522d0a834c
[aryion] Unescape paths too ( #1414 )
...
Without this you'll get paths like this:
- Starcross - Ch. 2 "The Ins and Outs of Sarah"
This commit changes it to:
- Starcross - Ch. 2 "The Ins and Outs of Sarah"
4 years ago
5ad615f0db
fix PixivFavoriteExtractor regex ( #1405 )
...
* fix PixivFavoriteExtractor regex
* do not use lookbehind
4 years ago
62cfee4d28
[vk] initial support for albums ( #474 )
4 years ago
0e601de67b
[sankaku] simplify 'pool' tags ( #1388 )
...
normalize 'tags' and 'artist_tags' to a string-list
4 years ago
d085ade9d5
[sankaku] add 'tag_string' metadata field ( #1388 )
...
The 'join()'ed version of 'tags'.
Handling lists in format strings isn't properly supported yet.
4 years ago
2dffd231b7
[sankaku] add enumeration index for books ( #1388 )
4 years ago
139fb84108
[deviantart] fix username for 'watch' results ( #794 )
...
before it'd use "/" as username
4 years ago
91c2e15da9
[deviantart] add support for posts from watched users ( #794 )
4 years ago
03c20d8c8e
[deviantart] update 'watch' URL pattern ( #794 )
4 years ago
2846235669
[twitter] allow specifying a custom format for user results
...
(#1337 )
4 years ago
bf241811dd
allow '_extractor' fields to be None or empty
4 years ago
dc23cfd684
[deviantart] use fallback for /intermediary/ URLs
...
instead of checking availability with HEAD requests
4 years ago
15daa62842
release version 1.17.1
4 years ago
b0438c8f99
Revert "[deviantart] extend 'extra' option"
...
This reverts commit
5ad2b9c82b5c32a7bf5883f465faca#1387 , #1356 )
4 years ago
58b93635ee
[architizer] add 'firm' extractor ( #1369 )
4 years ago
204523611c
[imgclick] use 'http://' for image URLs
...
The TLS certificate for main.imgclick.net is invalid.
4 years ago
0b55f5ad84
[imgur] fix/improve rate limit handling ( #1386 )
...
- also wait-and-retry on 429 status codes
- use infinite loop instead of recursive calls
- 'extractor.sleep()' -> 'extractor.wait()'
4 years ago
69ca4e29f1
[deviantart] add 'watch' extractor ( #794 )
4 years ago
fcdda6128c
[mangastream] remove module
4 years ago
c677ea19dd
[mangareader] remove module
4 years ago
71523aaab6
[architizer] add 'project' extractor ( #1369 )
4 years ago
3378b39719
[twitter] implement 'users' option ( #1337 )
4 years ago
847e9b0ed7
[philomena] support post URLs without '/images/'
...
e.g. 'derpibooru.org/1'
4 years ago
466966bf83
[hentaicafe] remove module
4 years ago
97641cd151
[hentainexus] remove module
4 years ago
23641742a3
improve 'parent-directory' ( #1364 )
...
Allow forwarding metadata from the top-level extractor to all children
if 'parent-directory' is enabled for all extractors along the way.
For example 'reddit' -> 'gfycat' -> 'redgifs'
4 years ago
c485d0a956
[philomena] add generalized extractors for philomena sites
...
(closes #1379 )
4 years ago
6be7df53da
[hentaifox] improve metadata extraction ( fixes #1378 )
4 years ago
72fe9ac0f3
[gelbooru_v01] support some more boorus by default
...
- https://drawfriends.booru.org/
- https://vidyart.booru.org/
- https://tlb.booru.org/
4 years ago
10c279f285
Weasyl: Drop the `&feature=submit` part of the favourite extractor URL ( #1374 )
...
It's optional and requiring it forces users to escape those URLs because
of the ampersand
4 years ago
df94182e11
implement 'parent-metadata' option ( #1364 )
...
experimental, might not work as expected, etc.
4 years ago
4be27ff0fe
[nozomi] support '/index-N.html' URLs ( closes #1365 )
...
and '/index-Popular-N.html'
4 years ago
780bac4c8a
[gelbooru] update video server ( fixes #1368 )
...
from 'https://img2.gelbooru.com ' to 'https://img3.gelbooru.com '
and provide fallback URLs
4 years ago
f8441e851a
[hentaifox] improve image extraction ( fixes #1366 )
...
build image URLs from embedded JSON data
instead 0f rewriting thumbnail URLs
4 years ago
c7c3fef0bc
[exhentai] support '/tag/' URLs ( closes #1363 )
4 years ago
90830daf85
[exhentai] improve 'favorites' extraction ( closes #1360 )
...
add special cases for when the favorite count is 0 (Never) or 1 (Once)
4 years ago
b6719becf1
ensure '-s/--simulate' always prints filenames ( #1360 )
...
by assuming a potentially wrong filename extension in cases where the
correct one would only get known after a download started
4 years ago
83f465faca
[deviantart] refactor 'extra' ( #1356 )
...
- change its expected type to string
- let users specify a list of sources (stash, posts) or 'all'
4 years ago
5c32a7bf58
[deviantart] allow selecting source for 'extra' ( #1356 )
...
Setting 'extra' to "stash" or "deviations" will only download embedded
sta.sh content or deviations. 'true' still downloads both.
4 years ago
a677123abb
[instagram] recognize 'reels' as option for 'include' ( #1329 )
4 years ago
94faf8c85a
add type check before applying 'browser' option ( fixes #1358 )
4 years ago
5cf593a00a
release version 1.17.0
4 years ago
7440d1f112
[pixiv] add 'translated-tags' option ( closes #1354 )
...
(a lot more straight forward than I thought ...)
4 years ago
2e8061091a
Adding handling of several input files ( #1353 )
...
* Adding handling of several input files
* Fixed flake8 error due to bad indenting
4 years ago
106cdc37c0
[instagram] support '/user/reels/' URLs ( closes #1329 )
4 years ago
524ebb133e
[instagram] refactor reel handling
4 years ago
9785c551bc
[500px] skip unavailable photos ( #1335 )
...
instead of crashing with a KeyError exception
4 years ago
6cfc9613fe
update some code in Extractor constructor
...
- combine '_init_headers' and '_emulate_browser' functionality
into new '_init_session'
- add 'headers' and 'ciphers' options
4 years ago
f59e63669b
[hentaicafe] add 'search' and 'tag' extractors ( #1345 )
4 years ago
38e66940c1
[tumblrgallery] simplify
4 years ago
665499924d
Support naver webtoon ( #1331 )
...
* Support naver webtoon (WIP)
* Apply patch
* Change filename format
* Fill test results
* Fill test result
4 years ago
a9119da4d4
support `tumblrgallery.xyz` ( #1298 )
...
* support `tumblrgallery.xyz`
* fix format issues
* Refactor and add post and search page support
* Fix warnings
* Few improvments
* Better file names
* Fix linting errors
* move id closer to the begining of the file name
Co-authored-by: topozorra <none>
4 years ago
c963741860
add '-E/--extractor-info' command-line option ( #875 )
4 years ago
Mike Fährmann
