6110e3f940
[common] fix Logger names of BaseCategory extractors
...
update of d11ec0090c178846
2 months ago
668219b36e
[exhentai] improve error message when temporarily banned ( #5845 )
2 months ago
f7383a56f8
wrap filters/conditionals in a try-except block
...
allows accessing undefined fields without exception or locals().get(…)
but hides mistakes/typos/etc by evaluating to False without feedback
performance loss compared to the previous version without try-except
is negligible (~20ns for me)
2 months ago
c83c812a1e
[instagram][twitter] rename 'profile' to 'info' ( #5262 , #3623 )
2 months ago
16e276fca4
[nijie] fix image URLs for single image posts ( #5842 )
...
fixes regression introduced in 2e11b6e7
2 months ago
eb3ef13d28
include 'zstd' in Accept-Encoding header when supported
...
… and slightly update optional dependency list
2 months ago
8aca0e6970
update default User-Agent header to Firefox 128 ESR
2 months ago
8e747b6dee
[twitter] send initial 'cursor' only when given via option
2 months ago
ff39d28ef4
[twitter] fix 'user' when providing 'cursor' ( #5833 )
2 months ago
65385e09cb
[twitter] remove break ( #5831 )
3 months ago
1656c9287d
[pp:metadata] fix 'directory' when file directory has conditions ( #5728 )
3 months ago
da9916c01f
[pp:metadata] implement format strings for 'directory' ( #5728 )
3 months ago
8f3f061daf
[hentainexus] fix error for spread pages ( #5827 )
3 months ago
b3aded1939
add '--print-traffic'
3 months ago
b376fa814e
[zerochan] handle "KeyError - 'items'" ( #5826 )
...
Zerochan sometimes sends an empty response when there are no more
accessible posts to be had.
3 months ago
97a50a23d2
[twitter] implement 'cursor' support ( #5753 )
3 months ago
4f3be2f571
Remove leftover split() statement
...
Forgot to remove this from a previous implementation attempt
3 months ago
162756b684
[toyhouse] fix Content Warning bypass ( #5820 )
3 months ago
f43bccb5be
[furaffinity] Add 'thumbnail' ( #1284 ) and 'folders' properties
...
Retrieve 'thumbnail' and 'folders' properties for each post.
'thumbnail' (#1284 ):
- Preview image used for search results, writing posts, music, etc.
- Filename format: <post_id>@600-<directory_containing_full_image>.jpg
'Folders' (related to #1817 ):
- A list of all gallery folders containing this post
- Folder name format: [<folder_category> - ]<folder_name>
- Only works on new layout; old layout does not show folders, so list will be empty
A test is included for each property.
3 months ago
e03b99ba0e
[actions] emit logging messages before waiting/exiting/etc
...
distinguish between actions that should run before or after
a logging message gets emitted
3 months ago
f7a6401031
[actions] move LoggerAdapter from 'output' to 'actions'
3 months ago
f41a5065b2
[actions] support multiple actions per pattern
3 months ago
8219a78b8d
[actions] extend 'wait'
...
allow specifying a duration like for --sleep
3 months ago
84b3da54aa
[actions] add 'exec' action ( #5619 )
3 months ago
c9860002ba
[actions] add 'abort' and 'terminate' actions ( #5778 )
3 months ago
d80f4fbc10
[actions] optimize empty patterns
3 months ago
d10bfa9065
[vipergirls] improve 'thread' URL pattern
...
allow for query parameters and fragments at tne end of URLs
3 months ago
ba51c1a8e4
[path] fix 1ca2cf51
3 months ago
1ca2cf51a2
merge #5807 : [path] fix moving files across drives on Windows
3 months ago
de61b3b546
Update path.py
3 months ago
91a989a172
path.py - fix system cannot move file to different drive
...
This fixes the below error message:
OSError(18, 'The system cannot move the file to a different disk drive')
3 months ago
29eb535e18
merge #5802 : [twitter] extract 'bookmark_count' and 'view_count'
3 months ago
ee93dfeb8d
Add `bookmark_count` and `view_count` to the X / Twitter extractor
...
- Parse `view_count` as integer in Twitter extractor
- Style fix
- optimize 'view_count' extraction
3 months ago
c2bf4ff99c
[twitter] add 'profile' extractor ( #3623 )
3 months ago
44896b0296
[instagram] add 'profile' extractor ( #5262 )
...
https://github.com/mikf/gallery-dl/issues/5262#issuecomment-2188915210
3 months ago
ea81fa985f
[archive] implement 'archive-event' option ( #5784 )
...
With this, IDs of skipped files will no longer be written to an archive
by default. Use "archive-event": "file,skip" to restore the previous
behavior.
3 months ago
51fdfbe6fc
[erome] extract 'date' metadata ( #5796 )
3 months ago
9329c30b41
merge #5783 : [subscribestar] fix 'date' parsing in updated posts
3 months ago
ea7ccb55a1
merge #5781 : [cookies] add 'thorium' support
3 months ago
b2a40a5b93
add thorium cookies support
...
- Update cookies.py
- preserve lexicographical order
3 months ago
03486599af
Fix subscribestar date parsing in udated posts
3 months ago
0bb8831853
[twitter] fix 'username-alt' option name ( #5715 )
3 months ago
8bb793e21b
release version 1.27.1
3 months ago
606202fb04
[fanbox] remove debug code ( #5759 )
3 months ago
8f50c04af2
[formatter] implement 'X' format specifier ( #5770 )
3 months ago
b707b31264
merge #5774 : [tcbscans] support other domains
3 months ago
ad25be3420
[fanbox] fix 8452d04a ( #5759 )
...
The previous code would raise an exception when there are no or
more than 1 plans with a fee higher than 'feeRequired'.
3 months ago
5adbfe526d
[tcbscans] support other domains
3 months ago
f58b0e6fc7
[twitter] ignore 'Unavailable' media ( #5736 )
...
… including geo-restricted content.
add 'unavailable' option to allow re-enabling them again
3 months ago
8452d04a33
[fanbox] handle KeyError for no longer existing plans ( #5759 )
...
return the plan of the next higher tier instead
3 months ago
11421cf940
[skeb] fix '429 Too Many Requests' errors ( #5766 )
...
Introduce '_handle_429' method to make it easier for Extractors to react
to 429 errors regardless of 'sleep-429' settings.
3 months ago
60b4541199
improve a1bb3279, fix oauth:pixiv ( #5757 )
...
Check 'input' option only when required.
This also fixes an exception in oauth:pixiv caused by using the same
'_input' name as a method defined there.
3 months ago
9671bd6d40
[actions] fix exception when 'msg' is not a string ( #5683 )
3 months ago
4727ad6f57
[pp:ugoira] include traceback in exception debug log ( #5683 )
3 months ago
a1bb32792b
do not try to read from stdin when it is non-interactive ( #5733 )
...
add '--no-input' command-line option and 'input' config file option
to allow users to manually configure this
3 months ago
76385f5f18
re-categorize some command-line options in --help
3 months ago
ae3706286a
[speakerdeck] inherit from GalleryExtractor
3 months ago
9c65db2a92
consistent 'with open(…) as fp:' syntax
3 months ago
3fc2e61818
add '--config-open' and '--config-status' ( #5713 )
...
adapted from 119a25a59c
3 months ago
a92c0bef96
merge #5730 : [speakerdeck] fix extraction
3 months ago
f272b5d2ac
fix: extractor for speakerdeck.com
3 months ago
c699ce8ebb
[twitter] add 'username-alt' option ( #5715 )
3 months ago
c6fc0281e8
[newgrounds] extend 'format' option ( #5709 )
...
- check more extensions for original formats (mp4, webm, m4v, mov, mkv)
- allow specifying which extensions and recoded formats to check
3 months ago
86f0c3baaf
[szurubooru] support empty tag searches ( #5711 )
3 months ago
40c685dc43
merge #5712 : [hentainexus] update key
3 months ago
87859ff200
fix: static method defines hostname, update url link
3 months ago
68540aef2a
update nexus key
3 months ago
2e11b6e756
[nijie] support downloading videos ( #5707 , #5617 )
3 months ago
e17b1d5073
[deviantart] fix 2c8aa9b2 ( #5696 )
3 months ago
f160859c5c
[hitomi] extract 'title_jpn' metadata ( #5706 )
3 months ago
722b4b513d
merge #5609 : [instagram] extract 'liked' metadata
4 months ago
7b0c07ee69
[instagram] rename to 'liked', enable for GraphQL results
4 months ago
aa9be75d44
[twitter] fix duplicate ArkoseLogin check
...
forgot to replace this in 9e5d65fbf3
4 months ago
2c8aa9b2de
[deviantart] fix watching module ID extraction ( #5696 )
4 months ago
9abeab5ecf
[shimmie2] support 'vidya.pics' ( #5632 )
4 months ago
162d4269ec
[twitter] extend 'ratelimit' option ( #5532 )
...
allow waiting for a set amount of seconds
4 months ago
2cab87c5b6
[kemonoparty:favorite] fix exception when sorting None ( #5692 )
4 months ago
7614bc458e
[util] extend CustomNone with comparison operators
4 months ago
1ce5de0290
[formatter] implement 'C' format specifier ( #5647 )
...
to apply a conversion after ':' or
to apply multiple conversions
for example {tags:CSl} or {tags:J - /Cl}
to convert list to string and lowercase it
4 months ago
9b99d2c886
[philomena] support downloading SVG files ( #5643 )
4 months ago
8fce9ea6d5
[hentainexus] restore module ( #5275 )
...
revert 97641cd151
4 months ago
3ec7ee6baa
[downloader:ytdl] fix exception due to missing 'ext' ( #5675 )
4 months ago
5d3d03a1f1
fix 6cfbc107
...
the former condition would return True for 2.31.*
6cfbc1071f (commitcomment-142642913)
4 months ago
4d11cd9ffb
[vichan] remove 'wikieat.club'
...
redirects to some non-vichan site
4 months ago
31133b97fb
[nitter] remove instances
4 months ago
ce228ee163
[photobucket] remove module
...
had been broken for years and the new site is payed access only
4 months ago
612435480a
[workflows:tests] restore Python 3.5 usage
...
https://github.com/actions/setup-python/issues/866
4 months ago
31bdb288ef
release version 1.27.0
4 months ago
6cfbc1071f
workaround for requests 2.32.3 ( #5665 )
...
manually call 'load_default_certs()' for SSLContexts
in custom HTTPAdapter instances
4 months ago
807e2f7094
[realbooru] fix videos and provide fallback URLs ( #2530 )
...
revert acc94ac187
4 months ago
deacbcbfdf
[tbscans] remove excess whitespace in 'manga'
4 months ago
00440b8fb1
[slideshare] fix extraction
4 months ago
281ab33f87
[4archive] fix extraction
4 months ago
a53db09ca0
[artstation] disable TLS 1.2 ciphers by default ( #5564 , #5658 )
4 months ago
7ba2b209f1
fix exit status of --clear-cache/--list-modules/--list-extractors
...
All three would always cause gallery-dl to exit with status 1,
regardless of error or success.
4 months ago
3d033a48ea
[workflows:executables] change dev/linux binary name to …_linux
...
This reverts commit 05f8435e09
4 months ago
53aadb0c44
[update] implement '--update-to'
...
to switch between update channels (stable or dev (alias nightly & master))
or to install a specific version
Examples:
--update-to dev
--update-to dev@2024.05.25
--update-to v1.25.2
4 months ago
5018bd42f8
[update] implement '--update-check'
...
to check if a new update is available without downloading it
4 months ago
2e323ef0a2
[update] implement '-U/--update'
...
to auto-update executable files in-place
4 months ago
601f5becc8
[version] add __variant__
...
Specifies origin and OS of executable files.
For example 'stable/windows'.
4 months ago
020050ea8b
merge #5641 : [pixeldrain] add support for single file album download
4 months ago
154a890399
[pixeldrain] integrate into 'album' extractor
4 months ago
24e70b956b
[pixeldrain] add support for single file album download
4 months ago
0761b22a7f
[hiperdex] update domain to 'hiperdex.top' ( #5635 )
4 months ago
ea434963ae
[subscribestar] fix file URLs ( #5631 )
4 months ago
6198dfa424
[seiga] re-implement login with username & password
4 months ago
329da4db5e
[newgrounds] update and fix login procedure ( #5109 )
4 months ago
895e633c44
implement 'keywords-eval' option ( #5621 )
...
to allow evaluating 'keywords' values as format strings
4 months ago
bc25636e63
[patreon] fix bootstrap data extraction ( #5624 )
...
again
4 months ago
141efc2ad3
[twitter] transfer 'twitter.com' cookies to 'x.com' ( #5597 )
4 months ago
ece3837408
Enable `user_has_liked` as IG download filter
...
Background and context:
1. Instagram by default requires you to be logged in.
2. Your account can LIKE posts.
3. Getting all of your likes from IG can be a pain.
As an active IG user, I want to be able to download only posts I've liked.
The information of whether or not your logged-in user has liked the target post is available in the API results, but was not available to the end-user for filtering.
This small change allows `user_has_liked` to be used to e.g. download only liked posts of an entire IG account.
Example use:
```
$ gallery-dl --cookies-from-browser FIREFOX--filter 'user_has_liked' https://www.instagram.com/imgur/
```
This will only download the posts that have `has_liked: True` set in the returned REST API data.
Note: it will still need to loop through the entire account's posts since this is done per-post basis instead of looping just the liked posts, but at least you are able to use this IG feature as a download marker.
4 months ago
6276f5cd94
[twitter] update domain to x.com ( #5597 )
4 months ago
bfcb66af05
[poipiku] send LANG cookie to ensure English UI ( #5590 )
4 months ago
f651b3b6ab
merge #5601 : [twitter] match '/video/' Tweet URLs
4 months ago
2dec4bef60
merge #5591 : [poipoku] avoid language-specific extraction
4 months ago
7f1ed909d5
[imgur] match gallery/album/image URLs with title slugs ( #5593 )
4 months ago
8ba73e2ec9
[twitter] match /video/ tweet urls
4 months ago
831f922c1c
[poipoku] avoid language-specific extr
4 months ago
33b07c4603
[twitter] wait for rate limit reset before 429 error ( #5532 )
4 months ago
3b6f306a8b
[exhentai] fix blank page detection
4 months ago
90b3741643
[cookies] set proper 'expires' value for Chrome session cookies
...
https://github.com/yt-dlp/yt-dlp/pull/9747
4 months ago
f178839fc2
[exhentai] fix multi-page viewer detection ( #4969 )
4 months ago
5b6b5dd81b
[8chan] fix downloaded files by sending 'TOS' cookie ( #5578 )
4 months ago
d2f50ecf09
add 'skip-filter' option ( #5255 )
4 months ago
fd734b9222
[archive] add 'archive-mode' option ( #5255 )
4 months ago
28039229fe
[common] use 'create_urllib3_context' for creating SSLContexts
...
enables dumping TLS session keys by setting SSLKEYLOGFILE (#5215 )
as well as other potentially useful settings.
4 months ago
33006fe126
[common] disable 'check_hostname' for non-urllib3 SSLContexts
...
e.g. when 'browser' is set to a non-empty value and gallery-dl creates
its own SSLContext instance instead of using requests' and urllib3's
defaults.
urllib3 disables this option for its default contexts,
since it does this check on its own.
Fixes "ValueError: Cannot set verify_mode to CERT_NONE when
check_hostname is enabled" when using --no-check-certificate.
(#3614 , #4891 , #5576 )
4 months ago
215abbc3e4
[archive] implement DownloadArchiveMemory class ( #5255 )
...
keeps archive IDs in memory
and only writes them to disk in a 'finalize' step
4 months ago
88f94190f4
[archive] move DownloadArchive into its own module
4 months ago
2ee9ffeed6
merge #5568 : [furaffinity] match 'xfuraffinity' URLs
4 months ago
b27ffd790a
[poipiku] fix downloading R-18 posts ( #5567 )
...
… by automatically sending a `POIPIKU_CONTENTS_VIEW_MODE=1` cookie
to enable "adult" mode.
4 months ago
11109d5bad
[furaffinity] match xfuraffinity.com
4 months ago
dc9d83e64b
[output] support 'NO_COLOR' environment variable
5 months ago
625740c3d2
merge #5538 : [cookies] optimize _find_most_recently_used_file for exact profiles
5 months ago
06d102f19a
optimize _find_most_recently_used_file for exact profile
...
When reading cookies from the browser, the user is able to give either just the browser name, or also provide profile/container information.
If an exact profile is provided, there is no need to find the latest profile with `os.walk` which is very expensive.
This change optimizes that case and the performance increase is significant (~8 sec to 0.6 sec).
```
$ time gallery-dl --config-ignore -d . -D . --cookies-from-browser FIREFOX https://imgur.com/OO4UNqJ
[cookies][info] Extracted 16 cookies from Firefox
./imgur_OO4UNqJ.jpg
real 0m8.429s
user 0m0.216s
sys 0m0.431s
$ time gallery-dl --config-ignore -d . -D . --cookies-from-browser FIREFOX:bgamf5r6.default-release https://imgur.com/OO4UNqJ
[cookies][info] Extracted 16 cookies from Firefox
./imgur_OO4UNqJ.jpg
real 0m0.456s
user 0m0.183s
sys 0m0.011s
$ gallery-dl --version
1.26.9
```
5 months ago
bd8e4797e5
[vsco] add 'avatar' extractor ( #5341 )
5 months ago
d0cead105b
[formatter] allow dots etc in '…' literals ( #5539 )
...
don't parse fields starting with '
this disables the ability to directly apply […] to '…' literals,
but that's not really useful anyway and can still be done with _lit
5 months ago
619bf5c644
[inkbunny] retry API calls with a loop instead of recursion
5 months ago
d11ec00908
[common] fix _cfgpath for BaseExtractor objects
...
After the changes in 0c17884673
5 months ago
6db1837a6d
[bluesky] filter reposts only for user timelines ( #5528 )
5 months ago
3cf5366143
[mastodon] add support for card images
5 months ago
9b1995dda3
[mastodon] add 'favorite', 'list', and 'hashtag' extractors ( #5529 )
5 months ago
7fd31aaf8e
merge #5516 : [mastodon] send canonical 'true'/'false' boolean values
...
instead of '1'/'0' to be compatible with Mitra instances or others that
do not fully implement Mastodon's boolean value semantics.
5 months ago
3ba5fd9efd
[mastodon] Use boolean instead of integer keys for accounts/statuses endpoint
5 months ago
6c57958806
merge #5511 : [twitter] [furaffinity] match fixvx.com and fxfuraffinity/fxraffinity.net URLs
5 months ago
5aefa9465c
[furaffinity] simplify pattern, match original domain first
5 months ago
6969963125
[furaffinity] support fxfuraffinity and fxraffinity urls
5 months ago
85550a37c4
[twitter] support fixvx urls
5 months ago
cd241bea0a
[downloader:http] add MIME type and signature for .m4v files ( #5505 )
5 months ago
bef0bd3b49
add '--no-colors' command-line option
5 months ago
20e2c0042b
[output] enable colors by default
5 months ago
14b38264e0
[output] extend 'output.colors' ( #2566 )
...
allow specifying ANSI colors for all loglevels
(debug, info, warning, error)
5 months ago
ddffeeaa7b
[exhentai] detect CAPTCHAs during login ( #5492 )
5 months ago
347af7f5c8
[twitter] handle missing 'expanded_url' fields ( #5463 , #5490 )
5 months ago
c9d3b5e5d9
[pixiv] change 'sanity_level' debug message to a warning ( #5180 )
5 months ago
bffadf35b7
[hotleak] download files with 404 status code ( #5395 )
5 months ago
a7d8cbab0e
[common] show full URL in Extractor.request() error messages
5 months ago
a5071c9ca0
[common] fix NameError in Extractor.request()
...
… when accessing 'code' after an requests exception was raised.
Caused by the changes in 566472f080
5 months ago
566472f080
[common] add 'sleep-429' option ( #5160 )
5 months ago
923c6f3214
[common] simplify 'status_code' check in Extractor.request()
5 months ago
68f4208251
[common] update Extractor.wait() message format
5 months ago
9e5d65fbf3
[twitter] improve username & password login procedure ( #5445 )
...
- handle more subtasks
- support 2FA
- support email verification codes
5 months ago
b38a917355
[common] add Extractor.input() method
5 months ago
257e9fb435
[gelbooru] improve pagination logic for meta tags ( #5478 )
...
similar to 494acabd38
5 months ago
a2affdcef3
re-order verbose and warning options
5 months ago
a3f580254a
Add warnings flag
...
This commit adds a warnings flag
It can be combined with -q / --quiet to display warnings.
The intent is to provide a silent option that still surfaces
warning and error messages so that they are visible in logs.
5 months ago
0d72789aa3
merge #5461 : [cookies] use tempfile when saving cookies.txt files
5 months ago
63ac06643f
compute tempfile path only once
5 months ago
5227bb6b1d
[text] catch general Exceptions
5 months ago
141a93c8fd
[docs] update docs/configuration links ( #5059 , #5369 , #5423 )
5 months ago
85bbb59483
[twitter] implement 'relogin' option ( #5445 )
5 months ago
e02d2ff45d
[tapas] add 'creator' extractor ( #5306 )
5 months ago
35d4a706ae
[pixiv:novel] add 'covers' option ( #5373 )
5 months ago
b57051719f
[wikimedia] support wiki.gg wikis
5 months ago
40c1a8e471
[wikimedia] fix exception for files with empty 'metadata'
5 months ago
0e730ba980
[pp:mtime] do not overwrite '_mtime' for None values ( #5439 )
6 months ago
647a87d17c
[twitter] match '/photo/' Tweet URLs ( #5443 )
...
fixes regression introduced in 40c05535
6 months ago
3346a377b3
[pp:ugoira] log errors for general exceptions
6 months ago
40bd145637
remove 'contextlib' imports
6 months ago
9a8403917a
restore LD_LIBRARY_PATH for PyInstaller builds ( #5421 )
6 months ago
86a97d8e27
[kemonoparty] fix crash on posts with missing datetime info ( #5422 )
6 months ago
ef8f02c3d9
[oauth] use Extractor.request() for HTTP requests ( #5433 )
...
Enables using proxies and general network options.
6 months ago
095e5ded6f
[reddit] support comment embeds ( #5366 )
6 months ago
64948f2c09
[foolfuuka] improve 'board' pattern & support pages ( #5408 )
6 months ago
0c17884673
store 'match' and 'groups' in Extractor objects
6 months ago
bdbc8be5f3
[twitter] prevent crash when extracting 'birdwatch' metadata ( #5403 )
6 months ago
ef0c90414c
[wikimedia] suppress exception for entries without 'imageinfo' ( #5384 )
6 months ago
9cce461627
[kemonoparty] add 'announcements' option ( #5262 )
...
https://github.com/mikf/gallery-dl/issues/5262#issuecomment-2015919188
6 months ago
72ac2c750d
[kemonoparty:favorite] support 'sort' and 'order' query params ( #5375 )
6 months ago
d1d017ab5d
merge #5372 : [twitter] match Tweet URLs with query parameters ( #5371 )
...
fixes regression introduced in 40c05535
6 months ago
423599ce95
[twitter] fix pattern for single tweet ( #5371 )
...
- Add optional slash
- Update tests to include some non-standard tweet URLs
6 months ago
15a4bc2584
[kemonoparty] fix KeyError for empty files ( #5368 )
6 months ago
c3bafd6a2b
release version 1.26.9
6 months ago
925123e007
[deviantart] handle CloudFront blocks in general ( #5363 )
...
This was already done for non-OAuth requests (#655 )
but CF is now blocking OAuth API requests as well.
6 months ago
31e7ca73b6
[gelbooru] add 'order-posts' option for favorites ( #5220 )
6 months ago
4b6f47e571
[pornhub:gif] extract 'viewkey' and 'timestamp' metadata ( #4463 )
...
https://github.com/mikf/gallery-dl/issues/4463#issuecomment-2014550302
6 months ago
9f73fac5ef
[steamgriddb] raise proper exception for deleted assets
6 months ago
fe9171508f
[kemonoparty] fix exception for '/revision/' URLs
...
caused by 03a9ce98
6 months ago
a650fd3177
[deviantart] improve 'index' extraction for stash files ( #5335 )
6 months ago
1d6260f456
[bunkr] remove 'description' metadata
...
album descriptions are no longer available on album pages
and the previous code erroneously returned just '0'
6 months ago
Mike Fährmann