Mike Fährmann
fd734b9222
[archive] add 'archive-mode' option ( #5255 )
4 months ago
Mike Fährmann
28039229fe
[common] use 'create_urllib3_context' for creating SSLContexts
...
enables dumping TLS session keys by setting SSLKEYLOGFILE (#5215 )
as well as other potentially useful settings.
4 months ago
Mike Fährmann
33006fe126
[common] disable 'check_hostname' for non-urllib3 SSLContexts
...
e.g. when 'browser' is set to a non-empty value and gallery-dl creates
its own SSLContext instance instead of using requests' and urllib3's
defaults.
urllib3 disables this option for its default contexts,
since it does this check on its own.
Fixes "ValueError: Cannot set verify_mode to CERT_NONE when
check_hostname is enabled" when using --no-check-certificate.
(#3614 , #4891 , #5576 )
4 months ago
Mike Fährmann
215abbc3e4
[archive] implement DownloadArchiveMemory class ( #5255 )
...
keeps archive IDs in memory
and only writes them to disk in a 'finalize' step
4 months ago
Mike Fährmann
88f94190f4
[archive] move DownloadArchive into its own module
4 months ago
Mike Fährmann
2ee9ffeed6
merge #5568 : [furaffinity] match 'xfuraffinity' URLs
4 months ago
Mike Fährmann
b27ffd790a
[poipiku] fix downloading R-18 posts ( #5567 )
...
… by automatically sending a `POIPIKU_CONTENTS_VIEW_MODE=1` cookie
to enable "adult" mode.
4 months ago
Delphox
11109d5bad
[furaffinity] match xfuraffinity.com
4 months ago
Mike Fährmann
dc9d83e64b
[output] support 'NO_COLOR' environment variable
5 months ago
Mike Fährmann
625740c3d2
merge #5538 : [cookies] optimize _find_most_recently_used_file for exact profiles
5 months ago
Jan Wikholm
06d102f19a
optimize _find_most_recently_used_file for exact profile
...
When reading cookies from the browser, the user is able to give either just the browser name, or also provide profile/container information.
If an exact profile is provided, there is no need to find the latest profile with `os.walk` which is very expensive.
This change optimizes that case and the performance increase is significant (~8 sec to 0.6 sec).
```
$ time gallery-dl --config-ignore -d . -D . --cookies-from-browser FIREFOX https://imgur.com/OO4UNqJ
[cookies][info] Extracted 16 cookies from Firefox
./imgur_OO4UNqJ.jpg
real 0m8.429s
user 0m0.216s
sys 0m0.431s
$ time gallery-dl --config-ignore -d . -D . --cookies-from-browser FIREFOX:bgamf5r6.default-release https://imgur.com/OO4UNqJ
[cookies][info] Extracted 16 cookies from Firefox
./imgur_OO4UNqJ.jpg
real 0m0.456s
user 0m0.183s
sys 0m0.011s
$ gallery-dl --version
1.26.9
```
5 months ago
Mike Fährmann
bd8e4797e5
[vsco] add 'avatar' extractor ( #5341 )
5 months ago
Mike Fährmann
d0cead105b
[formatter] allow dots etc in '…' literals ( #5539 )
...
don't parse fields starting with '
this disables the ability to directly apply […] to '…' literals,
but that's not really useful anyway and can still be done with _lit
5 months ago
Mike Fährmann
619bf5c644
[inkbunny] retry API calls with a loop instead of recursion
5 months ago
Mike Fährmann
d11ec00908
[common] fix _cfgpath for BaseExtractor objects
...
After the changes in 0c17884673
,
_cfgpath was mssing its 'category' value
since that hadn't been initialized yet.
5 months ago
Mike Fährmann
6db1837a6d
[bluesky] filter reposts only for user timelines ( #5528 )
5 months ago
Mike Fährmann
3cf5366143
[mastodon] add support for card images
5 months ago
Mike Fährmann
9b1995dda3
[mastodon] add 'favorite', 'list', and 'hashtag' extractors ( #5529 )
5 months ago
Mike Fährmann
7fd31aaf8e
merge #5516 : [mastodon] send canonical 'true'/'false' boolean values
...
instead of '1'/'0' to be compatible with Mitra instances or others that
do not fully implement Mastodon's boolean value semantics.
5 months ago
cenodis
3ba5fd9efd
[mastodon] Use boolean instead of integer keys for accounts/statuses endpoint
5 months ago
Mike Fährmann
6c57958806
merge #5511 : [twitter] [furaffinity] match fixvx.com and fxfuraffinity/fxraffinity.net URLs
5 months ago
Mike Fährmann
5aefa9465c
[furaffinity] simplify pattern, match original domain first
5 months ago
Delphox
6969963125
[furaffinity] support fxfuraffinity and fxraffinity urls
5 months ago
Delphox
85550a37c4
[twitter] support fixvx urls
5 months ago
Mike Fährmann
cd241bea0a
[downloader:http] add MIME type and signature for .m4v files ( #5505 )
5 months ago
Mike Fährmann
bef0bd3b49
add '--no-colors' command-line option
5 months ago
Mike Fährmann
20e2c0042b
[output] enable colors by default
5 months ago
Mike Fährmann
14b38264e0
[output] extend 'output.colors' ( #2566 )
...
allow specifying ANSI colors for all loglevels
(debug, info, warning, error)
5 months ago
Mike Fährmann
ddffeeaa7b
[exhentai] detect CAPTCHAs during login ( #5492 )
5 months ago
Mike Fährmann
347af7f5c8
[twitter] handle missing 'expanded_url' fields ( #5463 , #5490 )
5 months ago
Mike Fährmann
c9d3b5e5d9
[pixiv] change 'sanity_level' debug message to a warning ( #5180 )
5 months ago
Mike Fährmann
bffadf35b7
[hotleak] download files with 404 status code ( #5395 )
5 months ago
Mike Fährmann
a7d8cbab0e
[common] show full URL in Extractor.request() error messages
5 months ago
Mike Fährmann
a5071c9ca0
[common] fix NameError in Extractor.request()
...
… when accessing 'code' after an requests exception was raised.
Caused by the changes in 566472f080
5 months ago
Mike Fährmann
566472f080
[common] add 'sleep-429' option ( #5160 )
5 months ago
Mike Fährmann
923c6f3214
[common] simplify 'status_code' check in Extractor.request()
5 months ago
Mike Fährmann
68f4208251
[common] update Extractor.wait() message format
5 months ago
Mike Fährmann
9e5d65fbf3
[twitter] improve username & password login procedure ( #5445 )
...
- handle more subtasks
- support 2FA
- support email verification codes
5 months ago
Mike Fährmann
b38a917355
[common] add Extractor.input() method
5 months ago
Mike Fährmann
257e9fb435
[gelbooru] improve pagination logic for meta tags ( #5478 )
...
similar to 494acabd38
5 months ago
Aidan Harris
a2affdcef3
re-order verbose and warning options
5 months ago
Aidan Harris
a3f580254a
Add warnings flag
...
This commit adds a warnings flag
It can be combined with -q / --quiet to display warnings.
The intent is to provide a silent option that still surfaces
warning and error messages so that they are visible in logs.
5 months ago
Mike Fährmann
0d72789aa3
merge #5461 : [cookies] use tempfile when saving cookies.txt files
5 months ago
Mike Fährmann
63ac06643f
compute tempfile path only once
5 months ago
Mike Fährmann
5227bb6b1d
[text] catch general Exceptions
5 months ago
Mike Fährmann
141a93c8fd
[docs] update docs/configuration links ( #5059 , #5369 , #5423 )
5 months ago
Mike Fährmann
85bbb59483
[twitter] implement 'relogin' option ( #5445 )
5 months ago
Mike Fährmann
e02d2ff45d
[tapas] add 'creator' extractor ( #5306 )
5 months ago
Mike Fährmann
35d4a706ae
[pixiv:novel] add 'covers' option ( #5373 )
5 months ago
Mike Fährmann
b57051719f
[wikimedia] support wiki.gg wikis
5 months ago
Mike Fährmann
40c1a8e471
[wikimedia] fix exception for files with empty 'metadata'
5 months ago
Mike Fährmann
0e730ba980
[pp:mtime] do not overwrite '_mtime' for None values ( #5439 )
6 months ago
Mike Fährmann
647a87d17c
[twitter] match '/photo/' Tweet URLs ( #5443 )
...
fixes regression introduced in 40c05535
6 months ago
Mike Fährmann
3346a377b3
[pp:ugoira] log errors for general exceptions
6 months ago
Mike Fährmann
40bd145637
remove 'contextlib' imports
6 months ago
Mike Fährmann
9a8403917a
restore LD_LIBRARY_PATH for PyInstaller builds ( #5421 )
6 months ago
Mike Fährmann
86a97d8e27
[kemonoparty] fix crash on posts with missing datetime info ( #5422 )
6 months ago
Mike Fährmann
ef8f02c3d9
[oauth] use Extractor.request() for HTTP requests ( #5433 )
...
Enables using proxies and general network options.
6 months ago
Mike Fährmann
095e5ded6f
[reddit] support comment embeds ( #5366 )
6 months ago
Mike Fährmann
64948f2c09
[foolfuuka] improve 'board' pattern & support pages ( #5408 )
6 months ago
Mike Fährmann
0c17884673
store 'match' and 'groups' in Extractor objects
6 months ago
Mike Fährmann
bdbc8be5f3
[twitter] prevent crash when extracting 'birdwatch' metadata ( #5403 )
6 months ago
Mike Fährmann
ef0c90414c
[wikimedia] suppress exception for entries without 'imageinfo' ( #5384 )
6 months ago
Mike Fährmann
9cce461627
[kemonoparty] add 'announcements' option ( #5262 )
...
https://github.com/mikf/gallery-dl/issues/5262#issuecomment-2015919188
6 months ago
Mike Fährmann
72ac2c750d
[kemonoparty:favorite] support 'sort' and 'order' query params ( #5375 )
6 months ago
Mike Fährmann
d1d017ab5d
merge #5372 : [twitter] match Tweet URLs with query parameters ( #5371 )
...
fixes regression introduced in 40c05535
6 months ago
fireattack
423599ce95
[twitter] fix pattern for single tweet ( #5371 )
...
- Add optional slash
- Update tests to include some non-standard tweet URLs
6 months ago
Mike Fährmann
15a4bc2584
[kemonoparty] fix KeyError for empty files ( #5368 )
6 months ago
Mike Fährmann
c3bafd6a2b
release version 1.26.9
6 months ago
Mike Fährmann
925123e007
[deviantart] handle CloudFront blocks in general ( #5363 )
...
This was already done for non-OAuth requests (#655 )
but CF is now blocking OAuth API requests as well.
6 months ago
Mike Fährmann
31e7ca73b6
[gelbooru] add 'order-posts' option for favorites ( #5220 )
6 months ago
Mike Fährmann
4b6f47e571
[pornhub:gif] extract 'viewkey' and 'timestamp' metadata ( #4463 )
...
https://github.com/mikf/gallery-dl/issues/4463#issuecomment-2014550302
6 months ago
Mike Fährmann
9f73fac5ef
[steamgriddb] raise proper exception for deleted assets
6 months ago
Mike Fährmann
fe9171508f
[kemonoparty] fix exception for '/revision/' URLs
...
caused by 03a9ce98
6 months ago
Mike Fährmann
a650fd3177
[deviantart] improve 'index' extraction for stash files ( #5335 )
6 months ago
Mike Fährmann
1d6260f456
[bunkr] remove 'description' metadata
...
album descriptions are no longer available on album pages
and the previous code erroneously returned just '0'
6 months ago
Mike Fährmann
32262a048b
[idolcomplex] fix metadata extraction
...
- replace legacy 'id' vales with alphanumeric ones, since the former are
no longer available
- approximate 'vote_average', since the real value is no longer
available
- fix 'vote_count'
6 months ago
Mike Fährmann
77ab015df2
[idolcomplex] support new pool URLs
6 months ago
Mike Fährmann
1b34d5ac40
[subscribestar] fix 'date' metadata
6 months ago
Mike Fährmann
ddb2edfd32
[formatter] fix local DST datetime offsets for ':O'
...
'O' would get the *current* local UTC offset and apply it to all
'datetime' objects it gets applied to.
This would result in a wrong offset if the current offset includes
DST and the target 'datetime' does not or vice-versa.
'O' now determines the correct local UTC offset while respecting DST for
each individual 'datetime'.
6 months ago
wankio
56f2d5a5f2
fapello.py Fullsize image
...
by remove ".md" and ".th" in image url, it will download fullsize of images
6 months ago
wankio
eb673a7204
Update fapello.py
...
get fullsize image instead resized
6 months ago
Mike Fährmann
6d93295fea
[gelbooru] add 'date_favorited' metadata field
6 months ago
Mike Fährmann
0d69af94d5
[gelbooru] detect returned favorites order ( #5220 )
6 months ago
Mike Fährmann
e1c51c0dfb
[vipergirls] add 'domain' option ( #4166 )
6 months ago
Mike Fährmann
ae116812cc
[vipergirls] add 'like' option ( #4166 )
6 months ago
Mike Fährmann
da6ba60331
[bluesky] add 'instance' metadata field ( #4438 )
6 months ago
Mike Fährmann
5c1f5861b6
[flickr] add 'contexts' option ( #5324 )
6 months ago
Mike Fährmann
26bc2d55f4
[hiperdex] update URL patterns & fix 'manga' metadata ( #5340 )
6 months ago
Mike Fährmann
8e694d85c4
[twitter] add 'birdwatch' metadata field ( #5317 )
...
should probably get a better name,
but this is what it's called internally by Twitter
6 months ago
Mike Fährmann
b8e7be225c
merge #5333 : [imagefap] fix folder extractor
6 months ago
Herp
99c53f7fa8
Fix imagefap extrcator
6 months ago
Mike Fährmann
03a9ce9832
[kemonoparty] add 'order-revisions' option ( #5334 )
6 months ago
Mike Fährmann
1418c0ce38
[kemonoparty] add 'revision_count' metadata field ( #5334 )
6 months ago
Mike Fährmann
ddee5ae7c4
[gofile] fix extraction
6 months ago
Mike Fährmann
5716430c35
[deviantart:stash] recognize 'deviantart.com/stash/…' URLs
6 months ago
Mike Fährmann
76683c5f5c
[deviantart:stash] fix 'index' metadata ( #5335 )
6 months ago
Mike Fährmann
5158cbb4c1
[weibo] rework pagination logic ( #4168 )
...
don't automatically stop when receiving an empty status list
shouldn't improve 'tabtype=feed' results, but at least 'tabtype=album'
ones and others using cursors won't end prematurely
6 months ago
Mike Fährmann
d53db6e11a
[twitter] handle "account is temporarily locked" errors ( #5300 )
...
- display proper error message
- add 'locked' option
6 months ago
Mike Fährmann
108abab537
[twitter] add 'protected' metadata field ( #5327 )
...
for 'author' and 'user'
6 months ago
Mike Fährmann
dde822e69a
[instagram] change 'posts are private' exception to a warning ( #5322 )
6 months ago
Mike Fährmann
e40973118d
merge #5323 : [instagram] default 'likes' to '0'
6 months ago
Mike Fährmann
b188cb2729
merge #5321 : [mastodon] fix handling null 'moved' account field
6 months ago
Martin Contento
c210c7a4ed
[instagram] default posts like_count to zero
...
I don't know when/why this happens and for private account reasons can't
add an example here, but i had this error out and it feels like a safe
fix to default to `0` here.
6 months ago
blankie
225d849139
[mastodon] fix handling null 'moved' account field
6 months ago
Mike Fährmann
106dfdb4c3
cleanup sleep-request retry delay code
...
more lines but easier to read I'd say
6 months ago
Mike Fährmann
4565de3c8e
[reddit] provide 'gif' and 'mp4' variant fallbacks ( #5315 )
6 months ago
Mike Fährmann
5842e4928d
[imgur] fail downloads when redirected to 'removed.png' ( #5308 )
6 months ago
Mike Fährmann
ac4e29f70a
[lensdump] support more direct link formats ( #5293 )
6 months ago
Mike Fährmann
146459056c
[reddit] provide 'fallback_url' as video fallback ( #5296 )
7 months ago
Mike Fährmann
d3003f8531
merge #5270 : [imagefap] add 'folder' metadata
7 months ago
Mike Fährmann
05331f9cf1
[imagefap] flake8, cleanup, tests
7 months ago
Mike Fährmann
40c0553523
[twitter] add 'quotes' extractor ( #5262 )
...
https://github.com/mikf/gallery-dl/issues/5262#issuecomment-1981571924
It's implemented as a search for 'quoted_tweet_id:…' on Twitter.
7 months ago
Mike Fährmann
790c0ffb8d
[lensdump] recognize direct image links ( #5293
7 months ago
Mike Fährmann
6d9e3c0eb1
[skeb] add extractor for followed users ( #5290 )
...
needs 'Authorization' header from browser session
-o headers.Authorization="Bearer ey…"
7 months ago
Mike Fährmann
ace16f00f5
[weibo] fix retweets ( #2825 , #3874 , #5263 )
...
- handle 快转 retweets
- disable 'retweets' by default
- skip all retweet media when 'retweets' are disabled
- extract all retweet media when 'retweets' is set to "original"
7 months ago
Mike Fährmann
0676a9d6ec
[weibo] fix 'livephoto' filename extensions ( #5287 )
7 months ago
Mike Fährmann
a8027745e3
[downloader:http] add MIME type and signature for .mov files ( #5287 )
7 months ago
Mike Fährmann
db507e30c7
[pixiv] fix novel text extraction ( #5285 )
...
change to '/webview/v2/novel'
since '/v1/novel/text' does not work anymore
7 months ago
Mike Fährmann
296f20e630
[warosu] fix 'board_name' metadata
7 months ago
Mike Fährmann
24873c2724
[warosu] fix crash for threads with deleted posts ( #5289 )
7 months ago
Mike Fährmann
f296067797
[naver] unescape post 'title' and 'description'
7 months ago
Mike Fährmann
a71cdab53e
merge #5126 : [naver] fix EUC-KR encoding issue in old image URLs
7 months ago
Mike Fährmann
a8d3efbb99
[naver] simplify code + add test
7 months ago
Johann Hong
f64fb8f239
[naver] EUC-KR encoding issue in old image URLs Fix
...
Around October 2010, the image server URL format and file name
encoding changed from EUC-KR to UTF-8.
Modified to detect old URL format and decode image URLs into EUC-KR
- (lint with flake8) Customize conditions
Wrap lines smaller than 79 characters
- (lint with flake8) Customize conditions (2nd try)
- One import per line
- Indent on consecutive lines
- (lint with flake8) Customize conditions (3rd try)
- E128 continuation line under-indented for visual indent
- E123 closing bracket does not match indentation of opening bracket's line
- Update naver.py
Check encoding for all image URLs
7 months ago
Mike Fährmann
7b28418f69
[naver] recognize '.naver' URLs
...
https://blog.naver.com/PostView.naver ?…
7 months ago
Mike Fährmann
a767832332
[deviantart:avatar] ignore default avatars ( #5276 )
7 months ago
Mike Fährmann
0cbc910905
[deviantart:avatar] fix 'index' for avatars without '?' ( #5276 )
7 months ago
Mike Fährmann
6482bbc525
[bluesky] handle different 'embed' structure
7 months ago
Mike Fährmann
1115dccd0d
[bluesky] fix feeds and lists
...
bug introduced in 495c9ee1
7 months ago
Mike Fährmann
1a9b9aa310
[artstation] support video clips ( #2566 , #3309 , #3911 )
...
- add 'videos' and 'previews' options
- fix 403 errors for video previews
7 months ago
termvacycurtocs
f8b037ed40
[Imagefap] Add folder metadata
...
[Imagefap] Add "folder" metadata when downloading a folder or user profile.
No additional request is made to the server.
Use for example with the following configuration :
"parent-metadata": true
"directory":["{category}", "{uploader}", "{folder}", "{gallery_id} {title}"]
7 months ago
Mike Fährmann
982880615d
[deviantart] prevent unnecessary API requests ( #4995 )
...
… when using 'comments-avatars'
This also has the added benefit of making it possible to download
comment avatars from users without a valid user profile entry,
like deleted users.
7 months ago
Mike Fährmann
25d2854272
[deviantart] add 'comments-avatars' option ( #4995 )
7 months ago
Mike Fährmann
218ec1a9ee
[instagram] raise proper error for missing 'reels_media' ( #5257 )
7 months ago
Mike Fährmann
82c73c77b0
[redgifs] make 'date' available for directories ( #5262 )
...
https://github.com/mikf/gallery-dl/issues/5262#issuecomment-1973975415
7 months ago
Mike Fährmann
cf9e99c07b
[artstation] support collections ( #146 )
...
https://github.com/mikf/gallery-dl/issues/146#issuecomment-1972101003
7 months ago
Mike Fährmann
32ec695195
merge #5256 : [wikimedia] add azurlane.koumakan.jp
7 months ago
thatfuckingbird
88a06df165
[wikimedia] add azurlane.koumakan.jp to presets
7 months ago
Mike Fährmann
1db0a587f3
[nitter] ignore invalid Tweets ( #5253 )
...
like "Load newest"
7 months ago
Mike Fährmann
76581c13f7
handle URLs without '/' after their TLD ( #5252 )
7 months ago
Mike Fährmann
a00b171d4e
[bluesky] wait unitl 'RateLimit-Reset' on 429 responses
7 months ago
Mike Fährmann
7d874e2497
[bluesky] improve API error messages
7 months ago
Mike Fährmann
d921d860f1
automatically create directory path for logging files ( #5249 )
7 months ago
Mike Fährmann
495c9ee126
[bluesky] add 'reposts' option ( #4438 , #5248 )
7 months ago
Mike Fährmann
c8b591303f
[paheal] cleanup
7 months ago
Mike Fährmann
8a11b72253
remove extractor/test.py ( #4504 )
7 months ago
Mike Fährmann
fde9e25c9f
[tests:kemonoparty] '.party' -> '.su'
7 months ago
Mike Fährmann
311a21bfb2
[bluesky] fix '/follows' not spawning child extractors ( #5246 )
7 months ago
Mike Fährmann
d3dca68225
[xvideos] fix galleries with more than 500 images ( #5244 )
7 months ago