Mike Fährmann
6e830ffc9e
[kemonoparty] support post searches ( #3385 , #4057 )
11 months ago
Mike Fährmann
aaf539009b
[kemonoparty] initial support for post revisions ( #4498 , #4597 )
...
- single revision
https://kemono.party/SERVICE/user/12345/post/12345/revision/12345
- all revisions
https://kemono.party/SERVICE/user/12345/post/12345/revisions
11 months ago
Mike Fährmann
174191cb79
[kemonoparty] restore discord pagination ( #4676 )
11 months ago
Mike Fährmann
c9a976d8a6
[kemonoparty] various updates and fixes ( #4676 , #4681 )
...
- fix pagination
- fix 'date' metadata
- fix discord channel API endpoint
11 months ago
Klion Xu
dc1c2139b1
fix line too long
11 months ago
Klion Xu
6b22af9720
[kemonoparty] update API endpoint ( #4676 )
11 months ago
Mike Fährmann
bfdc07632a
[deviantart] expand nested comment replies ( #4653 )
11 months ago
Mike Fährmann
390d14dbcc
[chevereto] support 'img.kiwi' and 'deltaporno.com' ( #4664 , #1381 )
11 months ago
Mike Fährmann
727c8eec6c
merge #4667 : [redgifs] fix 'niches' extraction ( #4666 )
11 months ago
Mike Fährmann
2911ed1240
[chevereto] add generic extractors ( #4664 )
...
- support jpgfish
- support pixl.li / pixl.is (#3179 , #4357 )
11 months ago
enduser420
db3363ac0b
[redgifs] fix 'niches' extraction
11 months ago
Mike Fährmann
ade8347ead
[kemonoparty] fix DM dates
11 months ago
Mike Fährmann
6dfe200ae4
[kemonoparty] support discord URLs with channel IDs ( #4662 )
11 months ago
Mike Fährmann
c6a3892210
[imgbb] update username extraction ( #4626 )
11 months ago
Mike Fährmann
830a48bca4
[fantia] bad workaround for 833dce14
( #4627 )
...
at least this makes "filter": "content_num == content_count+1"
with "event": "post-after" work
11 months ago
Mike Fährmann
13ce3a9acb
[warosu] fix extraction ( #4634 )
11 months ago
Mike Fährmann
c4c4e4d2f4
[newgrounds] improve 'art-image' extraction ( #4642 )
...
- download files in original resolution
- replace .webp with extension of first file
11 months ago
Mike Fährmann
833dce141f
[fantia] add 'content_count' and 'content_num' metadata fields ( #4627 )
11 months ago
Mike Fährmann
2d41702762
[deviantart] implement '"group": "skip"' ( #4630 )
11 months ago
Mike Fährmann
992e86ec94
[deviantart] disable 'jwt' ( #4652 )
11 months ago
Mike Fährmann
2974b8e3c8
[moebooru] add 'metadata' option ( #4646 )
...
for extended 'pool' metadata
11 months ago
Mike Fährmann
d194ea68a9
[cookies] open cookie databases in read-only mode
...
bypasses the need to copy the entire database file
might solve #4195
11 months ago
Mike Fährmann
8bb7243c10
[reddit] fix wrong previews ( #4649 )
...
caused by a failed comment URL
using the main submission's preview as fallback
14af15bd
4963bb9b
12 months ago
Mike Fährmann
08bdde5aac
merge #4619 : [twitter] add 'sensitive' metadata field
12 months ago
Mike Fährmann
f3d6aaff13
[twitter] rename to 'sensitive'; use 'tget()'
12 months ago
Mike Fährmann
95c280c59b
[imgbb] update pagination end condition ( #4626 )
12 months ago
Mike Fährmann
2e350dd82a
merge #4626 : [imgbb] fix 'user' extraction, add 'displayname'
12 months ago
Mike Fährmann
a2daa9befe
[imgbb] fix flake8 and username order
12 months ago
Mike Fährmann
67ba4ee842
[pp:exec] support more replacement fields for '--exec' ( #4633 )
...
- {_directory}
- {_filename}
- {_path} (alias for {})
12 months ago
Mike Fährmann
9a008523ac
[hentaifoundry] fix '.swf' file downloads ( #4641 )
12 months ago
Mike Fährmann
15f940819b
[newgrounds] support 'art-image' files ( #4642 )
12 months ago
Mike Fährmann
63db54b905
[patreon] update 'campaign_id' path ( #4639 )
12 months ago
HRXN
b846f56c3a
[imgbb] Fix `user` extraction, add `displayname`
12 months ago
Mike Fährmann
efaab4fbfa
[twitter] fix crash due to missing 'source' ( #4620 )
...
regression caused by 06aaedde
12 months ago
Nahida
3438a3098d
[twitter] add possible_sensitive field
12 months ago
Mike Fährmann
85357c1ef8
release version 1.26.0
12 months ago
Mike Fährmann
64dbc58a5a
[deviantart] update Eclipse API endpoints 2 ( #4615 )
12 months ago
Mike Fährmann
84fbbd96aa
[shimmie2] remove 'meme.museum'
12 months ago
Mike Fährmann
aa77fda78c
[instagram] better error message for invalid users ( #4606 )
12 months ago
Mike Fährmann
482f002e1f
[nsfwalbum] detect '/error.jpg' images ( #4598 )
12 months ago
Mike Fährmann
eb230e4b77
[nsfwalbum] disable Referer headers by default ( #4598 )
12 months ago
Mike Fährmann
b92645cd37
[bunkr] fix extraction ( #4514 , #4532 , #4529 , #4540 )
12 months ago
Mike Fährmann
4477808d1c
fix symlink resolution in __main__.py
...
adapt ytdl order
12 months ago
Mike Fährmann
be17103e21
[regifs] support 'order' parameter for user URLs ( #4583 )
12 months ago
Mike Fährmann
7150c4c76c
fix imports when using the gallery_dl directory as argument ( #4581 )
12 months ago
HRXN
ec91eeb7ef
Update gallery_dl/extractor/reddit.py
...
Co-authored-by: Mike Fährmann <mike_faehrmann@web.de>
12 months ago
HRXN
66613c3a32
[reddit] ignore '/message/compose' URLs without www subdomain
1 year ago
Mike Fährmann
bb39779e1a
[deviantart] use private tokens for 'is_mature' posts ( #4563 )
1 year ago
Mike Fährmann
0c5d8b1505
[deviantart] re-add 'quality' option and 'intermediary' transform
1 year ago
Mike Fährmann
20d1683c47
[deviantart] fix JWT replacement ( #293 , #4548 , #4563 )
...
And again, a huge thank you to @Ironchest337
for discovering this.
1 year ago
Mike Fährmann
d7aac9fc06
[reddit] ignore '/message/compose' URLs ( #4482 )
1 year ago
Mike Fährmann
1e31fce37b
[pillowfort] support '/tagged/' URLs ( #4570 )
1 year ago
Mike Fährmann
1d2fd0b831
[pillowfort] extract 'b2_lg_url' media ( #4570 )
1 year ago
Mike Fährmann
50e2ebaff0
[danbooru] support 'donmai.moe' URLs
1 year ago
Mike Fährmann
82296b1f05
[reddit] add 'previews' option ( #4322 )
...
another way to disable this new behavior
1 year ago
Mike Fährmann
918ba4f847
[redgifs] match gfycat image URLs ( #4558 )
1 year ago
Mike Fährmann
2ad75bab05
[deviantart] add 'is_original' metadata field ( #4559 )
...
true for 'downloadable' content, journals, flash animations,
and images without '/v1/' in their URL; false otherwise
1 year ago
Mike Fährmann
9d8317d963
[deviantart] disable JWT updates ( #4548 , #4563 )
...
back to lowres images ...
1 year ago
Mike Fährmann
8064663bda
[deviantart] update Eclipse API endpoints ( #4553 )
1 year ago
Mike Fährmann
2cd801232b
fix --range causing crashes ( #4557 )
...
regression caused by a383eca7
1 year ago
Mike Fährmann
3528974459
[instagram] handle exceptions due to missing media ( #4555 )
1 year ago
Mike Fährmann
4963bb9b30
[reddit] improve comment metadata v2 ( #4482 )
...
provide main submission metadata at the top level
and comment metadata inside the 'comment' field,
i.e. the other way round than in 1710f1e9
1 year ago
Mike Fährmann
7592c5e566
[patreon] fix extraction ( #4547 )
1 year ago
Mike Fährmann
0655ce1bae
[mangakakalot] update domain
...
the old one still works, but it incurs a redirect
1 year ago
Mike Fährmann
3ecb512722
send Referer headers by default
1 year ago
Mike Fährmann
cb4798f07a
[architizer] fix extraction ( #4537 )
1 year ago
Mike Fährmann
6178177227
[twitter] fix '_extractor' of following results ( #4536 )
...
regression from 20ed647f
1 year ago
Mike Fährmann
d13c82eff1
[kemonoparty] update favorites API endpoint ( #4522 )
1 year ago
Mike Fährmann
27ec653991
fix bug in test_init and update example URLs
1 year ago
Mike Fährmann
24a1d46391
[mastodon] support '/@USER/following' URLs
...
Previously, only '/users/USER/following' got matched.
1 year ago
Mike Fährmann
9f75713e00
[recursive] simplify
1 year ago
Mike Fährmann
899df8f237
remove another '*' for keyword-only arguments
...
076380e0
1 year ago
Mike Fährmann
6ae92da57e
Merge branch 'tests'
1 year ago
Mike Fährmann
32da3c70d3
[behance] handle videos without 'renditions' ( #4523 )
1 year ago
Mike Fährmann
ae5e049c4f
[redgifs] provide 'collection' metadata in a separate field ( #4508 )
...
instead of overwriting the actual metadata
1 year ago
Mike Fährmann
1710f1e983
[reddit] improve comment metadata ( #4482 )
...
- provide 'date'
- make metadata of the main submission available as 'submission[…]'
1 year ago
Mike Fährmann
4cdab8074e
update/fix --list-extractors
1 year ago
Mike Fährmann
a453335a9f
remove test results in extractor modules
...
and add generic example URLs
1 year ago
Mike Fährmann
1d2b5d0c60
update test comment positions
...
always put them above the test they're referring to
1 year ago
Mike Fährmann
93a7a89cf6
[formatter] use value of last alternative ( #4492 )
...
fixes {fieldname|''} evaluating to the value of 'keywords-default'
instead of an empty string
1 year ago
Mike Fährmann
f856987297
[subscribestar] fix preview detection ( #4468 )
...
and show a warning message when posts contain previews
1 year ago
Mike Fährmann
4c0b3d5dc5
[twitter] fix crash when 'sortIndex' is None ( #4499 )
1 year ago
Mike Fährmann
f2de70f254
[gfycat] remove module
1 year ago
Mike Fährmann
6eca1fab9b
[gelbooru_v02] support 'xbooru.com' ( #4493 )
1 year ago
Mike Fährmann
23bac772f2
[jpgfish] update domain to 'jpg1.su' ( #4494 )
1 year ago
Mike Fährmann
ceb59e176f
fix default Firefox user agent string
...
note to self: do not trust some random third-party website
1 year ago
Mike Fährmann
8259a5abe4
flake8
1 year ago
Mike Fährmann
0b6e5b8161
[hiperdex] send Referer headers during file downloads ( #4490 )
1 year ago
Mike Fährmann
a05821f8b4
[hiperdex] fix 'manga' metadata
...
remove trailing ' Manga'
1 year ago
Mike Fährmann
03d471a0d4
merge #4481 : [pixiv] handle errors for private novels
1 year ago
Cisney-Gassai
8c477f7146
[bunkr] Fixes media-files-pizza.bunkr.ru failed to resolve.
1 year ago
johnsmith1202gmail
c7e31b2724
Update pixiv.py
1 year ago
johnsmith1202gmail
d3046561d4
continue downloading when the item is made private on pixiv
1 year ago
Mike Fährmann
28798594e8
[gfycat] update pagination logic ( #4479 )
...
Some searches do not use cursor based pagination
but an offset based one.
1 year ago
Mike Fährmann
a783c4f0fe
[pornhub] add 'gif' support ( #4463 )
1 year ago
Mike Fährmann
ba842981af
[imagevenue] fix extraction ( #4473 )
1 year ago
Mike Fährmann
7defb24e1e
[reddit] provide video previews if available ( #4322 )
1 year ago
Mike Fährmann
fd65f27ede
[reddit] fix 'preview.redd.it' URLs ( #4470 )
1 year ago
Mike Fährmann
06aaedded5
[twitter] extract 'source' metadata ( #4459 )
1 year ago
Mike Fährmann
14af15bd18
[reddit] download preview for 404ed imgur links ( #4322 )
...
This is a pretty ugly hack as the internal infrastructure doesn't
really support switching from external URL to regular download in
case the former fails, but it kind of works ...
Can be disabled by setting 'reddit.fallback' to 'false'.
1 year ago
Mike Fährmann
d12a5e440a
update docs/supportedsites
1 year ago
Mike Fährmann
3a27150479
[instagram] add 'following' extractor ( #1848 )
1 year ago
Mike Fährmann
e0829ff0fd
[twitter] add 'date_original' metadata for retweets ( #4337 , #4443 )
1 year ago
Mike Fährmann
5ed245317d
[exhentai] add 'fav' option ( #4409 )
...
The name 'favorite' is already taken as extractor subcategory
1 year ago
Mike Fährmann
fd6b413f3c
[exhentai] fix 'domain' option ( #4458 )
...
regression from a383eca7
1 year ago
Mike Fährmann
fdfb22c91f
[instagram] fix video preview archive IDs ( #2135 , #4455 )
1 year ago
Mike Fährmann
92f98e6f5e
'sys.exit' -> 'SystemExit'
1 year ago
Mike Fährmann
410f783a33
implement 'subconfigs' option ( #4440 )
1 year ago
Mike Fährmann
2b88ad19e9
[twitter] accept 'x.com' URLs ( #4452 )
1 year ago
Mike Fährmann
c1c73c0b0e
[pp:ugoira] add '"framerate": "uniform"' ( #4421 )
1 year ago
Mike Fährmann
2a3acd318a
[pp:ugoira] fix high frame rates ( #4421 )
...
only return an output frame rate for non-uniform ugoira
when the frame delay gcd is >= 10, i.e. 100 fps
1 year ago
Mike Fährmann
70bdf32a88
[pp:ugoira] extend 'ffmpeg-output' ( #4421 )
...
- when setting this option to a string value,
pass -hide-banner and -loglevel to FFmpeg
- change default to "error"
1 year ago
Mike Fährmann
8dceea3384
[shimme2] move 'giantessbooru' back into shimmie module ( #4373 )
...
Do the same thing as for 'realbooru' and override 'posts()'
insteadd of using a separate module.
1 year ago
Mike Fährmann
6482f9453b
[behance] fix cookie usage ( #4417 )
1 year ago
Mike Fährmann
d34195b41d
[behance] fix and update 'user' extractor ( #4417 )
1 year ago
Mike Fährmann
4d3cf709da
[behance] add 'date' metadata field ( #4417 )
1 year ago
Mike Fährmann
c689cd9720
[behance] show error for mature content ( #4417 )
1 year ago
Mike Fährmann
33d912490f
merge #4419 : [bunkr] Fix extracting wmv files
1 year ago
Mike Fährmann
01610a6e9e
merge #4412 : [bunkr] fix media domain for cdn9
1 year ago
Mike Fährmann
b19d62263b
merge #4420 : [issuu] fix extraction
1 year ago
ClosedPort22
6dc8be5e48
[issuu] fix extraction
1 year ago
Luc Ritchie
85a070b9e6
[bunkr] Fix extracting wmv files
1 year ago
Mike Fährmann
3f8ff692a7
[bunkr] fix media domain for cdn9
...
Fixes #4386
1 year ago
Mike Fährmann
d8b21a97bf
[formatter] use 'rpartition' for \fM format strings
...
fixes using absolute module paths like C:\path\module.py on Windows
1 year ago
Mike Fährmann
f9fb276e81
[postprocessor] add 'prepare-after' event ( #4083 )
1 year ago
Mike Fährmann
0ef1fcab20
[postprocessor] update 'finalize' events
...
Add 'finalize-error' and 'finalize-success' events that trigger
depending on whether error(s) did or did not happen.
'finalize' itself now always triggers regardless of error status.
(was supposed to have the same behavior as the new 'finalize-success')
1 year ago
Mike Fährmann
af4bdb62a7
merge #4403 : [downloader:http] close connection when file already exists
1 year ago
Mike Fährmann
15275b3524
[postprocessor:ugoira] restore 'libx264-prevent-odd' ( #4407 )
...
was accidentally removed in commit be9547a5
1 year ago
Mike Fährmann
391a7d74c8
[giantessbooru] fix and move to separate module ( #4373 )
...
too many differences to the other shimmie2 sites
1 year ago
ClosedPort22
5448268d5c
[downloader:http] close connection when file already exists ( #3748 )
1 year ago
Mike Fährmann
3963dbe5e4
extend 'parent>child' categories
...
continuation of ed21908f
allow for children to have an arbitrary distance from their parent,
e.g. reddit -> danbooru -> imgur:gallery -> imgur:album
would still be covered by 'reddit>imgur' or even 'danbooru>imgur'
1 year ago
Mike Fährmann
089d1a4f67
[twitter] fix 'TweetWithVisibilityResults' ( #4369 )
1 year ago
Mike Fährmann
a4f7f7da17
add '_dump()' convenience method to Extractor
1 year ago
Mike Fährmann
df5c7ee03e
[deviantart] fix search ( #4384 )
...
send correct usernames instead of 'u'
1 year ago
Mike Fährmann
a60db454af
[sankaku] update/fix API headers
...
'Referer' and 'Origin' were both empty
1 year ago
Mike Fährmann
fb3f0453db
[twitter] improve error messages for single Tweets ( #4369 )
...
also fixes '"quoted": false' not having any effect
1 year ago
Mike Fährmann
541bff5a37
[pururin] fix extraction ( #4375 )
...
- rename 'title_jp' to 'title_ja'
- change type of 'collection', 'convention', and 'scanlator' to list
1 year ago
Mike Fährmann
6a87c314af
[instagram] fix private posts with long shortcodes ( #4362 )
1 year ago
Mike Fährmann
f899fac4c5
[giantessbooru] fix extraction ( #4373 )
...
This does not fix anything Cloudflare related,
just other things caused by a site update.
1 year ago
Mike Fährmann
136283d402
[shimmie2] update base URL pattern
...
to match new giantessbooru URLs
1 year ago
Mike Fährmann
9d67655397
add "ascii+" as a special 'path-restrict' value ( #4371 )
1 year ago
Mike Fährmann
c79359eb3a
[fantia] improve metadata extraction ( #4126 )
...
extract all metadata and URLs before starting to download
1 year ago
Mike Fährmann
48ef062867
fix issues with 'Extractor.finalize()'
...
- prevent crash in InstagramUserExtractor (#4359 )
- call it at the end of every DownloadJob
- add it to tests
1 year ago
Mike Fährmann
ed21908fda
initial support for child extractor options
...
Using "parent-category>child-category" as extractor category in a config
file allows to set options for a child extractor when it was spawned by
that parent.
For example "reddit>gfycat" to set gfycat options for when it was found
in a reddit post.
{
"extractor": {
"gfycat": {
"filename": "regular filename"
},
"reddit>gfycat": {
"filename": "reddit-specific filename"
}
}
}
Note: This does currently not work for most imgur links due to how its
extractor hierarchy is structured.
1 year ago
Mike Fährmann
255d08b79e
add test for 'Extractor.initialize()' ( #4359 )
1 year ago
Mike Fährmann
2bcf0a4c49
[instagram] fix initialization order ( #4359 )
...
regression caused by the changes in a383eca7
1 year ago
Mike Fährmann
7eab101144
[acidimg] fix extraction
...
swap ' and " again (2e309a13
)
and add a fallback in case this happens yet another time
1 year ago
Mike Fährmann
62fce6a75f
[imagehosts] adjust variable names ( #4358 )
...
prefix them with underscores to prevent a clash
with the new 'self.cookies' from d97b8c2f
1 year ago
Mike Fährmann
e8299b459a
[moebooru] match search URLs with empty 'tags' ( #4354 )
1 year ago
Mike Fährmann
7fbc304ae9
[twitter] fix crash on private user ( #4349 )
1 year ago
Mike Fährmann
1ece3b92ff
[mangadex] allow multiple values for 'lang' ( #4093 )
...
This was already possible by setting 'lang' to a list of strings,
but now it can also be done as a more command-line friendly string.
-o lang=fr,it
1 year ago
Mike Fährmann
52053b58f0
[lensdump] fix extraction ( #4352 )
1 year ago
Mike Fährmann
11f71a9cba
remove 'mememuseum' module
...
This was forgotten when adding generic Shimmie2 support in 7865067d
1 year ago
Mike Fährmann
a383eca7f6
decouple extractor initialization
...
Introduce an 'initialize()' function that does the actual init
(session, cookies, config options) and can called separately from
the constructor __init__().
This allows, for example, to adjust config access inside a Job
before most of it already happened when calling 'extractor.find()'.
1 year ago
Mike Fährmann
6c9432165e
add return value to 'PostProcessor._init_archive()'
1 year ago
Mike Fährmann
54d974deb0
add 'python' post processor
...
similar to 'exec' but calls a Python function
1 year ago
Mike Fährmann
1baf83a9e5
[hiperdex] fix for unicode titles ( #4325 )
1 year ago
Mike Fährmann
7da954f810
[flickr] update default API credentials ( #4332 )
...
and add a delay between API requests
1 year ago
Mike Fährmann
a45a17ddb7
[pixiv] ignore 'limit_sanity_level' images ( #4328 )
1 year ago
Mike Fährmann
088e8d5fcf
[pornhub] fix extraction ( #4301 )
1 year ago
Mike Fährmann
d97b8c2fba
consistent cookie-related names
...
- rename every cookie variable or method to 'cookies_*'
- simplify '.session.cookies' to just '.cookies'
- more consistent 'login()' structure
1 year ago
Mike Fährmann
ceebacc9e1
remove 'pyopenssl' option
1 year ago
Mike Fährmann
3c2c7e21dd
merge #4319 : [zerochan] fix 'tags' extraction
1 year ago
Mike Fährmann
0ba8d1f168
merge #4312 : [redgifs] add 'niches' extractor
1 year ago
Mike Fährmann
c5565f79f7
merge #4096 : [danbooru] add support for booru.borvar.art instance
1 year ago
Mike Fährmann
63326e3168
[danbooru] add tests for booruvar
1 year ago
Mike Fährmann
5171d8975c
[E621] support 'e6ai.net' ( #4320 )
1 year ago
Mike Fährmann
a996d936d2
[imagefap] fix pagination ( #3013 )
1 year ago
Mike Fährmann
22099422ca
[deviantart] fix shortened URLs ( #4316 )
1 year ago
Mike Fährmann
90231f2d5a
[twitter] add 'tweet-endpoint' option ( #4307 )
...
use the newer TweetResultByRestId only for guests by default
1 year ago
Mike Fährmann
20ed647f6f
[twitter] add 'user' extractor and 'include' option ( #4275 )
1 year ago
Mike Fährmann
86be197d11
[twitter] remove '/search/adaptive.json'
1 year ago
enduser420
d52ed2bc5a
[zerochan] fix 'tags' extraction
1 year ago
enduser420
12cd85658b
[redgifs] add 'niches' extractor
1 year ago
Mike Fährmann
248e8bc699
release version 1.25.8
1 year ago
Mike Fährmann
bc9123cfee
[naverwebtoon] fix 'comic' metadata extraction
1 year ago
Mike Fährmann
ab5dde7221
[mangaread] fix 'tags' extraction
1 year ago
Mike Fährmann
c9a82c9313
[erome] ignore duplicate album IDs
1 year ago
Mike Fährmann
c84397023a
[slideshare] fix extraction
1 year ago
Mike Fährmann
ffbbbd3baf
[gelbooru_v01] 'vidyart' -> 'vidyart2'
1 year ago
Mike Fährmann
e40b90e137
merge #4303 : [gelbooru_v01] fix 'source' ( #4302 )
1 year ago
Mike Fährmann
c6b31a2169
[reddit] set default 0.6s delay between requests ( #4292 )
...
to limit API requests to 100 per minute
https://www.reddit.com/r/redditdev/comments/14nbw6g/
1 year ago
Mike Fährmann
20da41018d
[pornhub] set 'accessAgeDisclaimerPH' cookie ( #4301 )
1 year ago
ncaat
75757c4ace
[gelbooru_v01] fix 'source' ( #4302 )
1 year ago
Mike Fährmann
2dd6942d1c
[jpgfish] update domain to 'jpeg.pet'
1 year ago
Mike Fährmann
1137b89ed4
[lineblog] remove module
...
"LINE BLOGは2023年6月29日をもちましてサービスを終了いたしました"
1 year ago
Mike Fährmann
86560fe0cd
[bcy] remove module
...
"The website was shut down on July 12, 2023"
https://danbooru.donmai.us/wiki_pages/bcy
1 year ago
Mike Fährmann
fceabee433
[philomena] use API interface class
...
handle 429 errors and retry after 10min (#4288 )
1 year ago
Mike Fährmann
f079d9a703
[reddit] notify users about registering an oauth application
...
(#4292 , #4253 , #3943 )
1 year ago
Mike Fährmann
fb3d1462b1
merge #4291 : [wikifeet] fix 'tag' extraction
1 year ago
Mike Fährmann
0b08e2e8a8
merge #4287 : [twitter] Fix following extractor not getting all users
1 year ago
Mike Fährmann
f6553ffd2f
[twitter] simplify '_pagination_users'
...
- remove 'stop' variable
- call 'cursor.startswith()' only once
1 year ago
Mike Fährmann
1590124aae
[twibooru] fix '--range'
1 year ago
enduser420
a2111dd025
[wikifeet] fix 'tag' extraction
1 year ago
Mike Fährmann
a1ffa1ff09
[philomena] fix '--range' ( #4288 )
1 year ago
Mike Fährmann
a27dbe8c82
[twitter] use 'TweetResultByRestId' endpoint ( #4250 )
...
allows accessing single Tweets without login
1 year ago
Mike Fährmann
d3d639a159
[twitter] don't treat missing 'TimelineAddEntries' as fatal ( #4278 )
1 year ago
ActuallyKit
c321c773f2
make the code less ugly
1 year ago
ActuallyKit
a437a34bcf
fix lint i guess?
1 year ago
ActuallyKit
6cbc434b54
Fix users pagination
1 year ago
Mike Fährmann
d5b6802774
[seiga] set 'skip_fetish_warning' cookie ( #4242 )
1 year ago
Mike Fährmann
88d1e29401
[bunkr] use '.la' TLD for 'media-files12' servers ( #4147 , #4276 )
1 year ago
Mike Fährmann
f0cb951566
[paheal] unescape 'source'
1 year ago
Mike Fährmann
b480b7076a
[paheal] fix a78f8ce5
for enabled 'metadata' ( #4262 )
1 year ago
Mike Fährmann
384337d3dd
[fantia] send 'X-Requested-With' header only for API requests ( #4273 )
1 year ago
Mike Fährmann
c2ac665ff7
[fantia] send 'X-Requested-With' header ( #4273 )
1 year ago
Mike Fährmann
7444fc125b
[gfycat] implement login support ( #3770 , #4271 )
...
For the record: '/webtoken' and '/weblogin' are not the same ...
1 year ago
Mike Fährmann
e9b9f751bf
[gfycat] support '@me' user ( #3770 , #4271 )
1 year ago
Mike Fährmann
5b59a0d143
update default User-Agent header to Firefox 115 ESR
1 year ago
Mike Fährmann
0556e1ad45
merge #4268 : [newgrounds] extract & pass auth token for login
1 year ago
Mike Fährmann
a16d7c59cb
[newgrounds] access 'response.text' only once
1 year ago
Mike Fährmann
1bf9f52c99
[twitter] add 'ratelimit' option ( #4251 )
1 year ago
Mike Fährmann
f86fdf64a6
[twitter] use GraphQL search by default ( #4264 )
1 year ago
Mike Fährmann
1d4db83d49
[weibo] fix end of cursor based pagination
1 year ago
Mike Fährmann
a78f8ce5b0
[paheal] fix extraction ( #4262 )
...
swap ' and "
1 year ago
FrostTheFox
9576652fa5
extract & pass auth token for newgrounds
1 year ago
Mike Fährmann
5457007dd3
release version 1.25.7
1 year ago
Mike Fährmann
3d8de383bf
[mangapark] extract 'source_id' for manga
...
forgot to add this to 6ae3101f
1 year ago
Mike Fährmann
6ae3101fd0
[mangapark] add 'source' option ( #3969 )
1 year ago
Mike Fährmann
c45a913bfd
[flickr] add 'exif' option
1 year ago
Mike Fährmann
3845c0256d
[sankaku] improve warnings for unavailable posts
1 year ago
Mike Fährmann
46cae04aa3
[piczel] update API server ( #4244 )
1 year ago
Mike Fährmann
3479646f65
[mangapark] update and fix 'manga' extractor ( #3969 )
...
TODO:
- non-English chapters
- 'source' option
1 year ago
Mike Fährmann
10786c657e
[mangapark] update and fix 'chapter' extractor ( #3969 )
1 year ago
Mike Fährmann
9c31c2daef
[poipiku] improve error detection ( #4206 )
1 year ago
Mike Fährmann
260ff55e19
[senmanga] ensure download URLs have a scheme ( #4235 )
1 year ago
Mike Fährmann
ccbc1a1d55
[flickr] add 'metadata' option ( #4227 )
1 year ago
Mike Fährmann
c1cce4a80b
[twitter] extend 'conversations' option ( #4211 )
1 year ago
Mike Fährmann
b6c959744d
[furaffinity] improve 'description' HTML ( #4224 )
...
- ignore header
- include footer and closing <div> if present
1 year ago
Mike Fährmann
8357acf359
[gelbooru_v01] replace 'extract_all()' with 'extract_from()'
...
It's even slightly faster, especially on Python before 3.11
1 year ago
Mike Fährmann
068aa26c3e
[gelbooru_v01] fix '--range' ( #4167 )
1 year ago
Mike Fährmann
2052e7ce59
[hentaifox] fix titles containing '@' ( #4201 )
1 year ago
Mike Fährmann
92d98697b2
[wallhaven] update API error message
1 year ago
Mike Fährmann
a673998b1e
release version 1.25.6
1 year ago
Mike Fährmann
339fcdb8ad
[wallhaven] handle '429 Too Many Requests' errors ( #4192 )
...
- set 1.4s delay between API requests
(WH allows 45 requests per minute)
- wait and retry on 429 errors
1 year ago
Mike Fährmann
ef9891ec9d
[fantia] extract 'plan' metadata ( #2477 , #4128 )
1 year ago
Mike Fährmann
f8452984fa
[fantia] emit warning for non-visible contents ( #4128 )
1 year ago
Mike Fährmann
dc7af00014
[fantia] refactor
...
- embed response data as hidden '_data' field
(instead of returning/passing 'resp')
- split _get_urls_from_post()
1 year ago
Mike Fährmann
6c8bf9a762
[pornhub] improve redirect handling ( #4188 )
1 year ago
Mike Fährmann
654267a335
[weibo] fix 'json' extension for some videos
1 year ago
Mike Fährmann
ce93c460a6
[formatter] implement 'H' conversion ( #4164 )
...
to remove HTML tags and unescape HTML entities
1 year ago
Mike Fährmann
deff3b434d
[vipergirls] implement login support ( #4166 )
1 year ago
Mike Fährmann
db20a645c5
[vipergirls] use API endpoints ( #4166 )
1 year ago
Mike Fährmann
0b34a444e0
[pixiv:novel] only detect Pixiv embeds ( #4175 )
1 year ago
Mike Fährmann
9f1aee3884
[vipergirls] limit number of requests per second ( #4166 )
1 year ago
Mike Fährmann
21c75d03a3
merge #4133 : [furaffinity] extract 'favorite_id' metadata
1 year ago
Mike Fährmann
5e3a1749c8
[furaffinity] simplify 'favorite_id' assignment
1 year ago
Mike Fährmann
ad882291d3
[instagram] fix retrieving '/tagged' posts ( #4122 )
...
reduce number of retrieved posts per API request from 50 to 20
1 year ago
Mike Fährmann
0a9aaa7a8d
[weibo] prevent fatal exception due to missing video ( #4150 )
1 year ago
Mike Fährmann
ac651c604c
[senmanga] fix and update ( #4160 )
1 year ago
Mike Fährmann
df106fb58b
[bunkr] fix video downloads
1 year ago
Mike Fährmann
aad5e6490c
merge #4159 : [bunkr] update domain to bunkrr.su
1 year ago
Mike Fährmann
e0522ffb3d
[bunkr] update
1 year ago
Mike Fährmann
e04796e04b
merge #3447 : [jschan] add generic extractors for jschan imageboards
1 year ago
Mike Fährmann
b9692341fe
[jschan] update
1 year ago
Stephan
a7c066cbac
Update bunkr.py
1 year ago
Stephan
72e697b8b5
Update bunkr.py
...
Support bunkrr.su
1 year ago
Mike Fährmann
4ae925c88f
[kemonoparty] support '.su' TLD ( #4139 )
1 year ago
Mike Fährmann
2d9e3093ca
merge #4134 : [postimage] add gallery support, update image extractor
1 year ago
Mike Fährmann
e64b521287
merge #4136 : [acidimg] fix extractor
1 year ago
Mike Fährmann
a90974178d
[jpgfish] update domain to 'jpg.pet' ( #4138 )
1 year ago
Mike Fährmann
ee959052ac
merge #4138 : add jpg.pet as alias for jpgfish
1 year ago
Mike Fährmann
0281cc7d08
[fanbox] skip 404ed fanbox embeds ( #4088 )
...
continuation of 4fc9675d
1 year ago
Prinz23
97c0d13cbb
add jpg.pet as alias for jpgfish
1 year ago
chio0hai
2e309a13a7
[acidimg] fix extractor
1 year ago
chio0hai
92178b369c
[postimage] add gallery support, update image extractor to download
...
original image instead of main image
1 year ago
Bad Manners
952c03bc9e
Add fav_id data to FuraffinityFavoriteExtractor
...
An extra field is collected when paginating favorites, and saved to
a temporary cache variable. This field is identical for both the old
and the new page layouts for FurAffinity, but can only be collected
during pagination, hence the cache variable. Other FurAffinity
extractors should be unaffected by this change.
1 year ago
Mike Fährmann
54cf1fa3e7
[twitter] use GraphQL search endpoint ( #3942 )
...
for guest users; selectable with 'search-endpoint' option.
adapted from 9c7b888ffa
1 year ago
Mike Fährmann
864a654b25
[twitter] update query hashes
1 year ago
Mike Fährmann
45cc7cee1a
[twitter] better error message for guest searches ( #3942 )
1 year ago
Mike Fährmann
271f23d971
[twitter] extract 'conversation_id' metadata ( #3839 )
1 year ago
Mike Fährmann
94b6a67666
[reddit] fix crash with empty 'crosspost_parent_lists' ( #4120 )
1 year ago
Mike Fährmann
0cf7282fa0
[pixiv] add 'full-series' option for novels ( #4111 )
1 year ago
Mike Fährmann
bab13402df
[redgifs] update 'search' URL pattern ( #4115 )
1 year ago
Mike Fährmann
5a6fd8027d
[redgifs] support galleries ( #4021 )
1 year ago
Mike Fährmann
0ad59c92b1
[blogger] download files from 'lh*.googleusercontent.com' (4070)
1 year ago
Mike Fährmann
ffed7efb6f
[pixiv] use BASE_PATTERN
1 year ago
Mike Fährmann
b286efefcc
[pixiv] add 'novel-bookmark' extractor ( #4111 )
1 year ago
Mike Fährmann
5283db1aae
release version 1.25.5
1 year ago
Mike Fährmann
28f6487c64
[instagram] add 'metadata' option ( #3107 )
1 year ago
Mike Fährmann
8cf13f8696
merge #4104 : [lensdump] add lensdump.com extractors
1 year ago
Mike Fährmann
58f7480d46
[lensdump] update
...
- update docs/supportedsites.md
- add GPL2 header
- use BASE_PATTERN
- improve LensdumpImageExtractor
1 year ago
Mike Fährmann
3516fdae74
[kemonoparty] fix kemono and coomer logins using the same cache
...
(#4098 )
1 year ago
chio0hai
d5300cf381
[lensdump] subcategory
1 year ago
chio0hai
82ba6bfdc0
[lensdump] f-string fix
1 year ago
chio0hai
9b2326e4e1
[lensdump] add lensdump.com extractor
1 year ago
Mike Fährmann
a5d0b03bde
[ytdl] fix crash due to removed 'no_color' attribute
...
8417f26b8a
1 year ago
Mike Fährmann
148bdc04a4
merge #2719 : [jpgfish] add 'jpgfish' extractors
1 year ago
Mike Fährmann
609c4f3e07
[jpgfish] simplify and improve
1 year ago
Mike Fährmann
2b1f875ef4
[jpgchurch] update to 'jpgfish'
1 year ago
Mike Fährmann
3d29c42142
[mangaread] fix 'tags' extraction
1 year ago
Mike Fährmann
5f86527cbe
merge #2781 : [mangaread] Add Mangaread extractor
1 year ago
Mike Fährmann
cdc6549fd2
merge #3329 : [8muses] Add 'parts' to album data
...
and fix 'album[url]'
1 year ago
Mike Fährmann
ad760429b1
[8muses] update
1 year ago
Mike Fährmann
d0184fddcf
[twitter] optimize '_extract_twitpic()'
...
- use findall instead of finditer
- store URLs in a dict to discard duplicates
1 year ago
Mike Fährmann
3dc862c7fc
merge #3796 : [twitter] extract TwitPic URLs in text ( #3792 )
1 year ago
Mike Fährmann
243de697b9
merge #3976 : [reddit] support cross-posted media ( #887 , #3586 )
1 year ago
Mike Fährmann
f8c4c5eef9
[reddit] simplify and add tests
1 year ago
thatfuckingbird
822a77d846
[danbooru] add support for booru.borvar.art instance
1 year ago
Mike Fährmann
f3cca50b9e
[mangadex] update links to API docs
1 year ago
Mike Fährmann
65a9f4b124
merge #3950 : [misskey] add 'favorite' extractor
1 year ago
Mike Fährmann
c76f0f3a1b
[misskey] update
...
- rename to 'MisskeyFavoriteExtractor'
- add 'access-token' option to docs
- add test URLs for other instances
- simplify 'pattern'
1 year ago
Mike Fährmann
3fca455b82
[pixiv] add 'embeds' option ( #1241 )
1 year ago
Mike Fährmann
d1f2ef3b7b
[imagechest] update
...
- don't load HTML page when using API
- restructure some code
- add more methods to ImagechestAPI
1 year ago
Mike Fährmann
856f6c10cd
allow for GalleryExtractors to skip loading gallery_url
1 year ago
Mike Fährmann
4fc9675d48
[fanbox] skip 404ed or otherwise invalid posts ( #4088 )
1 year ago
Mike Fährmann
69865dcc05
[formatter] implement slicing strings as bytes ( #4087 )
...
prefixing a slice '[10:30]' with a lowercase b '[b10:30]' encodes
the string to bytes in filesystem encoding before applying the slice
1 year ago
Mike Fährmann
56b8b8cd36
[pixiv] support short novel URLs
...
https://www.pixiv.net/n/ <ID>
1 year ago
Mike Fährmann
e6f55d1555
[imagechest] add API support and 'access-token' option ( #4065 )
1 year ago
Mike Fährmann
77abcf5ab3
[gofile] automatically fetch 'website-token' by default
...
the hardcoded token changed yet again
1 year ago
Mike Fährmann
e3fed9bd17
[tcbscans] update domain to 'tcbscans.com' ( #4080 )
1 year ago
Mike Fährmann
a83983c651
[instagram] add 'order-posts' option ( #4017 , #3993 )
1 year ago
Mike Fährmann
d680623db3
[instagram] add 'order-files' option ( #4017 , #3993 )
1 year ago
Naatie
f9b7a033e0
[misskey] refactor misskey extractor
1 year ago
Naatie
04dbfd994e
[misskey] add my favorites extractor
1 year ago
Mike Fährmann
82a12d6126
[nsfwalbum] detect placeholder images
...
patch by an anonymous contributor
1 year ago
Mike Fährmann
011e4607c3
[poipiku] extract full 'descriptions' ( #4066 )
...
don't cut it off after the first line
1 year ago
Mike Fährmann
5037013e2b
[gofile] update 'website-token' ( #4056 )
1 year ago
Mike Fährmann
6b6bb4be73
[weibo] require numeric IDs to have length >= 10 ( #4059 )
1 year ago
Mike Fährmann
494acabd38
[danbooru] refactor pagination logic ( #4002 )
...
- only use 'b<ID>' when no other order is specified
- support 'a<ID>' when using 'order:id' as tag
1 year ago
Mike Fährmann
fd0e1ffd6e
[danbooru] improve 75666cf9
( #4002 )
...
Search for direct post IDs instead of trying to
replicate the same results as the initial request.
1 year ago
Mike Fährmann
e41e45ff6b
[gofile] add basic password support ( #4056 )
1 year ago
Mike Fährmann
790dd365e1
[postprocessor:exec] support tilde expansion for 'command'
...
https://github.com/mikf/gallery-dl/issues/146#issuecomment-1544733532
1 year ago
Mike Fährmann
2e6cea95db
[cookies] update logging behavior ( #4050 )
...
- only show the same warning/error once
- simplify and capitalize logging messages
1 year ago
Mike Fährmann
20dc13f832
[pixiv] initial 'novel' support ( #1241 , #4044 )
...
supported URLs are
- https://www.pixiv.net/novel/show.php?id= <ID>
- https://www.pixiv.net/novel/series/ <ID>
- https://www.pixiv.net/en/users/ <ID>/novels
1 year ago
Mike Fährmann
c698c3de44
[newgrounds] add default delay between requests ( #4046 )
1 year ago
Mike Fährmann
708f478d15
[danbooru][e621] add 'date' metadata field ( #4047 )
1 year ago
Mike Fährmann
306e13a4d4
release version 1.25.4
1 year ago
Mike Fährmann
35c23a2fd8
merge #4031 : [mangadex] add 'status' and 'tags' metadata
1 year ago
Mike Fährmann
2266fc8cc5
[mangadex] update and extend test results
1 year ago
Janne Alaranta
1ce5dc9e18
fix whitespaces
1 year ago
Janne Alaranta
13dedae09f
add status and tags info to mangadex extractor
1 year ago
Mike Fährmann
be0fa94b2e
[imagechest] load all images when a 'Load More' button is present
...
(#4028 )
1 year ago
Mike Fährmann
7eadcbea70
[4chanarchives] add end condition for 'board' extractor ( #4012 )
1 year ago
Mike Fährmann
1406f7125f
[4chanarchives] add 'thread' and 'board' extractors ( #4012 )
1 year ago
Mike Fährmann
285391df43
add '-C' as short option for '--cookies'
...
and put cookie options into their own section
1 year ago
Mike Fährmann
b9b1cdd71b
add '--cookies-export' command-line option
1 year ago
Mike Fährmann
d12dd3813c
[imgur] fix internal image/album URLs
...
URLs from "link" attributes of newer images/albums were all returned
as 'https://imgur.com/gallery/ ...' instead of the expected format,
causing them to be ignored.
1 year ago
Mike Fährmann
8520de57f0
[imgur] add 'favorite-folder' extractor ( #4016 )
1 year ago
Mike Fährmann
4c1f3b2160
[cookies] simplify '_mac_absolute_time_to_posix()'
...
hardcode UNIX timestamp of 2001-01-01
1 year ago
Mike Fährmann
a14b63d941
support selecting a domain for '--cookies-from-browser'
...
for example 'gallery-dl --cookies-from-browser firefox/twitter.com'
1 year ago
Mike Fährmann
3ca5dac8b6
extend 'cookies-update' functionality
...
Allow writing cookies to a different file than a given cookies.txt,
making it possible to export cookies imported with --cookies-from-browser
To convert browser cookies to cookies.txt format:
gallery-dl --cookies-fr chromium \
-o cookies-update=cookies.txt \
--no-download \
http://example.org/file.jpg
1 year ago
Mike Fährmann
bc6d65d203
implement 'Extractor.config_deprecated()'
...
a version of 'Extractor.config()'
that logs a warning when using a deprecated option name
1 year ago
Mike Fährmann
850df34c31
remove '&' from URL patterns part 2
...
follow-up on 968d3e8465
1 year ago
Mike Fährmann
4d415376d1
[pinterest] fix 'pin.it' extractor
...
it really was just the single '/' at the end of the url_shortener URL
1 year ago
Mike Fährmann
657b6a9100
[pinterest] update endpoint for related board pins
1 year ago
Mike Fährmann
79f47f98dd
[nana] remove module
...
permanently gone since 2023-03-13
1 year ago
Mike Fährmann
0e74df1de8
[420chan] remove module
...
offline since 2022-06-01
1 year ago
Mike Fährmann
7499fa7075
[exhentai] remove and update sad panda check
...
there hasn't been a sad panda in several years
1 year ago
Mike Fährmann
076380e079
remove '*' indicating keyword-only arguments
...
they are kind of unnecessary and
cause a non-insignificant function call overhead (~10%)
1 year ago
Mike Fährmann
0c46758a93
[foolslide] remove 'sensescans.com'
...
group moved to mangadex
https://mangadex.org/group/1071e71d-cc55-4fa6-81d1-4b5913a2fde5/sense-scans
1 year ago
Mike Fährmann
a08fdfac6e
[foolfuuka] add 'archive.palanq.win'
1 year ago
Mike Fährmann
1870df8b23
[foolfuuka] remove 'tokyochronos.net'
1 year ago
Mike Fährmann
ef4e2d8178
[foolfuuka] remove 'archive.alice.al'
1 year ago
Mike Fährmann
57cf942bb1
[config] include exception type in error message
1 year ago
Mike Fährmann
aa731c4298
[ytdl] run yt-dlp tests with latest code from master ( #3989 )
...
Only use PyPI version for Python 3.6, since that's no longer supported
by the current codebase.
1 year ago
Mike Fährmann
6a860876bc
release version 1.25.3
1 year ago
Mike Fährmann
b12dad8df5
[pixiv] fix 'pixivision' extraction
1 year ago
Mike Fährmann
5fb7107f2b
[imxto] fix 'gallery' extraction
...
support both single and double quotes
1 year ago
Mike Fährmann
15d7c5a199
[behance] 'items()' -> 'values()'
...
we only need 'size', 'name' is unnecessary
1 year ago
Mike Fährmann
61a65d5bb9
[ytdl] fix crash due to --geo-bypass deprecation ( #3975 )
1 year ago
Mike Fährmann
0fb580135d
[behance] fix extraction ( #3980 )
1 year ago
Alexandru Vasilescu
d4f8b2fe22
fix: linter issues
1 year ago
Alexandru Vasilescu
1b918bd937
fix(extractor): fix extraction for cross-posted reddit videos and galleries
1 year ago
Mike Fährmann
215028a462
[manganelo] match more minor version separators ( #3972 )
1 year ago
Mike Fährmann
c182094ebf
merge #3748 : [downloader:http] add 'consume-content' option
1 year ago
thatfuckingbird
9f76783ac0
[pixiv] allow sorting by popularity (requires pixiv premium)
1 year ago
Mike Fährmann
7865067d19
[shimmie2] add generic extractors for Shimmie2 sites ( #3734 )
...
add support for
- loudbooru.com (#3734 )
- booru.cavemanon.xyz (#3734 )
- giantessbooru.com (#943 )
- tentaclerape.net
1 year ago
Mike Fährmann
28419bf45a
[itchio] add 'game' extractor ( #3923 )
1 year ago
Mike Fährmann
3905f05f00
[postprocessor:metadata] support putting keys in quotes
...
for mode 'modify' and 'delete'
based on fe41a2b1
1 year ago
Mike Fährmann
7459e4abce
[postprocessor:metadata] fix traversing more than 1 level deep
...
for mode 'modify' and 'delete'
1 year ago
Mike Fährmann
5297ee0cd9
[tumblr] add 'day' extractor ( #3951 )
1 year ago
Mike Fährmann
de670bd7de
[tumblr] update pagination logic ( #2191 )
1 year ago
ClosedPort22
6f4a843fba
[downloader:http] release connection before logging messages
...
This allows connections to be properly released when using 'actions'
feature.
1 year ago
Mike Fährmann
98c9fdb414
[deviantart] revert e9353c63; retry downloads with private token
1 year ago
Mike Fährmann
5d7435e803
[nitter] extract user IDs from encoded banner URLs
...
still requires a banner to be present to begin with
1 year ago
Mike Fährmann
7f25cab56e
[sankaku] support post URLs with MD5 hashes ( #3952 )
1 year ago
Mike Fährmann
a05120412a
[oauth] catch exception from 'webbrowser.get()' ( #3947 )
...
It raises an exception instead of returning None
when no runnable browser is available.
1 year ago
Mike Fährmann
3fc2223893
merge #3935 : [reddit] match 'preview.redd.it' URLs
1 year ago
Mike Fährmann
1d505b39f8
[twitter] support 'profile-conversation' entries ( #3938 )
1 year ago
Mike Fährmann
aaf58a1259
[imgur] document 'client-id' option ( #3937 )
1 year ago
Mike Fährmann
202f5d86a7
[reddit] ignore 'id-max' value "zik0zj"/2147483647
...
(#3939 , #3862 , #3697 , #3606 , #3546 , #3521 , #3412 )
1 year ago
Mike Fährmann
8586ee81be
[nana] fix 'keyword' tests
1 year ago
ClosedPort22
cd4bfb0dd1
[reddit] match 'preview.redd.it' URLs
1 year ago
Mike Fährmann
faca32a850
[sankaku] sanitize 'date:…' tags ( #1790 )
1 year ago
Mike Fährmann
6f1e34ec69
[vipergirls] add 'thread' and 'post' extractors
...
(#731 , #2720 , #3812 )
1 year ago
Mike Fährmann
81bd2af83e
[2chen] update domain to sturdychan.help
1 year ago
Mike Fährmann
f500b45b5e
[twitter] improve 480bc34e
...
only check for double user assignment where necessary
1 year ago
Mike Fährmann
5b635f2317
[imxto] add 'gallery' extractor ( #1289 )
1 year ago
Mike Fährmann
359e31e462
[nozomi] update file URLs ( #3925 )
...
Static images are now only available in WebP format over the 'w'
subdomain. GIFs also got their own 'g' subdomain.
1 year ago
Mike Fährmann
2dfd4a3de2
[imagefap] extract 'categories' metadata and fix empty 'tags'
1 year ago
Mike Fährmann
480bc34e54
[twitter] do not overwrite previously assigned users ( #3922 )
1 year ago
Mike Fährmann
02ec5bb8e5
[imagefap] extract 'description' metadata ( #3905 )
1 year ago
Mike Fährmann
842f964c49
release version 1.25.2
1 year ago
Mike Fährmann
d253a3c542
merge #3841 : [urlshortener] add support for bit.ly & t.co
1 year ago
Mike Fährmann
5e63942b37
[urlshortener] update
1 year ago
Mike Fährmann
2edcdee32f
[downloader:http] add MIME type and signature for .heic files
...
(#3915 )
https://github.com/strukturag/libheif/issues/83
1 year ago
Mike Fährmann
c45f09d2a8
[imagechest] fix extraction ( #3914 )
1 year ago
Mike Fährmann
2cd4411ff8
[nitter] extract videos from 'source' elements ( #3912 )
1 year ago
Mike Fährmann
9501579279
[sexcom] fix fetching HD videos
1 year ago
Mike Fährmann
a2f7274eae
[sexcom] fix pagination ( #3906 )
1 year ago
Mike Fährmann
e9353c63d6
[deviantart] keep using private access tokens
...
for deviations returned from a private API call
also fixes a bug from 0a7eee3e
where '_pagination()'
would never switch from unspecified (None) to private access token
1 year ago
Mike Fährmann
e70af6a550
[hentaifoundry] do not update filters when cookies are provided
1 year ago
Mike Fährmann
9c29c904c7
[mastodon] try to get account IDs without access token
...
Try to query the public '/api/v1/accounts/lookup' endpoint
and fall back to '/v1/accounts/search' if it returns an error.
'/api/v1/accounts/lookup' is available since Mastodon v3.4.0.
The version of an instance can be found at '/api/v1/instance'.
1 year ago
Mike Fährmann
1614c5c4bf
[generic] write regular expressions without 'x' flags
1 year ago
Mike Fährmann
d84a617273
[hentaifoundry] fix setting content filters ( #3887 )
1 year ago
ClosedPort22
875485313f
[urlshortener] force HTTPS
1 year ago
Mike Fährmann
0a7eee3ee0
[deviantart] add 'public' option
1 year ago
Mike Fährmann
f5a59c4170
[twitter] add 'date_bookmarked' metadata ( #3816 )
1 year ago
Mike Fährmann
1c1f6fdc80
[twitter] fix regression from 160335ad
...
Tweets from 'homeConversation' or 'conversationthread' entries do not
contain a 'sortIndex' field. Accessing it raises a KeyError and would
erroneously get them labeled as 'deleted'.
1 year ago
Mike Fährmann
160335ad44
[twitter] add 'date_liked' metadata for liked Tweets ( #3816 )
1 year ago
Mike Fährmann
6d850ce629
[twitter] calculate 'date' from Tweet IDs
...
20 times faster than parsing 'created_at'
1 year ago
Mike Fährmann
25949bd767
merge #3871 : [hotleak] Fix downloading of creators whose name starts with a category name
1 year ago
Mike Fährmann
dbe06cdba1
[twitter] warn about 'withheld' Tweets and users ( #3864 )
1 year ago
Mike Fährmann
3cc1dd1572
[twitter] update query hashes
1 year ago
Mike Fährmann
3846ce0de5
[twitter] update to bookmark timeline v2 ( #3859 )
1 year ago
Mike Fährmann
34699fbf64
[deviantart:search] detect login redirects ( #3860 )
1 year ago
Mike Fährmann
e6cb92864a
[twitter] allow setting custom features per API endpoint
1 year ago
Balgden
4b141cce66
Fix indentation
1 year ago
Balgden
bbc5977121
Fix line length
1 year ago
Balgden
ffd30abcb3
[hotleak] Fix downloading of creators whose name starts with a category name
...
E.g. `hot4lexi` would start downloading the `hot` section by mistake
This happened because the regex had a negative lookahead for the category names, but didn't ensure that they where followed by either end-of-string or a slash.
1 year ago
Mike Fährmann
5ca9d55595
merge #3870 : [blogger] update 'sub' regex to get the highest resolution url
1 year ago
Mike Fährmann
fd7ce4c081
merge #3868 : [shopify] fix 'collection' extractor
1 year ago
Mike Fährmann
135ac9c302
merge #3854 : [twitter] fix: graphql_timeline_v2_bookmark_timeline cannot be null
1 year ago
enduser420
bbb1e34c34
[blogger] update sub regex
1 year ago
enduser420
96e3dd2128
[shopify] fix 'collection' extractor
1 year ago
Mike Fährmann
ac97aca99c
[realbooru] fix extraction
...
get file URLs from HTML pages
1 year ago
Mike Fährmann
75666cf9c3
[danbooru] reduce API requests for fetching extended 'metadata'
...
Instead of using one additional API request per post object (N+1),
this requires only one request per 200-post batch.
1 year ago
ClosedPort22
775d2ac999
[downloader:http] improve error logging when releasing connection
1 year ago
Amer Jazaerli
bebbff6578
fix: graphql_timeline_v2_bookmark_timeline cannot be null
...
twitter: 400 Bad Request (The following features cannot be null: graphql_timeline_v2_bookmark_timeline)
1 year ago
ClosedPort22
71b26adb9b
[urlshortener] add tinyurl.com as an example
2 years ago
Mike Fährmann
421db26aff
[bunkr] update domain to 'bunkr.la'
2 years ago
ClosedPort22
9e2a945013
[urlshortener] add support for bit.ly & t.co
2 years ago
Mike Fährmann
82f83c18e8
release version 1.25.1
2 years ago
Mike Fährmann
9b5e7ce8b9
[hiperdex] fix extraction
2 years ago
Mike Fährmann
89a67c45e0
[nitter] support nitter.it ( #3819 )
2 years ago
Mike Fährmann
88f29a751d
[nitter] skip broadcasts
...
instead of downloading an "Unsupported feature" HTML page
2 years ago
Mike Fährmann
1e013eba5a
[nitter] fix extraction for instances without user banners
2 years ago
Mike Fährmann
d94aa1ee02
[gelbooru] fix --range for favorites ( #3704 )
2 years ago
Mike Fährmann
1f82b00b8f
[gelbooru] fix and improve --range for pools
2 years ago
ClosedPort22
1a977f0f62
[downloader:http] handle exceptions in 'validate'
...
This isn't strictly necessary for 'exhentai.py', but it improves
efficiency when the adapter is reused
2 years ago
Mike Fährmann
197882cf12
[twitter] add 'hashtag' extractor ( #3783 )
2 years ago
Mike Fährmann
082d55de16
fix circular reference detection for -K
2 years ago
Mike Fährmann
2ab66ad899
update -K output to include quotes around keys
2 years ago
Mike Fährmann
fe41a2b159
[formatter] support putting keys in quotes
...
i.e. obj["key"] or obj['key']
as in f-strings
2 years ago
Mike Fährmann
46fdf46f21
[formatter] support loading an f-string from a template file
...
"\fTF ~/path/to/file.txt"
2 years ago
Mike Fährmann
1a4d4a799b
[formatter] support filesystem paths for \fM
2 years ago
Mike Fährmann
9789ebac52
[naverwebtoon] fix extraction ( #3729 )
2 years ago
Mike Fährmann
72f1f16eb2
[weibo] support 'mix_media_info' entries ( #3793 )
2 years ago
ClosedPort22
d4fb4ff47f
[twitter] extract TwitPic URLs in text ( #3792 )
...
also ignore previously seen URLs
2 years ago
Mike Fährmann
00f0233b28
[postprocessor:metadata] add 'skip' option ( #3786 )
2 years ago
Mike Fährmann
2bb937014f
[twitter] fall back to legacy /media endpoint when not logged in
2 years ago
Mike Fährmann
b68094d326
[twitter] support 'note_tweet's
2 years ago
Mike Fährmann
3dcabc97ed
[twitter] update API endpoints and parameters
2 years ago
Mike Fährmann
a1ca2404f9
add 'globals' instead of overwriting the default ( #3773 )
2 years ago
Mike Fährmann
dcb8af659a
[gelbooru] extract favorites without needing cookies ( #3704 )
...
TODO: fix --range
2 years ago
Mike Fährmann
b756dc13aa
[gelbooru] warn about missing cookies for favorites ( #3704 )
...
and add docstring so it shows up in --list-extractors
2 years ago
Mike Fährmann
17bd053d94
[hiperdex] fix extraction ( #3768 )
2 years ago
Mike Fährmann
f7ce33c85c
[output] set 'errors=replace' for output streams ( #3765 )
...
fixes regression from e480a933
2 years ago
Mike Fährmann
a14a2d6e59
release version 1.25.0
2 years ago
ClosedPort22
fcaeaf539c
[downloader:http] handle exceptions while consuming content
2 years ago
Mike Fährmann
4235d412c4
implement 'actions'
...
continuation of d37e7f48
but more versatile and extendable
Example:
"actions": [
# change debug messages to info
["debug", "level ~info"],
# change exit status to a non-zero value
["info:^No results for", "status |= 1"],
# exit with status 2 on 429
["warning:429", "exit 2"],
# restart extractor when no cookies found
["warning:^[Nn]o .*cookies", "restart"]
]
2 years ago
Mike Fährmann
817fc0fbd1
[nitter] remove nitter.pussthecat.org
...
"Shutdown"
2 years ago
Mike Fährmann
67ec91cdbd
[downloader:http] change '_http_retry' to accept a Python function
...
and rename '_http_retry_codes' to '_http_retry'
(#3569 )
2 years ago
Mike Fährmann
175822e065
merge #3738 : [generic] add tests
2 years ago
Mike Fährmann
4883420e67
[generic] revert pattern change
2 years ago
ClosedPort22
df77271438
[downloader:http] add 'consume-content' option
...
* fix connection not being released when the response is neither
successful nor retried
* add the ability to consume the HTTP response body instead of closing
the connection
reference:
https://docs.python-requests.org/en/latest/user/advanced/#body-content-workflow
2 years ago
Mike Fährmann
9037128315
[twitter] fix some 'original' retweets not downloading ( #3744 )
2 years ago
Mike Fährmann
ea3d95e7e8
merge #3740 : [deviantart] add support for fxdeviantart.com URLs
2 years ago
Mike Fährmann
9abcb2b6e5
update headers and ciphers for '"browser": "chrome"'
2 years ago
ClosedPort22
c489aecb3e
[deviantart] add support for fxdeviantart.com URLs
...
fxdeviantart.com is a service that fixes embeds on Discord, similar to
fxtwitter.com
2 years ago
ClosedPort22
34a7fab0e2
[generic] add support for IDNs
...
(internationalized domain name)
2 years ago
Mike Fährmann
c9a7345228
[newgrounds] prevent archive ID overlap ( #3681 )
...
add an 'i' and 'a' prefix to image and audio files
(/art/view/, /audio/listen/)
since their numeric ID may conflict with movies and other media
2 years ago
Mike Fährmann
8148c2a097
[downloader:ytdl] prevent exception on empty results
...
a7c7953107 (commitcomment-92042240)
2 years ago
Mike Fährmann
da9840a39d
[reddit] update 'videos' option ( #3712 )
...
- add 'dash' to directly extract DASH manifest URLs
(was default behavior since a7c79531
)
- change default strategy back to before a7c79531
- disable 'Falling back on generic information extractor' warning
2 years ago
Mike Fährmann
8f8b4de0e8
[ytdl] fix '--parse-metadata' ( #3663 )
2 years ago
Mike Fährmann
11df3a021d
[formatter] enclose f-strings with """ instead of '''
2 years ago
Mike Fährmann
baf41d7437
[misskey] update ( #3717 )
...
- add module docstring
- add options to docs/gallery-dl.conf
2 years ago
Mike Fährmann
7610d9cf82
merge #3675 : [pixiv] fix --write-tags for '"tags": "original"'
2 years ago
Mike Fährmann
6762d99515
merge #3717 : [misskey] add misskey extractors
2 years ago
Mike Fährmann
b8a702929d
[oauth] import extractor modules on demand
2 years ago
Mike Fährmann
dd88740ec7
replace remaining instances of base64 with binascii
2 years ago
enduser420
e1867cf5eb
[misskey] add 'renotes' and 'replies' options
2 years ago
enduser420
a95b5e0d8e
[misskey] add misskey extractors
2 years ago
Mike Fährmann
0d142e403c
[szurubooru] add 'tag' and 'post' extractors ( #3583 , #3713 )
2 years ago
Mike Fährmann
075c965512
add '--config-create' command-line option
...
(#2333 )
2 years ago
Mike Fährmann
26d06e0bb2
move executable check into util.py
2 years ago
Mike Fährmann
de2f35d068
simplify config.load()
2 years ago
Mike Fährmann
632d5d7745
allow loading config files in TOML format with --config-toml
2 years ago
Mike Fährmann
9e870eb930
rename --ignore-config to --config-ignore
...
--ignore-config still works as before,
but is no longer shown by --help
2 years ago
Mike Fährmann
d66257f2c8
improve option.Formatter performance
...
as always, only a very marginal difference,
but it still uses less resources than before
2 years ago
Mike Fährmann
d788e6c60c
implement 'globals' option
2 years ago
Mike Fährmann
b14f8d5817
[gelbooru] add 'favorite' extractor ( #3704 )
...
requires logged in cookies to work
2 years ago
Mike Fährmann
e480a93337
add 'output.stdout', '.stdin', and '.stderr' options
...
(#1621 , #2152 , #2529 )
Allow setting custom input/output encodings and options
without having to rely on Python's defaults.
2 years ago
Mike Fährmann
a70a3e5da6
[mangasee] extract 'author' and 'genre' metadata ( #3703 )
...
Both are lists/arrays. Use {author!S} or {genre:J, } to format them.
2 years ago
Mike Fährmann
6b03506655
[deviantart] allow searching when not logged in
2 years ago
Mike Fährmann
511a051705
[fanbox] fix crash with missing images ( #3673 )
2 years ago
Mike Fährmann
3fa456d989
[deviantart] remove mature scraps warning ( #3691 )
...
warn about private deviations
when paginating over eclipse results
2 years ago
Mike Fährmann
51301e0c31
replace remaining time.sleep() calls
...
with Extractor.sleep() or request_interval
2 years ago
Mike Fährmann
6ed4309aba
[deviantart] add 'gallery-search' extractor ( #1695 )
2 years ago
Mike Fährmann
3d8777fbc1
move user agent string to util.py
2 years ago
Mike Fährmann
56039d2456
add 'hash_md5' and 'hash_sha1' functions ( #3679 )
...
... to global eval namespace
2 years ago
Mike Fährmann
e1df7f73b1
[deviantart] add 'search' extractor
...
(#538 , #1264 , #2954 , #2970 , #3577 )
Requires login to fetch any results, since the API endpoint raises an
error for not logged in requests.
TODO: parse HTML search results
2 years ago
Gray Manley
f33ac885a6
[pixiv] fix tag write when set to original
2 years ago
Mike Fährmann
4f029ab38b
[pornpics] support '/pornstar' and '/channels' listings
...
- fix docstring (#3671 )
- simplify code
2 years ago
Mike Fährmann
cbe4769246
[danbooru] use gallery-dl UA ( #3665 )
...
this removes the ability to set a custom UA via 'user-agent' option
for extractor requests
2 years ago
Mike Fährmann
253ac08203
pre-define and use 'gallery-dö/<version>' UA string
2 years ago
Mike Fährmann
b4899c266f
merge #3656 : [deviantart] fix crash when handling deleted deviations in status updates
2 years ago
Mike Fährmann
bb11c2a576
merge #3662 : [redgifs] add 'collection' extractors
2 years ago
Mike Fährmann
884f1848d6
[redgifs] fix syntax for older Python versions
...
and update docs/supportedsites
2 years ago
Mike Fährmann
725baedad3
[deviantart] use '/collections/all' endpoint for favorites
...
(#3666 ,#3668)
2 years ago
Mike Fährmann
2bd8f2f4bd
[pornpics] add 'search' and 'tag' extractors
...
(#263 , #3544 , #3654 )
2 years ago
Mike Fährmann
79bc82884c
[pornpics] add 'gallery' extractor ( #263 , #3544 , #3654 )
2 years ago
Mike Fährmann
7bdc1d6d3d
[manganelo] update and fix metadata extraction
2 years ago
Mike Fährmann
363bb76dff
[manganelo] simplify URL pattern
2 years ago
enduser420
b28bd9789e
[redgifs] add 'collection' extractors
2 years ago
ClosedPort22
f4e211356d
[deviantart] slight refactor
2 years ago
Mike Fährmann
bd5d08abbc
[catbox] add 'file' extractor ( #3570 )
2 years ago
Mike Fährmann
8e1e8a5bea
[soundgasm] rewrite ( #3578 )
...
use a more standard extractor structure to make -A work as expected
2 years ago
Mike Fährmann
0b93420a81
[pinterest] unescape search terms ( #3621 )
2 years ago
Mike Fährmann
ad96e70546
[bunkr] fix extraction ( #3636 , #3655 )
2 years ago
Mike Fährmann
9335d55bbc
[manganelo] support mobile-only chapters
2 years ago
ClosedPort22
a74114ef7a
[deviantart] fix crash when handling deleted deviations
...
in status updates
2 years ago
Mike Fährmann
75570ad3f1
[oauth] remove stray 'exit()' ( #3628 )
...
- bug from 70ce45d9
- broke oauth:tumblr, oauth:flickr, and oauth:smugmug
2 years ago
Mike Fährmann
d37e7f4898
add 'hooks' option
...
Very much a work in progress.
At the moment, it allows to
- wait and restart an extractor (#3338 )
- change the exit code (#3630 )
- change the log level of a logging message
based on the contents of a logging message
2 years ago
Mike Fährmann
8fb043e8ff
[tumblr] raise more detailed errors for dashboard-only blogs
...
(#3628 )
2 years ago
Mike Fährmann
d4232f3a8b
implement restarting an extractor ( #3338 )
2 years ago
Mike Fährmann
ce996dd21b
[poipiku] warn about incorrect passwords ( #3646 )
2 years ago
Mike Fährmann
70ce45d965
[oauth] use default name for browsers without 'name' attribute
...
(#3645 )
Seem to only be an issue for MacOSXOSAScript before Python 3.11.
d12bec6993
2 years ago
Mike Fährmann
1aae72773f
put argument init on separate lines
2 years ago
Mike Fährmann
2a53e6445c
[bunkr] update domain ( #3636 )
2 years ago
Mike Fährmann
5503ac4d5e
replace json.dumps with direct calls to JSONEncoder.encode
2 years ago
Mike Fährmann
dd884b02ee
replace json.loads with direct calls to JSONDecoder.decode
2 years ago
Mike Fährmann
b7337d810e
[postprocessor:metadata] add 'sort' and 'separators' options
2 years ago
Mike Fährmann
8805bd38ab
merge #3622 : [imagetwist] add phun.imagetwist.com and imagehaha.com support
2 years ago
Mike Fährmann
706ec70e89
[imagetwist] simplify pattern and add tests
2 years ago
Mike Fährmann
f2e91732ae
[instagram] add 'user' metadata field ( #3107 )
...
at the moment only for URLs that need to translate user name to ID
2 years ago
Mike Fährmann
3436c6b117
[postprocessor:metadata] speed up JSON encoding
2 years ago
Prinz23
29f0830b53
[imagetwist] add phun.imagetwist.com and imagehaha.com alias to imagetwist extractor
2 years ago
Mike Fährmann
762a68996b
implement 'archive-pragma' option
2 years ago
Mike Fährmann
bbf0911a46
[e621] implement 'notes' and 'pools' metadata extraction
...
(#3425 )
2 years ago
Mike Fährmann
925b467496
split e621 from danbooru module ( #3425 )
2 years ago
Mike Fährmann
1ae48a54f8
[twitter] add 'transform' option
2 years ago
Mike Fährmann
78d3960a31
[postprocessor:exec] implement archive options ( #3584 )
2 years ago
Mike Fährmann
489c51cecc
[telegraph] fix extraction when images not in <figure> ( #3590 )
2 years ago
Mike Fährmann
0f7e6c422a
merge #3596 : [shopify] support ohpolly.com
2 years ago
enduser420
fcf7030b85
[shopify] support ohpolly.com
2 years ago
Mike Fährmann
a6a631f992
merge #3589 : [redgifs] support v3 URLs
2 years ago
Mike Fährmann
137a395ae0
[imagefap] fix infinite pagination loop ( #3594 )
2 years ago
Mike Fährmann
3c708ade8f
[imagefap] fix metadata extraction
2 years ago
Mike Fährmann
17e24eacf0
[imagefap] update 'gallery' URLs ( #3595 )
2 years ago
Mike Fährmann
d16873941c
[downloader:http] use 'time.monotonic()'
2 years ago
Mike Fährmann
c2bc70593e
implement ability to load external extractor classes
...
- -X/--extractors
- extractor.module-sources
2 years ago
enduser420
a18f627bfc
[redgifs] support v3 URLs
2 years ago
Mike Fährmann
9ec627c760
release version 1.24.5
2 years ago
Mike Fährmann
13a90969c7
merge #3575 : [nudecollect] add 'image' and 'album' extractors
2 years ago
Mike Fährmann
aacd27e4ef
merge #3581 : [hotleak] fix video URLs
2 years ago
Mike Fährmann
abc3619feb
[lexica] add 'search' extractor ( #3567 )
2 years ago
Mike Fährmann
7c9b1ec830
[hotleak] optimize decoding video URLs
...
- use binascii module
- combine slice and reverse step
2 years ago
nifnat
f14dbfe079
Make decode_video_url static (used in both post and creator extractor).
2 years ago
nifnat
bd23a701f3
Tidy up code.
2 years ago
nifnat
7f34f99a26
Reverse engineered obfuscated JS function and reimplemented in python.
2 years ago
Mike Fährmann
0d818d3540
[fantia] send 'X-CSRF-Token' headers ( #3576 )
2 years ago
Mike Fährmann
f58215705a
add '-O/--postprocessor-option' command-line option ( #3565 )
2 years ago
enduser420
2a5903dc16
[nudecollect] add 'image' and 'album' extractors
2 years ago
Mike Fährmann
c8fdd5096e
merge #3571 : [bunkr] Fix extracting mkv and ts files
2 years ago
Mike Fährmann
58c008e30a
[hiperdex] update domain ( #3572 )
2 years ago
Luc Ritchie
842064e597
[bunkr] Fix extracting ts files
2 years ago
Luc Ritchie
99ca0437e4
[bunkr] Fix extracting mkv files
2 years ago
Mike Fährmann
76b01b64cf
[kemonoparty] remove MD5 hash extraction ( #3531 )
...
This partially reverts commit 20d6194ffa
.
2 years ago
Mike Fährmann
09fb212414
[philomena] match URLs with www subdomain
2 years ago
Mike Fährmann
7e2fd2e573
merge #3560 : [deviantart] add support for /deviation/ and fav.me URLs
2 years ago
Mike Fährmann
caae8fefe1
merge #3541 : [deviantart] add extractor for status updates
2 years ago
ClosedPort22
c90b4ea8d9
[deviantart] add support for fav.me URLs
2 years ago
Mike Fährmann
d63af4f3d3
merge #3555 : [generic] fix regex for non-src image URLs
2 years ago
Mike Fährmann
8993b10751
[mastodon] add 'num' and 'count' metadata fields ( #3517 )
2 years ago
Mike Fährmann
d817d23ccb
[instagram] update csrf token handling
...
- update internal value according to cookie
- do not send a second 'csrftoken' cookie
2 years ago
Mike Fährmann
00b94946b3
[instagram] show -o cursor=… after every error ( #3440 )
2 years ago
ClosedPort22
674c719646
[deviantart] refactor base36 conversion
2 years ago
ClosedPort22
293abb8921
[deviantart] add support for /deviation/ URLs
2 years ago
thatfuckingbird
8cfeed78b1
[generic] fix regex for non-src image URLs
2 years ago
Mike Fährmann
fc6ea8ee5c
[instagram] update API domain and headers
2 years ago
ClosedPort22
597b89245e
[deviantart] misc improvements to status extractor
...
- relax regex pattern
- handle invalid 'items' field
- add a test for shared sta.sh item
Co-authored-by: Mike Fährmann <mike_faehrmann@web.de>
2 years ago
Mike Fährmann
137de090dd
merge #3549 : [twitter] fix search ( #3536 )
2 years ago
Mike Fährmann
02e314c1b6
merge #3537 : [wikifeet/wikifeetx] add 'gallery' extractor
2 years ago
Mike Fährmann
568112dfbb
[oauth] improve output
...
- show which api key / client id gets used (#3518 )
- show in which browser authorization URLs gets opened in
2 years ago
ClosedPort22
ab58c375b4
[twitter] fix search ( #3536 )
...
- partially revert 18fe4b334d
- properly search for cursor when processing 'replaceEntry'
2 years ago
Mike Fährmann
df91ebb945
[oauth] simplify OAuth 1.0a init
2 years ago
ClosedPort22
013733c9e9
[deviantart] fix index fields for embedded/shared images
2 years ago
ClosedPort22
c4aeca7a5a
[deviantart] improve handling of statuses
...
- recursively yield statuses
- ignore items with missing or unexpected field(s)
2 years ago
ClosedPort22
3b32671fbd
[deviantart] add extractor for status updates
...
extract user status updates using the '/user/statuses/' endpoint
2 years ago
Mike Fährmann
107c60c973
[sankaku] update URL pattern ( #3523 )
...
match tag searches with language codes without a trailing slash
2 years ago
enduser420
5cb263fdd2
[wikifeet/wikifeetx] add 'gallery' extractor
2 years ago
Mike Fährmann
35a30498bc
merge #3531 : [kemonoparty] improve hash extraction
...
- extract md5 hashes if available
- extract discord file hashes
2 years ago
Mike Fährmann
ec9ff7640d
merge #3535 : [downloader:http] add signature checks for .blend, .obj, and .clip files
2 years ago
Mike Fährmann
9683d79bb7
[twitter] "fix" search pagination ( #3536 , #3534 )
...
- properly process instructions
- do not expect a predetermined instruction order
2 years ago
Mike Fährmann
4fec848858
[twitter] use "browser": "firefox" by default ( #3522 )
...
and reenable TLS 1.2 ciphers
2 years ago
Mike Fährmann
78937564fd
[twitter] fix login after 32b03433
2 years ago
ClosedPort22
b6706b373a
[downloader:http] add signature checks for some formats
...
also add the MIME type for .obj files
2 years ago
ClosedPort22
20d6194ffa
[kemonoparty] improve hash extraction
...
- extract MD5 hash from URLs
- extract MD5 and SHA256 hash from Discord URLs (kemono.party only)
- minor optimization (do not call 'hashes.add' when 'duplicates' is
true)
- update tests accordingly
Co-authored-by: Mike Fährmann <mike_faehrmann@web.de>
2 years ago
Mike Fährmann
80a2ff2d38
support setting 'write-pages' to "ALL"
...
to show authentication header, cookies, etc
2 years ago
Mike Fährmann
d6793b2c7d
include request body in 'write-pages=all' output
2 years ago
Mike Fährmann
c881548a27
add 'extractor.retry-codes' option ( #3313 )
...
do not retry 429 and 430 by default
2 years ago
Mike Fährmann
e30e8aeef7
[mastodon] rename '_check_move' -> '_check_moved'
2 years ago
Mike Fährmann
32b0343334
[twitter] refresh guest tokens ( #3445 , #3458 )
2 years ago
Mike Fährmann
512abeb4ae
[booru] add 'url' option
2 years ago
Mike Fährmann
c87bd1a752
[danbooru] extend 'metadata' option
...
make it possible to specify a custom list of metadata includes
2 years ago
Mike Fährmann
26c3292538
[twitter] disable TLS 1.2 ciphers by default ( #3522 )
2 years ago
Mike Fährmann
18fe4b334d
[twitter] remove 'tweet_search_mode' from search parameters ( #3522 )
...
and update API root and general query parameters
2 years ago
Mike Fährmann
ec04c97075
release version 1.24.4
2 years ago
Mike Fährmann
c0d7d2be35
[downloader:http] add 'validate' option
2 years ago
Mike Fährmann
85bd1cbc89
[kemonoparty] fix regression from 473bd380
( #3519 )
...
- do not access 'response.content' unless necessary
- only validate responses if filename extensions differ
2 years ago
Mike Fährmann
805a5663ec
release version 1.24.3
2 years ago
Mike Fährmann
473bd380c8
[kemonoparty] reject invalid/empty files ( #3510 )
2 years ago
Mike Fährmann
4833ec323e
[imagefap] add 'folder' extractor ( #3504 )
2 years ago
Mike Fährmann
362cd6991b
[pixiv] implement 'metadata-bookmark' option ( #3417 )
2 years ago
Mike Fährmann
0895e6afee
merge #3462 : [docs] Update links and fix field typo
2 years ago
Mike Fährmann
2142b9c7ae
merge #3503 : [myhentaigallery] handle whitespace before title tag
2 years ago
Mike Fährmann
3a0450adbf
[behance] use default delay between requests ( #2507 )
2 years ago
Mike Fährmann
2cae4567ba
[telegraph] fix file URLs ( #3506 )
2 years ago
Mike Fährmann
cbaeee9533
[imagefap] warn about redirects to '/human-verification' ( #1140 )
2 years ago
Mike Fährmann
435de1329a
[imagefap] use default delay between requests ( #1140 )
2 years ago
Erik Rimskog
a8a982359e
[myhentaigallery] handle whitespace before the title tag
2 years ago
Mike Fährmann
d1dd52349a
merge #3189 : [tcbscans] add 'chapter' and 'manga' extractors
2 years ago
Mike Fährmann
2f31d21509
merge #3455 : [twitter] apply tweet type checks before uniqueness check
2 years ago
enduser420
e8541a131d
[tcbscans] add 'chapter' and 'manga' extractors
2 years ago
Mike Fährmann
9695c4e88d
emit debug logging message when loading cookies from file
...
attempt nr. 2
no idea how I managed to remove 6514828d
in a918ce29
2 years ago
Mike Fährmann
30a31836e7
merge #3449 : [twitter] force HTTPS for TwitPic URLs
2 years ago
Mike Fährmann
e18482e9ae
[twitter] improve 'http' -> 'https' replacement
2 years ago
Mike Fährmann
4fd6da474f
merge #3473 : [twitter] fix crash when using 'expand' and 'syndication'
2 years ago
Mike Fährmann
a918ce29b5
run tests on ubuntu-20.04
...
and remove Python 3.4, since that's no longer available
on this test runner
2 years ago
Mike Fährmann
6514828d4e
emit debug logging message when loading cookies from file
2 years ago
Mike Fährmann
3a238fd490
[poipiku] warn about login requirements
2 years ago
Mike Fährmann
fa144f38ed
[ytdl} fix dfe4f00c
for legacy yt-dlp
2 years ago
Mike Fährmann
f29ba089ff
merge #3474 : [fanleaks] add 'post' and 'model' extractors
2 years ago
Mike Fährmann
6933727b45
merge #3483 : [twitter] implement 'syndication=extended'
2 years ago
Mike Fährmann
07ed3a1fbf
merge #3460 : [poipiku] fix extraction for a different warning button style
...
(#3493 , #3492 )
2 years ago
Mike Fährmann
9116398c1c
[pinterest] add 'domain' option ( #3484 )
...
use input URL domain by default
2 years ago
Mike Fährmann
6f6af36cad
use double quotes for --help examples
2 years ago
Mike Fährmann
dfe4f00ca2
[ytdl] update for yt-dlp changes
2 years ago
blankie
2f985bcddb
[poipiku] fix extraction for a different warning button style
2 years ago
Mike Fährmann
294108c90a
[pinterest] support 'All Pins' boards ( #2855 , #3484 )
2 years ago
Mike Fährmann
77df8d3116
[deviantart] implement username&password login for scraps ( #1029 )
...
re-login when getting prematurely logged out by dA
is missing at the moment
2 years ago
Mike Fährmann
ed2d715019
fix 'keywords' in extractor tests ( #3491 )
2 years ago
Mike Fährmann
3f29b8fe91
[cookies] convert browser names to lowercase
2 years ago
ClosedPort22
6853b14be3
[twitter] apply suggestions from code review
...
Co-authored-by: Mike Fährmann <mike_faehrmann@web.de>
2 years ago
Mike Fährmann
4611237f8c
merge #3457 : [danbooru] extract uploader metadata (if option is set)
2 years ago
Mike Fährmann
e7522482bb
merge #3463 : [lynxchan] support 'bbw-chan.nl'
2 years ago
Mike Fährmann
7d6c846176
[fanbox] return 'imageMap' files in order ( #2718 )
2 years ago
Mike Fährmann
dc8e7ff54e
[bunkr] fix URLs returned by API ( #3481 )
2 years ago