33d912490f
merge #4419 : [bunkr] Fix extracting wmv files
1 year ago
01610a6e9e
merge #4412 : [bunkr] fix media domain for cdn9
1 year ago
b19d62263b
merge #4420 : [issuu] fix extraction
1 year ago
6dc8be5e48
[issuu] fix extraction
1 year ago
85a070b9e6
[bunkr] Fix extracting wmv files
1 year ago
3f8ff692a7
[bunkr] fix media domain for cdn9
...
Fixes #4386
1 year ago
d8b21a97bf
[formatter] use 'rpartition' for \fM format strings
...
fixes using absolute module paths like C:\path\module.py on Windows
1 year ago
f9fb276e81
[postprocessor] add 'prepare-after' event ( #4083 )
1 year ago
0ef1fcab20
[postprocessor] update 'finalize' events
...
Add 'finalize-error' and 'finalize-success' events that trigger
depending on whether error(s) did or did not happen.
'finalize' itself now always triggers regardless of error status.
(was supposed to have the same behavior as the new 'finalize-success')
1 year ago
af4bdb62a7
merge #4403 : [downloader:http] close connection when file already exists
1 year ago
15275b3524
[postprocessor:ugoira] restore 'libx264-prevent-odd' ( #4407 )
...
was accidentally removed in commit be9547a5
1 year ago
391a7d74c8
[giantessbooru] fix and move to separate module ( #4373 )
...
too many differences to the other shimmie2 sites
1 year ago
5448268d5c
[downloader:http] close connection when file already exists ( #3748 )
1 year ago
3963dbe5e4
extend 'parent>child' categories
...
continuation of ed21908f
1 year ago
089d1a4f67
[twitter] fix 'TweetWithVisibilityResults' ( #4369 )
1 year ago
a4f7f7da17
add '_dump()' convenience method to Extractor
1 year ago
df5c7ee03e
[deviantart] fix search ( #4384 )
...
send correct usernames instead of 'u'
1 year ago
a60db454af
[sankaku] update/fix API headers
...
'Referer' and 'Origin' were both empty
1 year ago
fb3f0453db
[twitter] improve error messages for single Tweets ( #4369 )
...
also fixes '"quoted": false' not having any effect
1 year ago
541bff5a37
[pururin] fix extraction ( #4375 )
...
- rename 'title_jp' to 'title_ja'
- change type of 'collection', 'convention', and 'scanlator' to list
1 year ago
6a87c314af
[instagram] fix private posts with long shortcodes ( #4362 )
1 year ago
f899fac4c5
[giantessbooru] fix extraction ( #4373 )
...
This does not fix anything Cloudflare related,
just other things caused by a site update.
1 year ago
136283d402
[shimmie2] update base URL pattern
...
to match new giantessbooru URLs
1 year ago
9d67655397
add "ascii+" as a special 'path-restrict' value ( #4371 )
1 year ago
c79359eb3a
[fantia] improve metadata extraction ( #4126 )
...
extract all metadata and URLs before starting to download
1 year ago
48ef062867
fix issues with 'Extractor.finalize()'
...
- prevent crash in InstagramUserExtractor (#4359 )
- call it at the end of every DownloadJob
- add it to tests
1 year ago
ed21908fda
initial support for child extractor options
...
Using "parent-category>child-category" as extractor category in a config
file allows to set options for a child extractor when it was spawned by
that parent.
For example "reddit>gfycat" to set gfycat options for when it was found
in a reddit post.
{
"extractor": {
"gfycat": {
"filename": "regular filename"
},
"reddit>gfycat": {
"filename": "reddit-specific filename"
}
}
}
Note: This does currently not work for most imgur links due to how its
extractor hierarchy is structured.
1 year ago
255d08b79e
add test for 'Extractor.initialize()' ( #4359 )
1 year ago
2bcf0a4c49
[instagram] fix initialization order ( #4359 )
...
regression caused by the changes in a383eca7
1 year ago
7eab101144
[acidimg] fix extraction
...
swap ' and " again (2e309a13
1 year ago
62fce6a75f
[imagehosts] adjust variable names ( #4358 )
...
prefix them with underscores to prevent a clash
with the new 'self.cookies' from d97b8c2f
1 year ago
e8299b459a
[moebooru] match search URLs with empty 'tags' ( #4354 )
1 year ago
7fbc304ae9
[twitter] fix crash on private user ( #4349 )
1 year ago
1ece3b92ff
[mangadex] allow multiple values for 'lang' ( #4093 )
...
This was already possible by setting 'lang' to a list of strings,
but now it can also be done as a more command-line friendly string.
-o lang=fr,it
1 year ago
52053b58f0
[lensdump] fix extraction ( #4352 )
1 year ago
11f71a9cba
remove 'mememuseum' module
...
This was forgotten when adding generic Shimmie2 support in 7865067d
1 year ago
a383eca7f6
decouple extractor initialization
...
Introduce an 'initialize()' function that does the actual init
(session, cookies, config options) and can called separately from
the constructor __init__().
This allows, for example, to adjust config access inside a Job
before most of it already happened when calling 'extractor.find()'.
1 year ago
6c9432165e
add return value to 'PostProcessor._init_archive()'
1 year ago
54d974deb0
add 'python' post processor
...
similar to 'exec' but calls a Python function
1 year ago
1baf83a9e5
[hiperdex] fix for unicode titles ( #4325 )
1 year ago
7da954f810
[flickr] update default API credentials ( #4332 )
...
and add a delay between API requests
1 year ago
a45a17ddb7
[pixiv] ignore 'limit_sanity_level' images ( #4328 )
1 year ago
088e8d5fcf
[pornhub] fix extraction ( #4301 )
1 year ago
d97b8c2fba
consistent cookie-related names
...
- rename every cookie variable or method to 'cookies_*'
- simplify '.session.cookies' to just '.cookies'
- more consistent 'login()' structure
1 year ago
ceebacc9e1
remove 'pyopenssl' option
1 year ago
3c2c7e21dd
merge #4319 : [zerochan] fix 'tags' extraction
1 year ago
0ba8d1f168
merge #4312 : [redgifs] add 'niches' extractor
1 year ago
c5565f79f7
merge #4096 : [danbooru] add support for booru.borvar.art instance
1 year ago
63326e3168
[danbooru] add tests for booruvar
1 year ago
5171d8975c
[E621] support 'e6ai.net' ( #4320 )
1 year ago
a996d936d2
[imagefap] fix pagination ( #3013 )
1 year ago
22099422ca
[deviantart] fix shortened URLs ( #4316 )
1 year ago
90231f2d5a
[twitter] add 'tweet-endpoint' option ( #4307 )
...
use the newer TweetResultByRestId only for guests by default
1 year ago
20ed647f6f
[twitter] add 'user' extractor and 'include' option ( #4275 )
1 year ago
86be197d11
[twitter] remove '/search/adaptive.json'
1 year ago
d52ed2bc5a
[zerochan] fix 'tags' extraction
1 year ago
12cd85658b
[redgifs] add 'niches' extractor
1 year ago
248e8bc699
release version 1.25.8
1 year ago
bc9123cfee
[naverwebtoon] fix 'comic' metadata extraction
1 year ago
ab5dde7221
[mangaread] fix 'tags' extraction
1 year ago
c9a82c9313
[erome] ignore duplicate album IDs
1 year ago
c84397023a
[slideshare] fix extraction
1 year ago
ffbbbd3baf
[gelbooru_v01] 'vidyart' -> 'vidyart2'
1 year ago
e40b90e137
merge #4303 : [gelbooru_v01] fix 'source' ( #4302 )
1 year ago
c6b31a2169
[reddit] set default 0.6s delay between requests ( #4292 )
...
to limit API requests to 100 per minute
https://www.reddit.com/r/redditdev/comments/14nbw6g/
1 year ago
20da41018d
[pornhub] set 'accessAgeDisclaimerPH' cookie ( #4301 )
1 year ago
75757c4ace
[gelbooru_v01] fix 'source' ( #4302 )
1 year ago
2dd6942d1c
[jpgfish] update domain to 'jpeg.pet'
1 year ago
1137b89ed4
[lineblog] remove module
...
"LINE BLOGは2023年6月29日をもちましてサービスを終了いたしました"
1 year ago
86560fe0cd
[bcy] remove module
...
"The website was shut down on July 12, 2023"
https://danbooru.donmai.us/wiki_pages/bcy
1 year ago
fceabee433
[philomena] use API interface class
...
handle 429 errors and retry after 10min (#4288 )
1 year ago
f079d9a703
[reddit] notify users about registering an oauth application
...
(#4292 , #4253 , #3943 )
1 year ago
fb3d1462b1
merge #4291 : [wikifeet] fix 'tag' extraction
1 year ago
0b08e2e8a8
merge #4287 : [twitter] Fix following extractor not getting all users
1 year ago
f6553ffd2f
[twitter] simplify '_pagination_users'
...
- remove 'stop' variable
- call 'cursor.startswith()' only once
1 year ago
1590124aae
[twibooru] fix '--range'
1 year ago
a2111dd025
[wikifeet] fix 'tag' extraction
1 year ago
a1ffa1ff09
[philomena] fix '--range' ( #4288 )
1 year ago
a27dbe8c82
[twitter] use 'TweetResultByRestId' endpoint ( #4250 )
...
allows accessing single Tweets without login
1 year ago
d3d639a159
[twitter] don't treat missing 'TimelineAddEntries' as fatal ( #4278 )
1 year ago
c321c773f2
make the code less ugly
1 year ago
a437a34bcf
fix lint i guess?
1 year ago
6cbc434b54
Fix users pagination
1 year ago
d5b6802774
[seiga] set 'skip_fetish_warning' cookie ( #4242 )
1 year ago
88d1e29401
[bunkr] use '.la' TLD for 'media-files12' servers ( #4147 , #4276 )
1 year ago
f0cb951566
[paheal] unescape 'source'
1 year ago
b480b7076a
[paheal] fix a78f8ce5 for enabled 'metadata' ( #4262 )
1 year ago
384337d3dd
[fantia] send 'X-Requested-With' header only for API requests ( #4273 )
1 year ago
c2ac665ff7
[fantia] send 'X-Requested-With' header ( #4273 )
1 year ago
7444fc125b
[gfycat] implement login support ( #3770 , #4271 )
...
For the record: '/webtoken' and '/weblogin' are not the same ...
1 year ago
e9b9f751bf
[gfycat] support '@me' user ( #3770 , #4271 )
1 year ago
5b59a0d143
update default User-Agent header to Firefox 115 ESR
1 year ago
0556e1ad45
merge #4268 : [newgrounds] extract & pass auth token for login
1 year ago
a16d7c59cb
[newgrounds] access 'response.text' only once
1 year ago
1bf9f52c99
[twitter] add 'ratelimit' option ( #4251 )
1 year ago
f86fdf64a6
[twitter] use GraphQL search by default ( #4264 )
1 year ago
1d4db83d49
[weibo] fix end of cursor based pagination
1 year ago
a78f8ce5b0
[paheal] fix extraction ( #4262 )
...
swap ' and "
1 year ago
9576652fa5
extract & pass auth token for newgrounds
1 year ago
5457007dd3
release version 1.25.7
1 year ago
Mike Fährmann