82296b1f05
[reddit] add 'previews' option ( #4322 )
...
another way to disable this new behavior
1 year ago
918ba4f847
[redgifs] match gfycat image URLs ( #4558 )
1 year ago
2ad75bab05
[deviantart] add 'is_original' metadata field ( #4559 )
...
true for 'downloadable' content, journals, flash animations,
and images without '/v1/' in their URL; false otherwise
1 year ago
9d8317d963
[deviantart] disable JWT updates ( #4548 , #4563 )
...
back to lowres images ...
1 year ago
8064663bda
[deviantart] update Eclipse API endpoints ( #4553 )
1 year ago
2cd801232b
fix --range causing crashes ( #4557 )
...
regression caused by a383eca7
1 year ago
3528974459
[instagram] handle exceptions due to missing media ( #4555 )
1 year ago
4963bb9b30
[reddit] improve comment metadata v2 ( #4482 )
...
provide main submission metadata at the top level
and comment metadata inside the 'comment' field,
i.e. the other way round than in 1710f1e9
1 year ago
7592c5e566
[patreon] fix extraction ( #4547 )
1 year ago
0655ce1bae
[mangakakalot] update domain
...
the old one still works, but it incurs a redirect
1 year ago
3ecb512722
send Referer headers by default
1 year ago
cb4798f07a
[architizer] fix extraction ( #4537 )
1 year ago
6178177227
[twitter] fix '_extractor' of following results ( #4536 )
...
regression from 20ed647f
1 year ago
d13c82eff1
[kemonoparty] update favorites API endpoint ( #4522 )
1 year ago
27ec653991
fix bug in test_init and update example URLs
1 year ago
24a1d46391
[mastodon] support '/@USER/following' URLs
...
Previously, only '/users/USER/following' got matched.
1 year ago
9f75713e00
[recursive] simplify
1 year ago
899df8f237
remove another '*' for keyword-only arguments
...
076380e0
1 year ago
6ae92da57e
Merge branch 'tests'
1 year ago
32da3c70d3
[behance] handle videos without 'renditions' ( #4523 )
1 year ago
ae5e049c4f
[redgifs] provide 'collection' metadata in a separate field ( #4508 )
...
instead of overwriting the actual metadata
1 year ago
1710f1e983
[reddit] improve comment metadata ( #4482 )
...
- provide 'date'
- make metadata of the main submission available as 'submission[…]'
1 year ago
4cdab8074e
update/fix --list-extractors
1 year ago
a453335a9f
remove test results in extractor modules
...
and add generic example URLs
1 year ago
1d2b5d0c60
update test comment positions
...
always put them above the test they're referring to
1 year ago
93a7a89cf6
[formatter] use value of last alternative ( #4492 )
...
fixes {fieldname|''} evaluating to the value of 'keywords-default'
instead of an empty string
1 year ago
f856987297
[subscribestar] fix preview detection ( #4468 )
...
and show a warning message when posts contain previews
1 year ago
4c0b3d5dc5
[twitter] fix crash when 'sortIndex' is None ( #4499 )
1 year ago
f2de70f254
[gfycat] remove module
1 year ago
6eca1fab9b
[gelbooru_v02] support 'xbooru.com' ( #4493 )
1 year ago
23bac772f2
[jpgfish] update domain to 'jpg1.su' ( #4494 )
1 year ago
ceb59e176f
fix default Firefox user agent string
...
note to self: do not trust some random third-party website
1 year ago
8259a5abe4
flake8
1 year ago
0b6e5b8161
[hiperdex] send Referer headers during file downloads ( #4490 )
1 year ago
a05821f8b4
[hiperdex] fix 'manga' metadata
...
remove trailing ' Manga'
1 year ago
03d471a0d4
merge #4481 : [pixiv] handle errors for private novels
1 year ago
8c477f7146
[bunkr] Fixes media-files-pizza.bunkr.ru failed to resolve.
1 year ago
c7e31b2724
Update pixiv.py
1 year ago
d3046561d4
continue downloading when the item is made private on pixiv
1 year ago
28798594e8
[gfycat] update pagination logic ( #4479 )
...
Some searches do not use cursor based pagination
but an offset based one.
1 year ago
a783c4f0fe
[pornhub] add 'gif' support ( #4463 )
1 year ago
ba842981af
[imagevenue] fix extraction ( #4473 )
1 year ago
7defb24e1e
[reddit] provide video previews if available ( #4322 )
1 year ago
fd65f27ede
[reddit] fix 'preview.redd.it' URLs ( #4470 )
1 year ago
06aaedded5
[twitter] extract 'source' metadata ( #4459 )
1 year ago
14af15bd18
[reddit] download preview for 404ed imgur links ( #4322 )
...
This is a pretty ugly hack as the internal infrastructure doesn't
really support switching from external URL to regular download in
case the former fails, but it kind of works ...
Can be disabled by setting 'reddit.fallback' to 'false'.
1 year ago
d12a5e440a
update docs/supportedsites
1 year ago
3a27150479
[instagram] add 'following' extractor ( #1848 )
1 year ago
e0829ff0fd
[twitter] add 'date_original' metadata for retweets ( #4337 , #4443 )
1 year ago
5ed245317d
[exhentai] add 'fav' option ( #4409 )
...
The name 'favorite' is already taken as extractor subcategory
1 year ago
fd6b413f3c
[exhentai] fix 'domain' option ( #4458 )
...
regression from a383eca7
1 year ago
fdfb22c91f
[instagram] fix video preview archive IDs ( #2135 , #4455 )
1 year ago
92f98e6f5e
'sys.exit' -> 'SystemExit'
1 year ago
410f783a33
implement 'subconfigs' option ( #4440 )
1 year ago
2b88ad19e9
[twitter] accept 'x.com' URLs ( #4452 )
1 year ago
c1c73c0b0e
[pp:ugoira] add '"framerate": "uniform"' ( #4421 )
1 year ago
2a3acd318a
[pp:ugoira] fix high frame rates ( #4421 )
...
only return an output frame rate for non-uniform ugoira
when the frame delay gcd is >= 10, i.e. 100 fps
1 year ago
70bdf32a88
[pp:ugoira] extend 'ffmpeg-output' ( #4421 )
...
- when setting this option to a string value,
pass -hide-banner and -loglevel to FFmpeg
- change default to "error"
1 year ago
8dceea3384
[shimme2] move 'giantessbooru' back into shimmie module ( #4373 )
...
Do the same thing as for 'realbooru' and override 'posts()'
insteadd of using a separate module.
1 year ago
6482f9453b
[behance] fix cookie usage ( #4417 )
1 year ago
d34195b41d
[behance] fix and update 'user' extractor ( #4417 )
1 year ago
4d3cf709da
[behance] add 'date' metadata field ( #4417 )
1 year ago
c689cd9720
[behance] show error for mature content ( #4417 )
1 year ago
33d912490f
merge #4419 : [bunkr] Fix extracting wmv files
1 year ago
01610a6e9e
merge #4412 : [bunkr] fix media domain for cdn9
1 year ago
b19d62263b
merge #4420 : [issuu] fix extraction
1 year ago
6dc8be5e48
[issuu] fix extraction
1 year ago
85a070b9e6
[bunkr] Fix extracting wmv files
1 year ago
3f8ff692a7
[bunkr] fix media domain for cdn9
...
Fixes #4386
1 year ago
d8b21a97bf
[formatter] use 'rpartition' for \fM format strings
...
fixes using absolute module paths like C:\path\module.py on Windows
1 year ago
f9fb276e81
[postprocessor] add 'prepare-after' event ( #4083 )
1 year ago
0ef1fcab20
[postprocessor] update 'finalize' events
...
Add 'finalize-error' and 'finalize-success' events that trigger
depending on whether error(s) did or did not happen.
'finalize' itself now always triggers regardless of error status.
(was supposed to have the same behavior as the new 'finalize-success')
1 year ago
af4bdb62a7
merge #4403 : [downloader:http] close connection when file already exists
1 year ago
15275b3524
[postprocessor:ugoira] restore 'libx264-prevent-odd' ( #4407 )
...
was accidentally removed in commit be9547a5
1 year ago
391a7d74c8
[giantessbooru] fix and move to separate module ( #4373 )
...
too many differences to the other shimmie2 sites
1 year ago
5448268d5c
[downloader:http] close connection when file already exists ( #3748 )
1 year ago
3963dbe5e4
extend 'parent>child' categories
...
continuation of ed21908f
1 year ago
089d1a4f67
[twitter] fix 'TweetWithVisibilityResults' ( #4369 )
1 year ago
a4f7f7da17
add '_dump()' convenience method to Extractor
1 year ago
df5c7ee03e
[deviantart] fix search ( #4384 )
...
send correct usernames instead of 'u'
1 year ago
a60db454af
[sankaku] update/fix API headers
...
'Referer' and 'Origin' were both empty
1 year ago
fb3f0453db
[twitter] improve error messages for single Tweets ( #4369 )
...
also fixes '"quoted": false' not having any effect
1 year ago
541bff5a37
[pururin] fix extraction ( #4375 )
...
- rename 'title_jp' to 'title_ja'
- change type of 'collection', 'convention', and 'scanlator' to list
1 year ago
6a87c314af
[instagram] fix private posts with long shortcodes ( #4362 )
1 year ago
f899fac4c5
[giantessbooru] fix extraction ( #4373 )
...
This does not fix anything Cloudflare related,
just other things caused by a site update.
1 year ago
136283d402
[shimmie2] update base URL pattern
...
to match new giantessbooru URLs
1 year ago
9d67655397
add "ascii+" as a special 'path-restrict' value ( #4371 )
1 year ago
c79359eb3a
[fantia] improve metadata extraction ( #4126 )
...
extract all metadata and URLs before starting to download
1 year ago
48ef062867
fix issues with 'Extractor.finalize()'
...
- prevent crash in InstagramUserExtractor (#4359 )
- call it at the end of every DownloadJob
- add it to tests
1 year ago
ed21908fda
initial support for child extractor options
...
Using "parent-category>child-category" as extractor category in a config
file allows to set options for a child extractor when it was spawned by
that parent.
For example "reddit>gfycat" to set gfycat options for when it was found
in a reddit post.
{
"extractor": {
"gfycat": {
"filename": "regular filename"
},
"reddit>gfycat": {
"filename": "reddit-specific filename"
}
}
}
Note: This does currently not work for most imgur links due to how its
extractor hierarchy is structured.
1 year ago
255d08b79e
add test for 'Extractor.initialize()' ( #4359 )
1 year ago
2bcf0a4c49
[instagram] fix initialization order ( #4359 )
...
regression caused by the changes in a383eca7
1 year ago
7eab101144
[acidimg] fix extraction
...
swap ' and " again (2e309a13
1 year ago
62fce6a75f
[imagehosts] adjust variable names ( #4358 )
...
prefix them with underscores to prevent a clash
with the new 'self.cookies' from d97b8c2f
1 year ago
e8299b459a
[moebooru] match search URLs with empty 'tags' ( #4354 )
1 year ago
7fbc304ae9
[twitter] fix crash on private user ( #4349 )
1 year ago
1ece3b92ff
[mangadex] allow multiple values for 'lang' ( #4093 )
...
This was already possible by setting 'lang' to a list of strings,
but now it can also be done as a more command-line friendly string.
-o lang=fr,it
1 year ago
52053b58f0
[lensdump] fix extraction ( #4352 )
1 year ago
11f71a9cba
remove 'mememuseum' module
...
This was forgotten when adding generic Shimmie2 support in 7865067d
1 year ago
a383eca7f6
decouple extractor initialization
...
Introduce an 'initialize()' function that does the actual init
(session, cookies, config options) and can called separately from
the constructor __init__().
This allows, for example, to adjust config access inside a Job
before most of it already happened when calling 'extractor.find()'.
1 year ago
6c9432165e
add return value to 'PostProcessor._init_archive()'
1 year ago
54d974deb0
add 'python' post processor
...
similar to 'exec' but calls a Python function
1 year ago
1baf83a9e5
[hiperdex] fix for unicode titles ( #4325 )
1 year ago
7da954f810
[flickr] update default API credentials ( #4332 )
...
and add a delay between API requests
1 year ago
a45a17ddb7
[pixiv] ignore 'limit_sanity_level' images ( #4328 )
1 year ago
088e8d5fcf
[pornhub] fix extraction ( #4301 )
1 year ago
d97b8c2fba
consistent cookie-related names
...
- rename every cookie variable or method to 'cookies_*'
- simplify '.session.cookies' to just '.cookies'
- more consistent 'login()' structure
1 year ago
ceebacc9e1
remove 'pyopenssl' option
1 year ago
3c2c7e21dd
merge #4319 : [zerochan] fix 'tags' extraction
1 year ago
0ba8d1f168
merge #4312 : [redgifs] add 'niches' extractor
1 year ago
c5565f79f7
merge #4096 : [danbooru] add support for booru.borvar.art instance
1 year ago
63326e3168
[danbooru] add tests for booruvar
1 year ago
5171d8975c
[E621] support 'e6ai.net' ( #4320 )
1 year ago
a996d936d2
[imagefap] fix pagination ( #3013 )
1 year ago
22099422ca
[deviantart] fix shortened URLs ( #4316 )
1 year ago
90231f2d5a
[twitter] add 'tweet-endpoint' option ( #4307 )
...
use the newer TweetResultByRestId only for guests by default
1 year ago
20ed647f6f
[twitter] add 'user' extractor and 'include' option ( #4275 )
1 year ago
86be197d11
[twitter] remove '/search/adaptive.json'
1 year ago
d52ed2bc5a
[zerochan] fix 'tags' extraction
1 year ago
12cd85658b
[redgifs] add 'niches' extractor
1 year ago
248e8bc699
release version 1.25.8
1 year ago
bc9123cfee
[naverwebtoon] fix 'comic' metadata extraction
1 year ago
ab5dde7221
[mangaread] fix 'tags' extraction
1 year ago
c9a82c9313
[erome] ignore duplicate album IDs
1 year ago
c84397023a
[slideshare] fix extraction
1 year ago
ffbbbd3baf
[gelbooru_v01] 'vidyart' -> 'vidyart2'
1 year ago
e40b90e137
merge #4303 : [gelbooru_v01] fix 'source' ( #4302 )
1 year ago
c6b31a2169
[reddit] set default 0.6s delay between requests ( #4292 )
...
to limit API requests to 100 per minute
https://www.reddit.com/r/redditdev/comments/14nbw6g/
1 year ago
20da41018d
[pornhub] set 'accessAgeDisclaimerPH' cookie ( #4301 )
1 year ago
75757c4ace
[gelbooru_v01] fix 'source' ( #4302 )
1 year ago
2dd6942d1c
[jpgfish] update domain to 'jpeg.pet'
1 year ago
1137b89ed4
[lineblog] remove module
...
"LINE BLOGは2023年6月29日をもちましてサービスを終了いたしました"
1 year ago
86560fe0cd
[bcy] remove module
...
"The website was shut down on July 12, 2023"
https://danbooru.donmai.us/wiki_pages/bcy
1 year ago
fceabee433
[philomena] use API interface class
...
handle 429 errors and retry after 10min (#4288 )
1 year ago
f079d9a703
[reddit] notify users about registering an oauth application
...
(#4292 , #4253 , #3943 )
1 year ago
fb3d1462b1
merge #4291 : [wikifeet] fix 'tag' extraction
1 year ago
0b08e2e8a8
merge #4287 : [twitter] Fix following extractor not getting all users
1 year ago
f6553ffd2f
[twitter] simplify '_pagination_users'
...
- remove 'stop' variable
- call 'cursor.startswith()' only once
1 year ago
1590124aae
[twibooru] fix '--range'
1 year ago
a2111dd025
[wikifeet] fix 'tag' extraction
1 year ago
a1ffa1ff09
[philomena] fix '--range' ( #4288 )
1 year ago
a27dbe8c82
[twitter] use 'TweetResultByRestId' endpoint ( #4250 )
...
allows accessing single Tweets without login
1 year ago
d3d639a159
[twitter] don't treat missing 'TimelineAddEntries' as fatal ( #4278 )
1 year ago
c321c773f2
make the code less ugly
1 year ago
a437a34bcf
fix lint i guess?
1 year ago
6cbc434b54
Fix users pagination
1 year ago
d5b6802774
[seiga] set 'skip_fetish_warning' cookie ( #4242 )
1 year ago
88d1e29401
[bunkr] use '.la' TLD for 'media-files12' servers ( #4147 , #4276 )
1 year ago
f0cb951566
[paheal] unescape 'source'
1 year ago
b480b7076a
[paheal] fix a78f8ce5 for enabled 'metadata' ( #4262 )
1 year ago
384337d3dd
[fantia] send 'X-Requested-With' header only for API requests ( #4273 )
1 year ago
c2ac665ff7
[fantia] send 'X-Requested-With' header ( #4273 )
1 year ago
7444fc125b
[gfycat] implement login support ( #3770 , #4271 )
...
For the record: '/webtoken' and '/weblogin' are not the same ...
1 year ago
e9b9f751bf
[gfycat] support '@me' user ( #3770 , #4271 )
1 year ago
5b59a0d143
update default User-Agent header to Firefox 115 ESR
1 year ago
0556e1ad45
merge #4268 : [newgrounds] extract & pass auth token for login
1 year ago
a16d7c59cb
[newgrounds] access 'response.text' only once
1 year ago
1bf9f52c99
[twitter] add 'ratelimit' option ( #4251 )
1 year ago
f86fdf64a6
[twitter] use GraphQL search by default ( #4264 )
1 year ago
1d4db83d49
[weibo] fix end of cursor based pagination
1 year ago
a78f8ce5b0
[paheal] fix extraction ( #4262 )
...
swap ' and "
1 year ago
9576652fa5
extract & pass auth token for newgrounds
1 year ago
5457007dd3
release version 1.25.7
1 year ago
3d8de383bf
[mangapark] extract 'source_id' for manga
...
forgot to add this to 6ae3101f
1 year ago
6ae3101fd0
[mangapark] add 'source' option ( #3969 )
1 year ago
c45a913bfd
[flickr] add 'exif' option
1 year ago
3845c0256d
[sankaku] improve warnings for unavailable posts
1 year ago
46cae04aa3
[piczel] update API server ( #4244 )
1 year ago
3479646f65
[mangapark] update and fix 'manga' extractor ( #3969 )
...
TODO:
- non-English chapters
- 'source' option
1 year ago
10786c657e
[mangapark] update and fix 'chapter' extractor ( #3969 )
1 year ago
9c31c2daef
[poipiku] improve error detection ( #4206 )
1 year ago
260ff55e19
[senmanga] ensure download URLs have a scheme ( #4235 )
1 year ago
ccbc1a1d55
[flickr] add 'metadata' option ( #4227 )
1 year ago
c1cce4a80b
[twitter] extend 'conversations' option ( #4211 )
1 year ago
b6c959744d
[furaffinity] improve 'description' HTML ( #4224 )
...
- ignore header
- include footer and closing <div> if present
1 year ago
8357acf359
[gelbooru_v01] replace 'extract_all()' with 'extract_from()'
...
It's even slightly faster, especially on Python before 3.11
1 year ago
068aa26c3e
[gelbooru_v01] fix '--range' ( #4167 )
1 year ago
2052e7ce59
[hentaifox] fix titles containing '@' ( #4201 )
1 year ago
92d98697b2
[wallhaven] update API error message
1 year ago
a673998b1e
release version 1.25.6
1 year ago
339fcdb8ad
[wallhaven] handle '429 Too Many Requests' errors ( #4192 )
...
- set 1.4s delay between API requests
(WH allows 45 requests per minute)
- wait and retry on 429 errors
1 year ago
ef9891ec9d
[fantia] extract 'plan' metadata ( #2477 , #4128 )
1 year ago
f8452984fa
[fantia] emit warning for non-visible contents ( #4128 )
1 year ago
dc7af00014
[fantia] refactor
...
- embed response data as hidden '_data' field
(instead of returning/passing 'resp')
- split _get_urls_from_post()
1 year ago
6c8bf9a762
[pornhub] improve redirect handling ( #4188 )
1 year ago
654267a335
[weibo] fix 'json' extension for some videos
1 year ago
ce93c460a6
[formatter] implement 'H' conversion ( #4164 )
...
to remove HTML tags and unescape HTML entities
1 year ago
deff3b434d
[vipergirls] implement login support ( #4166 )
1 year ago
db20a645c5
[vipergirls] use API endpoints ( #4166 )
1 year ago
0b34a444e0
[pixiv:novel] only detect Pixiv embeds ( #4175 )
1 year ago
9f1aee3884
[vipergirls] limit number of requests per second ( #4166 )
1 year ago
21c75d03a3
merge #4133 : [furaffinity] extract 'favorite_id' metadata
1 year ago
5e3a1749c8
[furaffinity] simplify 'favorite_id' assignment
1 year ago
ad882291d3
[instagram] fix retrieving '/tagged' posts ( #4122 )
...
reduce number of retrieved posts per API request from 50 to 20
1 year ago
0a9aaa7a8d
[weibo] prevent fatal exception due to missing video ( #4150 )
1 year ago
ac651c604c
[senmanga] fix and update ( #4160 )
1 year ago
df106fb58b
[bunkr] fix video downloads
1 year ago
aad5e6490c
merge #4159 : [bunkr] update domain to bunkrr.su
1 year ago
e0522ffb3d
[bunkr] update
1 year ago
e04796e04b
merge #3447 : [jschan] add generic extractors for jschan imageboards
1 year ago
b9692341fe
[jschan] update
1 year ago
a7c066cbac
Update bunkr.py
1 year ago
72e697b8b5
Update bunkr.py
...
Support bunkrr.su
1 year ago
4ae925c88f
[kemonoparty] support '.su' TLD ( #4139 )
1 year ago
2d9e3093ca
merge #4134 : [postimage] add gallery support, update image extractor
1 year ago
e64b521287
merge #4136 : [acidimg] fix extractor
1 year ago
a90974178d
[jpgfish] update domain to 'jpg.pet' ( #4138 )
1 year ago
ee959052ac
merge #4138 : add jpg.pet as alias for jpgfish
1 year ago
0281cc7d08
[fanbox] skip 404ed fanbox embeds ( #4088 )
...
continuation of 4fc9675d
1 year ago
97c0d13cbb
add jpg.pet as alias for jpgfish
1 year ago
2e309a13a7
[acidimg] fix extractor
1 year ago
92178b369c
[postimage] add gallery support, update image extractor to download
...
original image instead of main image
1 year ago
952c03bc9e
Add fav_id data to FuraffinityFavoriteExtractor
...
An extra field is collected when paginating favorites, and saved to
a temporary cache variable. This field is identical for both the old
and the new page layouts for FurAffinity, but can only be collected
during pagination, hence the cache variable. Other FurAffinity
extractors should be unaffected by this change.
1 year ago
54cf1fa3e7
[twitter] use GraphQL search endpoint ( #3942 )
...
for guest users; selectable with 'search-endpoint' option.
adapted from 9c7b888ffa
1 year ago
864a654b25
[twitter] update query hashes
1 year ago
45cc7cee1a
[twitter] better error message for guest searches ( #3942 )
1 year ago
271f23d971
[twitter] extract 'conversation_id' metadata ( #3839 )
1 year ago
94b6a67666
[reddit] fix crash with empty 'crosspost_parent_lists' ( #4120 )
1 year ago
0cf7282fa0
[pixiv] add 'full-series' option for novels ( #4111 )
1 year ago
bab13402df
[redgifs] update 'search' URL pattern ( #4115 )
1 year ago
5a6fd8027d
[redgifs] support galleries ( #4021 )
1 year ago
0ad59c92b1
[blogger] download files from 'lh*.googleusercontent.com' (4070)
1 year ago
ffed7efb6f
[pixiv] use BASE_PATTERN
1 year ago
b286efefcc
[pixiv] add 'novel-bookmark' extractor ( #4111 )
1 year ago
5283db1aae
release version 1.25.5
1 year ago
28f6487c64
[instagram] add 'metadata' option ( #3107 )
1 year ago
8cf13f8696
merge #4104 : [lensdump] add lensdump.com extractors
1 year ago
58f7480d46
[lensdump] update
...
- update docs/supportedsites.md
- add GPL2 header
- use BASE_PATTERN
- improve LensdumpImageExtractor
1 year ago
3516fdae74
[kemonoparty] fix kemono and coomer logins using the same cache
...
(#4098 )
1 year ago
d5300cf381
[lensdump] subcategory
1 year ago
82ba6bfdc0
[lensdump] f-string fix
1 year ago
9b2326e4e1
[lensdump] add lensdump.com extractor
1 year ago
a5d0b03bde
[ytdl] fix crash due to removed 'no_color' attribute
...
8417f26b8a
1 year ago
148bdc04a4
merge #2719 : [jpgfish] add 'jpgfish' extractors
1 year ago
609c4f3e07
[jpgfish] simplify and improve
1 year ago
2b1f875ef4
[jpgchurch] update to 'jpgfish'
1 year ago
3d29c42142
[mangaread] fix 'tags' extraction
1 year ago
5f86527cbe
merge #2781 : [mangaread] Add Mangaread extractor
1 year ago
cdc6549fd2
merge #3329 : [8muses] Add 'parts' to album data
...
and fix 'album[url]'
1 year ago
ad760429b1
[8muses] update
1 year ago
d0184fddcf
[twitter] optimize '_extract_twitpic()'
...
- use findall instead of finditer
- store URLs in a dict to discard duplicates
1 year ago
3dc862c7fc
merge #3796 : [twitter] extract TwitPic URLs in text ( #3792 )
1 year ago
243de697b9
merge #3976 : [reddit] support cross-posted media ( #887 , #3586 )
1 year ago
f8c4c5eef9
[reddit] simplify and add tests
1 year ago
822a77d846
[danbooru] add support for booru.borvar.art instance
1 year ago
f3cca50b9e
[mangadex] update links to API docs
1 year ago
65a9f4b124
merge #3950 : [misskey] add 'favorite' extractor
1 year ago
c76f0f3a1b
[misskey] update
...
- rename to 'MisskeyFavoriteExtractor'
- add 'access-token' option to docs
- add test URLs for other instances
- simplify 'pattern'
1 year ago
3fca455b82
[pixiv] add 'embeds' option ( #1241 )
1 year ago
d1f2ef3b7b
[imagechest] update
...
- don't load HTML page when using API
- restructure some code
- add more methods to ImagechestAPI
1 year ago
856f6c10cd
allow for GalleryExtractors to skip loading gallery_url
1 year ago
4fc9675d48
[fanbox] skip 404ed or otherwise invalid posts ( #4088 )
1 year ago
69865dcc05
[formatter] implement slicing strings as bytes ( #4087 )
...
prefixing a slice '[10:30]' with a lowercase b '[b10:30]' encodes
the string to bytes in filesystem encoding before applying the slice
1 year ago
56b8b8cd36
[pixiv] support short novel URLs
...
https://www.pixiv.net/n/ <ID>
1 year ago
e6f55d1555
[imagechest] add API support and 'access-token' option ( #4065 )
1 year ago
77abcf5ab3
[gofile] automatically fetch 'website-token' by default
...
the hardcoded token changed yet again
1 year ago
e3fed9bd17
[tcbscans] update domain to 'tcbscans.com' ( #4080 )
1 year ago
a83983c651
[instagram] add 'order-posts' option ( #4017 , #3993 )
1 year ago
d680623db3
[instagram] add 'order-files' option ( #4017 , #3993 )
1 year ago
f9b7a033e0
[misskey] refactor misskey extractor
1 year ago
04dbfd994e
[misskey] add my favorites extractor
1 year ago
82a12d6126
[nsfwalbum] detect placeholder images
...
patch by an anonymous contributor
1 year ago
011e4607c3
[poipiku] extract full 'descriptions' ( #4066 )
...
don't cut it off after the first line
1 year ago
5037013e2b
[gofile] update 'website-token' ( #4056 )
1 year ago
6b6bb4be73
[weibo] require numeric IDs to have length >= 10 ( #4059 )
1 year ago
494acabd38
[danbooru] refactor pagination logic ( #4002 )
...
- only use 'b<ID>' when no other order is specified
- support 'a<ID>' when using 'order:id' as tag
1 year ago
fd0e1ffd6e
[danbooru] improve 75666cf9 ( #4002 )
...
Search for direct post IDs instead of trying to
replicate the same results as the initial request.
1 year ago
e41e45ff6b
[gofile] add basic password support ( #4056 )
1 year ago
790dd365e1
[postprocessor:exec] support tilde expansion for 'command'
...
https://github.com/mikf/gallery-dl/issues/146#issuecomment-1544733532
1 year ago
2e6cea95db
[cookies] update logging behavior ( #4050 )
...
- only show the same warning/error once
- simplify and capitalize logging messages
1 year ago
20dc13f832
[pixiv] initial 'novel' support ( #1241 , #4044 )
...
supported URLs are
- https://www.pixiv.net/novel/show.php?id= <ID>
- https://www.pixiv.net/novel/series/ <ID>
- https://www.pixiv.net/en/users/ <ID>/novels
1 year ago
c698c3de44
[newgrounds] add default delay between requests ( #4046 )
1 year ago
708f478d15
[danbooru][e621] add 'date' metadata field ( #4047 )
1 year ago
306e13a4d4
release version 1.25.4
1 year ago
35c23a2fd8
merge #4031 : [mangadex] add 'status' and 'tags' metadata
1 year ago
2266fc8cc5
[mangadex] update and extend test results
1 year ago
1ce5dc9e18
fix whitespaces
1 year ago
13dedae09f
add status and tags info to mangadex extractor
1 year ago
be0fa94b2e
[imagechest] load all images when a 'Load More' button is present
...
(#4028 )
1 year ago
7eadcbea70
[4chanarchives] add end condition for 'board' extractor ( #4012 )
1 year ago
1406f7125f
[4chanarchives] add 'thread' and 'board' extractors ( #4012 )
1 year ago
285391df43
add '-C' as short option for '--cookies'
...
and put cookie options into their own section
1 year ago
b9b1cdd71b
add '--cookies-export' command-line option
1 year ago
d12dd3813c
[imgur] fix internal image/album URLs
...
URLs from "link" attributes of newer images/albums were all returned
as 'https://imgur.com/gallery/ ...' instead of the expected format,
causing them to be ignored.
1 year ago
8520de57f0
[imgur] add 'favorite-folder' extractor ( #4016 )
1 year ago
4c1f3b2160
[cookies] simplify '_mac_absolute_time_to_posix()'
...
hardcode UNIX timestamp of 2001-01-01
1 year ago
a14b63d941
support selecting a domain for '--cookies-from-browser'
...
for example 'gallery-dl --cookies-from-browser firefox/twitter.com'
1 year ago
3ca5dac8b6
extend 'cookies-update' functionality
...
Allow writing cookies to a different file than a given cookies.txt,
making it possible to export cookies imported with --cookies-from-browser
To convert browser cookies to cookies.txt format:
gallery-dl --cookies-fr chromium \
-o cookies-update=cookies.txt \
--no-download \
http://example.org/file.jpg
1 year ago
bc6d65d203
implement 'Extractor.config_deprecated()'
...
a version of 'Extractor.config()'
that logs a warning when using a deprecated option name
1 year ago
850df34c31
remove '&' from URL patterns part 2
...
follow-up on 968d3e8465
1 year ago
4d415376d1
[pinterest] fix 'pin.it' extractor
...
it really was just the single '/' at the end of the url_shortener URL
1 year ago
657b6a9100
[pinterest] update endpoint for related board pins
1 year ago
79f47f98dd
[nana] remove module
...
permanently gone since 2023-03-13
1 year ago
0e74df1de8
[420chan] remove module
...
offline since 2022-06-01
1 year ago
7499fa7075
[exhentai] remove and update sad panda check
...
there hasn't been a sad panda in several years
1 year ago
076380e079
remove '*' indicating keyword-only arguments
...
they are kind of unnecessary and
cause a non-insignificant function call overhead (~10%)
1 year ago
0c46758a93
[foolslide] remove 'sensescans.com'
...
group moved to mangadex
https://mangadex.org/group/1071e71d-cc55-4fa6-81d1-4b5913a2fde5/sense-scans
1 year ago
a08fdfac6e
[foolfuuka] add 'archive.palanq.win'
1 year ago
1870df8b23
[foolfuuka] remove 'tokyochronos.net'
1 year ago
ef4e2d8178
[foolfuuka] remove 'archive.alice.al'
1 year ago
57cf942bb1
[config] include exception type in error message
1 year ago
aa731c4298
[ytdl] run yt-dlp tests with latest code from master ( #3989 )
...
Only use PyPI version for Python 3.6, since that's no longer supported
by the current codebase.
1 year ago
6a860876bc
release version 1.25.3
1 year ago
b12dad8df5
[pixiv] fix 'pixivision' extraction
1 year ago
5fb7107f2b
[imxto] fix 'gallery' extraction
...
support both single and double quotes
1 year ago
15d7c5a199
[behance] 'items()' -> 'values()'
...
we only need 'size', 'name' is unnecessary
1 year ago
61a65d5bb9
[ytdl] fix crash due to --geo-bypass deprecation ( #3975 )
1 year ago
0fb580135d
[behance] fix extraction ( #3980 )
1 year ago
d4f8b2fe22
fix: linter issues
1 year ago
1b918bd937
fix(extractor): fix extraction for cross-posted reddit videos and galleries
1 year ago
215028a462
[manganelo] match more minor version separators ( #3972 )
1 year ago
c182094ebf
merge #3748 : [downloader:http] add 'consume-content' option
1 year ago
9f76783ac0
[pixiv] allow sorting by popularity (requires pixiv premium)
1 year ago
7865067d19
[shimmie2] add generic extractors for Shimmie2 sites ( #3734 )
...
add support for
- loudbooru.com (#3734 )
- booru.cavemanon.xyz (#3734 )
- giantessbooru.com (#943 )
- tentaclerape.net
1 year ago
28419bf45a
[itchio] add 'game' extractor ( #3923 )
1 year ago
3905f05f00
[postprocessor:metadata] support putting keys in quotes
...
for mode 'modify' and 'delete'
based on fe41a2b1
1 year ago
7459e4abce
[postprocessor:metadata] fix traversing more than 1 level deep
...
for mode 'modify' and 'delete'
1 year ago
5297ee0cd9
[tumblr] add 'day' extractor ( #3951 )
1 year ago
de670bd7de
[tumblr] update pagination logic ( #2191 )
1 year ago
6f4a843fba
[downloader:http] release connection before logging messages
...
This allows connections to be properly released when using 'actions'
feature.
1 year ago
98c9fdb414
[deviantart] revert e9353c63; retry downloads with private token
1 year ago
5d7435e803
[nitter] extract user IDs from encoded banner URLs
...
still requires a banner to be present to begin with
1 year ago
7f25cab56e
[sankaku] support post URLs with MD5 hashes ( #3952 )
1 year ago
a05120412a
[oauth] catch exception from 'webbrowser.get()' ( #3947 )
...
It raises an exception instead of returning None
when no runnable browser is available.
1 year ago
3fc2223893
merge #3935 : [reddit] match 'preview.redd.it' URLs
1 year ago
1d505b39f8
[twitter] support 'profile-conversation' entries ( #3938 )
1 year ago
aaf58a1259
[imgur] document 'client-id' option ( #3937 )
1 year ago
202f5d86a7
[reddit] ignore 'id-max' value "zik0zj"/2147483647
...
(#3939 , #3862 , #3697 , #3606 , #3546 , #3521 , #3412 )
1 year ago
8586ee81be
[nana] fix 'keyword' tests
1 year ago
cd4bfb0dd1
[reddit] match 'preview.redd.it' URLs
1 year ago
faca32a850
[sankaku] sanitize 'date:…' tags ( #1790 )
1 year ago
6f1e34ec69
[vipergirls] add 'thread' and 'post' extractors
...
(#731 , #2720 , #3812 )
1 year ago
81bd2af83e
[2chen] update domain to sturdychan.help
1 year ago
f500b45b5e
[twitter] improve 480bc34e
...
only check for double user assignment where necessary
1 year ago
5b635f2317
[imxto] add 'gallery' extractor ( #1289 )
1 year ago
359e31e462
[nozomi] update file URLs ( #3925 )
...
Static images are now only available in WebP format over the 'w'
subdomain. GIFs also got their own 'g' subdomain.
1 year ago
2dfd4a3de2
[imagefap] extract 'categories' metadata and fix empty 'tags'
1 year ago
480bc34e54
[twitter] do not overwrite previously assigned users ( #3922 )
1 year ago
02ec5bb8e5
[imagefap] extract 'description' metadata ( #3905 )
1 year ago
842f964c49
release version 1.25.2
1 year ago
d253a3c542
merge #3841 : [urlshortener] add support for bit.ly & t.co
1 year ago
5e63942b37
[urlshortener] update
1 year ago
2edcdee32f
[downloader:http] add MIME type and signature for .heic files
...
(#3915 )
https://github.com/strukturag/libheif/issues/83
1 year ago
c45f09d2a8
[imagechest] fix extraction ( #3914 )
1 year ago
2cd4411ff8
[nitter] extract videos from 'source' elements ( #3912 )
1 year ago
9501579279
[sexcom] fix fetching HD videos
1 year ago
a2f7274eae
[sexcom] fix pagination ( #3906 )
1 year ago
e9353c63d6
[deviantart] keep using private access tokens
...
for deviations returned from a private API call
also fixes a bug from 0a7eee3e
1 year ago
e70af6a550
[hentaifoundry] do not update filters when cookies are provided
1 year ago
9c29c904c7
[mastodon] try to get account IDs without access token
...
Try to query the public '/api/v1/accounts/lookup' endpoint
and fall back to '/v1/accounts/search' if it returns an error.
'/api/v1/accounts/lookup' is available since Mastodon v3.4.0.
The version of an instance can be found at '/api/v1/instance'.
1 year ago
Mike Fährmann
