Mike Fährmann
ade8347ead
[kemonoparty] fix DM dates
11 months ago
Mike Fährmann
6dfe200ae4
[kemonoparty] support discord URLs with channel IDs ( #4662 )
11 months ago
Mike Fährmann
c6a3892210
[imgbb] update username extraction ( #4626 )
11 months ago
Mike Fährmann
830a48bca4
[fantia] bad workaround for 833dce14
( #4627 )
...
at least this makes "filter": "content_num == content_count+1"
with "event": "post-after" work
11 months ago
Mike Fährmann
13ce3a9acb
[warosu] fix extraction ( #4634 )
11 months ago
Mike Fährmann
c4c4e4d2f4
[newgrounds] improve 'art-image' extraction ( #4642 )
...
- download files in original resolution
- replace .webp with extension of first file
11 months ago
Mike Fährmann
833dce141f
[fantia] add 'content_count' and 'content_num' metadata fields ( #4627 )
11 months ago
Mike Fährmann
2d41702762
[deviantart] implement '"group": "skip"' ( #4630 )
11 months ago
Mike Fährmann
992e86ec94
[deviantart] disable 'jwt' ( #4652 )
11 months ago
Mike Fährmann
2974b8e3c8
[moebooru] add 'metadata' option ( #4646 )
...
for extended 'pool' metadata
11 months ago
Mike Fährmann
d194ea68a9
[cookies] open cookie databases in read-only mode
...
bypasses the need to copy the entire database file
might solve #4195
11 months ago
Mike Fährmann
8bb7243c10
[reddit] fix wrong previews ( #4649 )
...
caused by a failed comment URL
using the main submission's preview as fallback
14af15bd
4963bb9b
12 months ago
Mike Fährmann
08bdde5aac
merge #4619 : [twitter] add 'sensitive' metadata field
12 months ago
Mike Fährmann
f3d6aaff13
[twitter] rename to 'sensitive'; use 'tget()'
12 months ago
Mike Fährmann
95c280c59b
[imgbb] update pagination end condition ( #4626 )
12 months ago
Mike Fährmann
2e350dd82a
merge #4626 : [imgbb] fix 'user' extraction, add 'displayname'
12 months ago
Mike Fährmann
a2daa9befe
[imgbb] fix flake8 and username order
12 months ago
Mike Fährmann
67ba4ee842
[pp:exec] support more replacement fields for '--exec' ( #4633 )
...
- {_directory}
- {_filename}
- {_path} (alias for {})
12 months ago
Mike Fährmann
9a008523ac
[hentaifoundry] fix '.swf' file downloads ( #4641 )
12 months ago
Mike Fährmann
15f940819b
[newgrounds] support 'art-image' files ( #4642 )
12 months ago
Mike Fährmann
63db54b905
[patreon] update 'campaign_id' path ( #4639 )
12 months ago
HRXN
b846f56c3a
[imgbb] Fix `user` extraction, add `displayname`
12 months ago
Mike Fährmann
efaab4fbfa
[twitter] fix crash due to missing 'source' ( #4620 )
...
regression caused by 06aaedde
12 months ago
Nahida
3438a3098d
[twitter] add possible_sensitive field
12 months ago
Mike Fährmann
85357c1ef8
release version 1.26.0
12 months ago
Mike Fährmann
64dbc58a5a
[deviantart] update Eclipse API endpoints 2 ( #4615 )
12 months ago
Mike Fährmann
84fbbd96aa
[shimmie2] remove 'meme.museum'
12 months ago
Mike Fährmann
aa77fda78c
[instagram] better error message for invalid users ( #4606 )
12 months ago
Mike Fährmann
482f002e1f
[nsfwalbum] detect '/error.jpg' images ( #4598 )
12 months ago
Mike Fährmann
eb230e4b77
[nsfwalbum] disable Referer headers by default ( #4598 )
12 months ago
Mike Fährmann
b92645cd37
[bunkr] fix extraction ( #4514 , #4532 , #4529 , #4540 )
12 months ago
Mike Fährmann
4477808d1c
fix symlink resolution in __main__.py
...
adapt ytdl order
12 months ago
Mike Fährmann
be17103e21
[regifs] support 'order' parameter for user URLs ( #4583 )
12 months ago
Mike Fährmann
7150c4c76c
fix imports when using the gallery_dl directory as argument ( #4581 )
12 months ago
HRXN
ec91eeb7ef
Update gallery_dl/extractor/reddit.py
...
Co-authored-by: Mike Fährmann <mike_faehrmann@web.de>
12 months ago
HRXN
66613c3a32
[reddit] ignore '/message/compose' URLs without www subdomain
1 year ago
Mike Fährmann
bb39779e1a
[deviantart] use private tokens for 'is_mature' posts ( #4563 )
1 year ago
Mike Fährmann
0c5d8b1505
[deviantart] re-add 'quality' option and 'intermediary' transform
1 year ago
Mike Fährmann
20d1683c47
[deviantart] fix JWT replacement ( #293 , #4548 , #4563 )
...
And again, a huge thank you to @Ironchest337
for discovering this.
1 year ago
Mike Fährmann
d7aac9fc06
[reddit] ignore '/message/compose' URLs ( #4482 )
1 year ago
Mike Fährmann
1e31fce37b
[pillowfort] support '/tagged/' URLs ( #4570 )
1 year ago
Mike Fährmann
1d2fd0b831
[pillowfort] extract 'b2_lg_url' media ( #4570 )
1 year ago
Mike Fährmann
50e2ebaff0
[danbooru] support 'donmai.moe' URLs
1 year ago
Mike Fährmann
82296b1f05
[reddit] add 'previews' option ( #4322 )
...
another way to disable this new behavior
1 year ago
Mike Fährmann
918ba4f847
[redgifs] match gfycat image URLs ( #4558 )
1 year ago
Mike Fährmann
2ad75bab05
[deviantart] add 'is_original' metadata field ( #4559 )
...
true for 'downloadable' content, journals, flash animations,
and images without '/v1/' in their URL; false otherwise
1 year ago
Mike Fährmann
9d8317d963
[deviantart] disable JWT updates ( #4548 , #4563 )
...
back to lowres images ...
1 year ago
Mike Fährmann
8064663bda
[deviantart] update Eclipse API endpoints ( #4553 )
1 year ago
Mike Fährmann
2cd801232b
fix --range causing crashes ( #4557 )
...
regression caused by a383eca7
1 year ago
Mike Fährmann
3528974459
[instagram] handle exceptions due to missing media ( #4555 )
1 year ago
Mike Fährmann
4963bb9b30
[reddit] improve comment metadata v2 ( #4482 )
...
provide main submission metadata at the top level
and comment metadata inside the 'comment' field,
i.e. the other way round than in 1710f1e9
1 year ago
Mike Fährmann
7592c5e566
[patreon] fix extraction ( #4547 )
1 year ago
Mike Fährmann
0655ce1bae
[mangakakalot] update domain
...
the old one still works, but it incurs a redirect
1 year ago
Mike Fährmann
3ecb512722
send Referer headers by default
1 year ago
Mike Fährmann
cb4798f07a
[architizer] fix extraction ( #4537 )
1 year ago
Mike Fährmann
6178177227
[twitter] fix '_extractor' of following results ( #4536 )
...
regression from 20ed647f
1 year ago
Mike Fährmann
d13c82eff1
[kemonoparty] update favorites API endpoint ( #4522 )
1 year ago
Mike Fährmann
27ec653991
fix bug in test_init and update example URLs
1 year ago
Mike Fährmann
24a1d46391
[mastodon] support '/@USER/following' URLs
...
Previously, only '/users/USER/following' got matched.
1 year ago
Mike Fährmann
9f75713e00
[recursive] simplify
1 year ago
Mike Fährmann
899df8f237
remove another '*' for keyword-only arguments
...
076380e0
1 year ago
Mike Fährmann
6ae92da57e
Merge branch 'tests'
1 year ago
Mike Fährmann
32da3c70d3
[behance] handle videos without 'renditions' ( #4523 )
1 year ago
Mike Fährmann
ae5e049c4f
[redgifs] provide 'collection' metadata in a separate field ( #4508 )
...
instead of overwriting the actual metadata
1 year ago
Mike Fährmann
1710f1e983
[reddit] improve comment metadata ( #4482 )
...
- provide 'date'
- make metadata of the main submission available as 'submission[…]'
1 year ago
Mike Fährmann
4cdab8074e
update/fix --list-extractors
1 year ago
Mike Fährmann
a453335a9f
remove test results in extractor modules
...
and add generic example URLs
1 year ago
Mike Fährmann
1d2b5d0c60
update test comment positions
...
always put them above the test they're referring to
1 year ago
Mike Fährmann
93a7a89cf6
[formatter] use value of last alternative ( #4492 )
...
fixes {fieldname|''} evaluating to the value of 'keywords-default'
instead of an empty string
1 year ago
Mike Fährmann
f856987297
[subscribestar] fix preview detection ( #4468 )
...
and show a warning message when posts contain previews
1 year ago
Mike Fährmann
4c0b3d5dc5
[twitter] fix crash when 'sortIndex' is None ( #4499 )
1 year ago
Mike Fährmann
f2de70f254
[gfycat] remove module
1 year ago
Mike Fährmann
6eca1fab9b
[gelbooru_v02] support 'xbooru.com' ( #4493 )
1 year ago
Mike Fährmann
23bac772f2
[jpgfish] update domain to 'jpg1.su' ( #4494 )
1 year ago
Mike Fährmann
ceb59e176f
fix default Firefox user agent string
...
note to self: do not trust some random third-party website
1 year ago
Mike Fährmann
8259a5abe4
flake8
1 year ago
Mike Fährmann
0b6e5b8161
[hiperdex] send Referer headers during file downloads ( #4490 )
1 year ago
Mike Fährmann
a05821f8b4
[hiperdex] fix 'manga' metadata
...
remove trailing ' Manga'
1 year ago
Mike Fährmann
03d471a0d4
merge #4481 : [pixiv] handle errors for private novels
1 year ago
Cisney-Gassai
8c477f7146
[bunkr] Fixes media-files-pizza.bunkr.ru failed to resolve.
1 year ago
johnsmith1202gmail
c7e31b2724
Update pixiv.py
1 year ago
johnsmith1202gmail
d3046561d4
continue downloading when the item is made private on pixiv
1 year ago
Mike Fährmann
28798594e8
[gfycat] update pagination logic ( #4479 )
...
Some searches do not use cursor based pagination
but an offset based one.
1 year ago
Mike Fährmann
a783c4f0fe
[pornhub] add 'gif' support ( #4463 )
1 year ago
Mike Fährmann
ba842981af
[imagevenue] fix extraction ( #4473 )
1 year ago
Mike Fährmann
7defb24e1e
[reddit] provide video previews if available ( #4322 )
1 year ago
Mike Fährmann
fd65f27ede
[reddit] fix 'preview.redd.it' URLs ( #4470 )
1 year ago
Mike Fährmann
06aaedded5
[twitter] extract 'source' metadata ( #4459 )
1 year ago
Mike Fährmann
14af15bd18
[reddit] download preview for 404ed imgur links ( #4322 )
...
This is a pretty ugly hack as the internal infrastructure doesn't
really support switching from external URL to regular download in
case the former fails, but it kind of works ...
Can be disabled by setting 'reddit.fallback' to 'false'.
1 year ago
Mike Fährmann
d12a5e440a
update docs/supportedsites
1 year ago
Mike Fährmann
3a27150479
[instagram] add 'following' extractor ( #1848 )
1 year ago
Mike Fährmann
e0829ff0fd
[twitter] add 'date_original' metadata for retweets ( #4337 , #4443 )
1 year ago
Mike Fährmann
5ed245317d
[exhentai] add 'fav' option ( #4409 )
...
The name 'favorite' is already taken as extractor subcategory
1 year ago
Mike Fährmann
fd6b413f3c
[exhentai] fix 'domain' option ( #4458 )
...
regression from a383eca7
1 year ago
Mike Fährmann
fdfb22c91f
[instagram] fix video preview archive IDs ( #2135 , #4455 )
1 year ago
Mike Fährmann
92f98e6f5e
'sys.exit' -> 'SystemExit'
1 year ago
Mike Fährmann
410f783a33
implement 'subconfigs' option ( #4440 )
1 year ago
Mike Fährmann
2b88ad19e9
[twitter] accept 'x.com' URLs ( #4452 )
1 year ago
Mike Fährmann
c1c73c0b0e
[pp:ugoira] add '"framerate": "uniform"' ( #4421 )
1 year ago
Mike Fährmann
2a3acd318a
[pp:ugoira] fix high frame rates ( #4421 )
...
only return an output frame rate for non-uniform ugoira
when the frame delay gcd is >= 10, i.e. 100 fps
1 year ago
Mike Fährmann
70bdf32a88
[pp:ugoira] extend 'ffmpeg-output' ( #4421 )
...
- when setting this option to a string value,
pass -hide-banner and -loglevel to FFmpeg
- change default to "error"
1 year ago
Mike Fährmann
8dceea3384
[shimme2] move 'giantessbooru' back into shimmie module ( #4373 )
...
Do the same thing as for 'realbooru' and override 'posts()'
insteadd of using a separate module.
1 year ago
Mike Fährmann
6482f9453b
[behance] fix cookie usage ( #4417 )
1 year ago
Mike Fährmann
d34195b41d
[behance] fix and update 'user' extractor ( #4417 )
1 year ago
Mike Fährmann
4d3cf709da
[behance] add 'date' metadata field ( #4417 )
1 year ago
Mike Fährmann
c689cd9720
[behance] show error for mature content ( #4417 )
1 year ago
Mike Fährmann
33d912490f
merge #4419 : [bunkr] Fix extracting wmv files
1 year ago
Mike Fährmann
01610a6e9e
merge #4412 : [bunkr] fix media domain for cdn9
1 year ago
Mike Fährmann
b19d62263b
merge #4420 : [issuu] fix extraction
1 year ago
ClosedPort22
6dc8be5e48
[issuu] fix extraction
1 year ago
Luc Ritchie
85a070b9e6
[bunkr] Fix extracting wmv files
1 year ago
Mike Fährmann
3f8ff692a7
[bunkr] fix media domain for cdn9
...
Fixes #4386
1 year ago
Mike Fährmann
d8b21a97bf
[formatter] use 'rpartition' for \fM format strings
...
fixes using absolute module paths like C:\path\module.py on Windows
1 year ago
Mike Fährmann
f9fb276e81
[postprocessor] add 'prepare-after' event ( #4083 )
1 year ago
Mike Fährmann
0ef1fcab20
[postprocessor] update 'finalize' events
...
Add 'finalize-error' and 'finalize-success' events that trigger
depending on whether error(s) did or did not happen.
'finalize' itself now always triggers regardless of error status.
(was supposed to have the same behavior as the new 'finalize-success')
1 year ago
Mike Fährmann
af4bdb62a7
merge #4403 : [downloader:http] close connection when file already exists
1 year ago
Mike Fährmann
15275b3524
[postprocessor:ugoira] restore 'libx264-prevent-odd' ( #4407 )
...
was accidentally removed in commit be9547a5
1 year ago
Mike Fährmann
391a7d74c8
[giantessbooru] fix and move to separate module ( #4373 )
...
too many differences to the other shimmie2 sites
1 year ago
ClosedPort22
5448268d5c
[downloader:http] close connection when file already exists ( #3748 )
1 year ago
Mike Fährmann
3963dbe5e4
extend 'parent>child' categories
...
continuation of ed21908f
allow for children to have an arbitrary distance from their parent,
e.g. reddit -> danbooru -> imgur:gallery -> imgur:album
would still be covered by 'reddit>imgur' or even 'danbooru>imgur'
1 year ago
Mike Fährmann
089d1a4f67
[twitter] fix 'TweetWithVisibilityResults' ( #4369 )
1 year ago
Mike Fährmann
a4f7f7da17
add '_dump()' convenience method to Extractor
1 year ago
Mike Fährmann
df5c7ee03e
[deviantart] fix search ( #4384 )
...
send correct usernames instead of 'u'
1 year ago
Mike Fährmann
a60db454af
[sankaku] update/fix API headers
...
'Referer' and 'Origin' were both empty
1 year ago
Mike Fährmann
fb3f0453db
[twitter] improve error messages for single Tweets ( #4369 )
...
also fixes '"quoted": false' not having any effect
1 year ago
Mike Fährmann
541bff5a37
[pururin] fix extraction ( #4375 )
...
- rename 'title_jp' to 'title_ja'
- change type of 'collection', 'convention', and 'scanlator' to list
1 year ago
Mike Fährmann
6a87c314af
[instagram] fix private posts with long shortcodes ( #4362 )
1 year ago
Mike Fährmann
f899fac4c5
[giantessbooru] fix extraction ( #4373 )
...
This does not fix anything Cloudflare related,
just other things caused by a site update.
1 year ago
Mike Fährmann
136283d402
[shimmie2] update base URL pattern
...
to match new giantessbooru URLs
1 year ago
Mike Fährmann
9d67655397
add "ascii+" as a special 'path-restrict' value ( #4371 )
1 year ago
Mike Fährmann
c79359eb3a
[fantia] improve metadata extraction ( #4126 )
...
extract all metadata and URLs before starting to download
1 year ago
Mike Fährmann
48ef062867
fix issues with 'Extractor.finalize()'
...
- prevent crash in InstagramUserExtractor (#4359 )
- call it at the end of every DownloadJob
- add it to tests
1 year ago
Mike Fährmann
ed21908fda
initial support for child extractor options
...
Using "parent-category>child-category" as extractor category in a config
file allows to set options for a child extractor when it was spawned by
that parent.
For example "reddit>gfycat" to set gfycat options for when it was found
in a reddit post.
{
"extractor": {
"gfycat": {
"filename": "regular filename"
},
"reddit>gfycat": {
"filename": "reddit-specific filename"
}
}
}
Note: This does currently not work for most imgur links due to how its
extractor hierarchy is structured.
1 year ago
Mike Fährmann
255d08b79e
add test for 'Extractor.initialize()' ( #4359 )
1 year ago
Mike Fährmann
2bcf0a4c49
[instagram] fix initialization order ( #4359 )
...
regression caused by the changes in a383eca7
1 year ago
Mike Fährmann
7eab101144
[acidimg] fix extraction
...
swap ' and " again (2e309a13
)
and add a fallback in case this happens yet another time
1 year ago
Mike Fährmann
62fce6a75f
[imagehosts] adjust variable names ( #4358 )
...
prefix them with underscores to prevent a clash
with the new 'self.cookies' from d97b8c2f
1 year ago
Mike Fährmann
e8299b459a
[moebooru] match search URLs with empty 'tags' ( #4354 )
1 year ago
Mike Fährmann
7fbc304ae9
[twitter] fix crash on private user ( #4349 )
1 year ago
Mike Fährmann
1ece3b92ff
[mangadex] allow multiple values for 'lang' ( #4093 )
...
This was already possible by setting 'lang' to a list of strings,
but now it can also be done as a more command-line friendly string.
-o lang=fr,it
1 year ago
Mike Fährmann
52053b58f0
[lensdump] fix extraction ( #4352 )
1 year ago
Mike Fährmann
11f71a9cba
remove 'mememuseum' module
...
This was forgotten when adding generic Shimmie2 support in 7865067d
1 year ago
Mike Fährmann
a383eca7f6
decouple extractor initialization
...
Introduce an 'initialize()' function that does the actual init
(session, cookies, config options) and can called separately from
the constructor __init__().
This allows, for example, to adjust config access inside a Job
before most of it already happened when calling 'extractor.find()'.
1 year ago
Mike Fährmann
6c9432165e
add return value to 'PostProcessor._init_archive()'
1 year ago
Mike Fährmann
54d974deb0
add 'python' post processor
...
similar to 'exec' but calls a Python function
1 year ago
Mike Fährmann
1baf83a9e5
[hiperdex] fix for unicode titles ( #4325 )
1 year ago
Mike Fährmann
7da954f810
[flickr] update default API credentials ( #4332 )
...
and add a delay between API requests
1 year ago
Mike Fährmann
a45a17ddb7
[pixiv] ignore 'limit_sanity_level' images ( #4328 )
1 year ago
Mike Fährmann
088e8d5fcf
[pornhub] fix extraction ( #4301 )
1 year ago
Mike Fährmann
d97b8c2fba
consistent cookie-related names
...
- rename every cookie variable or method to 'cookies_*'
- simplify '.session.cookies' to just '.cookies'
- more consistent 'login()' structure
1 year ago