aee5580c62
[idolcomplex] extract 'id_alnum' metadata ( #5171 )
7 months ago
cf7d6be2d4
[bluesky] initial support ( #4438 , #4708 , #4722 , #5047 )
8 months ago
6ef143ea31
[idolcomplex] support alphanumeric post IDs ( #5171 )
8 months ago
d7823b9f81
[pinterest] fix section URLs for boards with /?# in name ( #5104 )
8 months ago
de752eb7b1
[naverwebtoon] support '/webtoon/' paths for all comics ( #5123 )
8 months ago
22647c2626
[naverwebtoon] fix 'title' for comics with empty tags ( #5120 )
8 months ago
1f7101d606
[archivedmoe] fix thebarchive webm URLs ( #5116 )
8 months ago
34a4ddc399
[sankaku] add 'id-format' option ( #5073 )
8 months ago
afd20ef42c
[kemonoparty] implement filtering duplicate revisions ( #5013 )
...
set 'revisions' to '"unique"' to have it ignore duplicate revisions
8 months ago
c28475d325
[kemonoparty] fix deleting 'name' in orginal objects ( #5103 )
...
... when computing 'revision_hash'
regression caused by 3d68eda4
8 months ago
beacfa7436
[bunkr] update domain to 'bunkr.sk' ( #5114 )
8 months ago
0d3af0d35b
[tests] ignore 'ytdl' categories when import fails ( #5095 )
8 months ago
f3ad91b44f
[bunkr] update domain ( #5088 )
8 months ago
c7a42880ab
[wikimedia] support fandom wikis ( #1443 , #2677 , #3378 )
...
Wikis hosted on fandom.com are just wikimedia instances
and support its API.
8 months ago
df718887c2
[webtoons] fix extracting comic and episode name with commas
8 months ago
0d367ce1b9
[tests] update extractor results
8 months ago
9ca6117c67
[hbrowse] remove module
...
website gone
8 months ago
375eefb886
[chevereto] remove 'pixl.li'
...
"Pixl is closing down"
"All images will be deleted January 1st."
8 months ago
b0a441f1e3
[nitter] remove 'nitter.lacontrevoie.fr'
...
"Fermeture de Nitter / Closing down Nitter"
8 months ago
a1c1e80f67
[giantessbooru] update domain
8 months ago
2007cb2f59
[tests] check extractor category values
8 months ago
93b4120e77
[gelbooru] support 'all' and empty tag ( #5076 )
8 months ago
a416d4c3d5
[sankaku] support post URLs with alphanumeric IDs ( #5073 )
8 months ago
ea553a1d55
[wikimedia] generalize ( #1443 )
...
- support mediawiki.org
- support mariowiki.com (#3660 )
- combine code into a single extractor
(use prefix as subcategory)
- handle non-wiki instances
- unescape titles
8 months ago
c3c1635ef3
[wikimedia] update
...
- rewrite using BaseExtractor
- support most Wiki* domains
- update docs/supportedsites
- add tests
8 months ago
3d68eda4ab
[kemonoparty] add 'revision_hash' metadata ( #4706 , #4727 , #5013 )
...
A SHA1 hexdigest of other relevant metadata fields like
title, content, file and attachment URLs.
This value does NOT reflect which revisions are listed on the website.
Neither does 'edited' or any other metadata field (combinations).
8 months ago
799a8206ad
merge #5061 : [webtoons] extract more metadata
...
- author_name
- comic_name
- episode_name
- username
8 months ago
8ffa0cd3c8
[webtoons] small optimization
...
don't extract the entire 'author_area' and
avoid creating a second 'text.extract_from()' object
8 months ago
68196589c4
[2ch] update
...
- simplify extractor code
- more metadata
- add tests
8 months ago
69726fc82c
[tests] skip tests requiring auth when non is provided
8 months ago
bb446b1598
[webtoons] extract more metadata
8 months ago
355b909f46
merge #5041 : [steamgriddb] add support ( #5033 )
8 months ago
71e2c3e5a2
merge #5037 : [hatenablog] add support ( #5036 )
8 months ago
b97af09e03
[tests] include URL in failure report
8 months ago
58e0665fbc
[tests] load config from external file
8 months ago
2dcfb012ea
[patreon] download 'm3u8' manifests with ytdl
8 months ago
2191e29e14
[nijie] fix image URL for single image posts ( #5049 )
8 months ago
39904c9e4e
[deviantart:avatar] add 'formats' option ( #4995 )
8 months ago
887ade30a5
[batoto] support more mirror domains ( #5042 )
8 months ago
2ccb7d3bd3
[steamgriddb] add support
8 months ago
2cfe788f93
[hatenablog] fix extractor naming errors
9 months ago
61f3b2f820
[hatenablog] add support
9 months ago
657ed93a22
[batoto] improve v2 manga URL pattern
...
and add tests
9 months ago
33f228756a
[mangadex] add 'list' extractor ( #5025 )
...
supports listing manga and chapters from list feed
9 months ago
c25bdbae91
[komikcast] fix 'manga' extractor ( #5027 )
9 months ago
8e1a2b5446
[komikcast] update domain to 'komikcast.lol' ( #5027 )
9 months ago
a441249ea2
merge #4979 : [batoto] add 'chapter' and 'manga' extractors ( #1434 , #2111 )
9 months ago
b11c352d66
[bato] rename to 'batoto'
...
to use the same category name as the previous bato.to site
9 months ago
3aa24c3744
[bato] simplify and update
9 months ago
11150a7d72
[nudecollect] remove module
9 months ago
c158927c38
merge #5016 : [zzup] add 'gallery' extractor ( #4517 , #4604 , #4659 , #4863 )
9 months ago
217fa7f8a1
include 'test/results' in flake8 checks
9 months ago
e61f016465
[szurubooru] support 'snootbooru.com' ( #5023 )
9 months ago
b4bcf40278
[weibo] fix AttributeError in 'user' extractor ( #5022 )
...
yet another bug caused by a383eca7
9 months ago
0ab0a10d2d
[jpgfish] update domain
9 months ago
0f30136109
[zzup] add 'gallery' extractor
9 months ago
7eaf648f2e
[fanbox] add 'metadata' option ( #4921 )
...
extracts 'plan' and extended 'user' metadata
9 months ago
4f3671458e
[deviantart] add 'avatar' and 'background' extractors ( #4995 )
9 months ago
63f649cd92
[idolcomplex] fix extraction & update URL patterns ( #5002 )
9 months ago
7aa1c9671b
[tests] fix 'invalid escape sequence' warnings
9 months ago
b6903a4c90
[nijie] add 'count' metadata field
...
https://github.com/mikf/gallery-dl/issues/146#issuecomment-1812849102
9 months ago
b93b351db9
merge #4962 : [poringa] add support ( #4675 )
9 months ago
9f21c839ad
[poringa] improvements and fixes
...
- add 'num' and 'count' metadata fields
- prevent crash for "private" posts
- prevent crash when there's no 'main-info'
- update tests
9 months ago
caceb14fc2
[tests] fail when a results file contains syntax errors
...
or is otherwise not importable
9 months ago
085411f3f1
[rule34] recognize URLs with 'www' subdomain ( #4984 )
9 months ago
e348da7a06
[poringa] add support
9 months ago
74c225f94e
[bato] add support
9 months ago
f9544194c0
[paheal] restore 'extension' metadata ( #4976 )
9 months ago
77d46e6f0c
[lynxchan] update 'bbw-chan' domain ( #4970 )
9 months ago
108c978073
merge #4919 : [postmill] add support ( #4917 )
9 months ago
2a60645095
[deviantart] set 'is_original' for intermediary URLs to 'false'
9 months ago
01bb75f6cb
merge #4945 : {shimmie2[ support 'rule34hentai.net' ( #861 , #4789 )
9 months ago
79e4606893
[rule34hentai] cleanup
...
- fix using 'self._posts_rule34hentai'
- fix 'file_url' for posts
- update docs/supportedsites
- add tests
9 months ago
627ed794a2
[danbooru] provide 'tags' as list ( #4942 )
...
keep the old 'tag_string' values around, similar to sankaku
a lot of repeat code ...
would be a lot less bad if "".split(" ") returned an empty list
9 months ago
99aa923322
[inkbunny] improve '/submissionsviewall.php' patterns ( #4934 )
...
allow 'mode=…' to be in any position
don't require it to be somewhere in the middle
9 months ago
3f9c113d78
[mastodon] Support non-numeric status IDs ( #4936 )
9 months ago
2852404e49
[inkbunny] add 'unread' extractor ( #4934 )
9 months ago
a37b7759bc
[myhentaigallery] recognize '/g/' URLs ( #4920 )
9 months ago
fbe14a2745
[postmill] add support
9 months ago
bf74eb5c46
merge #4886 : [urlgalleries] add 'gallery' extractor ( #919 , #1184 , #2905 )
10 months ago
ade93c5397
[urlgalleries] add tests
10 months ago
4eb3590103
[nijie] fix image URLs of multi-image posts ( #4876 )
10 months ago
c83fbe6c2d
merge #4855 : [nitter] fix video extraction ( #4853 )
10 months ago
1137d72d48
[tests] skip test_init for BaseExtractor classes without instances
10 months ago
625e94fa7d
update extractor test results
...
still not everything, but good enough for now
10 months ago
1e9bacd169
[nitter] fix video extraction
10 months ago
95c1dfb089
[tests] swap assertEqual argument order
...
before this, it would show test failures as
+ test value
- extracted value
when it should be the other way round
10 months ago
bdb3ce7217
[foolslide] remove 'powermanga.org'
10 months ago
f9dac43be9
[warosu] fix file URLs
10 months ago
645b4627ef
[sankaku] update URL patterns
10 months ago
119755a5a3
[tests] implement skipping/failing tests when pressing ctrl+c
10 months ago
1ae43d8123
merge #4841 : [fapello] support '.su' TLD ( #4840 )
10 months ago
e1404827a6
[pixeldrain] add 'file' and 'album' extractors ( #4839 )
10 months ago
2402162e8a
[fapello] support '.su' TLD
10 months ago
725c8dd55a
[tmohentai] 'categories' -> 'genres'
...
quite likely that the site meant 'genres' by "Genders"
10 months ago
ce7c4cb544
merge #4832 : [tmohentai] add 'gallery' extractor ( #4808 )
10 months ago
c4a201ed42
[tmohentai] simplify + tests
10 months ago
e17a48fe56
[blogger] inherit from BaseExtractor
...
- support www.micmicidol.club (#4759 )
10 months ago
0fa85360a0
merge #4812 : [erome] add 'count' metadata field
10 months ago
a43cf78bb7
[erome] tests
10 months ago
07cb584231
[behance] add 'modules' option ( #4799 )
10 months ago
ea78f67860
[downloader:http] skip files not passing filesize-min/-max ( #4821 )
...
instead of failing the download
10 months ago
3f591d5a4e
[mastodon] update test results
10 months ago
6402f2950f
[pp:metadata] ignore non-string tag values ( #4764 )
11 months ago
007c433677
[patreon] support 'id:<campaign_id>' in place of a user name
...
https://patreon.com/id:12345
… and remove 'campaign-id' config option
11 months ago
43a3d93467
merge #4755 : [twitter] recognize fixupx.com URLs
11 months ago
cdf77e326f
[twitter] add test for fixupx.com
11 months ago
fc8f86bf24
[hitomi] recognize 'imageset' gallery URLs ( #4756 )
11 months ago
72b18d701f
represent util.NONE as 'null' in JSON output
...
was '"None"' before
11 months ago
68e72a836c
[exhentai] fix extraction ( #4730 )
...
- update to new API response layout
- use proper API server URL
- fix 'filesize' metadata
11 months ago
fd8f58ad76
[behance] unescape embed URLs ( #4742 )
11 months ago
c9a2be36d4
[sankaku] support '/posts/' tag search URLs ( #4740 )
11 months ago
218295a4c6
[twitter] fix avatars without 'date' information ( #4696 )
11 months ago
d0effcae20
[kemonoparty] add 'revision_index' metadata field ( #4727 )
11 months ago
3bbaa875f1
[kemonoparty] fix parsing of non-standard 'dates' ( #4676 )
11 months ago
a09df34bcf
merge #4714 : [4archive] add 'thread' and 'board' extractors
...
(#1262 , #2418 , #4400 , #4710 )
11 months ago
acb713b95a
[4archive] update
11 months ago
6766877524
merge #4693 : [reddit] support Reddit Mobile share links
11 months ago
1042278bec
[misskey] support 'misskey.design' ( #4713 )
11 months ago
c0714d5585
[4archive] add 'thread' and 'board' extractors
11 months ago
b68aad3dab
[reddit] implement Reddit Mobile share links
11 months ago
7958ab1946
[newgrounds] support 'imageData' files ( #4642 )
11 months ago
b52fd91ac6
[sankaku] support '/posts/' URLs ( #4688 )
11 months ago
b2c3db3e24
[bunkr] add extractor for media URLs ( #4684 )
11 months ago
6e830ffc9e
[kemonoparty] support post searches ( #3385 , #4057 )
11 months ago
aaf539009b
[kemonoparty] initial support for post revisions ( #4498 , #4597 )
...
- single revision
https://kemono.party/SERVICE/user/12345/post/12345/revision/12345
- all revisions
https://kemono.party/SERVICE/user/12345/post/12345/revisions
11 months ago
174191cb79
[kemonoparty] restore discord pagination ( #4676 )
11 months ago
c9a976d8a6
[kemonoparty] various updates and fixes ( #4676 , #4681 )
...
- fix pagination
- fix 'date' metadata
- fix discord channel API endpoint
11 months ago
bfdc07632a
[deviantart] expand nested comment replies ( #4653 )
11 months ago
9bc5ad4784
[tests] implement 'len:'
11 months ago
a1977a698e
[tests] fix spurious failures in '_assert_isotime()'
11 months ago
390d14dbcc
[chevereto] support 'img.kiwi' and 'deltaporno.com' ( #4664 , #1381 )
11 months ago
727c8eec6c
merge #4667 : [redgifs] fix 'niches' extraction ( #4666 )
11 months ago
2911ed1240
[chevereto] add generic extractors ( #4664 )
...
- support jpgfish
- support pixl.li / pixl.is (#3179 , #4357 )
11 months ago
db3363ac0b
[redgifs] fix 'niches' extraction
11 months ago
ade8347ead
[kemonoparty] fix DM dates
11 months ago
6dfe200ae4
[kemonoparty] support discord URLs with channel IDs ( #4662 )
11 months ago
c6a3892210
[imgbb] update username extraction ( #4626 )
11 months ago
13ce3a9acb
[warosu] fix extraction ( #4634 )
11 months ago
c4c4e4d2f4
[newgrounds] improve 'art-image' extraction ( #4642 )
...
- download files in original resolution
- replace .webp with extension of first file
11 months ago
833dce141f
[fantia] add 'content_count' and 'content_num' metadata fields ( #4627 )
11 months ago
2d41702762
[deviantart] implement '"group": "skip"' ( #4630 )
11 months ago
a9c3442d4e
[deviantart] add a couple 'deactivated account' test URLs
11 months ago
2974b8e3c8
[moebooru] add 'metadata' option ( #4646 )
...
for extended 'pool' metadata
11 months ago
67ba4ee842
[pp:exec] support more replacement fields for '--exec' ( #4633 )
...
- {_directory}
- {_filename}
- {_path} (alias for {})
12 months ago
9a008523ac
[hentaifoundry] fix '.swf' file downloads ( #4641 )
12 months ago
15f940819b
[newgrounds] support 'art-image' files ( #4642 )
12 months ago
efaab4fbfa
[twitter] fix crash due to missing 'source' ( #4620 )
...
regression caused by 06aaedde
12 months ago
84fbbd96aa
[shimmie2] remove 'meme.museum'
12 months ago
0b150d45db
[tests] add 'msg' arguments to assert statements
12 months ago
27da3f2958
[tests] re-implement filtering by basecategory
12 months ago
c7bd9925d9
[tests] use fallback URLs for content tests ( #3163 )
12 months ago
b92645cd37
[bunkr] fix extraction ( #4514 , #4532 , #4529 , #4540 )
12 months ago
bd3f7a5bbc
[tests] support one regex per URL for #pattern
12 months ago
0c5d8b1505
[deviantart] re-add 'quality' option and 'intermediary' transform
1 year ago
dbd820d7c5
[tests] allow checking for exact URL results
1 year ago
642998504d
[tests] support 'range()' for #count and metadata checks
1 year ago
1e31fce37b
[pillowfort] support '/tagged/' URLs ( #4570 )
1 year ago
1d2fd0b831
[pillowfort] extract 'b2_lg_url' media ( #4570 )
1 year ago
50e2ebaff0
[danbooru] support 'donmai.moe' URLs
1 year ago
918ba4f847
[redgifs] match gfycat image URLs ( #4558 )
1 year ago
2cd801232b
fix --range causing crashes ( #4557 )
...
regression caused by a383eca7
1 year ago
27ec653991
fix bug in test_init and update example URLs
1 year ago
24a1d46391
[mastodon] support '/@USER/following' URLs
...
Previously, only '/users/USER/following' got matched.
1 year ago
ac00d47a16
update test/test_results.py
1 year ago
65b6011cc5
update test/test_extractor.py
1 year ago
a833c244c8
add exported extractor results
1 year ago
93a7a89cf6
[formatter] use value of last alternative ( #4492 )
...
fixes {fieldname|''} evaluating to the value of 'keywords-default'
instead of an empty string
1 year ago
f2de70f254
[gfycat] remove module
1 year ago
d319777a24
[tests] skip 'test_init_ytdl' on Python<3.6
...
It passes without error in a Python 3.4/3.5 venv on my own machine,
but fails for some inexplicable reason on Github Actions.
1 year ago
0ef1fcab20
[postprocessor] update 'finalize' events
...
Add 'finalize-error' and 'finalize-success' events that trigger
depending on whether error(s) did or did not happen.
'finalize' itself now always triggers regardless of error status.
(was supposed to have the same behavior as the new 'finalize-success')
1 year ago
d50c312ff0
prevent test failure when there's no 'ytdl' module ( #4364 )
...
split of ytdl into its own test function and
skip it when there's an ImportError similar to test_ytdl.py
1 year ago
48ef062867
fix issues with 'Extractor.finalize()'
...
- prevent crash in InstagramUserExtractor (#4359 )
- call it at the end of every DownloadJob
- add it to tests
1 year ago
255d08b79e
add test for 'Extractor.initialize()' ( #4359 )
1 year ago
a383eca7f6
decouple extractor initialization
...
Introduce an 'initialize()' function that does the actual init
(session, cookies, config options) and can called separately from
the constructor __init__().
This allows, for example, to adjust config access inside a Job
before most of it already happened when calling 'extractor.find()'.
1 year ago
f0203b7559
[postprocessor:python] add tests
1 year ago
d97b8c2fba
consistent cookie-related names
...
- rename every cookie variable or method to 'cookies_*'
- simplify '.session.cookies' to just '.cookies'
- more consistent 'login()' structure
1 year ago
c5565f79f7
merge #4096 : [danbooru] add support for booru.borvar.art instance
1 year ago
63326e3168
[danbooru] add tests for booruvar
1 year ago
5171d8975c
[E621] support 'e6ai.net' ( #4320 )
1 year ago
7444fc125b
[gfycat] implement login support ( #3770 , #4271 )
...
For the record: '/webtoken' and '/weblogin' are not the same ...
1 year ago
25c5a6ffcb
no f-strings
1 year ago
ec64cbefeb
[postprocessor:exec] add tests
1 year ago
ce93c460a6
[formatter] implement 'H' conversion ( #4164 )
...
to remove HTML tags and unescape HTML entities
1 year ago
deff3b434d
[vipergirls] implement login support ( #4166 )
1 year ago
69865dcc05
[formatter] implement slicing strings as bytes ( #4087 )
...
prefixing a slice '[10:30]' with a lowercase b '[b10:30]' encodes
the string to bytes in filesystem encoding before applying the slice
1 year ago
df11214281
[ytdl] improve --xff/--geo-bypass detection ( #3989 )
...
check if --xff is supported in a try-except block
and select expected results accordingly
1 year ago
aa731c4298
[ytdl] run yt-dlp tests with latest code from master ( #3989 )
...
Only use PyPI version for Python 3.6, since that's no longer supported
by the current codebase.
1 year ago
43f4bd9faa
[ytdl] fix tests
...
tests pass with latest Git HEAD, but not with the current PyPI version
1 year ago
61a65d5bb9
[ytdl] fix crash due to --geo-bypass deprecation ( #3975 )
1 year ago
a96745368e
"fix" tests on Python 3.4 and 3.5
...
can't rely on dict insertion order
1 year ago
3905f05f00
[postprocessor:metadata] support putting keys in quotes
...
for mode 'modify' and 'delete'
based on fe41a2b1
1 year ago
7459e4abce
[postprocessor:metadata] fix traversing more than 1 level deep
...
for mode 'modify' and 'delete'
1 year ago
2edcdee32f
[downloader:http] add MIME type and signature for .heic files
...
(#3915 )
https://github.com/strukturag/libheif/issues/83
1 year ago
082d55de16
fix circular reference detection for -K
2 years ago
2ab66ad899
update -K output to include quotes around keys
2 years ago
fe41a2b159
[formatter] support putting keys in quotes
...
i.e. obj["key"] or obj['key']
as in f-strings
2 years ago
46fdf46f21
[formatter] support loading an f-string from a template file
...
"\fTF ~/path/to/file.txt"
2 years ago
1a4d4a799b
[formatter] support filesystem paths for \fM
2 years ago
00f0233b28
[postprocessor:metadata] add 'skip' option ( #3786 )
2 years ago
Mike Fährmann