6e10260fb0
release version 1.26.6
9 months ago
d0d199414f
modify useragent of pixiv
9 months ago
cbfb7bfdf1
[gelbooru] display error for invalid API responses ( #4903 )
9 months ago
c25bdbae91
[komikcast] fix 'manga' extractor ( #5027 )
9 months ago
8e1a2b5446
[komikcast] update domain to 'komikcast.lol' ( #5027 )
9 months ago
a441249ea2
merge #4979 : [batoto] add 'chapter' and 'manga' extractors ( #1434 , #2111 )
9 months ago
b11c352d66
[bato] rename to 'batoto'
...
to use the same category name as the previous bato.to site
9 months ago
3aa24c3744
[bato] simplify and update
9 months ago
11150a7d72
[nudecollect] remove module
9 months ago
c158927c38
merge #5016 : [zzup] add 'gallery' extractor ( #4517 , #4604 , #4659 , #4863 )
9 months ago
e61f016465
[szurubooru] support 'snootbooru.com' ( #5023 )
9 months ago
b4bcf40278
[weibo] fix AttributeError in 'user' extractor ( #5022 )
...
yet another bug caused by a383eca7
9 months ago
0ab0a10d2d
[jpgfish] update domain
9 months ago
0f30136109
[zzup] add 'gallery' extractor
9 months ago
a86775f617
[gelbooru] fix 'favorite' extractor ( #4903 )
...
lots of +1/-1 and </<= mistakes
9 months ago
7eaf648f2e
[fanbox] add 'metadata' option ( #4921 )
...
extracts 'plan' and extended 'user' metadata
9 months ago
00570028a3
[cookies] fix macOS Firefox profile path
...
85b33f5c16
9 months ago
4f3671458e
[deviantart] add 'avatar' and 'background' extractors ( #4995 )
9 months ago
9fa4f54c24
[twitter] raise error for invalid 'strategy' values ( #4953 )
9 months ago
516c69297d
[manganelo] fix extraction & recognize '.to' TLDs ( #5005 )
9 months ago
63f649cd92
[idolcomplex] fix extraction & update URL patterns ( #5002 )
9 months ago
b6903a4c90
[nijie] add 'count' metadata field
...
https://github.com/mikf/gallery-dl/issues/146#issuecomment-1812849102
9 months ago
b93b351db9
merge #4962 : [poringa] add support ( #4675 )
9 months ago
9f21c839ad
[poringa] improvements and fixes
...
- add 'num' and 'count' metadata fields
- prevent crash for "private" posts
- prevent crash when there's no 'main-info'
- update tests
9 months ago
00d83d9588
[rule34us] add fallback for 'video-cdn1' videos ( #4985 )
9 months ago
085411f3f1
[rule34] recognize URLs with 'www' subdomain ( #4984 )
9 months ago
9f5051e4ed
merge #4981 : [pinterest] add 'count' metadata field
9 months ago
f6ce870885
Better variable names
9 months ago
3553025584
Removed f-strings
9 months ago
f36dafad06
improve 'include' handling ( #4982 )
...
- remove spaces when given as string
- warn about invalid vales
9 months ago
375f2db4c2
[pinterest] add count metadata field
9 months ago
e348da7a06
[poringa] add support
9 months ago
2c3f171d65
Fix python 3.5 linting issue
9 months ago
06ff1d3a3c
Replace text.extract with extr
9 months ago
9c1ce28f68
[bato] Added mangatoto alias
9 months ago
663b8d789a
Fix linting
9 months ago
74c225f94e
[bato] add support
9 months ago
f9544194c0
[paheal] restore 'extension' metadata ( #4976 )
9 months ago
77d46e6f0c
[lynxchan] update 'bbw-chan' domain ( #4970 )
9 months ago
766316e436
[imagechest] fix loading more than 10 images in a gallery ( #4469 )
9 months ago
6840717745
release version 1.26.5
9 months ago
108c978073
merge #4919 : [postmill] add support ( #4917 )
9 months ago
8a42ea736a
[postmill] implement suggestions
9 months ago
c184454efb
[shimmie2] small optimizations
...
- unroll/remove loop
- avoid copy
9 months ago
7cd0211cc9
[shimmie2] autodetect single or double quotes
9 months ago
2a60645095
[deviantart] set 'is_original' for intermediary URLs to 'false'
9 months ago
01bb75f6cb
merge #4945 : {shimmie2[ support 'rule34hentai.net' ( #861 , #4789 )
9 months ago
79e4606893
[rule34hentai] cleanup
...
- fix using 'self._posts_rule34hentai'
- fix 'file_url' for posts
- update docs/supportedsites
- add tests
9 months ago
ef370df41d
[shimmie2] support 'rule34hentai.net'
...
- Add files via upload
- Update shimmie2.py
- Update shimme2.py
- Delete gallery_dl/extractor/shimme2.py
- spacefix
- Update shimmie2.py
- Update shimmie2.py
- flask warnings1
- Update shimmie2.py
- Update shimmie2.py
9 months ago
627ed794a2
[danbooru] provide 'tags' as list ( #4942 )
...
keep the old 'tag_string' values around, similar to sankaku
a lot of repeat code ...
would be a lot less bad if "".split(" ") returned an empty list
9 months ago
fbebc58189
[deviantart] add 'intermediary' option ( #4955 )
9 months ago
75fa1a5553
[pinterest] remove login code
...
this has been broken since forever
and is still "protected" by an invisible recaptcha check
9 months ago
92ff99c8e5
[twitter] remove 'syndication' option ( #3889 )
9 months ago
a75f85a2c2
[twitter] remove 'date_liked' ( #3850 , #4108 , #4657 )
...
Twitter's 'sortIndex' can't be used to calculate the timestamp
of when a Tweet was liked anymore.
9 months ago
a94f944148
[twitter] default to 'tweets' timeline when 'replies' are enabled ( #4953 )
9 months ago
a30a3e44d5
[nijie] move 'username required' out of _login_impl
9 months ago
57fc6fcf83
replace '24*3600' with '86400'
...
and generalize cache maxage values
9 months ago
1f9b16a70b
replace static 'sleep-request' defaults with dynamic ones
9 months ago
b127321b5c
[exhentai] only show 'using e-hentai.org' warning for exh domains
9 months ago
e097aaf64a
[exhentai] output continuation URL when interrupted ( #4782 )
9 months ago
99aa923322
[inkbunny] improve '/submissionsviewall.php' patterns ( #4934 )
...
allow 'mode=…' to be in any position
don't require it to be somewhere in the middle
9 months ago
3f9c113d78
[mastodon] Support non-numeric status IDs ( #4936 )
9 months ago
2852404e49
[inkbunny] add 'unread' extractor ( #4934 )
9 months ago
8b87a5330d
[inkbunny] stop pagination on empty results
9 months ago
6cd5e6adad
[patreon] fix bootstrap data extraction ( #4904 )
9 months ago
aac8bb4eae
[deviantart] simplify 9951c112
9 months ago
9951c112f8
[deviantart] workaround for integer client_id values ( #4924 )
9 months ago
a37b7759bc
[myhentaigallery] recognize '/g/' URLs ( #4920 )
9 months ago
da76e13e3b
[tumblr] fix exception after waiting for rate limit ( #4916 )
...
use a loop instead of recursive function calls
9 months ago
fbe14a2745
[postmill] add support
9 months ago
d59d4ebff4
[tumblr] support infinite 'fallback-retries'
9 months ago
2d5cda2b92
[exhentai] fix TypeError for infinite 'fallback-retries' ( #4911 )
9 months ago
a24b82e67d
add 'util.repeat()'
9 months ago
92fbf09643
remove single quotes in some logging messages ( #4908 )
...
('FileNotFoundError: [Errno 2] No such file or directory: ''')
->
(FileNotFoundError: [Errno 2] No such file or directory: '')
9 months ago
7dd79eee93
save cookies to tempfile, then rename
...
avoids wiping the cookies file if the disk is full
9 months ago
1d5ee4239d
[docker] let metadata-action automatically generate 'latest' tags
9 months ago
28d60e3546
release version 1.26.4
9 months ago
9a001fa6e4
merge #4906 : [patreon] fix bootstrap data extraction ( #4904 )
9 months ago
66cbe9da41
- fix style check failure "line to long"
9 months ago
244444b194
- adapt code to current code style
9 months ago
fd06255f93
- reformat and refactor to pass tests
9 months ago
5ff7106d4f
- add code for the situation when Patreon is using window.patreon = wrapInProxy({"bootstrap":' to store metadata
...
- refactor code to make it more readable
- output page content when the HTML structure is unknown (to make debugging easier)
9 months ago
75697dfb26
implement -e/--error-file as a logging handler
...
similar to --write-unsupported
10 months ago
ac22bbe80c
[twitter] retry API requests only for Timeout errors ( #4811 )
10 months ago
c55955db03
[twitter] quick and dirty fix for /media changes ( #4898 )
10 months ago
9a8dc6b02b
[exhentai] add 'fallback-retries' option ( #4792 )
10 months ago
bf74eb5c46
merge #4886 : [urlgalleries] add 'gallery' extractor ( #919 , #1184 , #2905 )
10 months ago
c29ae9af08
[urlgalleries] simplify + resolve redirects
10 months ago
042a9da451
add 'output.errorfile' config option
10 months ago
e256434c9e
use custom HTTPBasicAuth class
...
to support LazyPrompt as password
and to generate the Authorization header only once
instead of for every request
10 months ago
bdebe4597a
fix util.dump_response to work with bytes as header values
10 months ago
6a4218aa23
handle 'json' parameter in Extractor.request() manually
...
Mainly to allow passing custom classes like util.LazyPrompt,
but also to simplify and streamline how requests handles it.
10 months ago
9dd5cb8c8a
interactively prompt for passwords on login when none is provided
10 months ago
99b76628f7
implement '-e/--error-file' command-line option ( #4732 )
...
copying per-URL options from regular, read-only input files
does currently not work
10 months ago
4eb3590103
[nijie] fix image URLs of multi-image posts ( #4876 )
10 months ago
a4e6ea667b
[twitter] retry API calls when their response contains errors ( #4811 )
10 months ago
cf5702c843
[twitter] generalize "Login Required" error ( #4734 , #4324 )
10 months ago
ecaa0feb5d
[urlgalleries] add support
10 months ago
1770c31e63
[urlgalleries] add support
10 months ago
da0da0faaa
[exhentai] store more cookies when logging in ( #4881 )
...
include 'igneous', 'hath_perks', etc
and not just 'ipb_member_id' and 'ipb_pass_hash' like before
10 months ago
43ca49c1b4
[github] add workflow to build and push docker images
...
heavily inspired by and adapted from
https://github.com/danbooru/danbooru/blob/master/.github/workflows/docker-build.yaml
10 months ago
4dde36889c
release version 1.26.3
10 months ago
c83fbe6c2d
merge #4855 : [nitter] fix video extraction ( #4853 )
10 months ago
013ca21543
[idolcomplex] update to site layout changes
10 months ago
1e9bacd169
[nitter] fix video extraction
10 months ago
9f3368c46f
[pornhub] fix 'user' metadata for gifs
10 months ago
bdb3ce7217
[foolslide] remove 'powermanga.org'
10 months ago
d9734ce008
[cyberdrop] update to site layout changes
10 months ago
8ac68ffba2
[hentaicosplays] force 'https://' for download URLs
10 months ago
fc1101779c
[hiperdex] fix 'manga' metadata
10 months ago
d119507037
[imagefap] fix single image resolution
...
Downloading from a single image page like
https://www.imagefap.com/photo/123456789/
returned only the thumbnail URL.
10 months ago
311ec1d9ef
[mangaread] fix extraction
10 months ago
7608201a44
[tumblr] fix 'day' extractor
...
another bug caused by a383eca7
10 months ago
c8c744a7c0
[webtoons] fix pagination when receiving an HTTP redirect
10 months ago
23cd17997d
[wallpapercave] fix extraction
10 months ago
5b979b5706
[xvideos] fix metadata extraction
10 months ago
adc3aa0b77
[zerochan] fix metadata extraction
...
author, path, tags
10 months ago
f9dac43be9
[warosu] fix file URLs
10 months ago
645b4627ef
[sankaku] update URL patterns
10 months ago
1ae43d8123
merge #4841 : [fapello] support '.su' TLD ( #4840 )
10 months ago
b43be67206
[exhentai] add 'gp' option ( #4576 )
10 months ago
cb9a1176e6
[pixeldrain] add 'api-key' option ( #4839 )
10 months ago
e1404827a6
[pixeldrain] add 'file' and 'album' extractors ( #4839 )
10 months ago
2402162e8a
[fapello] support '.su' TLD
10 months ago
725c8dd55a
[tmohentai] 'categories' -> 'genres'
...
quite likely that the site meant 'genres' by "Genders"
10 months ago
ce7c4cb544
merge #4832 : [tmohentai] add 'gallery' extractor ( #4808 )
10 months ago
c4a201ed42
[tmohentai] simplify + tests
10 months ago
e17a48fe56
[blogger] inherit from BaseExtractor
...
- support www.micmicidol.club (#4759 )
10 months ago
714b1a7089
[tmohentai] simplify url matching
10 months ago
31963fa947
[tmohentai] inherit from GalleryExtractor. refactor metadata.
10 months ago
0fa85360a0
merge #4812 : [erome] add 'count' metadata field
10 months ago
a43cf78bb7
[erome] tests
10 months ago
aea15f6d17
add 'metadata-extractor' option ( #4549 )
10 months ago
34a387b6e2
support 'metadata-*' names for '*-metadata' options
...
For example, instead of 'url-metadata' it is now also possible to use
'metadata-url' as option name.
- metadata-url
- metadata-path
- metadata-http
- metadata-version
- metadata-parent
10 months ago
e97d7b1c85
[exhentai] fix empty api_url with '"source": "hitomi"' ( #4829 )
10 months ago
ed965eecbb
[tmohentai] refactor to str.format for backwards compatibility
10 months ago
dad7ba1d58
[tmohentai] fix edge cases. updated archive_fmt and filename_fmt
10 months ago
286d0cb098
[tmohentai] add support
10 months ago
b714df5a16
disable 'downloader.progress' when using -q/--quiet ( #4810 )
...
it didn't produce any output since output.mode is set to to "null",
but it caused some unnecessary function calls
10 months ago
07cb584231
[behance] add 'modules' option ( #4799 )
10 months ago
6a753d9ff3
[behance] support 'text' modules ( #4799 )
10 months ago
ea78f67860
[downloader:http] skip files not passing filesize-min/-max ( #4821 )
...
instead of failing the download
10 months ago
8bf161e574
reorder post processing options shown by --help
10 months ago
168331d147
replace '--ugoira-conv' etc with a general '--ugoira'
...
update --ugoira webm to use the same FFmpeg args as Danbooru
--ugoira-conv -> --ugoira vp8
--ugoira-conv-lossless -> --ugoira vp9-lossless
--ugoira-conv-copy -> --ugoira copy
(--ugoira-conv and co still work as before,
but --help now lists only --ugoira)
10 months ago
97357e65ee
replace '--mtime-from-date' with a more generic '--mtime'
...
--mtime-from-date -> --mtime date
for the same effect as before
(--mtime-from-date also still works,
but --help now lists only --mtime)
10 months ago
387c8b0950
reword some (internal) option text
10 months ago
c6ad9bcd9b
[erome] add "count" for albums
10 months ago
51e377e612
add '--cbz' command-line option
10 months ago
4700051562
rework and extend input file processing ( #4732 )
...
- add 2 command-line options to modify input file contents
- -I/--input-file-comment
- -x/--input-file-delete
- implement InputManager class
- move code from util.py to __init__.py
(mainly to avoid import cycles)
10 months ago
17e710c4bf
[oauth] warn when cache is enabled but not writeable ( #4771 )
10 months ago
2e4bf54644
[hentaifoundry] check for and update expired sessions ( #4694 )
10 months ago
0435c6e603
[exhentai] handle 'Downloading … requires GP' errors ( #4576 , #4763 )
10 months ago
4288cea94a
[mastodon] fix reblogs ( #4580 )
10 months ago
7a0f145cbe
[twitter] ignore promoted Tweets ( #4790 , #3894 )
...
add 'ads' option in case someone actually wants to
download promoted content for whatever reason
10 months ago
e8b5e59a08
[weibo] detect redirects to login page ( #4773 )
10 months ago
5e58d2b455
[instagram] fix exception on empty 'video_versions' ( #4795 )
10 months ago
807ddde7e1
release version 1.26.2
11 months ago
6402f2950f
[pp:metadata] ignore non-string tag values ( #4764 )
11 months ago
61d6558322
[exhentai] try to avoid 'DH_KEY_TOO_SMALL' errors ( #1021 , #4593 )
11 months ago
69b931b9bb
[exhentai] provide fallback URLs ( #1021 , #4745 )
11 months ago
007c433677
[patreon] support 'id:<campaign_id>' in place of a user name
...
https://patreon.com/id:12345
… and remove 'campaign-id' config option
11 months ago
3984a49abf
[nijie] set 1-2s delay between requests to avoid 429 errors
11 months ago
dd14adccf6
[pixiv] allow cookies for non-OAuth URLs ( #4760 )
11 months ago
caf31e751c
[kemonoparty] limit 'title' length ( #4741 )
11 months ago
43d0c49d7e
[exhentai] fix original image URLs ( #4754 )
11 months ago
43a3d93467
merge #4755 : [twitter] recognize fixupx.com URLs
11 months ago
fc8f86bf24
[hitomi] recognize 'imageset' gallery URLs ( #4756 )
11 months ago
91e20eb59b
[fantia] simplify 'tags' to a list of strings ( #4752 )
11 months ago
72b18d701f
represent util.NONE as 'null' in JSON output
...
was '"None"' before
11 months ago
44d7964c09
[twitter] recognize fixupx.com URLs
11 months ago
56cd9d408d
[weibo] fix Sina Visitor request
11 months ago
68e72a836c
[exhentai] fix extraction ( #4730 )
...
- update to new API response layout
- use proper API server URL
- fix 'filesize' metadata
11 months ago
fd8f58ad76
[behance] unescape embed URLs ( #4742 )
11 months ago
ca1d5c2c0c
merge #4738 : [patreon] parse new bootstrap data format ( #4736 )
11 months ago
4730de163f
[patreon] refactor _extract_bootstrap()
11 months ago
e46efbd5b5
prevent crash when 'stdout.line_buffering' is not defined ( #642 )
11 months ago
c9a2be36d4
[sankaku] support '/posts/' tag search URLs ( #4740 )
11 months ago
28ada11cba
Try to parse newer HTTP response from Patreon
11 months ago
fd36eafe32
[twitter] restore truncated retweet text ( #3430 , #4690 )
11 months ago
218295a4c6
[twitter] fix avatars without 'date' information ( #4696 )
11 months ago
969be65d0b
[instagram] update API headers
11 months ago
d0effcae20
[kemonoparty] add 'revision_index' metadata field ( #4727 )
11 months ago
3bbaa875f1
[kemonoparty] fix parsing of non-standard 'dates' ( #4676 )
11 months ago
75dec71253
[idolcomplex] disable Referer headers by default ( #4726 )
11 months ago
a09df34bcf
merge #4714 : [4archive] add 'thread' and 'board' extractors
...
(#1262 , #2418 , #4400 , #4710 )
11 months ago
acb713b95a
[4archive] update
11 months ago
6766877524
merge #4693 : [reddit] support Reddit Mobile share links
11 months ago
1042278bec
[misskey] support 'misskey.design' ( #4713 )
11 months ago
12a800ce21
[patreon] improve 'campaign_id' handling ( #4699 , #4715 )
...
- add ways to directly specify a 'campaign_id'
- 'campaign-id' config option
- 'c' or 'campaign_id' URL query parameter
- more descriptive error messages
- show 'campaign_id' value in debug log
11 months ago
31dbbffc0b
[twitter] cache 'user_by_…' results ( #4719 )
11 months ago
c0714d5585
[4archive] add 'thread' and 'board' extractors
11 months ago
b68aad3dab
[reddit] implement Reddit Mobile share links
11 months ago
95a74be2a5
release version 1.26.1
11 months ago
de224ef3e4
[cookies] include exception in fallback warning
11 months ago
7958ab1946
[newgrounds] support 'imageData' files ( #4642 )
11 months ago
b52fd91ac6
[sankaku] support '/posts/' URLs ( #4688 )
11 months ago
b8674776e9
[4chanarchives] disable Referer headers by default ( #4686 )
11 months ago
78493f0870
[bunkr] fix '/d/' file URLs ( #4685 )
11 months ago
b2c3db3e24
[bunkr] add extractor for media URLs ( #4684 )
11 months ago
0d52b775cb
[kemonoparty] add 'revisions' option ( #4498 , #4597 )
11 months ago
6e830ffc9e
[kemonoparty] support post searches ( #3385 , #4057 )
11 months ago
aaf539009b
[kemonoparty] initial support for post revisions ( #4498 , #4597 )
...
- single revision
https://kemono.party/SERVICE/user/12345/post/12345/revision/12345
- all revisions
https://kemono.party/SERVICE/user/12345/post/12345/revisions
11 months ago
174191cb79
[kemonoparty] restore discord pagination ( #4676 )
11 months ago
c9a976d8a6
[kemonoparty] various updates and fixes ( #4676 , #4681 )
...
- fix pagination
- fix 'date' metadata
- fix discord channel API endpoint
11 months ago
dc1c2139b1
fix line too long
11 months ago
6b22af9720
[kemonoparty] update API endpoint ( #4676 )
11 months ago
bfdc07632a
[deviantart] expand nested comment replies ( #4653 )
11 months ago
390d14dbcc
[chevereto] support 'img.kiwi' and 'deltaporno.com' ( #4664 , #1381 )
11 months ago
727c8eec6c
merge #4667 : [redgifs] fix 'niches' extraction ( #4666 )
11 months ago
2911ed1240
[chevereto] add generic extractors ( #4664 )
...
- support jpgfish
- support pixl.li / pixl.is (#3179 , #4357 )
11 months ago
db3363ac0b
[redgifs] fix 'niches' extraction
11 months ago
ade8347ead
[kemonoparty] fix DM dates
11 months ago
6dfe200ae4
[kemonoparty] support discord URLs with channel IDs ( #4662 )
11 months ago
c6a3892210
[imgbb] update username extraction ( #4626 )
11 months ago
830a48bca4
[fantia] bad workaround for 833dce14 ( #4627 )
...
at least this makes "filter": "content_num == content_count+1"
with "event": "post-after" work
11 months ago
13ce3a9acb
[warosu] fix extraction ( #4634 )
11 months ago
c4c4e4d2f4
[newgrounds] improve 'art-image' extraction ( #4642 )
...
- download files in original resolution
- replace .webp with extension of first file
11 months ago
833dce141f
[fantia] add 'content_count' and 'content_num' metadata fields ( #4627 )
11 months ago
2d41702762
[deviantart] implement '"group": "skip"' ( #4630 )
11 months ago
992e86ec94
[deviantart] disable 'jwt' ( #4652 )
11 months ago
2974b8e3c8
[moebooru] add 'metadata' option ( #4646 )
...
for extended 'pool' metadata
11 months ago
d194ea68a9
[cookies] open cookie databases in read-only mode
...
bypasses the need to copy the entire database file
might solve #4195
11 months ago
8bb7243c10
[reddit] fix wrong previews ( #4649 )
...
caused by a failed comment URL
using the main submission's preview as fallback
14af15bd4963bb9b
12 months ago
08bdde5aac
merge #4619 : [twitter] add 'sensitive' metadata field
12 months ago
f3d6aaff13
[twitter] rename to 'sensitive'; use 'tget()'
12 months ago
95c280c59b
[imgbb] update pagination end condition ( #4626 )
12 months ago
2e350dd82a
merge #4626 : [imgbb] fix 'user' extraction, add 'displayname'
12 months ago
a2daa9befe
[imgbb] fix flake8 and username order
12 months ago
67ba4ee842
[pp:exec] support more replacement fields for '--exec' ( #4633 )
...
- {_directory}
- {_filename}
- {_path} (alias for {})
12 months ago
9a008523ac
[hentaifoundry] fix '.swf' file downloads ( #4641 )
12 months ago
15f940819b
[newgrounds] support 'art-image' files ( #4642 )
12 months ago
63db54b905
[patreon] update 'campaign_id' path ( #4639 )
12 months ago
b846f56c3a
[imgbb] Fix `user` extraction, add `displayname`
12 months ago
efaab4fbfa
[twitter] fix crash due to missing 'source' ( #4620 )
...
regression caused by 06aaedde
12 months ago
3438a3098d
[twitter] add possible_sensitive field
12 months ago
85357c1ef8
release version 1.26.0
12 months ago
64dbc58a5a
[deviantart] update Eclipse API endpoints 2 ( #4615 )
12 months ago
84fbbd96aa
[shimmie2] remove 'meme.museum'
12 months ago
aa77fda78c
[instagram] better error message for invalid users ( #4606 )
12 months ago
482f002e1f
[nsfwalbum] detect '/error.jpg' images ( #4598 )
12 months ago
eb230e4b77
[nsfwalbum] disable Referer headers by default ( #4598 )
12 months ago
b92645cd37
[bunkr] fix extraction ( #4514 , #4532 , #4529 , #4540 )
12 months ago
4477808d1c
fix symlink resolution in __main__.py
...
adapt ytdl order
12 months ago
be17103e21
[regifs] support 'order' parameter for user URLs ( #4583 )
12 months ago
7150c4c76c
fix imports when using the gallery_dl directory as argument ( #4581 )
12 months ago
ec91eeb7ef
Update gallery_dl/extractor/reddit.py
...
Co-authored-by: Mike Fährmann <mike_faehrmann@web.de>
12 months ago
66613c3a32
[reddit] ignore '/message/compose' URLs without www subdomain
1 year ago
bb39779e1a
[deviantart] use private tokens for 'is_mature' posts ( #4563 )
1 year ago
0c5d8b1505
[deviantart] re-add 'quality' option and 'intermediary' transform
1 year ago
20d1683c47
[deviantart] fix JWT replacement ( #293 , #4548 , #4563 )
...
And again, a huge thank you to @Ironchest337
for discovering this.
1 year ago
d7aac9fc06
[reddit] ignore '/message/compose' URLs ( #4482 )
1 year ago
1e31fce37b
[pillowfort] support '/tagged/' URLs ( #4570 )
1 year ago
1d2fd0b831
[pillowfort] extract 'b2_lg_url' media ( #4570 )
1 year ago
50e2ebaff0
[danbooru] support 'donmai.moe' URLs
1 year ago
82296b1f05
[reddit] add 'previews' option ( #4322 )
...
another way to disable this new behavior
1 year ago
918ba4f847
[redgifs] match gfycat image URLs ( #4558 )
1 year ago
2ad75bab05
[deviantart] add 'is_original' metadata field ( #4559 )
...
true for 'downloadable' content, journals, flash animations,
and images without '/v1/' in their URL; false otherwise
1 year ago
9d8317d963
[deviantart] disable JWT updates ( #4548 , #4563 )
...
back to lowres images ...
1 year ago
8064663bda
[deviantart] update Eclipse API endpoints ( #4553 )
1 year ago
2cd801232b
fix --range causing crashes ( #4557 )
...
regression caused by a383eca7
1 year ago
3528974459
[instagram] handle exceptions due to missing media ( #4555 )
1 year ago
4963bb9b30
[reddit] improve comment metadata v2 ( #4482 )
...
provide main submission metadata at the top level
and comment metadata inside the 'comment' field,
i.e. the other way round than in 1710f1e9
1 year ago
7592c5e566
[patreon] fix extraction ( #4547 )
1 year ago
0655ce1bae
[mangakakalot] update domain
...
the old one still works, but it incurs a redirect
1 year ago
3ecb512722
send Referer headers by default
1 year ago
cb4798f07a
[architizer] fix extraction ( #4537 )
1 year ago
6178177227
[twitter] fix '_extractor' of following results ( #4536 )
...
regression from 20ed647f
1 year ago
d13c82eff1
[kemonoparty] update favorites API endpoint ( #4522 )
1 year ago
27ec653991
fix bug in test_init and update example URLs
1 year ago
24a1d46391
[mastodon] support '/@USER/following' URLs
...
Previously, only '/users/USER/following' got matched.
1 year ago
9f75713e00
[recursive] simplify
1 year ago
899df8f237
remove another '*' for keyword-only arguments
...
076380e0
1 year ago
6ae92da57e
Merge branch 'tests'
1 year ago
32da3c70d3
[behance] handle videos without 'renditions' ( #4523 )
1 year ago
ae5e049c4f
[redgifs] provide 'collection' metadata in a separate field ( #4508 )
...
instead of overwriting the actual metadata
1 year ago
1710f1e983
[reddit] improve comment metadata ( #4482 )
...
- provide 'date'
- make metadata of the main submission available as 'submission[…]'
1 year ago
4cdab8074e
update/fix --list-extractors
1 year ago
a453335a9f
remove test results in extractor modules
...
and add generic example URLs
1 year ago
1d2b5d0c60
update test comment positions
...
always put them above the test they're referring to
1 year ago
93a7a89cf6
[formatter] use value of last alternative ( #4492 )
...
fixes {fieldname|''} evaluating to the value of 'keywords-default'
instead of an empty string
1 year ago
f856987297
[subscribestar] fix preview detection ( #4468 )
...
and show a warning message when posts contain previews
1 year ago
4c0b3d5dc5
[twitter] fix crash when 'sortIndex' is None ( #4499 )
1 year ago
f2de70f254
[gfycat] remove module
1 year ago
6eca1fab9b
[gelbooru_v02] support 'xbooru.com' ( #4493 )
1 year ago
23bac772f2
[jpgfish] update domain to 'jpg1.su' ( #4494 )
1 year ago
ceb59e176f
fix default Firefox user agent string
...
note to self: do not trust some random third-party website
1 year ago
8259a5abe4
flake8
1 year ago
0b6e5b8161
[hiperdex] send Referer headers during file downloads ( #4490 )
1 year ago
a05821f8b4
[hiperdex] fix 'manga' metadata
...
remove trailing ' Manga'
1 year ago
03d471a0d4
merge #4481 : [pixiv] handle errors for private novels
1 year ago
8c477f7146
[bunkr] Fixes media-files-pizza.bunkr.ru failed to resolve.
1 year ago
c7e31b2724
Update pixiv.py
1 year ago
d3046561d4
continue downloading when the item is made private on pixiv
1 year ago
28798594e8
[gfycat] update pagination logic ( #4479 )
...
Some searches do not use cursor based pagination
but an offset based one.
1 year ago
a783c4f0fe
[pornhub] add 'gif' support ( #4463 )
1 year ago
ba842981af
[imagevenue] fix extraction ( #4473 )
1 year ago
7defb24e1e
[reddit] provide video previews if available ( #4322 )
1 year ago
fd65f27ede
[reddit] fix 'preview.redd.it' URLs ( #4470 )
1 year ago
06aaedded5
[twitter] extract 'source' metadata ( #4459 )
1 year ago
14af15bd18
[reddit] download preview for 404ed imgur links ( #4322 )
...
This is a pretty ugly hack as the internal infrastructure doesn't
really support switching from external URL to regular download in
case the former fails, but it kind of works ...
Can be disabled by setting 'reddit.fallback' to 'false'.
1 year ago
d12a5e440a
update docs/supportedsites
1 year ago
3a27150479
[instagram] add 'following' extractor ( #1848 )
1 year ago
e0829ff0fd
[twitter] add 'date_original' metadata for retweets ( #4337 , #4443 )
1 year ago
5ed245317d
[exhentai] add 'fav' option ( #4409 )
...
The name 'favorite' is already taken as extractor subcategory
1 year ago
fd6b413f3c
[exhentai] fix 'domain' option ( #4458 )
...
regression from a383eca7
1 year ago
fdfb22c91f
[instagram] fix video preview archive IDs ( #2135 , #4455 )
1 year ago
92f98e6f5e
'sys.exit' -> 'SystemExit'
1 year ago
410f783a33
implement 'subconfigs' option ( #4440 )
1 year ago
2b88ad19e9
[twitter] accept 'x.com' URLs ( #4452 )
1 year ago
c1c73c0b0e
[pp:ugoira] add '"framerate": "uniform"' ( #4421 )
1 year ago
2a3acd318a
[pp:ugoira] fix high frame rates ( #4421 )
...
only return an output frame rate for non-uniform ugoira
when the frame delay gcd is >= 10, i.e. 100 fps
1 year ago
70bdf32a88
[pp:ugoira] extend 'ffmpeg-output' ( #4421 )
...
- when setting this option to a string value,
pass -hide-banner and -loglevel to FFmpeg
- change default to "error"
1 year ago
8dceea3384
[shimme2] move 'giantessbooru' back into shimmie module ( #4373 )
...
Do the same thing as for 'realbooru' and override 'posts()'
insteadd of using a separate module.
1 year ago
6482f9453b
[behance] fix cookie usage ( #4417 )
1 year ago
d34195b41d
[behance] fix and update 'user' extractor ( #4417 )
1 year ago
4d3cf709da
[behance] add 'date' metadata field ( #4417 )
1 year ago
c689cd9720
[behance] show error for mature content ( #4417 )
1 year ago
33d912490f
merge #4419 : [bunkr] Fix extracting wmv files
1 year ago
01610a6e9e
merge #4412 : [bunkr] fix media domain for cdn9
1 year ago
b19d62263b
merge #4420 : [issuu] fix extraction
1 year ago
6dc8be5e48
[issuu] fix extraction
1 year ago
85a070b9e6
[bunkr] Fix extracting wmv files
1 year ago
3f8ff692a7
[bunkr] fix media domain for cdn9
...
Fixes #4386
1 year ago
d8b21a97bf
[formatter] use 'rpartition' for \fM format strings
...
fixes using absolute module paths like C:\path\module.py on Windows
1 year ago
f9fb276e81
[postprocessor] add 'prepare-after' event ( #4083 )
1 year ago
0ef1fcab20
[postprocessor] update 'finalize' events
...
Add 'finalize-error' and 'finalize-success' events that trigger
depending on whether error(s) did or did not happen.
'finalize' itself now always triggers regardless of error status.
(was supposed to have the same behavior as the new 'finalize-success')
1 year ago
af4bdb62a7
merge #4403 : [downloader:http] close connection when file already exists
1 year ago
15275b3524
[postprocessor:ugoira] restore 'libx264-prevent-odd' ( #4407 )
...
was accidentally removed in commit be9547a5
1 year ago
391a7d74c8
[giantessbooru] fix and move to separate module ( #4373 )
...
too many differences to the other shimmie2 sites
1 year ago
5448268d5c
[downloader:http] close connection when file already exists ( #3748 )
1 year ago
3963dbe5e4
extend 'parent>child' categories
...
continuation of ed21908f
1 year ago
089d1a4f67
[twitter] fix 'TweetWithVisibilityResults' ( #4369 )
1 year ago
a4f7f7da17
add '_dump()' convenience method to Extractor
1 year ago
df5c7ee03e
[deviantart] fix search ( #4384 )
...
send correct usernames instead of 'u'
1 year ago
a60db454af
[sankaku] update/fix API headers
...
'Referer' and 'Origin' were both empty
1 year ago
fb3f0453db
[twitter] improve error messages for single Tweets ( #4369 )
...
also fixes '"quoted": false' not having any effect
1 year ago
541bff5a37
[pururin] fix extraction ( #4375 )
...
- rename 'title_jp' to 'title_ja'
- change type of 'collection', 'convention', and 'scanlator' to list
1 year ago
6a87c314af
[instagram] fix private posts with long shortcodes ( #4362 )
1 year ago
f899fac4c5
[giantessbooru] fix extraction ( #4373 )
...
This does not fix anything Cloudflare related,
just other things caused by a site update.
1 year ago
136283d402
[shimmie2] update base URL pattern
...
to match new giantessbooru URLs
1 year ago
9d67655397
add "ascii+" as a special 'path-restrict' value ( #4371 )
1 year ago
c79359eb3a
[fantia] improve metadata extraction ( #4126 )
...
extract all metadata and URLs before starting to download
1 year ago
48ef062867
fix issues with 'Extractor.finalize()'
...
- prevent crash in InstagramUserExtractor (#4359 )
- call it at the end of every DownloadJob
- add it to tests
1 year ago
ed21908fda
initial support for child extractor options
...
Using "parent-category>child-category" as extractor category in a config
file allows to set options for a child extractor when it was spawned by
that parent.
For example "reddit>gfycat" to set gfycat options for when it was found
in a reddit post.
{
"extractor": {
"gfycat": {
"filename": "regular filename"
},
"reddit>gfycat": {
"filename": "reddit-specific filename"
}
}
}
Note: This does currently not work for most imgur links due to how its
extractor hierarchy is structured.
1 year ago
255d08b79e
add test for 'Extractor.initialize()' ( #4359 )
1 year ago
2bcf0a4c49
[instagram] fix initialization order ( #4359 )
...
regression caused by the changes in a383eca7
1 year ago
7eab101144
[acidimg] fix extraction
...
swap ' and " again (2e309a13
1 year ago
62fce6a75f
[imagehosts] adjust variable names ( #4358 )
...
prefix them with underscores to prevent a clash
with the new 'self.cookies' from d97b8c2f
1 year ago
e8299b459a
[moebooru] match search URLs with empty 'tags' ( #4354 )
1 year ago
7fbc304ae9
[twitter] fix crash on private user ( #4349 )
1 year ago
1ece3b92ff
[mangadex] allow multiple values for 'lang' ( #4093 )
...
This was already possible by setting 'lang' to a list of strings,
but now it can also be done as a more command-line friendly string.
-o lang=fr,it
1 year ago
52053b58f0
[lensdump] fix extraction ( #4352 )
1 year ago
11f71a9cba
remove 'mememuseum' module
...
This was forgotten when adding generic Shimmie2 support in 7865067d
1 year ago
a383eca7f6
decouple extractor initialization
...
Introduce an 'initialize()' function that does the actual init
(session, cookies, config options) and can called separately from
the constructor __init__().
This allows, for example, to adjust config access inside a Job
before most of it already happened when calling 'extractor.find()'.
1 year ago
6c9432165e
add return value to 'PostProcessor._init_archive()'
1 year ago
54d974deb0
add 'python' post processor
...
similar to 'exec' but calls a Python function
1 year ago
1baf83a9e5
[hiperdex] fix for unicode titles ( #4325 )
1 year ago
7da954f810
[flickr] update default API credentials ( #4332 )
...
and add a delay between API requests
1 year ago
a45a17ddb7
[pixiv] ignore 'limit_sanity_level' images ( #4328 )
1 year ago
088e8d5fcf
[pornhub] fix extraction ( #4301 )
1 year ago
d97b8c2fba
consistent cookie-related names
...
- rename every cookie variable or method to 'cookies_*'
- simplify '.session.cookies' to just '.cookies'
- more consistent 'login()' structure
1 year ago
ceebacc9e1
remove 'pyopenssl' option
1 year ago
3c2c7e21dd
merge #4319 : [zerochan] fix 'tags' extraction
1 year ago
0ba8d1f168
merge #4312 : [redgifs] add 'niches' extractor
1 year ago
c5565f79f7
merge #4096 : [danbooru] add support for booru.borvar.art instance
1 year ago
63326e3168
[danbooru] add tests for booruvar
1 year ago
5171d8975c
[E621] support 'e6ai.net' ( #4320 )
1 year ago
a996d936d2
[imagefap] fix pagination ( #3013 )
1 year ago
22099422ca
[deviantart] fix shortened URLs ( #4316 )
1 year ago
90231f2d5a
[twitter] add 'tweet-endpoint' option ( #4307 )
...
use the newer TweetResultByRestId only for guests by default
1 year ago
20ed647f6f
[twitter] add 'user' extractor and 'include' option ( #4275 )
1 year ago
86be197d11
[twitter] remove '/search/adaptive.json'
1 year ago
d52ed2bc5a
[zerochan] fix 'tags' extraction
1 year ago
12cd85658b
[redgifs] add 'niches' extractor
1 year ago
248e8bc699
release version 1.25.8
1 year ago
bc9123cfee
[naverwebtoon] fix 'comic' metadata extraction
1 year ago
ab5dde7221
[mangaread] fix 'tags' extraction
1 year ago
c9a82c9313
[erome] ignore duplicate album IDs
1 year ago
c84397023a
[slideshare] fix extraction
1 year ago
ffbbbd3baf
[gelbooru_v01] 'vidyart' -> 'vidyart2'
1 year ago
e40b90e137
merge #4303 : [gelbooru_v01] fix 'source' ( #4302 )
1 year ago
c6b31a2169
[reddit] set default 0.6s delay between requests ( #4292 )
...
to limit API requests to 100 per minute
https://www.reddit.com/r/redditdev/comments/14nbw6g/
1 year ago
20da41018d
[pornhub] set 'accessAgeDisclaimerPH' cookie ( #4301 )
1 year ago
75757c4ace
[gelbooru_v01] fix 'source' ( #4302 )
1 year ago
2dd6942d1c
[jpgfish] update domain to 'jpeg.pet'
1 year ago
1137b89ed4
[lineblog] remove module
...
"LINE BLOGは2023年6月29日をもちましてサービスを終了いたしました"
1 year ago
86560fe0cd
[bcy] remove module
...
"The website was shut down on July 12, 2023"
https://danbooru.donmai.us/wiki_pages/bcy
1 year ago
fceabee433
[philomena] use API interface class
...
handle 429 errors and retry after 10min (#4288 )
1 year ago
f079d9a703
[reddit] notify users about registering an oauth application
...
(#4292 , #4253 , #3943 )
1 year ago
fb3d1462b1
merge #4291 : [wikifeet] fix 'tag' extraction
1 year ago
0b08e2e8a8
merge #4287 : [twitter] Fix following extractor not getting all users
1 year ago
f6553ffd2f
[twitter] simplify '_pagination_users'
...
- remove 'stop' variable
- call 'cursor.startswith()' only once
1 year ago
1590124aae
[twibooru] fix '--range'
1 year ago
a2111dd025
[wikifeet] fix 'tag' extraction
1 year ago
a1ffa1ff09
[philomena] fix '--range' ( #4288 )
1 year ago
a27dbe8c82
[twitter] use 'TweetResultByRestId' endpoint ( #4250 )
...
allows accessing single Tweets without login
1 year ago
d3d639a159
[twitter] don't treat missing 'TimelineAddEntries' as fatal ( #4278 )
1 year ago
c321c773f2
make the code less ugly
1 year ago
a437a34bcf
fix lint i guess?
1 year ago
6cbc434b54
Fix users pagination
1 year ago
d5b6802774
[seiga] set 'skip_fetish_warning' cookie ( #4242 )
1 year ago
88d1e29401
[bunkr] use '.la' TLD for 'media-files12' servers ( #4147 , #4276 )
1 year ago
f0cb951566
[paheal] unescape 'source'
1 year ago
b480b7076a
[paheal] fix a78f8ce5 for enabled 'metadata' ( #4262 )
1 year ago
384337d3dd
[fantia] send 'X-Requested-With' header only for API requests ( #4273 )
1 year ago
c2ac665ff7
[fantia] send 'X-Requested-With' header ( #4273 )
1 year ago
7444fc125b
[gfycat] implement login support ( #3770 , #4271 )
...
For the record: '/webtoken' and '/weblogin' are not the same ...
1 year ago
e9b9f751bf
[gfycat] support '@me' user ( #3770 , #4271 )
1 year ago
5b59a0d143
update default User-Agent header to Firefox 115 ESR
1 year ago
0556e1ad45
merge #4268 : [newgrounds] extract & pass auth token for login
1 year ago
a16d7c59cb
[newgrounds] access 'response.text' only once
1 year ago
1bf9f52c99
[twitter] add 'ratelimit' option ( #4251 )
1 year ago
f86fdf64a6
[twitter] use GraphQL search by default ( #4264 )
1 year ago
1d4db83d49
[weibo] fix end of cursor based pagination
1 year ago
a78f8ce5b0
[paheal] fix extraction ( #4262 )
...
swap ' and "
1 year ago
9576652fa5
extract & pass auth token for newgrounds
1 year ago
5457007dd3
release version 1.25.7
1 year ago
3d8de383bf
[mangapark] extract 'source_id' for manga
...
forgot to add this to 6ae3101f
1 year ago
6ae3101fd0
[mangapark] add 'source' option ( #3969 )
1 year ago
c45a913bfd
[flickr] add 'exif' option
1 year ago
3845c0256d
[sankaku] improve warnings for unavailable posts
1 year ago
46cae04aa3
[piczel] update API server ( #4244 )
1 year ago
3479646f65
[mangapark] update and fix 'manga' extractor ( #3969 )
...
TODO:
- non-English chapters
- 'source' option
1 year ago
10786c657e
[mangapark] update and fix 'chapter' extractor ( #3969 )
1 year ago
9c31c2daef
[poipiku] improve error detection ( #4206 )
1 year ago
260ff55e19
[senmanga] ensure download URLs have a scheme ( #4235 )
1 year ago
ccbc1a1d55
[flickr] add 'metadata' option ( #4227 )
1 year ago
c1cce4a80b
[twitter] extend 'conversations' option ( #4211 )
1 year ago
b6c959744d
[furaffinity] improve 'description' HTML ( #4224 )
...
- ignore header
- include footer and closing <div> if present
1 year ago
8357acf359
[gelbooru_v01] replace 'extract_all()' with 'extract_from()'
...
It's even slightly faster, especially on Python before 3.11
1 year ago
068aa26c3e
[gelbooru_v01] fix '--range' ( #4167 )
1 year ago
2052e7ce59
[hentaifox] fix titles containing '@' ( #4201 )
1 year ago
92d98697b2
[wallhaven] update API error message
1 year ago
a673998b1e
release version 1.25.6
1 year ago
339fcdb8ad
[wallhaven] handle '429 Too Many Requests' errors ( #4192 )
...
- set 1.4s delay between API requests
(WH allows 45 requests per minute)
- wait and retry on 429 errors
1 year ago
ef9891ec9d
[fantia] extract 'plan' metadata ( #2477 , #4128 )
1 year ago
f8452984fa
[fantia] emit warning for non-visible contents ( #4128 )
1 year ago
dc7af00014
[fantia] refactor
...
- embed response data as hidden '_data' field
(instead of returning/passing 'resp')
- split _get_urls_from_post()
1 year ago
6c8bf9a762
[pornhub] improve redirect handling ( #4188 )
1 year ago
654267a335
[weibo] fix 'json' extension for some videos
1 year ago
ce93c460a6
[formatter] implement 'H' conversion ( #4164 )
...
to remove HTML tags and unescape HTML entities
1 year ago
deff3b434d
[vipergirls] implement login support ( #4166 )
1 year ago
db20a645c5
[vipergirls] use API endpoints ( #4166 )
1 year ago
0b34a444e0
[pixiv:novel] only detect Pixiv embeds ( #4175 )
1 year ago
9f1aee3884
[vipergirls] limit number of requests per second ( #4166 )
1 year ago
21c75d03a3
merge #4133 : [furaffinity] extract 'favorite_id' metadata
1 year ago
5e3a1749c8
[furaffinity] simplify 'favorite_id' assignment
1 year ago
ad882291d3
[instagram] fix retrieving '/tagged' posts ( #4122 )
...
reduce number of retrieved posts per API request from 50 to 20
1 year ago
0a9aaa7a8d
[weibo] prevent fatal exception due to missing video ( #4150 )
1 year ago
ac651c604c
[senmanga] fix and update ( #4160 )
1 year ago
Mike Fährmann