Mike Fährmann
cdc6549fd2
merge #3329 : [8muses] Add 'parts' to album data
...
and fix 'album[url]'
1 year ago
Mike Fährmann
ad760429b1
[8muses] update
1 year ago
Mike Fährmann
d0184fddcf
[twitter] optimize '_extract_twitpic()'
...
- use findall instead of finditer
- store URLs in a dict to discard duplicates
1 year ago
Mike Fährmann
3dc862c7fc
merge #3796 : [twitter] extract TwitPic URLs in text ( #3792 )
1 year ago
Mike Fährmann
243de697b9
merge #3976 : [reddit] support cross-posted media ( #887 , #3586 )
1 year ago
Mike Fährmann
f8c4c5eef9
[reddit] simplify and add tests
1 year ago
thatfuckingbird
822a77d846
[danbooru] add support for booru.borvar.art instance
1 year ago
Mike Fährmann
f3cca50b9e
[mangadex] update links to API docs
1 year ago
Mike Fährmann
65a9f4b124
merge #3950 : [misskey] add 'favorite' extractor
1 year ago
Mike Fährmann
c76f0f3a1b
[misskey] update
...
- rename to 'MisskeyFavoriteExtractor'
- add 'access-token' option to docs
- add test URLs for other instances
- simplify 'pattern'
1 year ago
Mike Fährmann
3fca455b82
[pixiv] add 'embeds' option ( #1241 )
1 year ago
Mike Fährmann
d1f2ef3b7b
[imagechest] update
...
- don't load HTML page when using API
- restructure some code
- add more methods to ImagechestAPI
1 year ago
Mike Fährmann
856f6c10cd
allow for GalleryExtractors to skip loading gallery_url
1 year ago
Mike Fährmann
4fc9675d48
[fanbox] skip 404ed or otherwise invalid posts ( #4088 )
1 year ago
Mike Fährmann
69865dcc05
[formatter] implement slicing strings as bytes ( #4087 )
...
prefixing a slice '[10:30]' with a lowercase b '[b10:30]' encodes
the string to bytes in filesystem encoding before applying the slice
1 year ago
Mike Fährmann
56b8b8cd36
[pixiv] support short novel URLs
...
https://www.pixiv.net/n/ <ID>
1 year ago
Mike Fährmann
e6f55d1555
[imagechest] add API support and 'access-token' option ( #4065 )
1 year ago
Mike Fährmann
77abcf5ab3
[gofile] automatically fetch 'website-token' by default
...
the hardcoded token changed yet again
1 year ago
Mike Fährmann
e3fed9bd17
[tcbscans] update domain to 'tcbscans.com' ( #4080 )
1 year ago
Mike Fährmann
a83983c651
[instagram] add 'order-posts' option ( #4017 , #3993 )
1 year ago
Mike Fährmann
d680623db3
[instagram] add 'order-files' option ( #4017 , #3993 )
1 year ago
Naatie
f9b7a033e0
[misskey] refactor misskey extractor
1 year ago
Naatie
04dbfd994e
[misskey] add my favorites extractor
1 year ago
Mike Fährmann
82a12d6126
[nsfwalbum] detect placeholder images
...
patch by an anonymous contributor
1 year ago
Mike Fährmann
011e4607c3
[poipiku] extract full 'descriptions' ( #4066 )
...
don't cut it off after the first line
1 year ago
Mike Fährmann
5037013e2b
[gofile] update 'website-token' ( #4056 )
1 year ago
Mike Fährmann
6b6bb4be73
[weibo] require numeric IDs to have length >= 10 ( #4059 )
1 year ago
Mike Fährmann
494acabd38
[danbooru] refactor pagination logic ( #4002 )
...
- only use 'b<ID>' when no other order is specified
- support 'a<ID>' when using 'order:id' as tag
1 year ago
Mike Fährmann
fd0e1ffd6e
[danbooru] improve 75666cf9
( #4002 )
...
Search for direct post IDs instead of trying to
replicate the same results as the initial request.
1 year ago
Mike Fährmann
e41e45ff6b
[gofile] add basic password support ( #4056 )
1 year ago
Mike Fährmann
790dd365e1
[postprocessor:exec] support tilde expansion for 'command'
...
https://github.com/mikf/gallery-dl/issues/146#issuecomment-1544733532
1 year ago
Mike Fährmann
2e6cea95db
[cookies] update logging behavior ( #4050 )
...
- only show the same warning/error once
- simplify and capitalize logging messages
1 year ago
Mike Fährmann
20dc13f832
[pixiv] initial 'novel' support ( #1241 , #4044 )
...
supported URLs are
- https://www.pixiv.net/novel/show.php?id= <ID>
- https://www.pixiv.net/novel/series/ <ID>
- https://www.pixiv.net/en/users/ <ID>/novels
1 year ago
Mike Fährmann
c698c3de44
[newgrounds] add default delay between requests ( #4046 )
1 year ago
Mike Fährmann
708f478d15
[danbooru][e621] add 'date' metadata field ( #4047 )
1 year ago
Mike Fährmann
306e13a4d4
release version 1.25.4
1 year ago
Mike Fährmann
35c23a2fd8
merge #4031 : [mangadex] add 'status' and 'tags' metadata
1 year ago
Mike Fährmann
2266fc8cc5
[mangadex] update and extend test results
1 year ago
Janne Alaranta
1ce5dc9e18
fix whitespaces
1 year ago
Janne Alaranta
13dedae09f
add status and tags info to mangadex extractor
1 year ago
Mike Fährmann
be0fa94b2e
[imagechest] load all images when a 'Load More' button is present
...
(#4028 )
1 year ago
Mike Fährmann
7eadcbea70
[4chanarchives] add end condition for 'board' extractor ( #4012 )
1 year ago
Mike Fährmann
1406f7125f
[4chanarchives] add 'thread' and 'board' extractors ( #4012 )
1 year ago
Mike Fährmann
285391df43
add '-C' as short option for '--cookies'
...
and put cookie options into their own section
1 year ago
Mike Fährmann
b9b1cdd71b
add '--cookies-export' command-line option
1 year ago
Mike Fährmann
d12dd3813c
[imgur] fix internal image/album URLs
...
URLs from "link" attributes of newer images/albums were all returned
as 'https://imgur.com/gallery/ ...' instead of the expected format,
causing them to be ignored.
1 year ago
Mike Fährmann
8520de57f0
[imgur] add 'favorite-folder' extractor ( #4016 )
1 year ago
Mike Fährmann
4c1f3b2160
[cookies] simplify '_mac_absolute_time_to_posix()'
...
hardcode UNIX timestamp of 2001-01-01
1 year ago
Mike Fährmann
a14b63d941
support selecting a domain for '--cookies-from-browser'
...
for example 'gallery-dl --cookies-from-browser firefox/twitter.com'
1 year ago
Mike Fährmann
3ca5dac8b6
extend 'cookies-update' functionality
...
Allow writing cookies to a different file than a given cookies.txt,
making it possible to export cookies imported with --cookies-from-browser
To convert browser cookies to cookies.txt format:
gallery-dl --cookies-fr chromium \
-o cookies-update=cookies.txt \
--no-download \
http://example.org/file.jpg
1 year ago
Mike Fährmann
bc6d65d203
implement 'Extractor.config_deprecated()'
...
a version of 'Extractor.config()'
that logs a warning when using a deprecated option name
1 year ago
Mike Fährmann
850df34c31
remove '&' from URL patterns part 2
...
follow-up on 968d3e8465
1 year ago
Mike Fährmann
4d415376d1
[pinterest] fix 'pin.it' extractor
...
it really was just the single '/' at the end of the url_shortener URL
1 year ago
Mike Fährmann
657b6a9100
[pinterest] update endpoint for related board pins
1 year ago
Mike Fährmann
79f47f98dd
[nana] remove module
...
permanently gone since 2023-03-13
1 year ago
Mike Fährmann
0e74df1de8
[420chan] remove module
...
offline since 2022-06-01
1 year ago
Mike Fährmann
7499fa7075
[exhentai] remove and update sad panda check
...
there hasn't been a sad panda in several years
1 year ago
Mike Fährmann
076380e079
remove '*' indicating keyword-only arguments
...
they are kind of unnecessary and
cause a non-insignificant function call overhead (~10%)
1 year ago
Mike Fährmann
0c46758a93
[foolslide] remove 'sensescans.com'
...
group moved to mangadex
https://mangadex.org/group/1071e71d-cc55-4fa6-81d1-4b5913a2fde5/sense-scans
1 year ago
Mike Fährmann
a08fdfac6e
[foolfuuka] add 'archive.palanq.win'
1 year ago
Mike Fährmann
1870df8b23
[foolfuuka] remove 'tokyochronos.net'
1 year ago
Mike Fährmann
ef4e2d8178
[foolfuuka] remove 'archive.alice.al'
1 year ago
Mike Fährmann
57cf942bb1
[config] include exception type in error message
1 year ago
Mike Fährmann
aa731c4298
[ytdl] run yt-dlp tests with latest code from master ( #3989 )
...
Only use PyPI version for Python 3.6, since that's no longer supported
by the current codebase.
1 year ago
Mike Fährmann
6a860876bc
release version 1.25.3
1 year ago
Mike Fährmann
b12dad8df5
[pixiv] fix 'pixivision' extraction
1 year ago
Mike Fährmann
5fb7107f2b
[imxto] fix 'gallery' extraction
...
support both single and double quotes
1 year ago
Mike Fährmann
15d7c5a199
[behance] 'items()' -> 'values()'
...
we only need 'size', 'name' is unnecessary
1 year ago
Mike Fährmann
61a65d5bb9
[ytdl] fix crash due to --geo-bypass deprecation ( #3975 )
1 year ago
Mike Fährmann
0fb580135d
[behance] fix extraction ( #3980 )
1 year ago
Alexandru Vasilescu
d4f8b2fe22
fix: linter issues
1 year ago
Alexandru Vasilescu
1b918bd937
fix(extractor): fix extraction for cross-posted reddit videos and galleries
1 year ago
Mike Fährmann
215028a462
[manganelo] match more minor version separators ( #3972 )
1 year ago
Mike Fährmann
c182094ebf
merge #3748 : [downloader:http] add 'consume-content' option
1 year ago
thatfuckingbird
9f76783ac0
[pixiv] allow sorting by popularity (requires pixiv premium)
1 year ago
Mike Fährmann
7865067d19
[shimmie2] add generic extractors for Shimmie2 sites ( #3734 )
...
add support for
- loudbooru.com (#3734 )
- booru.cavemanon.xyz (#3734 )
- giantessbooru.com (#943 )
- tentaclerape.net
1 year ago
Mike Fährmann
28419bf45a
[itchio] add 'game' extractor ( #3923 )
1 year ago
Mike Fährmann
3905f05f00
[postprocessor:metadata] support putting keys in quotes
...
for mode 'modify' and 'delete'
based on fe41a2b1
1 year ago
Mike Fährmann
7459e4abce
[postprocessor:metadata] fix traversing more than 1 level deep
...
for mode 'modify' and 'delete'
1 year ago
Mike Fährmann
5297ee0cd9
[tumblr] add 'day' extractor ( #3951 )
1 year ago
Mike Fährmann
de670bd7de
[tumblr] update pagination logic ( #2191 )
1 year ago
ClosedPort22
6f4a843fba
[downloader:http] release connection before logging messages
...
This allows connections to be properly released when using 'actions'
feature.
1 year ago
Mike Fährmann
98c9fdb414
[deviantart] revert e9353c63; retry downloads with private token
1 year ago
Mike Fährmann
5d7435e803
[nitter] extract user IDs from encoded banner URLs
...
still requires a banner to be present to begin with
1 year ago
Mike Fährmann
7f25cab56e
[sankaku] support post URLs with MD5 hashes ( #3952 )
1 year ago
Mike Fährmann
a05120412a
[oauth] catch exception from 'webbrowser.get()' ( #3947 )
...
It raises an exception instead of returning None
when no runnable browser is available.
1 year ago
Mike Fährmann
3fc2223893
merge #3935 : [reddit] match 'preview.redd.it' URLs
1 year ago
Mike Fährmann
1d505b39f8
[twitter] support 'profile-conversation' entries ( #3938 )
1 year ago
Mike Fährmann
aaf58a1259
[imgur] document 'client-id' option ( #3937 )
1 year ago
Mike Fährmann
202f5d86a7
[reddit] ignore 'id-max' value "zik0zj"/2147483647
...
(#3939 , #3862 , #3697 , #3606 , #3546 , #3521 , #3412 )
1 year ago
Mike Fährmann
8586ee81be
[nana] fix 'keyword' tests
1 year ago
ClosedPort22
cd4bfb0dd1
[reddit] match 'preview.redd.it' URLs
1 year ago
Mike Fährmann
faca32a850
[sankaku] sanitize 'date:…' tags ( #1790 )
1 year ago
Mike Fährmann
6f1e34ec69
[vipergirls] add 'thread' and 'post' extractors
...
(#731 , #2720 , #3812 )
1 year ago
Mike Fährmann
81bd2af83e
[2chen] update domain to sturdychan.help
1 year ago
Mike Fährmann
f500b45b5e
[twitter] improve 480bc34e
...
only check for double user assignment where necessary
1 year ago
Mike Fährmann
5b635f2317
[imxto] add 'gallery' extractor ( #1289 )
1 year ago
Mike Fährmann
359e31e462
[nozomi] update file URLs ( #3925 )
...
Static images are now only available in WebP format over the 'w'
subdomain. GIFs also got their own 'g' subdomain.
1 year ago
Mike Fährmann
2dfd4a3de2
[imagefap] extract 'categories' metadata and fix empty 'tags'
1 year ago
Mike Fährmann
480bc34e54
[twitter] do not overwrite previously assigned users ( #3922 )
1 year ago
Mike Fährmann
02ec5bb8e5
[imagefap] extract 'description' metadata ( #3905 )
1 year ago
Mike Fährmann
842f964c49
release version 1.25.2
1 year ago
Mike Fährmann
d253a3c542
merge #3841 : [urlshortener] add support for bit.ly & t.co
1 year ago
Mike Fährmann
5e63942b37
[urlshortener] update
1 year ago
Mike Fährmann
2edcdee32f
[downloader:http] add MIME type and signature for .heic files
...
(#3915 )
https://github.com/strukturag/libheif/issues/83
1 year ago
Mike Fährmann
c45f09d2a8
[imagechest] fix extraction ( #3914 )
1 year ago
Mike Fährmann
2cd4411ff8
[nitter] extract videos from 'source' elements ( #3912 )
1 year ago
Mike Fährmann
9501579279
[sexcom] fix fetching HD videos
1 year ago
Mike Fährmann
a2f7274eae
[sexcom] fix pagination ( #3906 )
1 year ago
Mike Fährmann
e9353c63d6
[deviantart] keep using private access tokens
...
for deviations returned from a private API call
also fixes a bug from 0a7eee3e
where '_pagination()'
would never switch from unspecified (None) to private access token
1 year ago
Mike Fährmann
e70af6a550
[hentaifoundry] do not update filters when cookies are provided
1 year ago
Mike Fährmann
9c29c904c7
[mastodon] try to get account IDs without access token
...
Try to query the public '/api/v1/accounts/lookup' endpoint
and fall back to '/v1/accounts/search' if it returns an error.
'/api/v1/accounts/lookup' is available since Mastodon v3.4.0.
The version of an instance can be found at '/api/v1/instance'.
1 year ago
Mike Fährmann
1614c5c4bf
[generic] write regular expressions without 'x' flags
1 year ago
Mike Fährmann
d84a617273
[hentaifoundry] fix setting content filters ( #3887 )
1 year ago
ClosedPort22
875485313f
[urlshortener] force HTTPS
1 year ago
Mike Fährmann
0a7eee3ee0
[deviantart] add 'public' option
1 year ago
Mike Fährmann
f5a59c4170
[twitter] add 'date_bookmarked' metadata ( #3816 )
1 year ago
Mike Fährmann
1c1f6fdc80
[twitter] fix regression from 160335ad
...
Tweets from 'homeConversation' or 'conversationthread' entries do not
contain a 'sortIndex' field. Accessing it raises a KeyError and would
erroneously get them labeled as 'deleted'.
1 year ago
Mike Fährmann
160335ad44
[twitter] add 'date_liked' metadata for liked Tweets ( #3816 )
1 year ago
Mike Fährmann
6d850ce629
[twitter] calculate 'date' from Tweet IDs
...
20 times faster than parsing 'created_at'
1 year ago
Mike Fährmann
25949bd767
merge #3871 : [hotleak] Fix downloading of creators whose name starts with a category name
1 year ago
Mike Fährmann
dbe06cdba1
[twitter] warn about 'withheld' Tweets and users ( #3864 )
1 year ago
Mike Fährmann
3cc1dd1572
[twitter] update query hashes
1 year ago
Mike Fährmann
3846ce0de5
[twitter] update to bookmark timeline v2 ( #3859 )
1 year ago
Mike Fährmann
34699fbf64
[deviantart:search] detect login redirects ( #3860 )
1 year ago
Mike Fährmann
e6cb92864a
[twitter] allow setting custom features per API endpoint
1 year ago
Balgden
4b141cce66
Fix indentation
1 year ago
Balgden
bbc5977121
Fix line length
1 year ago
Balgden
ffd30abcb3
[hotleak] Fix downloading of creators whose name starts with a category name
...
E.g. `hot4lexi` would start downloading the `hot` section by mistake
This happened because the regex had a negative lookahead for the category names, but didn't ensure that they where followed by either end-of-string or a slash.
1 year ago
Mike Fährmann
5ca9d55595
merge #3870 : [blogger] update 'sub' regex to get the highest resolution url
1 year ago
Mike Fährmann
fd7ce4c081
merge #3868 : [shopify] fix 'collection' extractor
1 year ago
Mike Fährmann
135ac9c302
merge #3854 : [twitter] fix: graphql_timeline_v2_bookmark_timeline cannot be null
1 year ago
enduser420
bbb1e34c34
[blogger] update sub regex
1 year ago
enduser420
96e3dd2128
[shopify] fix 'collection' extractor
1 year ago
Mike Fährmann
ac97aca99c
[realbooru] fix extraction
...
get file URLs from HTML pages
1 year ago
Mike Fährmann
75666cf9c3
[danbooru] reduce API requests for fetching extended 'metadata'
...
Instead of using one additional API request per post object (N+1),
this requires only one request per 200-post batch.
1 year ago
ClosedPort22
775d2ac999
[downloader:http] improve error logging when releasing connection
1 year ago
Amer Jazaerli
bebbff6578
fix: graphql_timeline_v2_bookmark_timeline cannot be null
...
twitter: 400 Bad Request (The following features cannot be null: graphql_timeline_v2_bookmark_timeline)
1 year ago
ClosedPort22
71b26adb9b
[urlshortener] add tinyurl.com as an example
2 years ago
Mike Fährmann
421db26aff
[bunkr] update domain to 'bunkr.la'
2 years ago
ClosedPort22
9e2a945013
[urlshortener] add support for bit.ly & t.co
2 years ago
Mike Fährmann
82f83c18e8
release version 1.25.1
2 years ago
Mike Fährmann
9b5e7ce8b9
[hiperdex] fix extraction
2 years ago
Mike Fährmann
89a67c45e0
[nitter] support nitter.it ( #3819 )
2 years ago
Mike Fährmann
88f29a751d
[nitter] skip broadcasts
...
instead of downloading an "Unsupported feature" HTML page
2 years ago
Mike Fährmann
1e013eba5a
[nitter] fix extraction for instances without user banners
2 years ago
Mike Fährmann
d94aa1ee02
[gelbooru] fix --range for favorites ( #3704 )
2 years ago
Mike Fährmann
1f82b00b8f
[gelbooru] fix and improve --range for pools
2 years ago
ClosedPort22
1a977f0f62
[downloader:http] handle exceptions in 'validate'
...
This isn't strictly necessary for 'exhentai.py', but it improves
efficiency when the adapter is reused
2 years ago
Mike Fährmann
197882cf12
[twitter] add 'hashtag' extractor ( #3783 )
2 years ago
Mike Fährmann
082d55de16
fix circular reference detection for -K
2 years ago
Mike Fährmann
2ab66ad899
update -K output to include quotes around keys
2 years ago
Mike Fährmann
fe41a2b159
[formatter] support putting keys in quotes
...
i.e. obj["key"] or obj['key']
as in f-strings
2 years ago
Mike Fährmann
46fdf46f21
[formatter] support loading an f-string from a template file
...
"\fTF ~/path/to/file.txt"
2 years ago
Mike Fährmann
1a4d4a799b
[formatter] support filesystem paths for \fM
2 years ago
Mike Fährmann
9789ebac52
[naverwebtoon] fix extraction ( #3729 )
2 years ago
Mike Fährmann
72f1f16eb2
[weibo] support 'mix_media_info' entries ( #3793 )
2 years ago
ClosedPort22
d4fb4ff47f
[twitter] extract TwitPic URLs in text ( #3792 )
...
also ignore previously seen URLs
2 years ago
Mike Fährmann
00f0233b28
[postprocessor:metadata] add 'skip' option ( #3786 )
2 years ago
Mike Fährmann
2bb937014f
[twitter] fall back to legacy /media endpoint when not logged in
2 years ago
Mike Fährmann
b68094d326
[twitter] support 'note_tweet's
2 years ago
Mike Fährmann
3dcabc97ed
[twitter] update API endpoints and parameters
2 years ago
Mike Fährmann
a1ca2404f9
add 'globals' instead of overwriting the default ( #3773 )
2 years ago
Mike Fährmann
dcb8af659a
[gelbooru] extract favorites without needing cookies ( #3704 )
...
TODO: fix --range
2 years ago
Mike Fährmann
b756dc13aa
[gelbooru] warn about missing cookies for favorites ( #3704 )
...
and add docstring so it shows up in --list-extractors
2 years ago
Mike Fährmann
17bd053d94
[hiperdex] fix extraction ( #3768 )
2 years ago
Mike Fährmann
f7ce33c85c
[output] set 'errors=replace' for output streams ( #3765 )
...
fixes regression from e480a933
2 years ago
Mike Fährmann
a14a2d6e59
release version 1.25.0
2 years ago
ClosedPort22
fcaeaf539c
[downloader:http] handle exceptions while consuming content
2 years ago
Mike Fährmann
4235d412c4
implement 'actions'
...
continuation of d37e7f48
but more versatile and extendable
Example:
"actions": [
# change debug messages to info
["debug", "level ~info"],
# change exit status to a non-zero value
["info:^No results for", "status |= 1"],
# exit with status 2 on 429
["warning:429", "exit 2"],
# restart extractor when no cookies found
["warning:^[Nn]o .*cookies", "restart"]
]
2 years ago
Mike Fährmann
817fc0fbd1
[nitter] remove nitter.pussthecat.org
...
"Shutdown"
2 years ago
Mike Fährmann
67ec91cdbd
[downloader:http] change '_http_retry' to accept a Python function
...
and rename '_http_retry_codes' to '_http_retry'
(#3569 )
2 years ago
Mike Fährmann
175822e065
merge #3738 : [generic] add tests
2 years ago
Mike Fährmann
4883420e67
[generic] revert pattern change
2 years ago
ClosedPort22
df77271438
[downloader:http] add 'consume-content' option
...
* fix connection not being released when the response is neither
successful nor retried
* add the ability to consume the HTTP response body instead of closing
the connection
reference:
https://docs.python-requests.org/en/latest/user/advanced/#body-content-workflow
2 years ago
Mike Fährmann
9037128315
[twitter] fix some 'original' retweets not downloading ( #3744 )
2 years ago
Mike Fährmann
ea3d95e7e8
merge #3740 : [deviantart] add support for fxdeviantart.com URLs
2 years ago
Mike Fährmann
9abcb2b6e5
update headers and ciphers for '"browser": "chrome"'
2 years ago
ClosedPort22
c489aecb3e
[deviantart] add support for fxdeviantart.com URLs
...
fxdeviantart.com is a service that fixes embeds on Discord, similar to
fxtwitter.com
2 years ago
ClosedPort22
34a7fab0e2
[generic] add support for IDNs
...
(internationalized domain name)
2 years ago
Mike Fährmann
c9a7345228
[newgrounds] prevent archive ID overlap ( #3681 )
...
add an 'i' and 'a' prefix to image and audio files
(/art/view/, /audio/listen/)
since their numeric ID may conflict with movies and other media
2 years ago
Mike Fährmann
8148c2a097
[downloader:ytdl] prevent exception on empty results
...
a7c7953107 (commitcomment-92042240)
2 years ago
Mike Fährmann
da9840a39d
[reddit] update 'videos' option ( #3712 )
...
- add 'dash' to directly extract DASH manifest URLs
(was default behavior since a7c79531
)
- change default strategy back to before a7c79531
- disable 'Falling back on generic information extractor' warning
2 years ago
Mike Fährmann
8f8b4de0e8
[ytdl] fix '--parse-metadata' ( #3663 )
2 years ago
Mike Fährmann
11df3a021d
[formatter] enclose f-strings with """ instead of '''
2 years ago
Mike Fährmann
baf41d7437
[misskey] update ( #3717 )
...
- add module docstring
- add options to docs/gallery-dl.conf
2 years ago
Mike Fährmann
7610d9cf82
merge #3675 : [pixiv] fix --write-tags for '"tags": "original"'
2 years ago
Mike Fährmann
6762d99515
merge #3717 : [misskey] add misskey extractors
2 years ago
Mike Fährmann
b8a702929d
[oauth] import extractor modules on demand
2 years ago
Mike Fährmann
dd88740ec7
replace remaining instances of base64 with binascii
2 years ago
enduser420
e1867cf5eb
[misskey] add 'renotes' and 'replies' options
2 years ago
enduser420
a95b5e0d8e
[misskey] add misskey extractors
2 years ago
Mike Fährmann
0d142e403c
[szurubooru] add 'tag' and 'post' extractors ( #3583 , #3713 )
2 years ago
Mike Fährmann
075c965512
add '--config-create' command-line option
...
(#2333 )
2 years ago
Mike Fährmann
26d06e0bb2
move executable check into util.py
2 years ago
Mike Fährmann
de2f35d068
simplify config.load()
2 years ago
Mike Fährmann
632d5d7745
allow loading config files in TOML format with --config-toml
2 years ago
Mike Fährmann
9e870eb930
rename --ignore-config to --config-ignore
...
--ignore-config still works as before,
but is no longer shown by --help
2 years ago
Mike Fährmann
d66257f2c8
improve option.Formatter performance
...
as always, only a very marginal difference,
but it still uses less resources than before
2 years ago
Mike Fährmann
d788e6c60c
implement 'globals' option
2 years ago
Mike Fährmann
b14f8d5817
[gelbooru] add 'favorite' extractor ( #3704 )
...
requires logged in cookies to work
2 years ago
Mike Fährmann
e480a93337
add 'output.stdout', '.stdin', and '.stderr' options
...
(#1621 , #2152 , #2529 )
Allow setting custom input/output encodings and options
without having to rely on Python's defaults.
2 years ago
Mike Fährmann
a70a3e5da6
[mangasee] extract 'author' and 'genre' metadata ( #3703 )
...
Both are lists/arrays. Use {author!S} or {genre:J, } to format them.
2 years ago
Mike Fährmann
6b03506655
[deviantart] allow searching when not logged in
2 years ago
Mike Fährmann
511a051705
[fanbox] fix crash with missing images ( #3673 )
2 years ago
Mike Fährmann
3fa456d989
[deviantart] remove mature scraps warning ( #3691 )
...
warn about private deviations
when paginating over eclipse results
2 years ago
Mike Fährmann
51301e0c31
replace remaining time.sleep() calls
...
with Extractor.sleep() or request_interval
2 years ago
Mike Fährmann
6ed4309aba
[deviantart] add 'gallery-search' extractor ( #1695 )
2 years ago
Mike Fährmann
3d8777fbc1
move user agent string to util.py
2 years ago
Mike Fährmann
56039d2456
add 'hash_md5' and 'hash_sha1' functions ( #3679 )
...
... to global eval namespace
2 years ago
Mike Fährmann
e1df7f73b1
[deviantart] add 'search' extractor
...
(#538 , #1264 , #2954 , #2970 , #3577 )
Requires login to fetch any results, since the API endpoint raises an
error for not logged in requests.
TODO: parse HTML search results
2 years ago
Gray Manley
f33ac885a6
[pixiv] fix tag write when set to original
2 years ago
Mike Fährmann
4f029ab38b
[pornpics] support '/pornstar' and '/channels' listings
...
- fix docstring (#3671 )
- simplify code
2 years ago
Mike Fährmann
cbe4769246
[danbooru] use gallery-dl UA ( #3665 )
...
this removes the ability to set a custom UA via 'user-agent' option
for extractor requests
2 years ago
Mike Fährmann
253ac08203
pre-define and use 'gallery-dö/<version>' UA string
2 years ago
Mike Fährmann
b4899c266f
merge #3656 : [deviantart] fix crash when handling deleted deviations in status updates
2 years ago
Mike Fährmann
bb11c2a576
merge #3662 : [redgifs] add 'collection' extractors
2 years ago
Mike Fährmann
884f1848d6
[redgifs] fix syntax for older Python versions
...
and update docs/supportedsites
2 years ago
Mike Fährmann
725baedad3
[deviantart] use '/collections/all' endpoint for favorites
...
(#3666 ,#3668)
2 years ago
Mike Fährmann
2bd8f2f4bd
[pornpics] add 'search' and 'tag' extractors
...
(#263 , #3544 , #3654 )
2 years ago
Mike Fährmann
79bc82884c
[pornpics] add 'gallery' extractor ( #263 , #3544 , #3654 )
2 years ago
Mike Fährmann
7bdc1d6d3d
[manganelo] update and fix metadata extraction
2 years ago
Mike Fährmann
363bb76dff
[manganelo] simplify URL pattern
2 years ago
enduser420
b28bd9789e
[redgifs] add 'collection' extractors
2 years ago
ClosedPort22
f4e211356d
[deviantart] slight refactor
2 years ago
Mike Fährmann
bd5d08abbc
[catbox] add 'file' extractor ( #3570 )
2 years ago
Mike Fährmann
8e1e8a5bea
[soundgasm] rewrite ( #3578 )
...
use a more standard extractor structure to make -A work as expected
2 years ago
Mike Fährmann
0b93420a81
[pinterest] unescape search terms ( #3621 )
2 years ago
Mike Fährmann
ad96e70546
[bunkr] fix extraction ( #3636 , #3655 )
2 years ago
Mike Fährmann
9335d55bbc
[manganelo] support mobile-only chapters
2 years ago
ClosedPort22
a74114ef7a
[deviantart] fix crash when handling deleted deviations
...
in status updates
2 years ago
Mike Fährmann
75570ad3f1
[oauth] remove stray 'exit()' ( #3628 )
...
- bug from 70ce45d9
- broke oauth:tumblr, oauth:flickr, and oauth:smugmug
2 years ago
Mike Fährmann
d37e7f4898
add 'hooks' option
...
Very much a work in progress.
At the moment, it allows to
- wait and restart an extractor (#3338 )
- change the exit code (#3630 )
- change the log level of a logging message
based on the contents of a logging message
2 years ago
Mike Fährmann
8fb043e8ff
[tumblr] raise more detailed errors for dashboard-only blogs
...
(#3628 )
2 years ago
Mike Fährmann
d4232f3a8b
implement restarting an extractor ( #3338 )
2 years ago
Mike Fährmann
ce996dd21b
[poipiku] warn about incorrect passwords ( #3646 )
2 years ago
Mike Fährmann
70ce45d965
[oauth] use default name for browsers without 'name' attribute
...
(#3645 )
Seem to only be an issue for MacOSXOSAScript before Python 3.11.
d12bec6993
2 years ago
Mike Fährmann
1aae72773f
put argument init on separate lines
2 years ago
Mike Fährmann
2a53e6445c
[bunkr] update domain ( #3636 )
2 years ago
Mike Fährmann
5503ac4d5e
replace json.dumps with direct calls to JSONEncoder.encode
2 years ago
Mike Fährmann
dd884b02ee
replace json.loads with direct calls to JSONDecoder.decode
2 years ago
Mike Fährmann
b7337d810e
[postprocessor:metadata] add 'sort' and 'separators' options
2 years ago
Mike Fährmann
8805bd38ab
merge #3622 : [imagetwist] add phun.imagetwist.com and imagehaha.com support
2 years ago
Mike Fährmann
706ec70e89
[imagetwist] simplify pattern and add tests
2 years ago
Mike Fährmann
f2e91732ae
[instagram] add 'user' metadata field ( #3107 )
...
at the moment only for URLs that need to translate user name to ID
2 years ago
Mike Fährmann
3436c6b117
[postprocessor:metadata] speed up JSON encoding
2 years ago
Prinz23
29f0830b53
[imagetwist] add phun.imagetwist.com and imagehaha.com alias to imagetwist extractor
2 years ago
Mike Fährmann
762a68996b
implement 'archive-pragma' option
2 years ago
Mike Fährmann
bbf0911a46
[e621] implement 'notes' and 'pools' metadata extraction
...
(#3425 )
2 years ago
Mike Fährmann
925b467496
split e621 from danbooru module ( #3425 )
2 years ago
Mike Fährmann
1ae48a54f8
[twitter] add 'transform' option
2 years ago
Mike Fährmann
78d3960a31
[postprocessor:exec] implement archive options ( #3584 )
2 years ago
Mike Fährmann
489c51cecc
[telegraph] fix extraction when images not in <figure> ( #3590 )
2 years ago
Mike Fährmann
0f7e6c422a
merge #3596 : [shopify] support ohpolly.com
2 years ago
enduser420
fcf7030b85
[shopify] support ohpolly.com
2 years ago
Mike Fährmann
a6a631f992
merge #3589 : [redgifs] support v3 URLs
2 years ago
Mike Fährmann
137a395ae0
[imagefap] fix infinite pagination loop ( #3594 )
2 years ago
Mike Fährmann
3c708ade8f
[imagefap] fix metadata extraction
2 years ago
Mike Fährmann
17e24eacf0
[imagefap] update 'gallery' URLs ( #3595 )
2 years ago
Mike Fährmann
d16873941c
[downloader:http] use 'time.monotonic()'
2 years ago
Mike Fährmann
c2bc70593e
implement ability to load external extractor classes
...
- -X/--extractors
- extractor.module-sources
2 years ago
enduser420
a18f627bfc
[redgifs] support v3 URLs
2 years ago
Mike Fährmann
9ec627c760
release version 1.24.5
2 years ago
Mike Fährmann
13a90969c7
merge #3575 : [nudecollect] add 'image' and 'album' extractors
2 years ago
Mike Fährmann
aacd27e4ef
merge #3581 : [hotleak] fix video URLs
2 years ago
Mike Fährmann
abc3619feb
[lexica] add 'search' extractor ( #3567 )
2 years ago
Mike Fährmann
7c9b1ec830
[hotleak] optimize decoding video URLs
...
- use binascii module
- combine slice and reverse step
2 years ago
nifnat
f14dbfe079
Make decode_video_url static (used in both post and creator extractor).
2 years ago
nifnat
bd23a701f3
Tidy up code.
2 years ago
nifnat
7f34f99a26
Reverse engineered obfuscated JS function and reimplemented in python.
2 years ago
Mike Fährmann
0d818d3540
[fantia] send 'X-CSRF-Token' headers ( #3576 )
2 years ago
Mike Fährmann
f58215705a
add '-O/--postprocessor-option' command-line option ( #3565 )
2 years ago
enduser420
2a5903dc16
[nudecollect] add 'image' and 'album' extractors
2 years ago
Mike Fährmann
c8fdd5096e
merge #3571 : [bunkr] Fix extracting mkv and ts files
2 years ago
Mike Fährmann
58c008e30a
[hiperdex] update domain ( #3572 )
2 years ago
Luc Ritchie
842064e597
[bunkr] Fix extracting ts files
2 years ago
Luc Ritchie
99ca0437e4
[bunkr] Fix extracting mkv files
2 years ago
Mike Fährmann
76b01b64cf
[kemonoparty] remove MD5 hash extraction ( #3531 )
...
This partially reverts commit 20d6194ffa
.
2 years ago
Mike Fährmann
09fb212414
[philomena] match URLs with www subdomain
2 years ago
Mike Fährmann
7e2fd2e573
merge #3560 : [deviantart] add support for /deviation/ and fav.me URLs
2 years ago
Mike Fährmann
caae8fefe1
merge #3541 : [deviantart] add extractor for status updates
2 years ago
ClosedPort22
c90b4ea8d9
[deviantart] add support for fav.me URLs
2 years ago
Mike Fährmann
d63af4f3d3
merge #3555 : [generic] fix regex for non-src image URLs
2 years ago
Mike Fährmann
8993b10751
[mastodon] add 'num' and 'count' metadata fields ( #3517 )
2 years ago
Mike Fährmann
d817d23ccb
[instagram] update csrf token handling
...
- update internal value according to cookie
- do not send a second 'csrftoken' cookie
2 years ago
Mike Fährmann
00b94946b3
[instagram] show -o cursor=… after every error ( #3440 )
2 years ago
ClosedPort22
674c719646
[deviantart] refactor base36 conversion
2 years ago
ClosedPort22
293abb8921
[deviantart] add support for /deviation/ URLs
2 years ago
thatfuckingbird
8cfeed78b1
[generic] fix regex for non-src image URLs
2 years ago
Mike Fährmann
fc6ea8ee5c
[instagram] update API domain and headers
2 years ago
ClosedPort22
597b89245e
[deviantart] misc improvements to status extractor
...
- relax regex pattern
- handle invalid 'items' field
- add a test for shared sta.sh item
Co-authored-by: Mike Fährmann <mike_faehrmann@web.de>
2 years ago
Mike Fährmann
137de090dd
merge #3549 : [twitter] fix search ( #3536 )
2 years ago
Mike Fährmann
02e314c1b6
merge #3537 : [wikifeet/wikifeetx] add 'gallery' extractor
2 years ago
Mike Fährmann
568112dfbb
[oauth] improve output
...
- show which api key / client id gets used (#3518 )
- show in which browser authorization URLs gets opened in
2 years ago
ClosedPort22
ab58c375b4
[twitter] fix search ( #3536 )
...
- partially revert 18fe4b334d
- properly search for cursor when processing 'replaceEntry'
2 years ago
Mike Fährmann
df91ebb945
[oauth] simplify OAuth 1.0a init
2 years ago
ClosedPort22
013733c9e9
[deviantart] fix index fields for embedded/shared images
2 years ago
ClosedPort22
c4aeca7a5a
[deviantart] improve handling of statuses
...
- recursively yield statuses
- ignore items with missing or unexpected field(s)
2 years ago
ClosedPort22
3b32671fbd
[deviantart] add extractor for status updates
...
extract user status updates using the '/user/statuses/' endpoint
2 years ago
Mike Fährmann
107c60c973
[sankaku] update URL pattern ( #3523 )
...
match tag searches with language codes without a trailing slash
2 years ago