Mike Fährmann
ef4e2d8178
[foolfuuka] remove 'archive.alice.al'
1 year ago
Mike Fährmann
b12dad8df5
[pixiv] fix 'pixivision' extraction
1 year ago
Mike Fährmann
5fb7107f2b
[imxto] fix 'gallery' extraction
...
support both single and double quotes
1 year ago
Mike Fährmann
15d7c5a199
[behance] 'items()' -> 'values()'
...
we only need 'size', 'name' is unnecessary
1 year ago
Mike Fährmann
0fb580135d
[behance] fix extraction ( #3980 )
1 year ago
Alexandru Vasilescu
d4f8b2fe22
fix: linter issues
1 year ago
Alexandru Vasilescu
1b918bd937
fix(extractor): fix extraction for cross-posted reddit videos and galleries
1 year ago
Mike Fährmann
215028a462
[manganelo] match more minor version separators ( #3972 )
1 year ago
thatfuckingbird
9f76783ac0
[pixiv] allow sorting by popularity (requires pixiv premium)
1 year ago
Mike Fährmann
7865067d19
[shimmie2] add generic extractors for Shimmie2 sites ( #3734 )
...
add support for
- loudbooru.com (#3734 )
- booru.cavemanon.xyz (#3734 )
- giantessbooru.com (#943 )
- tentaclerape.net
1 year ago
Mike Fährmann
28419bf45a
[itchio] add 'game' extractor ( #3923 )
1 year ago
Mike Fährmann
5297ee0cd9
[tumblr] add 'day' extractor ( #3951 )
1 year ago
Mike Fährmann
de670bd7de
[tumblr] update pagination logic ( #2191 )
1 year ago
Mike Fährmann
98c9fdb414
[deviantart] revert e9353c63; retry downloads with private token
1 year ago
Mike Fährmann
5d7435e803
[nitter] extract user IDs from encoded banner URLs
...
still requires a banner to be present to begin with
1 year ago
Mike Fährmann
7f25cab56e
[sankaku] support post URLs with MD5 hashes ( #3952 )
1 year ago
Mike Fährmann
a05120412a
[oauth] catch exception from 'webbrowser.get()' ( #3947 )
...
It raises an exception instead of returning None
when no runnable browser is available.
1 year ago
Mike Fährmann
3fc2223893
merge #3935 : [reddit] match 'preview.redd.it' URLs
1 year ago
Mike Fährmann
1d505b39f8
[twitter] support 'profile-conversation' entries ( #3938 )
1 year ago
Mike Fährmann
aaf58a1259
[imgur] document 'client-id' option ( #3937 )
1 year ago
Mike Fährmann
202f5d86a7
[reddit] ignore 'id-max' value "zik0zj"/2147483647
...
(#3939 , #3862 , #3697 , #3606 , #3546 , #3521 , #3412 )
1 year ago
Mike Fährmann
8586ee81be
[nana] fix 'keyword' tests
1 year ago
ClosedPort22
cd4bfb0dd1
[reddit] match 'preview.redd.it' URLs
1 year ago
Mike Fährmann
faca32a850
[sankaku] sanitize 'date:…' tags ( #1790 )
1 year ago
Mike Fährmann
6f1e34ec69
[vipergirls] add 'thread' and 'post' extractors
...
(#731 , #2720 , #3812 )
1 year ago
Mike Fährmann
81bd2af83e
[2chen] update domain to sturdychan.help
1 year ago
Mike Fährmann
f500b45b5e
[twitter] improve 480bc34e
...
only check for double user assignment where necessary
1 year ago
Mike Fährmann
5b635f2317
[imxto] add 'gallery' extractor ( #1289 )
1 year ago
Mike Fährmann
359e31e462
[nozomi] update file URLs ( #3925 )
...
Static images are now only available in WebP format over the 'w'
subdomain. GIFs also got their own 'g' subdomain.
1 year ago
Mike Fährmann
2dfd4a3de2
[imagefap] extract 'categories' metadata and fix empty 'tags'
1 year ago
Mike Fährmann
480bc34e54
[twitter] do not overwrite previously assigned users ( #3922 )
1 year ago
Mike Fährmann
02ec5bb8e5
[imagefap] extract 'description' metadata ( #3905 )
1 year ago
Mike Fährmann
d253a3c542
merge #3841 : [urlshortener] add support for bit.ly & t.co
1 year ago
Mike Fährmann
5e63942b37
[urlshortener] update
1 year ago
Mike Fährmann
c45f09d2a8
[imagechest] fix extraction ( #3914 )
1 year ago
Mike Fährmann
2cd4411ff8
[nitter] extract videos from 'source' elements ( #3912 )
1 year ago
Mike Fährmann
9501579279
[sexcom] fix fetching HD videos
1 year ago
Mike Fährmann
a2f7274eae
[sexcom] fix pagination ( #3906 )
1 year ago
Mike Fährmann
e9353c63d6
[deviantart] keep using private access tokens
...
for deviations returned from a private API call
also fixes a bug from 0a7eee3e
where '_pagination()'
would never switch from unspecified (None) to private access token
1 year ago
Mike Fährmann
e70af6a550
[hentaifoundry] do not update filters when cookies are provided
1 year ago
Mike Fährmann
9c29c904c7
[mastodon] try to get account IDs without access token
...
Try to query the public '/api/v1/accounts/lookup' endpoint
and fall back to '/v1/accounts/search' if it returns an error.
'/api/v1/accounts/lookup' is available since Mastodon v3.4.0.
The version of an instance can be found at '/api/v1/instance'.
1 year ago
Mike Fährmann
1614c5c4bf
[generic] write regular expressions without 'x' flags
1 year ago
Mike Fährmann
d84a617273
[hentaifoundry] fix setting content filters ( #3887 )
1 year ago
ClosedPort22
875485313f
[urlshortener] force HTTPS
1 year ago
Mike Fährmann
0a7eee3ee0
[deviantart] add 'public' option
1 year ago
Mike Fährmann
f5a59c4170
[twitter] add 'date_bookmarked' metadata ( #3816 )
1 year ago
Mike Fährmann
1c1f6fdc80
[twitter] fix regression from 160335ad
...
Tweets from 'homeConversation' or 'conversationthread' entries do not
contain a 'sortIndex' field. Accessing it raises a KeyError and would
erroneously get them labeled as 'deleted'.
1 year ago
Mike Fährmann
160335ad44
[twitter] add 'date_liked' metadata for liked Tweets ( #3816 )
1 year ago
Mike Fährmann
6d850ce629
[twitter] calculate 'date' from Tweet IDs
...
20 times faster than parsing 'created_at'
1 year ago
Mike Fährmann
25949bd767
merge #3871 : [hotleak] Fix downloading of creators whose name starts with a category name
1 year ago
Mike Fährmann
dbe06cdba1
[twitter] warn about 'withheld' Tweets and users ( #3864 )
1 year ago
Mike Fährmann
3cc1dd1572
[twitter] update query hashes
1 year ago
Mike Fährmann
3846ce0de5
[twitter] update to bookmark timeline v2 ( #3859 )
1 year ago
Mike Fährmann
34699fbf64
[deviantart:search] detect login redirects ( #3860 )
1 year ago
Mike Fährmann
e6cb92864a
[twitter] allow setting custom features per API endpoint
1 year ago
Balgden
4b141cce66
Fix indentation
1 year ago
Balgden
bbc5977121
Fix line length
1 year ago
Balgden
ffd30abcb3
[hotleak] Fix downloading of creators whose name starts with a category name
...
E.g. `hot4lexi` would start downloading the `hot` section by mistake
This happened because the regex had a negative lookahead for the category names, but didn't ensure that they where followed by either end-of-string or a slash.
1 year ago
Mike Fährmann
5ca9d55595
merge #3870 : [blogger] update 'sub' regex to get the highest resolution url
1 year ago
Mike Fährmann
fd7ce4c081
merge #3868 : [shopify] fix 'collection' extractor
1 year ago
Mike Fährmann
135ac9c302
merge #3854 : [twitter] fix: graphql_timeline_v2_bookmark_timeline cannot be null
1 year ago
enduser420
bbb1e34c34
[blogger] update sub regex
1 year ago
enduser420
96e3dd2128
[shopify] fix 'collection' extractor
1 year ago
Mike Fährmann
ac97aca99c
[realbooru] fix extraction
...
get file URLs from HTML pages
1 year ago
Mike Fährmann
75666cf9c3
[danbooru] reduce API requests for fetching extended 'metadata'
...
Instead of using one additional API request per post object (N+1),
this requires only one request per 200-post batch.
1 year ago
Amer Jazaerli
bebbff6578
fix: graphql_timeline_v2_bookmark_timeline cannot be null
...
twitter: 400 Bad Request (The following features cannot be null: graphql_timeline_v2_bookmark_timeline)
1 year ago
ClosedPort22
71b26adb9b
[urlshortener] add tinyurl.com as an example
2 years ago
Mike Fährmann
421db26aff
[bunkr] update domain to 'bunkr.la'
2 years ago
ClosedPort22
9e2a945013
[urlshortener] add support for bit.ly & t.co
2 years ago
Mike Fährmann
9b5e7ce8b9
[hiperdex] fix extraction
2 years ago
Mike Fährmann
89a67c45e0
[nitter] support nitter.it ( #3819 )
2 years ago
Mike Fährmann
88f29a751d
[nitter] skip broadcasts
...
instead of downloading an "Unsupported feature" HTML page
2 years ago
Mike Fährmann
1e013eba5a
[nitter] fix extraction for instances without user banners
2 years ago
Mike Fährmann
d94aa1ee02
[gelbooru] fix --range for favorites ( #3704 )
2 years ago
Mike Fährmann
1f82b00b8f
[gelbooru] fix and improve --range for pools
2 years ago
Mike Fährmann
197882cf12
[twitter] add 'hashtag' extractor ( #3783 )
2 years ago
Mike Fährmann
9789ebac52
[naverwebtoon] fix extraction ( #3729 )
2 years ago
Mike Fährmann
72f1f16eb2
[weibo] support 'mix_media_info' entries ( #3793 )
2 years ago
ClosedPort22
d4fb4ff47f
[twitter] extract TwitPic URLs in text ( #3792 )
...
also ignore previously seen URLs
2 years ago
Mike Fährmann
2bb937014f
[twitter] fall back to legacy /media endpoint when not logged in
2 years ago
Mike Fährmann
b68094d326
[twitter] support 'note_tweet's
2 years ago
Mike Fährmann
3dcabc97ed
[twitter] update API endpoints and parameters
2 years ago
Mike Fährmann
dcb8af659a
[gelbooru] extract favorites without needing cookies ( #3704 )
...
TODO: fix --range
2 years ago
Mike Fährmann
b756dc13aa
[gelbooru] warn about missing cookies for favorites ( #3704 )
...
and add docstring so it shows up in --list-extractors
2 years ago
Mike Fährmann
17bd053d94
[hiperdex] fix extraction ( #3768 )
2 years ago
Mike Fährmann
817fc0fbd1
[nitter] remove nitter.pussthecat.org
...
"Shutdown"
2 years ago
Mike Fährmann
67ec91cdbd
[downloader:http] change '_http_retry' to accept a Python function
...
and rename '_http_retry_codes' to '_http_retry'
(#3569 )
2 years ago
Mike Fährmann
175822e065
merge #3738 : [generic] add tests
2 years ago
Mike Fährmann
4883420e67
[generic] revert pattern change
2 years ago
Mike Fährmann
9037128315
[twitter] fix some 'original' retweets not downloading ( #3744 )
2 years ago
Mike Fährmann
ea3d95e7e8
merge #3740 : [deviantart] add support for fxdeviantart.com URLs
2 years ago
Mike Fährmann
9abcb2b6e5
update headers and ciphers for '"browser": "chrome"'
2 years ago
ClosedPort22
c489aecb3e
[deviantart] add support for fxdeviantart.com URLs
...
fxdeviantart.com is a service that fixes embeds on Discord, similar to
fxtwitter.com
2 years ago
ClosedPort22
34a7fab0e2
[generic] add support for IDNs
...
(internationalized domain name)
2 years ago
Mike Fährmann
c9a7345228
[newgrounds] prevent archive ID overlap ( #3681 )
...
add an 'i' and 'a' prefix to image and audio files
(/art/view/, /audio/listen/)
since their numeric ID may conflict with movies and other media
2 years ago
Mike Fährmann
da9840a39d
[reddit] update 'videos' option ( #3712 )
...
- add 'dash' to directly extract DASH manifest URLs
(was default behavior since a7c79531
)
- change default strategy back to before a7c79531
- disable 'Falling back on generic information extractor' warning
2 years ago
Mike Fährmann
baf41d7437
[misskey] update ( #3717 )
...
- add module docstring
- add options to docs/gallery-dl.conf
2 years ago
Mike Fährmann
6762d99515
merge #3717 : [misskey] add misskey extractors
2 years ago
Mike Fährmann
b8a702929d
[oauth] import extractor modules on demand
2 years ago
Mike Fährmann
dd88740ec7
replace remaining instances of base64 with binascii
2 years ago
enduser420
e1867cf5eb
[misskey] add 'renotes' and 'replies' options
2 years ago
enduser420
a95b5e0d8e
[misskey] add misskey extractors
2 years ago
Mike Fährmann
0d142e403c
[szurubooru] add 'tag' and 'post' extractors ( #3583 , #3713 )
2 years ago
Mike Fährmann
b14f8d5817
[gelbooru] add 'favorite' extractor ( #3704 )
...
requires logged in cookies to work
2 years ago
Mike Fährmann
a70a3e5da6
[mangasee] extract 'author' and 'genre' metadata ( #3703 )
...
Both are lists/arrays. Use {author!S} or {genre:J, } to format them.
2 years ago
Mike Fährmann
6b03506655
[deviantart] allow searching when not logged in
2 years ago
Mike Fährmann
511a051705
[fanbox] fix crash with missing images ( #3673 )
2 years ago
Mike Fährmann
3fa456d989
[deviantart] remove mature scraps warning ( #3691 )
...
warn about private deviations
when paginating over eclipse results
2 years ago
Mike Fährmann
51301e0c31
replace remaining time.sleep() calls
...
with Extractor.sleep() or request_interval
2 years ago
Mike Fährmann
6ed4309aba
[deviantart] add 'gallery-search' extractor ( #1695 )
2 years ago
Mike Fährmann
3d8777fbc1
move user agent string to util.py
2 years ago
Mike Fährmann
e1df7f73b1
[deviantart] add 'search' extractor
...
(#538 , #1264 , #2954 , #2970 , #3577 )
Requires login to fetch any results, since the API endpoint raises an
error for not logged in requests.
TODO: parse HTML search results
2 years ago
Mike Fährmann
4f029ab38b
[pornpics] support '/pornstar' and '/channels' listings
...
- fix docstring (#3671 )
- simplify code
2 years ago
Mike Fährmann
cbe4769246
[danbooru] use gallery-dl UA ( #3665 )
...
this removes the ability to set a custom UA via 'user-agent' option
for extractor requests
2 years ago
Mike Fährmann
253ac08203
pre-define and use 'gallery-dö/<version>' UA string
2 years ago
Mike Fährmann
b4899c266f
merge #3656 : [deviantart] fix crash when handling deleted deviations in status updates
2 years ago
Mike Fährmann
bb11c2a576
merge #3662 : [redgifs] add 'collection' extractors
2 years ago
Mike Fährmann
884f1848d6
[redgifs] fix syntax for older Python versions
...
and update docs/supportedsites
2 years ago
Mike Fährmann
725baedad3
[deviantart] use '/collections/all' endpoint for favorites
...
(#3666 ,#3668)
2 years ago
Mike Fährmann
2bd8f2f4bd
[pornpics] add 'search' and 'tag' extractors
...
(#263 , #3544 , #3654 )
2 years ago
Mike Fährmann
79bc82884c
[pornpics] add 'gallery' extractor ( #263 , #3544 , #3654 )
2 years ago
Mike Fährmann
7bdc1d6d3d
[manganelo] update and fix metadata extraction
2 years ago
Mike Fährmann
363bb76dff
[manganelo] simplify URL pattern
2 years ago
enduser420
b28bd9789e
[redgifs] add 'collection' extractors
2 years ago
ClosedPort22
f4e211356d
[deviantart] slight refactor
2 years ago
Mike Fährmann
bd5d08abbc
[catbox] add 'file' extractor ( #3570 )
2 years ago
Mike Fährmann
8e1e8a5bea
[soundgasm] rewrite ( #3578 )
...
use a more standard extractor structure to make -A work as expected
2 years ago
Mike Fährmann
0b93420a81
[pinterest] unescape search terms ( #3621 )
2 years ago
Mike Fährmann
ad96e70546
[bunkr] fix extraction ( #3636 , #3655 )
2 years ago
Mike Fährmann
9335d55bbc
[manganelo] support mobile-only chapters
2 years ago
ClosedPort22
a74114ef7a
[deviantart] fix crash when handling deleted deviations
...
in status updates
2 years ago
Mike Fährmann
75570ad3f1
[oauth] remove stray 'exit()' ( #3628 )
...
- bug from 70ce45d9
- broke oauth:tumblr, oauth:flickr, and oauth:smugmug
2 years ago
Mike Fährmann
8fb043e8ff
[tumblr] raise more detailed errors for dashboard-only blogs
...
(#3628 )
2 years ago
Mike Fährmann
ce996dd21b
[poipiku] warn about incorrect passwords ( #3646 )
2 years ago
Mike Fährmann
70ce45d965
[oauth] use default name for browsers without 'name' attribute
...
(#3645 )
Seem to only be an issue for MacOSXOSAScript before Python 3.11.
d12bec6993
2 years ago
Mike Fährmann
2a53e6445c
[bunkr] update domain ( #3636 )
2 years ago
Mike Fährmann
5503ac4d5e
replace json.dumps with direct calls to JSONEncoder.encode
2 years ago
Mike Fährmann
dd884b02ee
replace json.loads with direct calls to JSONDecoder.decode
2 years ago
Mike Fährmann
8805bd38ab
merge #3622 : [imagetwist] add phun.imagetwist.com and imagehaha.com support
2 years ago
Mike Fährmann
706ec70e89
[imagetwist] simplify pattern and add tests
2 years ago
Mike Fährmann
f2e91732ae
[instagram] add 'user' metadata field ( #3107 )
...
at the moment only for URLs that need to translate user name to ID
2 years ago
Prinz23
29f0830b53
[imagetwist] add phun.imagetwist.com and imagehaha.com alias to imagetwist extractor
2 years ago
Mike Fährmann
bbf0911a46
[e621] implement 'notes' and 'pools' metadata extraction
...
(#3425 )
2 years ago
Mike Fährmann
925b467496
split e621 from danbooru module ( #3425 )
2 years ago
Mike Fährmann
1ae48a54f8
[twitter] add 'transform' option
2 years ago
Mike Fährmann
489c51cecc
[telegraph] fix extraction when images not in <figure> ( #3590 )
2 years ago
Mike Fährmann
0f7e6c422a
merge #3596 : [shopify] support ohpolly.com
2 years ago
enduser420
fcf7030b85
[shopify] support ohpolly.com
2 years ago
Mike Fährmann
a6a631f992
merge #3589 : [redgifs] support v3 URLs
2 years ago
Mike Fährmann
137a395ae0
[imagefap] fix infinite pagination loop ( #3594 )
2 years ago
Mike Fährmann
3c708ade8f
[imagefap] fix metadata extraction
2 years ago
Mike Fährmann
17e24eacf0
[imagefap] update 'gallery' URLs ( #3595 )
2 years ago
Mike Fährmann
c2bc70593e
implement ability to load external extractor classes
...
- -X/--extractors
- extractor.module-sources
2 years ago
enduser420
a18f627bfc
[redgifs] support v3 URLs
2 years ago
Mike Fährmann
13a90969c7
merge #3575 : [nudecollect] add 'image' and 'album' extractors
2 years ago
Mike Fährmann
aacd27e4ef
merge #3581 : [hotleak] fix video URLs
2 years ago
Mike Fährmann
abc3619feb
[lexica] add 'search' extractor ( #3567 )
2 years ago
Mike Fährmann
7c9b1ec830
[hotleak] optimize decoding video URLs
...
- use binascii module
- combine slice and reverse step
2 years ago
nifnat
f14dbfe079
Make decode_video_url static (used in both post and creator extractor).
2 years ago
nifnat
bd23a701f3
Tidy up code.
2 years ago
nifnat
7f34f99a26
Reverse engineered obfuscated JS function and reimplemented in python.
2 years ago
Mike Fährmann
0d818d3540
[fantia] send 'X-CSRF-Token' headers ( #3576 )
2 years ago
enduser420
2a5903dc16
[nudecollect] add 'image' and 'album' extractors
2 years ago
Mike Fährmann
c8fdd5096e
merge #3571 : [bunkr] Fix extracting mkv and ts files
2 years ago
Mike Fährmann
58c008e30a
[hiperdex] update domain ( #3572 )
2 years ago
Luc Ritchie
842064e597
[bunkr] Fix extracting ts files
2 years ago
Luc Ritchie
99ca0437e4
[bunkr] Fix extracting mkv files
2 years ago
Mike Fährmann
76b01b64cf
[kemonoparty] remove MD5 hash extraction ( #3531 )
...
This partially reverts commit 20d6194ffa
.
2 years ago
Mike Fährmann
09fb212414
[philomena] match URLs with www subdomain
2 years ago
Mike Fährmann
7e2fd2e573
merge #3560 : [deviantart] add support for /deviation/ and fav.me URLs
2 years ago
Mike Fährmann
caae8fefe1
merge #3541 : [deviantart] add extractor for status updates
2 years ago
ClosedPort22
c90b4ea8d9
[deviantart] add support for fav.me URLs
2 years ago
Mike Fährmann
d63af4f3d3
merge #3555 : [generic] fix regex for non-src image URLs
2 years ago
Mike Fährmann
8993b10751
[mastodon] add 'num' and 'count' metadata fields ( #3517 )
2 years ago
Mike Fährmann
d817d23ccb
[instagram] update csrf token handling
...
- update internal value according to cookie
- do not send a second 'csrftoken' cookie
2 years ago
Mike Fährmann
00b94946b3
[instagram] show -o cursor=… after every error ( #3440 )
2 years ago
ClosedPort22
674c719646
[deviantart] refactor base36 conversion
2 years ago
ClosedPort22
293abb8921
[deviantart] add support for /deviation/ URLs
2 years ago
thatfuckingbird
8cfeed78b1
[generic] fix regex for non-src image URLs
2 years ago
Mike Fährmann
fc6ea8ee5c
[instagram] update API domain and headers
2 years ago
ClosedPort22
597b89245e
[deviantart] misc improvements to status extractor
...
- relax regex pattern
- handle invalid 'items' field
- add a test for shared sta.sh item
Co-authored-by: Mike Fährmann <mike_faehrmann@web.de>
2 years ago
Mike Fährmann
137de090dd
merge #3549 : [twitter] fix search ( #3536 )
2 years ago
Mike Fährmann
02e314c1b6
merge #3537 : [wikifeet/wikifeetx] add 'gallery' extractor
2 years ago
Mike Fährmann
568112dfbb
[oauth] improve output
...
- show which api key / client id gets used (#3518 )
- show in which browser authorization URLs gets opened in
2 years ago
ClosedPort22
ab58c375b4
[twitter] fix search ( #3536 )
...
- partially revert 18fe4b334d
- properly search for cursor when processing 'replaceEntry'
2 years ago
Mike Fährmann
df91ebb945
[oauth] simplify OAuth 1.0a init
2 years ago
ClosedPort22
013733c9e9
[deviantart] fix index fields for embedded/shared images
2 years ago
ClosedPort22
c4aeca7a5a
[deviantart] improve handling of statuses
...
- recursively yield statuses
- ignore items with missing or unexpected field(s)
2 years ago
ClosedPort22
3b32671fbd
[deviantart] add extractor for status updates
...
extract user status updates using the '/user/statuses/' endpoint
2 years ago
Mike Fährmann
107c60c973
[sankaku] update URL pattern ( #3523 )
...
match tag searches with language codes without a trailing slash
2 years ago
enduser420
5cb263fdd2
[wikifeet/wikifeetx] add 'gallery' extractor
2 years ago
Mike Fährmann
35a30498bc
merge #3531 : [kemonoparty] improve hash extraction
...
- extract md5 hashes if available
- extract discord file hashes
2 years ago
Mike Fährmann
9683d79bb7
[twitter] "fix" search pagination ( #3536 , #3534 )
...
- properly process instructions
- do not expect a predetermined instruction order
2 years ago
Mike Fährmann
4fec848858
[twitter] use "browser": "firefox" by default ( #3522 )
...
and reenable TLS 1.2 ciphers
2 years ago
Mike Fährmann
78937564fd
[twitter] fix login after 32b03433
2 years ago
ClosedPort22
20d6194ffa
[kemonoparty] improve hash extraction
...
- extract MD5 hash from URLs
- extract MD5 and SHA256 hash from Discord URLs (kemono.party only)
- minor optimization (do not call 'hashes.add' when 'duplicates' is
true)
- update tests accordingly
Co-authored-by: Mike Fährmann <mike_faehrmann@web.de>
2 years ago
Mike Fährmann
80a2ff2d38
support setting 'write-pages' to "ALL"
...
to show authentication header, cookies, etc
2 years ago
Mike Fährmann
c881548a27
add 'extractor.retry-codes' option ( #3313 )
...
do not retry 429 and 430 by default
2 years ago
Mike Fährmann
e30e8aeef7
[mastodon] rename '_check_move' -> '_check_moved'
2 years ago
Mike Fährmann
32b0343334
[twitter] refresh guest tokens ( #3445 , #3458 )
2 years ago
Mike Fährmann
512abeb4ae
[booru] add 'url' option
2 years ago
Mike Fährmann
c87bd1a752
[danbooru] extend 'metadata' option
...
make it possible to specify a custom list of metadata includes
2 years ago
Mike Fährmann
26c3292538
[twitter] disable TLS 1.2 ciphers by default ( #3522 )
2 years ago
Mike Fährmann
18fe4b334d
[twitter] remove 'tweet_search_mode' from search parameters ( #3522 )
...
and update API root and general query parameters
2 years ago
Mike Fährmann
85bd1cbc89
[kemonoparty] fix regression from 473bd380
( #3519 )
...
- do not access 'response.content' unless necessary
- only validate responses if filename extensions differ
2 years ago
Mike Fährmann
473bd380c8
[kemonoparty] reject invalid/empty files ( #3510 )
2 years ago
Mike Fährmann
4833ec323e
[imagefap] add 'folder' extractor ( #3504 )
2 years ago
Mike Fährmann
362cd6991b
[pixiv] implement 'metadata-bookmark' option ( #3417 )
2 years ago
Mike Fährmann
2142b9c7ae
merge #3503 : [myhentaigallery] handle whitespace before title tag
2 years ago
Mike Fährmann
3a0450adbf
[behance] use default delay between requests ( #2507 )
2 years ago
Mike Fährmann
2cae4567ba
[telegraph] fix file URLs ( #3506 )
2 years ago
Mike Fährmann
cbaeee9533
[imagefap] warn about redirects to '/human-verification' ( #1140 )
2 years ago
Mike Fährmann
435de1329a
[imagefap] use default delay between requests ( #1140 )
2 years ago
Erik Rimskog
a8a982359e
[myhentaigallery] handle whitespace before the title tag
2 years ago
Mike Fährmann
d1dd52349a
merge #3189 : [tcbscans] add 'chapter' and 'manga' extractors
2 years ago
Mike Fährmann
2f31d21509
merge #3455 : [twitter] apply tweet type checks before uniqueness check
2 years ago
enduser420
e8541a131d
[tcbscans] add 'chapter' and 'manga' extractors
2 years ago
Mike Fährmann
9695c4e88d
emit debug logging message when loading cookies from file
...
attempt nr. 2
no idea how I managed to remove 6514828d
in a918ce29
2 years ago
Mike Fährmann
30a31836e7
merge #3449 : [twitter] force HTTPS for TwitPic URLs
2 years ago
Mike Fährmann
e18482e9ae
[twitter] improve 'http' -> 'https' replacement
2 years ago
Mike Fährmann
4fd6da474f
merge #3473 : [twitter] fix crash when using 'expand' and 'syndication'
2 years ago
Mike Fährmann
a918ce29b5
run tests on ubuntu-20.04
...
and remove Python 3.4, since that's no longer available
on this test runner
2 years ago
Mike Fährmann
6514828d4e
emit debug logging message when loading cookies from file
2 years ago
Mike Fährmann
3a238fd490
[poipiku] warn about login requirements
2 years ago
Mike Fährmann
f29ba089ff
merge #3474 : [fanleaks] add 'post' and 'model' extractors
2 years ago
Mike Fährmann
6933727b45
merge #3483 : [twitter] implement 'syndication=extended'
2 years ago
Mike Fährmann
07ed3a1fbf
merge #3460 : [poipiku] fix extraction for a different warning button style
...
(#3493 , #3492 )
2 years ago
Mike Fährmann
9116398c1c
[pinterest] add 'domain' option ( #3484 )
...
use input URL domain by default
2 years ago
blankie
2f985bcddb
[poipiku] fix extraction for a different warning button style
2 years ago
Mike Fährmann
294108c90a
[pinterest] support 'All Pins' boards ( #2855 , #3484 )
2 years ago
Mike Fährmann
77df8d3116
[deviantart] implement username&password login for scraps ( #1029 )
...
re-login when getting prematurely logged out by dA
is missing at the moment
2 years ago
Mike Fährmann
ed2d715019
fix 'keywords' in extractor tests ( #3491 )
2 years ago
ClosedPort22
6853b14be3
[twitter] apply suggestions from code review
...
Co-authored-by: Mike Fährmann <mike_faehrmann@web.de>
2 years ago
Mike Fährmann
4611237f8c
merge #3457 : [danbooru] extract uploader metadata (if option is set)
2 years ago
Mike Fährmann
e7522482bb
merge #3463 : [lynxchan] support 'bbw-chan.nl'
2 years ago
Mike Fährmann
7d6c846176
[fanbox] return 'imageMap' files in order ( #2718 )
2 years ago
Mike Fährmann
dc8e7ff54e
[bunkr] fix URLs returned by API ( #3481 )
2 years ago
enduser420
5fedef3a1a
[fanleaks] update 'model' URL pattern
2 years ago
enduser420
5a740ef78b
[fanleaks] add 'post' and 'model' extractors
2 years ago
ClosedPort22
7c8eab8d52
[twitter] implement 'syndication=extended'
...
to be able to fetch extended user metadata
2 years ago
ClosedPort22
be3286206a
[twitter] assume 'conversation_id' when using syndication
...
not possible to expand replies at the momemt
2 years ago
ClosedPort22
ce8dbb1ccc
[twitter] fix crash when using 'expand' and 'syndication'
...
caused by KeyError: 'conversation_id_str'
2 years ago
ClosedPort22
38786a9593
[twitter] refactor extraction of TwitPic URLs
...
flattening
2 years ago
enduser420
527bb2c4ab
[lynxchan/bbw-chan] add 'thread' and 'board' extractors
2 years ago
blankie
f82ee93676
[danbooru] extract uploader metadata (if metadata is set)
2 years ago
ClosedPort22
250d35107c
[twitter] prioritize tweet type checks ( #3439 )
...
Do not consider a tweet seen before applying 'retweet', 'quote' and
'reply' checks. Otherwise the original tweets will also be skipped if
the "derivative" tweets and the original tweets are from the same user.
2 years ago
ClosedPort22
3eb352fcb0
[twitter] force HTTPS for TwitPic URLs
2 years ago
lx30011
895b41f1ac
[jschan] add generic jschan extractor
2 years ago
Mike Fährmann
bee354c264
Merge pull request #3415 from enduser420/extractor/fapello
...
[fapello] add 'post', 'user' and 'path' extractors
2 years ago
Mike Fährmann
8d7585534e
Merge pull request #3367 from the-blank-x/deviantart-view
...
[deviantart] add /view URL support
2 years ago
blankie
6614d94b08
[deviantart] add /view URL support
2 years ago
Mike Fährmann
dd6eeb4336
Merge pull request #3366 from ClosedPort22/da-extra-stash
...
[deviantart] extract sta.sh URLs from `text_content`
2 years ago
Mike Fährmann
f36cbb3911
Merge pull request #3413 from ClosedPort22/e621-manual-pagination
...
[e621] implement manual pagination
2 years ago
ClosedPort22
dd4a4a3fa6
[e621] softcode the pagination threshold
2 years ago
ClosedPort22
9faa4ed738
[e621] refactor pagination control
...
as suggested by @mikf
2 years ago
Mike Fährmann
7851a2c520
[seiga] raise error when redirected to login page ( #3401 )
2 years ago
Mike Fährmann
68ce5f965d
[instagram] remove unused code
2 years ago
Mike Fährmann
4063563cd7
[zerochan] update for layout v3
...
- remove cookie disabling v3
- fix and improve metadata extraction
2 years ago
Mike Fährmann
1e6407ca98
Merge pull request #3414 from pubak42/master
...
[sex.com] Download videos from cdn (#3408 )
2 years ago
ClosedPort22
bf1649dadb
[imgur] add support for imgur.io URLs
2 years ago
enduser420
7e08e2d982
[fapello] set 'filename_fmt'
2 years ago
enduser420
e5076ba056
[fapello] add 'post', 'user' and 'path' extractors
2 years ago
pubak42
e7326cdf1d
[sex.com] Download videos from cdn ( #3408 )
...
The format of video sources was changed recently to be a full URL with https:// in the beginning.
The original extractor code appended the video source URL to root url of the website, thus yielding
invalid url in format ...sex.comhttps... that failed to resolve.
2 years ago
ClosedPort22
d0ad6d0e67
[e621] implement manual pagination mode
2 years ago
Mike Fährmann
6f0735568c
[2chen] fix file URLs
2 years ago
enduser420
a2be06d873
[2chen] add '.club' support ( #3406 )
2 years ago
Mike Fährmann
a6d4733e11
[pixiv] extract 'date_url' metadata ( #3405 )
...
i.e. the datetime encoded in each file URL.
https://i.pximg.net/img-master/img/2022/12/01/13/44/55/12345678_p0.jpg
->
2022-12-01 13:44:55 +09:00
->
2022-12-01 04:44:55
2 years ago
Mike Fährmann
1317625ec4
[webmshare] add 'video' extractor ( #2410 )
2 years ago
Mike Fährmann
90a9c0790f
[twitter] update 'search' pagination ( #544 )
...
Only stop when list of all returned Tweets is empty
instead of when no valid Tweet was found.
2 years ago
Mike Fährmann
1cbc234819
[mangafox] extract more metadata ( #3167 )
2 years ago
Mike Fährmann
3082544fff
misc fixes
...
- fix typo (#3399 )
- remove double assignment
- [bunkr] update things I forgot in 6b6f886d
- [soundgasm] adjust 'archive_fmt' (#3388 )
2 years ago
enduser420
41bf236d36
[lynxchan] add generic extractors for lynxchan imageboards ( #3394 )
...
* [lynxchan] add generic extractors for lynxchan imageboards
includes kohlchan.net, endchan.org:wq
* [lynxchan] set pop default to empty tuple
* Apply suggestions from code review
Co-authored-by: Mike Fährmann <mike_faehrmann@web.de>
2 years ago
Mike Fährmann
3c75c3bbc4
[soundgasm] add 'user' extractor ( #3384 )
...
based on code from PR #3388 by @enduser420
2 years ago
Mike Fährmann
2952add4a8
[reddit] increase 'id-max' default value ( #3397 )
...
to float("inf")
2 years ago
Mike Fährmann
a001c9c06f
[instagram] prevent post 'date' overwriting file 'date' ( #3392 )
2 years ago
Mike Fährmann
6b6f886dcf
[bunkr] update domain ( #3391 )
...
and improve bunkr/app.bunkr handling
2 years ago
ClosedPort22
bf3fd5951a
Merge branch 'master' into da-extra-stash
2 years ago
Mike Fährmann
eb94568e1f
[soundgasm] add 'audio' extractor ( #3384 )
2 years ago
Mike Fährmann
cd931e1139
update extractor test results
2 years ago
Mike Fährmann
989ec9fc79
[khinsider] fix metadata extraction
2 years ago
Mike Fährmann
1c25cc7a3e
[warosu] fix and update
2 years ago
Mike Fährmann
79e52f3539
[imgth] rewrite
...
- inherit from GalleryExtractor
- fix image URLs
- better metadata
2 years ago
Mike Fährmann
202c1210d5
[exhentai] fix pagination
2 years ago
Mike Fährmann
4a3a1f4c87
[komikcast] update domain and fix extraction
2 years ago
ClosedPort22
13d825731e
[deviantart] fix test for sta.sh URL extraction
...
Without the 'count' assertion, the test would be essentially useless.
2 years ago
ClosedPort22
6356c9be96
[deviantart] extract sta.sh URLs from 'text_content'
2 years ago
Mike Fährmann
5f57a27ba6
[imagetwist] fix extraction
2 years ago
Mike Fährmann
a42ba25ca1
[foolslide] remove 'kireicake'
...
site redirects to (unclaimed) mangadex group
2 years ago
Mike Fährmann
86f0597c95
[kissgoddess] remove module
...
site does not host albums anymore
2 years ago
Mike Fährmann
20e12b5d7c
[nitter] support '/i/user/' URLs ( #3310 )
...
as well as using 'id:<userid>' as username
not all nitter instances seem to support '/i/user/' ...
2 years ago
Mike Fährmann
fceaee3c4f
[lolisafe] remove zz.ht
2 years ago
Mike Fährmann
4554c43d5f
[bunkr] use 'media-files' servers for more file types
2 years ago
enduser420
4bc756dfe0
[2chen] fix extraction ( #3356 )
...
update 'archive_fmt'
update tests
update 'board' regex
2 years ago
enduser420
54844944ab
[pixhost] add 'gallery' support ( #3353 )
2 years ago
enduser420
213676c785
[fapachi] add 'post' and 'user' extractors ( #3347 )
...
* [fapachi] add 'post' and 'user' extractors
* [fapachi] add 'keyword' to test
* [fapachi] remove whitespaces
2 years ago
Mike Fährmann
a18511e346
[nitter] retry downloads on 404 ( #3313 )
2 years ago
Mike Fährmann
88610c3478
[patreon] update API query parameters
2 years ago
Mike Fährmann
c19b1f03b9
[patreon] fix '403 Forbidden' errors
...
send 'Content-Type' headers for API requests
2 years ago
Mike Fährmann
fc34f76cc5
[bunkr] fix video downloads ( #3326 )
...
by sending 'https://stream.bunkr.is/ ' as Referer header
2 years ago
Mike Fährmann
86a396e086
[bcy] fix JSONDecodeError ( #3321 )
2 years ago