8e1e8a5bea
[soundgasm] rewrite ( #3578 )
...
use a more standard extractor structure to make -A work as expected
2 years ago
0b93420a81
[pinterest] unescape search terms ( #3621 )
2 years ago
ad96e70546
[bunkr] fix extraction ( #3636 , #3655 )
2 years ago
9335d55bbc
[manganelo] support mobile-only chapters
2 years ago
a74114ef7a
[deviantart] fix crash when handling deleted deviations
...
in status updates
2 years ago
75570ad3f1
[oauth] remove stray 'exit()' ( #3628 )
...
- bug from 70ce45d9
2 years ago
d37e7f4898
add 'hooks' option
...
Very much a work in progress.
At the moment, it allows to
- wait and restart an extractor (#3338 )
- change the exit code (#3630 )
- change the log level of a logging message
based on the contents of a logging message
2 years ago
8fb043e8ff
[tumblr] raise more detailed errors for dashboard-only blogs
...
(#3628 )
2 years ago
d4232f3a8b
implement restarting an extractor ( #3338 )
2 years ago
ce996dd21b
[poipiku] warn about incorrect passwords ( #3646 )
2 years ago
70ce45d965
[oauth] use default name for browsers without 'name' attribute
...
(#3645 )
Seem to only be an issue for MacOSXOSAScript before Python 3.11.
d12bec6993
2 years ago
1aae72773f
put argument init on separate lines
2 years ago
2a53e6445c
[bunkr] update domain ( #3636 )
2 years ago
5503ac4d5e
replace json.dumps with direct calls to JSONEncoder.encode
2 years ago
dd884b02ee
replace json.loads with direct calls to JSONDecoder.decode
2 years ago
b7337d810e
[postprocessor:metadata] add 'sort' and 'separators' options
2 years ago
8805bd38ab
merge #3622 : [imagetwist] add phun.imagetwist.com and imagehaha.com support
2 years ago
706ec70e89
[imagetwist] simplify pattern and add tests
2 years ago
f2e91732ae
[instagram] add 'user' metadata field ( #3107 )
...
at the moment only for URLs that need to translate user name to ID
2 years ago
3436c6b117
[postprocessor:metadata] speed up JSON encoding
2 years ago
29f0830b53
[imagetwist] add phun.imagetwist.com and imagehaha.com alias to imagetwist extractor
2 years ago
762a68996b
implement 'archive-pragma' option
2 years ago
bbf0911a46
[e621] implement 'notes' and 'pools' metadata extraction
...
(#3425 )
2 years ago
925b467496
split e621 from danbooru module ( #3425 )
2 years ago
1ae48a54f8
[twitter] add 'transform' option
2 years ago
78d3960a31
[postprocessor:exec] implement archive options ( #3584 )
2 years ago
489c51cecc
[telegraph] fix extraction when images not in <figure> ( #3590 )
2 years ago
0f7e6c422a
merge #3596 : [shopify] support ohpolly.com
2 years ago
fcf7030b85
[shopify] support ohpolly.com
2 years ago
a6a631f992
merge #3589 : [redgifs] support v3 URLs
2 years ago
137a395ae0
[imagefap] fix infinite pagination loop ( #3594 )
2 years ago
3c708ade8f
[imagefap] fix metadata extraction
2 years ago
17e24eacf0
[imagefap] update 'gallery' URLs ( #3595 )
2 years ago
d16873941c
[downloader:http] use 'time.monotonic()'
2 years ago
c2bc70593e
implement ability to load external extractor classes
...
- -X/--extractors
- extractor.module-sources
2 years ago
a18f627bfc
[redgifs] support v3 URLs
2 years ago
9ec627c760
release version 1.24.5
2 years ago
13a90969c7
merge #3575 : [nudecollect] add 'image' and 'album' extractors
2 years ago
aacd27e4ef
merge #3581 : [hotleak] fix video URLs
2 years ago
abc3619feb
[lexica] add 'search' extractor ( #3567 )
2 years ago
7c9b1ec830
[hotleak] optimize decoding video URLs
...
- use binascii module
- combine slice and reverse step
2 years ago
f14dbfe079
Make decode_video_url static (used in both post and creator extractor).
2 years ago
bd23a701f3
Tidy up code.
2 years ago
7f34f99a26
Reverse engineered obfuscated JS function and reimplemented in python.
2 years ago
0d818d3540
[fantia] send 'X-CSRF-Token' headers ( #3576 )
2 years ago
f58215705a
add '-O/--postprocessor-option' command-line option ( #3565 )
2 years ago
2a5903dc16
[nudecollect] add 'image' and 'album' extractors
2 years ago
c8fdd5096e
merge #3571 : [bunkr] Fix extracting mkv and ts files
2 years ago
58c008e30a
[hiperdex] update domain ( #3572 )
2 years ago
842064e597
[bunkr] Fix extracting ts files
2 years ago
99ca0437e4
[bunkr] Fix extracting mkv files
2 years ago
76b01b64cf
[kemonoparty] remove MD5 hash extraction ( #3531 )
...
This partially reverts commit 20d6194ffa
2 years ago
09fb212414
[philomena] match URLs with www subdomain
2 years ago
7e2fd2e573
merge #3560 : [deviantart] add support for /deviation/ and fav.me URLs
2 years ago
caae8fefe1
merge #3541 : [deviantart] add extractor for status updates
2 years ago
c90b4ea8d9
[deviantart] add support for fav.me URLs
2 years ago
d63af4f3d3
merge #3555 : [generic] fix regex for non-src image URLs
2 years ago
8993b10751
[mastodon] add 'num' and 'count' metadata fields ( #3517 )
2 years ago
d817d23ccb
[instagram] update csrf token handling
...
- update internal value according to cookie
- do not send a second 'csrftoken' cookie
2 years ago
00b94946b3
[instagram] show -o cursor=… after every error ( #3440 )
2 years ago
674c719646
[deviantart] refactor base36 conversion
2 years ago
293abb8921
[deviantart] add support for /deviation/ URLs
2 years ago
8cfeed78b1
[generic] fix regex for non-src image URLs
2 years ago
fc6ea8ee5c
[instagram] update API domain and headers
2 years ago
597b89245e
[deviantart] misc improvements to status extractor
...
- relax regex pattern
- handle invalid 'items' field
- add a test for shared sta.sh item
Co-authored-by: Mike Fährmann <mike_faehrmann@web.de>
2 years ago
137de090dd
merge #3549 : [twitter] fix search ( #3536 )
2 years ago
02e314c1b6
merge #3537 : [wikifeet/wikifeetx] add 'gallery' extractor
2 years ago
568112dfbb
[oauth] improve output
...
- show which api key / client id gets used (#3518 )
- show in which browser authorization URLs gets opened in
2 years ago
ab58c375b4
[twitter] fix search ( #3536 )
...
- partially revert 18fe4b334d
2 years ago
df91ebb945
[oauth] simplify OAuth 1.0a init
2 years ago
013733c9e9
[deviantart] fix index fields for embedded/shared images
2 years ago
c4aeca7a5a
[deviantart] improve handling of statuses
...
- recursively yield statuses
- ignore items with missing or unexpected field(s)
2 years ago
3b32671fbd
[deviantart] add extractor for status updates
...
extract user status updates using the '/user/statuses/' endpoint
2 years ago
107c60c973
[sankaku] update URL pattern ( #3523 )
...
match tag searches with language codes without a trailing slash
2 years ago
5cb263fdd2
[wikifeet/wikifeetx] add 'gallery' extractor
2 years ago
35a30498bc
merge #3531 : [kemonoparty] improve hash extraction
...
- extract md5 hashes if available
- extract discord file hashes
2 years ago
ec9ff7640d
merge #3535 : [downloader:http] add signature checks for .blend, .obj, and .clip files
2 years ago
9683d79bb7
[twitter] "fix" search pagination ( #3536 , #3534 )
...
- properly process instructions
- do not expect a predetermined instruction order
2 years ago
4fec848858
[twitter] use "browser": "firefox" by default ( #3522 )
...
and reenable TLS 1.2 ciphers
2 years ago
78937564fd
[twitter] fix login after 32b03433
2 years ago
b6706b373a
[downloader:http] add signature checks for some formats
...
also add the MIME type for .obj files
2 years ago
20d6194ffa
[kemonoparty] improve hash extraction
...
- extract MD5 hash from URLs
- extract MD5 and SHA256 hash from Discord URLs (kemono.party only)
- minor optimization (do not call 'hashes.add' when 'duplicates' is
true)
- update tests accordingly
Co-authored-by: Mike Fährmann <mike_faehrmann@web.de>
2 years ago
80a2ff2d38
support setting 'write-pages' to "ALL"
...
to show authentication header, cookies, etc
2 years ago
d6793b2c7d
include request body in 'write-pages=all' output
2 years ago
c881548a27
add 'extractor.retry-codes' option ( #3313 )
...
do not retry 429 and 430 by default
2 years ago
e30e8aeef7
[mastodon] rename '_check_move' -> '_check_moved'
2 years ago
32b0343334
[twitter] refresh guest tokens ( #3445 , #3458 )
2 years ago
512abeb4ae
[booru] add 'url' option
2 years ago
c87bd1a752
[danbooru] extend 'metadata' option
...
make it possible to specify a custom list of metadata includes
2 years ago
26c3292538
[twitter] disable TLS 1.2 ciphers by default ( #3522 )
2 years ago
18fe4b334d
[twitter] remove 'tweet_search_mode' from search parameters ( #3522 )
...
and update API root and general query parameters
2 years ago
ec04c97075
release version 1.24.4
2 years ago
c0d7d2be35
[downloader:http] add 'validate' option
2 years ago
85bd1cbc89
[kemonoparty] fix regression from 473bd380 ( #3519 )
...
- do not access 'response.content' unless necessary
- only validate responses if filename extensions differ
2 years ago
805a5663ec
release version 1.24.3
2 years ago
473bd380c8
[kemonoparty] reject invalid/empty files ( #3510 )
2 years ago
4833ec323e
[imagefap] add 'folder' extractor ( #3504 )
2 years ago
362cd6991b
[pixiv] implement 'metadata-bookmark' option ( #3417 )
2 years ago
0895e6afee
merge #3462 : [docs] Update links and fix field typo
2 years ago
2142b9c7ae
merge #3503 : [myhentaigallery] handle whitespace before title tag
2 years ago
3a0450adbf
[behance] use default delay between requests ( #2507 )
2 years ago
2cae4567ba
[telegraph] fix file URLs ( #3506 )
2 years ago
cbaeee9533
[imagefap] warn about redirects to '/human-verification' ( #1140 )
2 years ago
435de1329a
[imagefap] use default delay between requests ( #1140 )
2 years ago
a8a982359e
[myhentaigallery] handle whitespace before the title tag
2 years ago
d1dd52349a
merge #3189 : [tcbscans] add 'chapter' and 'manga' extractors
2 years ago
2f31d21509
merge #3455 : [twitter] apply tweet type checks before uniqueness check
2 years ago
e8541a131d
[tcbscans] add 'chapter' and 'manga' extractors
2 years ago
9695c4e88d
emit debug logging message when loading cookies from file
...
attempt nr. 2
no idea how I managed to remove 6514828da918ce29
2 years ago
30a31836e7
merge #3449 : [twitter] force HTTPS for TwitPic URLs
2 years ago
e18482e9ae
[twitter] improve 'http' -> 'https' replacement
2 years ago
4fd6da474f
merge #3473 : [twitter] fix crash when using 'expand' and 'syndication'
2 years ago
a918ce29b5
run tests on ubuntu-20.04
...
and remove Python 3.4, since that's no longer available
on this test runner
2 years ago
6514828d4e
emit debug logging message when loading cookies from file
2 years ago
3a238fd490
[poipiku] warn about login requirements
2 years ago
fa144f38ed
[ytdl} fix dfe4f00c for legacy yt-dlp
2 years ago
f29ba089ff
merge #3474 : [fanleaks] add 'post' and 'model' extractors
2 years ago
6933727b45
merge #3483 : [twitter] implement 'syndication=extended'
2 years ago
07ed3a1fbf
merge #3460 : [poipiku] fix extraction for a different warning button style
...
(#3493 , #3492 )
2 years ago
9116398c1c
[pinterest] add 'domain' option ( #3484 )
...
use input URL domain by default
2 years ago
6f6af36cad
use double quotes for --help examples
2 years ago
dfe4f00ca2
[ytdl] update for yt-dlp changes
2 years ago
2f985bcddb
[poipiku] fix extraction for a different warning button style
2 years ago
294108c90a
[pinterest] support 'All Pins' boards ( #2855 , #3484 )
2 years ago
77df8d3116
[deviantart] implement username&password login for scraps ( #1029 )
...
re-login when getting prematurely logged out by dA
is missing at the moment
2 years ago
ed2d715019
fix 'keywords' in extractor tests ( #3491 )
2 years ago
3f29b8fe91
[cookies] convert browser names to lowercase
2 years ago
6853b14be3
[twitter] apply suggestions from code review
...
Co-authored-by: Mike Fährmann <mike_faehrmann@web.de>
2 years ago
4611237f8c
merge #3457 : [danbooru] extract uploader metadata (if option is set)
2 years ago
e7522482bb
merge #3463 : [lynxchan] support 'bbw-chan.nl'
2 years ago
7d6c846176
[fanbox] return 'imageMap' files in order ( #2718 )
2 years ago
dc8e7ff54e
[bunkr] fix URLs returned by API ( #3481 )
2 years ago
5fedef3a1a
[fanleaks] update 'model' URL pattern
2 years ago
5a740ef78b
[fanleaks] add 'post' and 'model' extractors
2 years ago
7c8eab8d52
[twitter] implement 'syndication=extended'
...
to be able to fetch extended user metadata
2 years ago
be3286206a
[twitter] assume 'conversation_id' when using syndication
...
not possible to expand replies at the momemt
2 years ago
ce8dbb1ccc
[twitter] fix crash when using 'expand' and 'syndication'
...
caused by KeyError: 'conversation_id_str'
2 years ago
d651d45239
implement specifying ranges in slice notation ( #918 , #2865 )
...
e.g.
- '1:101' or ':101' or ':101:' for files 1 to 100
- '1::2' or '::2' for every second file
- '1:101:5' or ':101:5' for files 1, 6, 11, ..., 91, 96
(the second argument specifies the first index NOT included)
2 years ago
38786a9593
[twitter] refactor extraction of TwitPic URLs
...
flattening
2 years ago
3616adfc75
implement '--range' with Python ranges
2 years ago
527bb2c4ab
[lynxchan/bbw-chan] add 'thread' and 'board' extractors
2 years ago
64902f518e
[docs] Update links and fix field typo
2 years ago
f82ee93676
[danbooru] extract uploader metadata (if metadata is set)
2 years ago
250d35107c
[twitter] prioritize tweet type checks ( #3439 )
...
Do not consider a tweet seen before applying 'retweet', 'quote' and
'reply' checks. Otherwise the original tweets will also be skipped if
the "derivative" tweets and the original tweets are from the same user.
2 years ago
1800bd7d14
allow '*-filter' options to be a list of expressions
2 years ago
3eb352fcb0
[twitter] force HTTPS for TwitPic URLs
2 years ago
73ab5d84c0
update docs/configuration.rst
2 years ago
2d7d80d302
release version 1.24.2
2 years ago
bee354c264
Merge pull request #3415 from enduser420/extractor/fapello
...
[fapello] add 'post', 'user' and 'path' extractors
2 years ago
8d7585534e
Merge pull request #3367 from the-blank-x/deviantart-view
...
[deviantart] add /view URL support
2 years ago
Mike Fährmann