Commit Graph

2697 Commits (e165e6c265e5238be81b2484d01f85e31dbfc792)

Author SHA1 Message Date
Mike Fährmann 371e9ca6df
[pinterest] implement video support (closes #1189)
4 years ago
Mike Fährmann 537742c0ee
[sankaku] normalize 'created_at' metadata (closes #1190)
4 years ago
Mike Fährmann ae6748996a
[pornhub] update tests
4 years ago
Mike Fährmann bf629a2818
[instagram] add 'include' option (closes #1180)
4 years ago
Mike Fährmann 78061658ea
[booru] reduce exceptions caught during _prepare_post()
4 years ago
Mike Fährmann 212ae0c399
[mangapanda] remove module
4 years ago
Mike Fährmann 337b118e25
[instagram] warn about private profiles (#1187)
4 years ago
Mike Fährmann e8c64dd961
[postprocessor:exec] do not auto-add '{}' to command (#1185)
4 years ago
Mike Fährmann 0a3bbc9c63
[postprocessor:exec] update output
4 years ago
Mike Fährmann 511d8d3fa3
increase SQLite connection timeouts (#1173)
4 years ago
Mike Fährmann 465015f75a
[sankaku] reimplement login support (#1176, #1182)
4 years ago
Mike Fährmann 8d2e4e5f13
[booru] improve error handling
4 years ago
Mike Fährmann 1f9121fecb
release version 1.16.0
4 years ago
Mike Fährmann 1d753542c2
[hentainexus] fix extraction (fixes #1166)
4 years ago
Mike Fährmann b6f1fe59cb
add deprecation warnings for exec.final and metadata.bypost
4 years ago
Mike Fährmann 476d563ec2
[downloader:http] add MIME type and signature for .swf files
4 years ago
Mike Fährmann a00b60fbe7
[twitter] update 'x-csrf-token' header (fixes #1170)
4 years ago
Mike Fährmann b88c97b873
[instagram] add 'cursor' option (#1149)
4 years ago
Mike Fährmann 0d406c8daf
[common] restrict values used in 'generate_extractors()'
4 years ago
Mike Fährmann fe0265c7a5
[downloader.http] small improvements to file signature list
4 years ago
Mike Fährmann b2c55f0a72
[sankaku] remove login support
4 years ago
Mike Fährmann 7f3d811d7b
[moebooru] inherit from BooruExtractor
4 years ago
Mike Fährmann a3a863fc13
[booru] add generalized extractors for *booru sites
4 years ago
Mike Fährmann 5f23441e12
[piczel] update API URLs
4 years ago
Mike Fährmann 47114339a2
[webtoons] update 'ageGate' cookie
4 years ago
Mike Fährmann 4225f12783
[nozomi] handle empty 'date' fields (fixes #1163)
4 years ago
Mike Fährmann 2b93515ee0
[instagram] reimplement support for stories (#1149)
4 years ago
Mike Fährmann ecdea799dd
[sankaku] use 'beta.sankakucomplex.com' API endpoints
4 years ago
Mike Fährmann b3ecc89a9a
[instagram] use double quotes for strings when possible
4 years ago
Mike Fährmann 76285eb60d
[instagram] reimplement support for story highlights (#1149)
4 years ago
Mike Fährmann 8ca7f54750
rename '_request_…' variables
4 years ago
Mike Fährmann 15a122aff3
[instagram] update 'X-IG-WWW-Claim' headers
4 years ago
Mike Fährmann e5d81bdc7b
[mangadex] handle 'external' chapters (closes #1154)
4 years ago
Mike Fährmann 447488fb18
[instagram] rewrite
4 years ago
Mike Fährmann cc15fbe71a
[moebooru] add generalized extractors for moebooru sites
4 years ago
Mike Fährmann 43120407cc
[paheal] create directory for each post (closes #1147)
4 years ago
Mike Fährmann 63e61a0932
[twitter] update image URL format (#1145)
4 years ago
Mike Fährmann 1a4b61f7eb
[downloader:http] fix issues with chunked transfer encoding
4 years ago
Mike Fährmann 536c088462
[downloader:http] improve 'adjust-extensions' (#776)
4 years ago
Mike Fährmann 46323ae6ff
initialize 'hooks' as empty tuple
4 years ago
Mike Fährmann 9c29fc4e55
always initialize DownloadJob.hooks (fixes #1135)
4 years ago
Mike Fährmann ae6a1d5fbc
[mangoxo] fix extraction 2
4 years ago
Mike Fährmann f6a684bc37
[hentainexus] update data decoding procedure (#1125)
4 years ago
Mike Fährmann c57a918f4a
[e621] implement delay via '_request_interval_min'
4 years ago
Mike Fährmann 93ce7466e2
[2chan] skip external links
4 years ago
Mike Fährmann b214e89b5c
[mangoxo] fix extraction
4 years ago
Mike Fährmann 578dcf805c
[mangapanda] don't force https://
4 years ago
Mike Fährmann 102c482f5e
[reddit] skip invalid/failed gallery items (fixes #1127)
4 years ago
Mike Fährmann 174945d2b2
[hentainexus] fix extraction (fixes #1125)
4 years ago
Mike Fährmann ca59bd691c
[postprocessor:metadata] add 'event' and 'filename' options
4 years ago
Mike Fährmann 9c3568c397
[postprocessor:exec] add 'event' option
4 years ago
Mike Fährmann 9fffa9c343
rework post processor callbacks
4 years ago
Mike Fährmann f99c6031e0
apply post processor blacklists/whitelists to basecategories
4 years ago
Mike Fährmann 1e3dd7330e
merge SharedConfigMixin functionality into Extractor
4 years ago
Mike Fährmann ddfb4fd07a
[twitter] use 'https://twitter.com/i/api/' for logged in users
4 years ago
Mike Fährmann 42ccae53c4
[mangadex] switch to API v2
4 years ago
Mike Fährmann ca44111726
[flickr] update
4 years ago
Mike Fährmann 9b1bd09454
change 'extension-map' default
4 years ago
Mike Fährmann e5438b8a29
release version 1.15.3
4 years ago
Mike Fährmann de0c57886d
[twitter] add 'list-members' extractor (closes #1096)
4 years ago
Mike Fährmann 904ba08568
[gfycat] fix default filename format
4 years ago
Mike Fährmann a46561bc16
[500px] update query hashes
4 years ago
Mike Fährmann 2e3a0dff21
[8kun] fix file URLs of older posts (fixes #1101)
4 years ago
Mike Fährmann 00825cddf5
[hentaifoundry] use scheme from input URL (fixes #1095)
4 years ago
Mike Fährmann 8a98d3549a
[weasyl] create directory for each favorite submission
4 years ago
Mike Fährmann 91db8df1c7
[deviantart] add 'index_base36' metadata field (closes #1099)
4 years ago
Mike Fährmann b9bfa4c675
update extractor test results
4 years ago
Mike Fährmann 1b5b789401
[mangoxo] fix metadata extraction
4 years ago
Mike Fährmann 41d4968866
[twitter] add 'list' extractor (#1096)
4 years ago
Mike Fährmann 5d10520f4c
[twitter] update GraphQL endpoint & fix width/height entries
4 years ago
Mike Fährmann 9b2e5f72d6
[exhentai] update image URL parsing (#1094)
4 years ago
Mike Fährmann e3480bc8de
implement 'extension-map' option (#318)
4 years ago
Mike Fährmann 98a4d86a01
[sankakucomplex] extract videos and embeds (closes #308)
4 years ago
Mike Fährmann c3f01dc4e6
implement 'util.unique()'
4 years ago
Mike Fährmann 558cde139c
[paheal] fix extraction (fixes #1088)
4 years ago
Mike Fährmann 0211af7ca8
[hentaifoundry] update 'YII_CSRF_TOKEN' cookie handling
4 years ago
Mike Fährmann d83b95fd28
[postprocessor:metadata] accept a string-list for 'content-format'
4 years ago
Mike Fährmann 198c33ec36
also collect post processors from 'basecategory' entries
4 years ago
Mike Fährmann 350b1afe1c
speed up _list_classes() after iterating over all modules once
4 years ago
Mike Fährmann 5bcf28de93
add a 'extractor.modules' option
4 years ago
Mike Fährmann 18213dc5ba
release version 1.15.2
4 years ago
Mike Fährmann de4a1e45c9
improve 'generate_csrf_token()'
4 years ago
Mike Fährmann b788712844
[fallenangels] fix extraction of '.5' chapters
4 years ago
Mike Fährmann 28d8541cb3
[mangafox] ensure download URLs have a scheme
4 years ago
Mike Fährmann 8e3a324c91
[mangakakalot] ignore "Go Home" buttons in chapter pages
4 years ago
Mike Fährmann c14c5d82d6
[newgrounds] use generator for fallback URLs
4 years ago
Mike Fährmann a09f42f6b3
improve filename_from_url() performance
4 years ago
Mike Fährmann 968d3e8465
remove '&' from URL patterns
4 years ago
Mike Fährmann 1686dc1757
[twitter] support media from Cards (#1005, #937)
4 years ago
Mike Fährmann ffd38215a4
[hitomi] fix image URLs and URL pattern
4 years ago
Mike Fährmann 286718950c
[mangahere] ensure download URLs have a scheme (fixes #1070)
4 years ago
Mike Fährmann 76dfa11a65
[reddit] add 'date' metadata field (closes #1068)
4 years ago
Mike Fährmann 3f2ba629ea
[newgrounds] provide fallback URLs for video downloads (#1042)
4 years ago
Mike Fährmann a3ca2f6080
update fallback URL handling
4 years ago
Mike Fährmann 43dab3a228
[mangadex] unescape more metadata fields (fixes #1066)
4 years ago
Mike Fährmann ec61696316
add 't' format string conversion (closes #1065)
4 years ago
Mike Fährmann 5565025221
[xhamster] fix user profile extraction
4 years ago
Mike Fährmann 07432d6262
[seiga] fix flake8 and cookie test (#1063)
4 years ago
Mike Fährmann b8daabc3ca
[pinterest] implement login support (closes #1055)
4 years ago
Mike Fährmann 1b1cf01d0d
add a general 'generate_csrf_token()' function
4 years ago
Mike Fährmann 7a0ba370d1
[gelbooru] rewrite mp4 video URLs (fixes #1048)
4 years ago
Mike Fährmann 6491db3eaf
[blogger] handle URLs with specified width/height (closes #1061)
4 years ago
Mike Fährmann 783e0af26d
[hentaifoundry] update and simplify
4 years ago
Mike Fährmann 5b844a72b7
[newgrounds] handle embeds without scheme (#1033)
4 years ago
kurumigi 7e0e872f4f
[seiga] Add metadata for single image downloads (#1063)
4 years ago
Zanny 3ec60e894a
[weasyl] api-key authentication (#1057)
4 years ago
Mike Fährmann 35056a07d1
release version 1.15.1
4 years ago
Mike Fährmann 844793847c
update extractor test results
4 years ago
Mike Fährmann ddd6840509
[behance] fix 'collection' extraction
4 years ago
Mike Fährmann c5e3971b18
[newgrounds] extract image embeds (closes #1033)
4 years ago
dawidsowa 43b156fb40
[reactor] match URLs without subdomain (#1053)
4 years ago
Mike Fährmann fd20093c96
allow blacklist/whitelist to be empty lists/strings (#1051)
4 years ago
Mike Fährmann 3ebb174f2c
add missing extractor info when spawning new ones (fixes #1051)
4 years ago
Mike Fährmann f9c1684af7
[newgrounds] restore original video URLs (#1042)
4 years ago
Mike Fährmann 73373c06ec
[weibo] handle posts with more than 9 images (closes #926)
4 years ago
Mike Fährmann dd1e545597
[hentaifoundry] rename GalleryExtractor to PicturesExtractor
4 years ago
Mike Fährmann c874071f5a
[kissmanga] remove module
4 years ago
Mike Fährmann 93e04bf9a9
[500px] update query hashes
4 years ago
Mike Fährmann 844502cad5
update extractor test results
4 years ago
Mike Fährmann fad7748b6b
[xvideos] fix 'title' extraction
4 years ago
Mike Fährmann 5b927c15df
[newgrounds] fix video extraction (closes #1042)
4 years ago
Mike Fährmann bdc6c8f074
improve message for 'oauth:deviantart' etc (closes #989)
4 years ago
Mike Fährmann 430b6d6e2e
[twitter] extend 'retweets' option (closes #1026)
4 years ago
Mike Fährmann b9bdd2c564
[hentaifoundry] add support for stories (closes #734)
4 years ago
Mike Fährmann 9a9d1924d8
[hentaicafe] add 'manga_id' metadata field (closes #1036)
4 years ago
Mike Fährmann cc4ac80302
[weasyl] add 'favorite' extractor (#1032)
4 years ago
Mike Fährmann e9cc719497
[weasyl] update and simplify
4 years ago
Mike Fährmann 6514312126
[nijie] add 'include' option (closes #1018)
4 years ago
Mike Fährmann 0d43456323
[hentaifoundry] add 'include' option
4 years ago
Zanny ebb7737b9b
Weasyl Extractor (#977)
4 years ago
Mike Fährmann d5fa716d89
fix crash when using 'skip=false' and archive (fixes #1023)
4 years ago
Mike Fährmann aeb0d32333
[twitter] improve twitpic extraction (fixes #1019)
4 years ago
Mike Fährmann 2184ec5d78
release version 1.15.0
4 years ago
Mike Fährmann 7cd383c0f9
update extractor test results
4 years ago
Mike Fährmann 1e313d5b84
implement 'sleep-request' option
4 years ago
Mike Fährmann 65744a7a31
use alternative for all falsey values in format strings
4 years ago
Mike Fährmann c43b3894be
[myhentaigallery] update and fix extraction (#1001)
4 years ago
choeronline 05b9ac8d37
[myhentaigallery] add extractor (#1001)
4 years ago
Mike Fährmann 2626629117
[danbooru] handle posts without 'id' (fixes #1004)
4 years ago
Mike Fährmann cc1fb0b4ea
[500px] update query hash
4 years ago
Mike Fährmann da87a5fb7e
[exhentai] fix accessing config before main constructor
4 years ago
Mike Fährmann f5b7ae01c1
update extractor test results
4 years ago
Mike Fährmann 136df52d1f
[deviantart] support watchers-only/paid deviations (#995)
4 years ago
Mike Fährmann 055c32e0f7
precompute extractor config paths
4 years ago
Mike Fährmann 231dd4c800
accumulate postprocessor objects (#994)
4 years ago
Mike Fährmann 392d022b04
implement 'config.accumulate()' (#994)
4 years ago
Mike Fährmann 3afd362e2e
add 'sleep-extractor' option (closes #964)
4 years ago
Mike Fährmann 3108e85b89
[worldthree] remove extractors
4 years ago
Mike Fährmann 8fed3eb8cb
[jaiminisbox] remove extractors
4 years ago
Mike Fährmann dcf3ad7eef
[furaffinity] update download URL extraction (fixes #988)
4 years ago
Mike Fährmann 3918b69677
remove 'extractor.blacklist' context manager
4 years ago
Mike Fährmann c78aa17506
add general 'blacklist' and 'whitelist' options (#492, #844)
4 years ago
Mike Fährmann abda352a5b
add '--no-skip' command-line option (closes #986)
4 years ago
Mike Fährmann 5912727b88
support format string replacement fields in archive paths
4 years ago
Mike Fährmann 2b8d57f0ab
[twitter] support '/intent/user?user_id=…' URLs (#980)
4 years ago
Mike Fährmann a3b473bd2f
[twitter] support specifying users by ID (#980)
4 years ago
Mike Fährmann a0d916ed41
[exhentai] update wait time before original image download (#978)
4 years ago
Mike Fährmann f6fd449b59
reduce wait time growth rate from exponential to linear
4 years ago
Mike Fährmann bc48514d84
[aryion] get post ID via gallery-item (fixes #981, closes #982)
4 years ago
Mike Fährmann 799ca07fc8
[imgur] update
4 years ago
Mike Fährmann b5243297ff
write skipped files to archive (closes #550)
4 years ago
Mike Fährmann ac3036ef56
add 'filesize-min' and 'filesize-max' options (closes #780)
4 years ago
Mike Fährmann 7876a03ece
[tumblr] create directories for each post (fixes #965)
4 years ago
Mike Fährmann fd0685d9b5
[postprocessor:zip] defer zip file creation (fixes #968)
4 years ago
Mike Fährmann 33fe67b594
release version 1.14.5
4 years ago
Mike Fährmann d50f3b333a
update extractor test results
4 years ago
Mike Fährmann 0f55b8e80a
[exhentai] fix type check from dbbbb21 (#940)
4 years ago
Mike Fährmann e33293fdd8
[hentaihand] update to new site layout
4 years ago
Mike Fährmann fda9e296dd
[gelbooru] fix extraction without API
4 years ago
Mike Fährmann 69e4871005
update extractor test results
4 years ago
Mike Fährmann ab1af66a97
[imgur] add 'search' extractor (#934)
4 years ago
Mike Fährmann e4bbc1fb5c
[imgur] add 'tag' extractor (#934)
4 years ago
Mike Fährmann deaacc70bb
[hitomi] update URL pattern for tag searches
4 years ago
ArtaxIsSleeping 0e941553ec
[aryion] Add username/password support (#960)
4 years ago
Mike Fährmann 84e04cc23b
[500px] fix extraction and update URL patterns (fixes #956)
4 years ago
Mike Fährmann d4ff767291
[reddit] improve gallery extraction (fixes #955)
4 years ago
Mike Fährmann 7140fe7e6d
[hitomi] fix redirect processing
4 years ago
Mike Fährmann a57b6b3c3a
[reddit] handle deleted galleries (fixes #953)
4 years ago
Mike Fährmann 063c71cd84
[furaffinity] add 'search' extractor (closes #915)
4 years ago
Mike Fährmann dbbbb21180
[exhentai] add ability to specify custom image limit (#940)
4 years ago
Mike Fährmann b2009ea39e
[aryion] update folder mime type list (fixes #945)
4 years ago
Mike Fährmann 688bd046fc
release version 1.14.4
4 years ago
Mike Fährmann d06ad148c7
[shopify] use alternate regex for products on collection pages
4 years ago
Mike Fährmann 7619152988
[reactor] sort 'tags'
4 years ago
Mike Fährmann cd9de613a2
[exhentai] adjust image limit costs (#940)
4 years ago
Mike Fährmann 2e6f6ee1c1
[mangoxo] fix login
4 years ago
Mike Fährmann a6a080656c
[pixnet] detect password-protected albums (#177)
4 years ago
Mike Fährmann 67ac6667af
[mangareader] fix extraction
4 years ago
Mike Fährmann 2b88c90f6f
[blogger] add search extractor (#925)
4 years ago
Mike Fährmann d5067c51c5
[instagram] support '/reel/' URLs
4 years ago
Mike Fährmann 2c9766b29f
fix UnboundLocalError in Extractor.request()
4 years ago
Mike Fährmann aa64149583
[blogger] support searching posts by labels (closes #925)
4 years ago
Mike Fährmann 60ba3cb946
[reddit] support gallery posts (closes #920)
4 years ago
Mike Fährmann 0d84d3af55
[subscribestar] extract attached media files (#852)
4 years ago
Mike Fährmann 19bf76bcf8
update extractor test results
4 years ago
Mike Fährmann 0762d6b29c
[inkbunny] add 'num' field (#283)
4 years ago
Mike Fährmann fbc4278fe4
[instagram] wait before GraphQL requests (#901)
4 years ago
Mike Fährmann ec5870576d
[imgur] handle 403 overcapacity responses (closes #910)
4 years ago
Mike Fährmann d6a271d2c7
add 'response' objects to 'HttpError's
4 years ago
Mike Fährmann 72c5578a27
[hentainexus] improve/simplify code
4 years ago