2524 Commits (5f23441e12cce8609bfe07b958b2ad30158324ff)

Author	SHA1	Message	Date
Mike Fährmann	c3f01dc4e6	implement 'util.unique()'	4 years ago
Mike Fährmann	558cde139c	[paheal] fix extraction (fixes #1088)	4 years ago
Mike Fährmann	0211af7ca8	[hentaifoundry] update 'YII_CSRF_TOKEN' cookie handling ... (fixes #1083)	4 years ago
Mike Fährmann	d83b95fd28	[postprocessor:metadata] accept a string-list for 'content-format' ... (closes #1080)	4 years ago
Mike Fährmann	198c33ec36	also collect post processors from 'basecategory' entries ... (fixes #1084)	4 years ago
Mike Fährmann	350b1afe1c	speed up _list_classes() after iterating over all modules once	4 years ago
Mike Fährmann	5bcf28de93	add a 'extractor.modules' option	4 years ago
Mike Fährmann	18213dc5ba	release version 1.15.2	4 years ago
Mike Fährmann	de4a1e45c9	improve 'generate_csrf_token()' ... no need to use hashlib.md5()	4 years ago
Mike Fährmann	b788712844	[fallenangels] fix extraction of '.5' chapters	4 years ago
Mike Fährmann	28d8541cb3	[mangafox] ensure download URLs have a scheme	4 years ago
Mike Fährmann	8e3a324c91	[mangakakalot] ignore "Go Home" buttons in chapter pages	4 years ago
Mike Fährmann	c14c5d82d6	[newgrounds] use generator for fallback URLs	4 years ago
Mike Fährmann	a09f42f6b3	improve filename_from_url() performance ... Manually extracting the part between the last '/' and '?' instead of relying on the standard libraries' 'urllib.parse.urlsplit()' increases performance by ~400%. urlsplit() : 3.64 secs per 1.000.000 iterations partition(): 0.87 secs per 1.000.000 iterations	4 years ago
Mike Fährmann	968d3e8465	remove '&' from URL patterns ... '/?&#' -> '/?#' and '?&#' -> '?#' According to https://www.ietf.org/rfc/rfc3986.txt, URLs are "organized hierarchically" by using "the slash ("/"), question mark ("?"), and number sign ("#") characters to delimit components"	4 years ago
Mike Fährmann	1686dc1757	[twitter] support media from Cards (#1005, #937) ... Can be enabled with 'extractor.twitter.cards', but for now disabled by default because cards can redirect to rather large videos from YouTube or Twitch.	4 years ago
Mike Fährmann	ffd38215a4	[hitomi] fix image URLs and URL pattern ... - non-webp files are now hosted on [a-c]b.hitomi.la - removed ampersand from invalid slug characters	4 years ago
Mike Fährmann	286718950c	[mangahere] ensure download URLs have a scheme (fixes #1070)	4 years ago
Mike Fährmann	76dfa11a65	[reddit] add 'date' metadata field (closes #1068)	4 years ago
Mike Fährmann	3f2ba629ea	[newgrounds] provide fallback URLs for video downloads (#1042)	4 years ago
Mike Fährmann	a3ca2f6080	update fallback URL handling ... remove Message.Urllist and use a '_fallback' field inside a kwdict	4 years ago
Mike Fährmann	43dab3a228	[mangadex] unescape more metadata fields (fixes #1066) ... like 'manga', 'author', 'artist', etc.	4 years ago
Mike Fährmann	ec61696316	add 't' format string conversion (closes #1065) ... to Trim whitespace from the beginning and end of strings. Example: '{field!t}' becomes 'foo' for 'field' == " \nfoo\t\r"	4 years ago
Mike Fährmann	5565025221	[xhamster] fix user profile extraction	4 years ago
Mike Fährmann	07432d6262	[seiga] fix flake8 and cookie test (#1063)	4 years ago
Mike Fährmann	b8daabc3ca	[pinterest] implement login support (closes #1055) ... being logged allows access to secret/protected boards	4 years ago
Mike Fährmann	1b1cf01d0d	add a general 'generate_csrf_token()' function	4 years ago
Mike Fährmann	7a0ba370d1	[gelbooru] rewrite mp4 video URLs (fixes #1048)	4 years ago
Mike Fährmann	6491db3eaf	[blogger] handle URLs with specified width/height (closes #1061) ... get highest quality for images with /wXXX-hXXX/ instead of the usual /sXXX/	4 years ago
Mike Fährmann	783e0af26d	[hentaifoundry] update and simplify	4 years ago
Mike Fährmann	5b844a72b7	[newgrounds] handle embeds without scheme (#1033)	4 years ago
kurumigi	7e0e872f4f	[seiga] Add metadata for single image downloads (#1063) ... * [seiga] Support image metadata. * [seiga] Update test data. * [seiga] Fix cookie check. * [test_cookies] [seiga] Fit test_cookies.py to the last commit.	4 years ago
Zanny	3ec60e894a	[weasyl] api-key authentication (#1057) ... * [weasyl] support api keys * [weasyl] document api-key authentication * [weasyl] usernames can contain ~	4 years ago
Mike Fährmann	35056a07d1	release version 1.15.1	4 years ago
Mike Fährmann	844793847c	update extractor test results	4 years ago
Mike Fährmann	ddd6840509	[behance] fix 'collection' extraction	4 years ago
Mike Fährmann	c5e3971b18	[newgrounds] extract image embeds (closes #1033)	4 years ago
dawidsowa	43b156fb40	[reactor] match URLs without subdomain (#1053)	4 years ago
Mike Fährmann	fd20093c96	allow blacklist/whitelist to be empty lists/strings (#1051)	4 years ago
Mike Fährmann	3ebb174f2c	add missing extractor info when spawning new ones (fixes #1051) ... Not having this information causes the blacklist/whitelist logic to trigger and prevents things from functioning as intended when using default settings. Fixes issues for 8muses, deviantart, exhentai, and mangoxo.	4 years ago
Mike Fährmann	f9c1684af7	[newgrounds] restore original video URLs (#1042)	4 years ago
Mike Fährmann	73373c06ec	[weibo] handle posts with more than 9 images (closes #926) ... Responses from '/api/container/getIndex' don't list more than 9 images per 'status' object, but the embedded JSON from a '/detail/<ID>' page does.	4 years ago
Mike Fährmann	dd1e545597	[hentaifoundry] rename GalleryExtractor to PicturesExtractor	4 years ago
Mike Fährmann	c874071f5a	[kissmanga] remove module	4 years ago
Mike Fährmann	93e04bf9a9	[500px] update query hashes	4 years ago
Mike Fährmann	844502cad5	update extractor test results	4 years ago
Mike Fährmann	fad7748b6b	[xvideos] fix 'title' extraction	4 years ago
Mike Fährmann	5b927c15df	[newgrounds] fix video extraction (closes #1042)	4 years ago
Mike Fährmann	bdc6c8f074	improve message for 'oauth:deviantart' etc (closes #989)	4 years ago
Mike Fährmann	430b6d6e2e	[twitter] extend 'retweets' option (closes #1026) ... Setting 'retweets' to '"original"' will use metadata from the original retweeted Tweets, and not from the Retweet entry.	4 years ago
Mike Fährmann	b9bdd2c564	[hentaifoundry] add support for stories (closes #734)	4 years ago
Mike Fährmann	9a9d1924d8	[hentaicafe] add 'manga_id' metadata field (closes #1036) ... This field is only available when using a non-foolslide URL like '/hc.fyi/9874' or '/hazuki-yuuto-summer-blues/'	4 years ago
Mike Fährmann	cc4ac80302	[weasyl] add 'favorite' extractor (#1032)	4 years ago
Mike Fährmann	e9cc719497	[weasyl] update and simplify ... - simplify 'pattern' regexps - parse 'posted_at' as 'date' - use unaltered 'title' ({title!l:R /_/} to lowercase and replace spaces)	4 years ago
Mike Fährmann	6514312126	[nijie] add 'include' option (closes #1018)	4 years ago
Mike Fährmann	0d43456323	[hentaifoundry] add 'include' option	4 years ago
Zanny	ebb7737b9b	Weasyl Extractor (#977) ... * weasyl extractor * @kattjevfel suggested changes * @mikf changes	4 years ago
Mike Fährmann	d5fa716d89	fix crash when using 'skip=false' and archive (fixes #1023) ... Separating the archive check from pathfmt.exists() in b5243297 had some unintended side effects. It is also not possible to monkey-patch a dunder method like __contains__ because of the special method lookup that gets performed for them.	4 years ago
Mike Fährmann	aeb0d32333	[twitter] improve twitpic extraction (fixes #1019) ... - ignore twitpic.com/photos/… URLs - ignore empty image URLs	4 years ago
Mike Fährmann	2184ec5d78	release version 1.15.0	4 years ago
Mike Fährmann	7cd383c0f9	update extractor test results	4 years ago
Mike Fährmann	1e313d5b84	implement 'sleep-request' option	4 years ago
Mike Fährmann	65744a7a31	use alternative for all falsey values in format strings ... … and not just None (#525) It would be better to consistently use None for all non-existent fields and/or fields without a valid value, but this is a good enough workaround for now.	4 years ago
Mike Fährmann	c43b3894be	[myhentaigallery] update and fix extraction (#1001) ... - extract more metadata - match "/show/" URLs - complete test results - fix missing images for lines starting with " <img" - fix missing comma in supportedsites.py	4 years ago
choeronline	05b9ac8d37	[myhentaigallery] add extractor (#1001) ... * adds support for myhentaigallery * fixes linting issues in myhentaigallery extractor	4 years ago
Mike Fährmann	2626629117	[danbooru] handle posts without 'id' (fixes #1004)	4 years ago
Mike Fährmann	cc1fb0b4ea	[500px] update query hash	4 years ago
Mike Fährmann	da87a5fb7e	[exhentai] fix accessing config before main constructor ... bug introduced with 055c32e0 Making 'Extractor.config()' quite a bit faster is worth the "cost" of having to set _cfgpath in exhentai constructors, I think.	4 years ago
Mike Fährmann	f5b7ae01c1	update extractor test results	4 years ago
Mike Fährmann	136df52d1f	[deviantart] support watchers-only/paid deviations (#995)	4 years ago
Mike Fährmann	055c32e0f7	precompute extractor config paths	4 years ago
Mike Fährmann	231dd4c800	accumulate postprocessor objects (#994) ... Instead of one 'postprocessors' setting overwriting all others lower in the hierarchy, all postprocessors along the config path will now get collected into one big list. For example '--mtime-from-date' will therefore no longer cause other postprocessor settings in a config file to get ignored.	4 years ago
Mike Fährmann	392d022b04	implement 'config.accumulate()' (#994)	4 years ago
Mike Fährmann	3afd362e2e	add 'sleep-extractor' option (closes #964) ... (would have been nice if this were possible without code duplication)	4 years ago
Mike Fährmann	3108e85b89	[worldthree] remove extractors ... http://www.slide.world-three.org/ hasn't been accessible for a long time.	4 years ago
Mike Fährmann	8fed3eb8cb	[jaiminisbox] remove extractors ... https://jaiminisbox.com/post.html	4 years ago
Mike Fährmann	dcf3ad7eef	[furaffinity] update download URL extraction (fixes #988) ... support the new 'd2.facdn.net' subdomain	4 years ago
Mike Fährmann	3918b69677	remove 'extractor.blacklist' context manager	4 years ago
Mike Fährmann	c78aa17506	add general 'blacklist' and 'whitelist' options (#492, #844)	4 years ago
Mike Fährmann	abda352a5b	add '--no-skip' command-line option (closes #986)	4 years ago
Mike Fährmann	5912727b88	support format string replacement fields in archive paths ... (closes #985)	4 years ago
Mike Fährmann	2b8d57f0ab	[twitter] support '/intent/user?user_id=…' URLs (#980)	4 years ago
Mike Fährmann	a3b473bd2f	[twitter] support specifying users by ID (#980) ... by using 'id:…' as their screen name, i.e. https://www.twitter.com/id:2976459548/media instead of https://twitter.com/supernaturepics/media The user ID can, for example, be obtained from the output of $ gallery-dl -j --range 1 https://twitter.com/<screen-name>	4 years ago
Mike Fährmann	a0d916ed41	[exhentai] update wait time before original image download (#978) ... depend on 'wait-max', don't use a hard-coded value	4 years ago
Mike Fährmann	f6fd449b59	reduce wait time growth rate from exponential to linear ... Waiting for 2**N seconds after each error grows too fast. Simply waiting N seconds seems far more reasonable.	4 years ago
Mike Fährmann	bc48514d84	[aryion] get post ID via gallery-item (fixes #981, closes #982) ... this even works when fetching post IDs from '/latest.php?id='	4 years ago
Mike Fährmann	799ca07fc8	[imgur] update ... - fix image/album detection for galleries - use new API endpoints for image/album data	4 years ago
Mike Fährmann	b5243297ff	write skipped files to archive (closes #550)	4 years ago
Mike Fährmann	ac3036ef56	add 'filesize-min' and 'filesize-max' options (closes #780)	4 years ago
Mike Fährmann	7876a03ece	[tumblr] create directories for each post (fixes #965) ... This changes the identifiers for directory format string fields. Everything blog related is now inside a 'blog' object and not at the "base level" anymore. E.g. '{name}' for directories is now '{blog[name]}' (or '{blog_name}', since that is also available)	4 years ago
Mike Fährmann	fd0685d9b5	[postprocessor:zip] defer zip file creation (fixes #968) ... don't try to create zip files on postprocessor construction, wait until directory creation during file download,	4 years ago
Mike Fährmann	33fe67b594	release version 1.14.5	4 years ago
Mike Fährmann	d50f3b333a	update extractor test results	4 years ago
Mike Fährmann	0f55b8e80a	[exhentai] fix type check from dbbbb21 (#940) ... 'bool' is a subclass of 'int', and therefore 'isinstance(self.limits, int)' also returns True when 'self.limits' has a boolean value	4 years ago
Mike Fährmann	e33293fdd8	[hentaihand] update to new site layout	4 years ago
Mike Fährmann	fda9e296dd	[gelbooru] fix extraction without API	4 years ago
Mike Fährmann	69e4871005	update extractor test results ... - sensescans: replace 404d chapters - mangapark: replace 404d chapters - subscribestar: update test for attached files	4 years ago
Mike Fährmann	ab1af66a97	[imgur] add 'search' extractor (#934)	4 years ago
Mike Fährmann	e4bbc1fb5c	[imgur] add 'tag' extractor (#934)	4 years ago
Mike Fährmann	deaacc70bb	[hitomi] update URL pattern for tag searches	4 years ago