2743 Commits (284623566928a4835a759e27237faebeea9abc54)

Author	SHA1	Message	Date
Mike Fährmann	ddfb4fd07a	[twitter] use 'https://twitter.com/i/api/' for logged in users ... Doesn't seem to make a difference from what I can tell, i.e. downloaded files are the same, but the website does it.	4 years ago
Mike Fährmann	42ccae53c4	[mangadex] switch to API v2 ... https://mangadex.org/api/v2/ https://mangadex.org/thread/351011	4 years ago
Mike Fährmann	ca44111726	[flickr] update ... - ensure every photo has an 'owner' (#828) - change default directories to a more consistent schema - create directory for each photo	4 years ago
Mike Fährmann	9b1bd09454	change 'extension-map' default ... Replace all JPEG filename extensions with 'jpg'.	4 years ago
Mike Fährmann	e5438b8a29	release version 1.15.3	4 years ago
Mike Fährmann	de0c57886d	[twitter] add 'list-members' extractor (closes #1096)	4 years ago
Mike Fährmann	904ba08568	[gfycat] fix default filename format	4 years ago
Mike Fährmann	a46561bc16	[500px] update query hashes	4 years ago
Mike Fährmann	2e3a0dff21	[8kun] fix file URLs of older posts (fixes #1101)	4 years ago
Mike Fährmann	00825cddf5	[hentaifoundry] use scheme from input URL (fixes #1095) ... Let the user choose between http and https, instead of always forcing https.	4 years ago
Mike Fährmann	8a98d3549a	[weasyl] create directory for each favorite submission ... (#1032)	4 years ago
Mike Fährmann	91db8df1c7	[deviantart] add 'index_base36' metadata field (closes #1099) ... This is the same ID as found in 'filename' without the 'd' in front, which is just 'index' encoded in base36.	4 years ago
Mike Fährmann	b9bfa4c675	update extractor test results	4 years ago
Mike Fährmann	1b5b789401	[mangoxo] fix metadata extraction	4 years ago
Mike Fährmann	41d4968866	[twitter] add 'list' extractor (#1096)	4 years ago
Mike Fährmann	5d10520f4c	[twitter] update GraphQL endpoint & fix width/height entries	4 years ago
Mike Fährmann	9b2e5f72d6	[exhentai] update image URL parsing (#1094)	4 years ago
Mike Fährmann	e3480bc8de	implement 'extension-map' option (#318)	4 years ago
Mike Fährmann	98a4d86a01	[sankakucomplex] extract videos and embeds (closes #308)	4 years ago
Mike Fährmann	c3f01dc4e6	implement 'util.unique()'	4 years ago
Mike Fährmann	558cde139c	[paheal] fix extraction (fixes #1088)	4 years ago
Mike Fährmann	0211af7ca8	[hentaifoundry] update 'YII_CSRF_TOKEN' cookie handling ... (fixes #1083)	4 years ago
Mike Fährmann	d83b95fd28	[postprocessor:metadata] accept a string-list for 'content-format' ... (closes #1080)	4 years ago
Mike Fährmann	198c33ec36	also collect post processors from 'basecategory' entries ... (fixes #1084)	4 years ago
Mike Fährmann	350b1afe1c	speed up _list_classes() after iterating over all modules once	4 years ago
Mike Fährmann	5bcf28de93	add a 'extractor.modules' option	4 years ago
Mike Fährmann	18213dc5ba	release version 1.15.2	4 years ago
Mike Fährmann	de4a1e45c9	improve 'generate_csrf_token()' ... no need to use hashlib.md5()	4 years ago
Mike Fährmann	b788712844	[fallenangels] fix extraction of '.5' chapters	4 years ago
Mike Fährmann	28d8541cb3	[mangafox] ensure download URLs have a scheme	4 years ago
Mike Fährmann	8e3a324c91	[mangakakalot] ignore "Go Home" buttons in chapter pages	4 years ago
Mike Fährmann	c14c5d82d6	[newgrounds] use generator for fallback URLs	4 years ago
Mike Fährmann	a09f42f6b3	improve filename_from_url() performance ... Manually extracting the part between the last '/' and '?' instead of relying on the standard libraries' 'urllib.parse.urlsplit()' increases performance by ~400%. urlsplit() : 3.64 secs per 1.000.000 iterations partition(): 0.87 secs per 1.000.000 iterations	4 years ago
Mike Fährmann	968d3e8465	remove '&' from URL patterns ... '/?&#' -> '/?#' and '?&#' -> '?#' According to https://www.ietf.org/rfc/rfc3986.txt, URLs are "organized hierarchically" by using "the slash ("/"), question mark ("?"), and number sign ("#") characters to delimit components"	4 years ago
Mike Fährmann	1686dc1757	[twitter] support media from Cards (#1005, #937) ... Can be enabled with 'extractor.twitter.cards', but for now disabled by default because cards can redirect to rather large videos from YouTube or Twitch.	4 years ago
Mike Fährmann	ffd38215a4	[hitomi] fix image URLs and URL pattern ... - non-webp files are now hosted on [a-c]b.hitomi.la - removed ampersand from invalid slug characters	4 years ago
Mike Fährmann	286718950c	[mangahere] ensure download URLs have a scheme (fixes #1070)	4 years ago
Mike Fährmann	76dfa11a65	[reddit] add 'date' metadata field (closes #1068)	4 years ago
Mike Fährmann	3f2ba629ea	[newgrounds] provide fallback URLs for video downloads (#1042)	4 years ago
Mike Fährmann	a3ca2f6080	update fallback URL handling ... remove Message.Urllist and use a '_fallback' field inside a kwdict	4 years ago
Mike Fährmann	43dab3a228	[mangadex] unescape more metadata fields (fixes #1066) ... like 'manga', 'author', 'artist', etc.	4 years ago
Mike Fährmann	ec61696316	add 't' format string conversion (closes #1065) ... to Trim whitespace from the beginning and end of strings. Example: '{field!t}' becomes 'foo' for 'field' == " \nfoo\t\r"	4 years ago
Mike Fährmann	5565025221	[xhamster] fix user profile extraction	4 years ago
Mike Fährmann	07432d6262	[seiga] fix flake8 and cookie test (#1063)	4 years ago
Mike Fährmann	b8daabc3ca	[pinterest] implement login support (closes #1055) ... being logged allows access to secret/protected boards	4 years ago
Mike Fährmann	1b1cf01d0d	add a general 'generate_csrf_token()' function	4 years ago
Mike Fährmann	7a0ba370d1	[gelbooru] rewrite mp4 video URLs (fixes #1048)	4 years ago
Mike Fährmann	6491db3eaf	[blogger] handle URLs with specified width/height (closes #1061) ... get highest quality for images with /wXXX-hXXX/ instead of the usual /sXXX/	4 years ago
Mike Fährmann	783e0af26d	[hentaifoundry] update and simplify	4 years ago
Mike Fährmann	5b844a72b7	[newgrounds] handle embeds without scheme (#1033)	4 years ago
kurumigi	7e0e872f4f	[seiga] Add metadata for single image downloads (#1063) ... * [seiga] Support image metadata. * [seiga] Update test data. * [seiga] Fix cookie check. * [test_cookies] [seiga] Fit test_cookies.py to the last commit.	4 years ago
Zanny	3ec60e894a	[weasyl] api-key authentication (#1057) ... * [weasyl] support api keys * [weasyl] document api-key authentication * [weasyl] usernames can contain ~	4 years ago
Mike Fährmann	35056a07d1	release version 1.15.1	4 years ago
Mike Fährmann	844793847c	update extractor test results	4 years ago
Mike Fährmann	ddd6840509	[behance] fix 'collection' extraction	4 years ago
Mike Fährmann	c5e3971b18	[newgrounds] extract image embeds (closes #1033)	4 years ago
dawidsowa	43b156fb40	[reactor] match URLs without subdomain (#1053)	4 years ago
Mike Fährmann	fd20093c96	allow blacklist/whitelist to be empty lists/strings (#1051)	4 years ago
Mike Fährmann	3ebb174f2c	add missing extractor info when spawning new ones (fixes #1051) ... Not having this information causes the blacklist/whitelist logic to trigger and prevents things from functioning as intended when using default settings. Fixes issues for 8muses, deviantart, exhentai, and mangoxo.	4 years ago
Mike Fährmann	f9c1684af7	[newgrounds] restore original video URLs (#1042)	4 years ago
Mike Fährmann	73373c06ec	[weibo] handle posts with more than 9 images (closes #926) ... Responses from '/api/container/getIndex' don't list more than 9 images per 'status' object, but the embedded JSON from a '/detail/<ID>' page does.	4 years ago
Mike Fährmann	dd1e545597	[hentaifoundry] rename GalleryExtractor to PicturesExtractor	4 years ago
Mike Fährmann	c874071f5a	[kissmanga] remove module	4 years ago
Mike Fährmann	93e04bf9a9	[500px] update query hashes	4 years ago
Mike Fährmann	844502cad5	update extractor test results	4 years ago
Mike Fährmann	fad7748b6b	[xvideos] fix 'title' extraction	4 years ago
Mike Fährmann	5b927c15df	[newgrounds] fix video extraction (closes #1042)	4 years ago
Mike Fährmann	bdc6c8f074	improve message for 'oauth:deviantart' etc (closes #989)	4 years ago
Mike Fährmann	430b6d6e2e	[twitter] extend 'retweets' option (closes #1026) ... Setting 'retweets' to '"original"' will use metadata from the original retweeted Tweets, and not from the Retweet entry.	4 years ago
Mike Fährmann	b9bdd2c564	[hentaifoundry] add support for stories (closes #734)	4 years ago
Mike Fährmann	9a9d1924d8	[hentaicafe] add 'manga_id' metadata field (closes #1036) ... This field is only available when using a non-foolslide URL like '/hc.fyi/9874' or '/hazuki-yuuto-summer-blues/'	4 years ago
Mike Fährmann	cc4ac80302	[weasyl] add 'favorite' extractor (#1032)	4 years ago
Mike Fährmann	e9cc719497	[weasyl] update and simplify ... - simplify 'pattern' regexps - parse 'posted_at' as 'date' - use unaltered 'title' ({title!l:R /_/} to lowercase and replace spaces)	4 years ago
Mike Fährmann	6514312126	[nijie] add 'include' option (closes #1018)	4 years ago
Mike Fährmann	0d43456323	[hentaifoundry] add 'include' option	4 years ago
Zanny	ebb7737b9b	Weasyl Extractor (#977) ... * weasyl extractor * @kattjevfel suggested changes * @mikf changes	4 years ago
Mike Fährmann	d5fa716d89	fix crash when using 'skip=false' and archive (fixes #1023) ... Separating the archive check from pathfmt.exists() in b5243297 had some unintended side effects. It is also not possible to monkey-patch a dunder method like __contains__ because of the special method lookup that gets performed for them.	4 years ago
Mike Fährmann	aeb0d32333	[twitter] improve twitpic extraction (fixes #1019) ... - ignore twitpic.com/photos/… URLs - ignore empty image URLs	4 years ago
Mike Fährmann	2184ec5d78	release version 1.15.0	4 years ago
Mike Fährmann	7cd383c0f9	update extractor test results	4 years ago
Mike Fährmann	1e313d5b84	implement 'sleep-request' option	4 years ago
Mike Fährmann	65744a7a31	use alternative for all falsey values in format strings ... … and not just None (#525) It would be better to consistently use None for all non-existent fields and/or fields without a valid value, but this is a good enough workaround for now.	4 years ago
Mike Fährmann	c43b3894be	[myhentaigallery] update and fix extraction (#1001) ... - extract more metadata - match "/show/" URLs - complete test results - fix missing images for lines starting with " <img" - fix missing comma in supportedsites.py	4 years ago
choeronline	05b9ac8d37	[myhentaigallery] add extractor (#1001) ... * adds support for myhentaigallery * fixes linting issues in myhentaigallery extractor	4 years ago
Mike Fährmann	2626629117	[danbooru] handle posts without 'id' (fixes #1004)	4 years ago
Mike Fährmann	cc1fb0b4ea	[500px] update query hash	4 years ago
Mike Fährmann	da87a5fb7e	[exhentai] fix accessing config before main constructor ... bug introduced with 055c32e0 Making 'Extractor.config()' quite a bit faster is worth the "cost" of having to set _cfgpath in exhentai constructors, I think.	4 years ago
Mike Fährmann	f5b7ae01c1	update extractor test results	4 years ago
Mike Fährmann	136df52d1f	[deviantart] support watchers-only/paid deviations (#995)	4 years ago
Mike Fährmann	055c32e0f7	precompute extractor config paths	4 years ago
Mike Fährmann	231dd4c800	accumulate postprocessor objects (#994) ... Instead of one 'postprocessors' setting overwriting all others lower in the hierarchy, all postprocessors along the config path will now get collected into one big list. For example '--mtime-from-date' will therefore no longer cause other postprocessor settings in a config file to get ignored.	4 years ago
Mike Fährmann	392d022b04	implement 'config.accumulate()' (#994)	4 years ago
Mike Fährmann	3afd362e2e	add 'sleep-extractor' option (closes #964) ... (would have been nice if this were possible without code duplication)	4 years ago
Mike Fährmann	3108e85b89	[worldthree] remove extractors ... http://www.slide.world-three.org/ hasn't been accessible for a long time.	4 years ago
Mike Fährmann	8fed3eb8cb	[jaiminisbox] remove extractors ... https://jaiminisbox.com/post.html	4 years ago
Mike Fährmann	dcf3ad7eef	[furaffinity] update download URL extraction (fixes #988) ... support the new 'd2.facdn.net' subdomain	4 years ago
Mike Fährmann	3918b69677	remove 'extractor.blacklist' context manager	4 years ago
Mike Fährmann	c78aa17506	add general 'blacklist' and 'whitelist' options (#492, #844)	4 years ago
Mike Fährmann	abda352a5b	add '--no-skip' command-line option (closes #986)	4 years ago
Mike Fährmann	5912727b88	support format string replacement fields in archive paths ... (closes #985)	4 years ago
Mike Fährmann	2b8d57f0ab	[twitter] support '/intent/user?user_id=…' URLs (#980)	4 years ago
Mike Fährmann	a3b473bd2f	[twitter] support specifying users by ID (#980) ... by using 'id:…' as their screen name, i.e. https://www.twitter.com/id:2976459548/media instead of https://twitter.com/supernaturepics/media The user ID can, for example, be obtained from the output of $ gallery-dl -j --range 1 https://twitter.com/<screen-name>	4 years ago
Mike Fährmann	a0d916ed41	[exhentai] update wait time before original image download (#978) ... depend on 'wait-max', don't use a hard-coded value	4 years ago
Mike Fährmann	f6fd449b59	reduce wait time growth rate from exponential to linear ... Waiting for 2**N seconds after each error grows too fast. Simply waiting N seconds seems far more reasonable.	4 years ago
Mike Fährmann	bc48514d84	[aryion] get post ID via gallery-item (fixes #981, closes #982) ... this even works when fetching post IDs from '/latest.php?id='	4 years ago
Mike Fährmann	799ca07fc8	[imgur] update ... - fix image/album detection for galleries - use new API endpoints for image/album data	4 years ago
Mike Fährmann	b5243297ff	write skipped files to archive (closes #550)	4 years ago
Mike Fährmann	ac3036ef56	add 'filesize-min' and 'filesize-max' options (closes #780)	4 years ago
Mike Fährmann	7876a03ece	[tumblr] create directories for each post (fixes #965) ... This changes the identifiers for directory format string fields. Everything blog related is now inside a 'blog' object and not at the "base level" anymore. E.g. '{name}' for directories is now '{blog[name]}' (or '{blog_name}', since that is also available)	4 years ago
Mike Fährmann	fd0685d9b5	[postprocessor:zip] defer zip file creation (fixes #968) ... don't try to create zip files on postprocessor construction, wait until directory creation during file download,	4 years ago
Mike Fährmann	33fe67b594	release version 1.14.5	4 years ago
Mike Fährmann	d50f3b333a	update extractor test results	4 years ago
Mike Fährmann	0f55b8e80a	[exhentai] fix type check from dbbbb21 (#940) ... 'bool' is a subclass of 'int', and therefore 'isinstance(self.limits, int)' also returns True when 'self.limits' has a boolean value	4 years ago
Mike Fährmann	e33293fdd8	[hentaihand] update to new site layout	4 years ago
Mike Fährmann	fda9e296dd	[gelbooru] fix extraction without API	4 years ago
Mike Fährmann	69e4871005	update extractor test results ... - sensescans: replace 404d chapters - mangapark: replace 404d chapters - subscribestar: update test for attached files	4 years ago
Mike Fährmann	ab1af66a97	[imgur] add 'search' extractor (#934)	4 years ago
Mike Fährmann	e4bbc1fb5c	[imgur] add 'tag' extractor (#934)	4 years ago
Mike Fährmann	deaacc70bb	[hitomi] update URL pattern for tag searches	4 years ago
ArtaxIsSleeping	0e941553ec	[aryion] Add username/password support (#960) ... * Add username/password support to aryion extractor * Update docs to match * Fix code style	4 years ago
Mike Fährmann	84e04cc23b	[500px] fix extraction and update URL patterns (fixes #956) ... - rewrite most API calls to GraphQL queries - match '500px.com/p/<user>' URLs	4 years ago
Mike Fährmann	d4ff767291	[reddit] improve gallery extraction (fixes #955)	4 years ago
Mike Fährmann	7140fe7e6d	[hitomi] fix redirect processing	4 years ago
Mike Fährmann	a57b6b3c3a	[reddit] handle deleted galleries (fixes #953)	4 years ago
Mike Fährmann	063c71cd84	[furaffinity] add 'search' extractor (closes #915)	4 years ago
Mike Fährmann	dbbbb21180	[exhentai] add ability to specify custom image limit (#940)	4 years ago
Mike Fährmann	b2009ea39e	[aryion] update folder mime type list (fixes #945)	4 years ago
Mike Fährmann	688bd046fc	release version 1.14.4	4 years ago
Mike Fährmann	d06ad148c7	[shopify] use alternate regex for products on collection pages ... when the first on doesn't yield any results	4 years ago
Mike Fährmann	7619152988	[reactor] sort 'tags' ... to ensure a consistent order for test results	4 years ago
Mike Fährmann	cd9de613a2	[exhentai] adjust image limit costs (#940) ... Each original file costs 10 points per 10^6 bytes, not 10 per 2^20 == 1048576 bytes.	4 years ago
Mike Fährmann	2e6f6ee1c1	[mangoxo] fix login	4 years ago
Mike Fährmann	a6a080656c	[pixnet] detect password-protected albums (#177)	4 years ago
Mike Fährmann	67ac6667af	[mangareader] fix extraction	4 years ago
Mike Fährmann	2b88c90f6f	[blogger] add search extractor (#925)	4 years ago
Mike Fährmann	d5067c51c5	[instagram] support '/reel/' URLs	4 years ago
Mike Fährmann	2c9766b29f	fix UnboundLocalError in Extractor.request() ... introduced in d6a271d	4 years ago
Mike Fährmann	aa64149583	[blogger] support searching posts by labels (closes #925)	4 years ago
Mike Fährmann	60ba3cb946	[reddit] support gallery posts (closes #920)	4 years ago
Mike Fährmann	0d84d3af55	[subscribestar] extract attached media files (#852)	4 years ago
Mike Fährmann	19bf76bcf8	update extractor test results	4 years ago
Mike Fährmann	0762d6b29c	[inkbunny] add 'num' field (#283)	4 years ago
Mike Fährmann	fbc4278fe4	[instagram] wait before GraphQL requests (#901)	4 years ago
Mike Fährmann	ec5870576d	[imgur] handle 403 overcapacity responses (closes #910)	4 years ago
Mike Fährmann	d6a271d2c7	add 'response' objects to 'HttpError's	4 years ago
Mike Fährmann	72c5578a27	[hentainexus] improve/simplify code	4 years ago
Mike Fährmann	627d2141d3	[xhamster] fix extraction (closes #917)	4 years ago
Mike Fährmann	3f73cc6855	allow 'parent-directory' to work recursively (fixes #905)	4 years ago
Mike Fährmann	27e31f4a16	[myportfolio] raise 'NotFoundError' for deleted posts	4 years ago
Mike Fährmann	f317a57c5e	[simplyhentai] fix 'gallery_id' extraction	4 years ago
Mike Fährmann	daeef8a5e3	[vsco] handle missing 'description' fields	4 years ago
Mike Fährmann	26a967cbd4	[pinterest] match 'pinterest.co.uk' URLs (fixes #914)	4 years ago
Mike Fährmann	c5aaa1de77	[inkbunny] simplify metadata structure (#283) ... Just put everything at the top level, instead of having a separate 'post' object.	4 years ago
Mike Fährmann	b921fee24d	[inkbunny] fix submission order (#283) ... Getting detailed submission info via /api_submissions.php reordered the input submissions and sorted them by ID. InkbunnyAPI.detail() now sorts them back and ensures they are returned in their original order. This commit also removes the 'metadata' option and always requests submission descriptions.	4 years ago
Mike Fährmann	e50c75628c	[subscribestar] update 'date' parsing	4 years ago
Mike Fährmann	c4ed9f4faa	[inkbunny] add 'metadata' option (#283)	4 years ago
Mike Fährmann	493cadb1e7	[inkbunny] add 'orderby' option (#283)	4 years ago
Mike Fährmann	336e682a7a	[inkbunny] handle gallery/scraps URLs (#283)	4 years ago
Mike Fährmann	8dbf827649	[bobx] remove module	4 years ago
Mike Fährmann	8f64585ff2	[twitter] handle 429 responses without x-rate-limit-reset header	4 years ago
Mike Fährmann	d2e17e16bf	[inkbunny] update tests (#283)	4 years ago
Mike Fährmann	57f7d9b790	[inkbunny] improve error handling (#283)	4 years ago
Mike Fährmann	baf5d0e3c1	[gfycat] skip malformed gfycat responses (closes #902)	4 years ago
Mike Fährmann	453f3bc519	[blogger] improve error messages for missing posts/blogs (#903)	4 years ago
Mike Fährmann	87202b8d74	[inkbunny] add 'user' and 'post' extractors (#283)	4 years ago
Mike Fährmann	b62ea72533	release version 1.14.3	4 years ago
Mike Fährmann	2ecf1efb16	update extractor test results ... - tumblr: remove deleted post - jaiminisbox: replace removed manga/chapters - smugmug: one inconsequential field got removed	4 years ago
Mike Fährmann	d5fcffcced	[subscribestar] add login capabilities (#852)	4 years ago
Mike Fährmann	ecaecc4064	[exhentai] add 'domain' option (#897)	4 years ago
Mike Fährmann	45c32213dc	[gfycat] retry 404'ed videos on redgifs (closes #874)	4 years ago
Mike Fährmann	cf44571fe0	[gfycat] add 'user' and 'search' extractors	4 years ago
Mike Fährmann	11b744d971	[mangakakalot] improve/fix chapter extraction	4 years ago
Mike Fährmann	2da71cb561	[twitter] raise proper exception if user doesn't exist (#891)	4 years ago
Leonardo Taccari	86e5a05e29	[twitter] add support for nitter.net URLs in pattern (#890) ... Please note that URLs are only "translated", all requests are still done always via the Twitter API.	4 years ago
Mike Fährmann	e17d4f44f6	[newgrounds] fix favorites extraction	4 years ago
Mike Fährmann	c51fbd72ba	update extractor test results	4 years ago
Mike Fährmann	9cd1bc6907	[mangakakalot] update URL patterns, fix flake8 errors (#876)	4 years ago
jakem72360	7dfdcc3fbf	[mangakakalot] Added extractors for MangaKakalot (#876)	4 years ago
Mike Fährmann	cb0132e441	[khinsider] add 'format' option (closes #840)	4 years ago
Mike Fährmann	d594977ca1	[artstation] add 'following' extractor (closes #888)	4 years ago
Mike Fährmann	3855d0dd3c	[twitter] add debug messages for all skipped Tweets (#867)	4 years ago
Mike Fährmann	27d163afb3	[imgur] support all '/t/...' URLs (closes #880) ... … instead of just '/t/unmuted/'	4 years ago
Mike Fährmann	f5c9f1d066	[subscribestar] use current date instead of hard-coded '2020' (#852)	4 years ago
Mike Fährmann	5a6e750704	[reddit] fix AttributeError when using 'recursion' (fixes #879)	4 years ago
Mike Fährmann	94a08f0bcb	[reddit] limit title length in default filenames (#873)	4 years ago
Mike Fährmann	3424fb96c3	[redgifs] support gifsdeliverynetwork.com URLs (#874)	4 years ago
Mike Fährmann	f1344fe552	[patreon] yield images and attachments before postfiles (#871) ... The reported filename of the 'postfile' entry of each post may differ from the corresponding entry in the list of images or attachments, and be outright "wrong".	4 years ago
Mike Fährmann	dbf841ebd1	prevent unhandled exception on Cloudflare challenges (#868) ... The relatively new v2 challenges aren't supported (*), but retrying often enough may yield a v1 challenge which can be solved. (*) and probably never will. They are far too complicated to do without a real browser.	4 years ago
Mike Fährmann	6e2af9a8d8	[twitter] improve error message formatting	4 years ago
Mike Fährmann	c28db7a6ea	[8muses] support 'comics.8muses.com' URLs	4 years ago
Mike Fährmann	4d8b3e4f70	defer directory creation (fixes #722) ... Only call os.makedirs() before a file is getting downloaded, and not immediately for every Directory message.	4 years ago
Mike Fährmann	d5bfb0b38c	set pseudo extension for Metadata messages (#865) ... This prevents pathfmt.filename from potentially being empty.	4 years ago
Mike Fährmann	821524e4ee	[subscribestar] add 'user' and 'post' extractors (#852)	4 years ago
Mike Fährmann	4f16fd37fe	release version 1.14.2	4 years ago
Mike Fährmann	e62ebb4643	update CHANGELOG before building sdist and wheel packages	4 years ago
Mike Fährmann	f1ddbff0b5	[aryion] add 'recursive' option (fixes #832) ... This is enabled by default and will recursively go through all (sub)folders in an artist's gallery. The old method of using "Latest Updates" lists can be restored by disabling this option.	4 years ago
Mike Fährmann	699062b91f	Revert "[kissmanga] workaround for CAPTCHAs (#818)" ... This reverts commit 4cf3d54718.	4 years ago
Mike Fährmann	0cac14c3bd	update extractor test results	4 years ago
Mike Fährmann	5e5be67c26	[tumblr] prevent KeyErrors when using reblogs=same-blog ... (fixes #851)	4 years ago
Mike Fährmann	9da2bc67f8	[twitter] add option to filter media from quoted tweets (#854)	4 years ago
Mike Fährmann	56ab5fb8f4	[twitter] improve handling of quoted tweets (#854) ... Split each "quote" into two parts: - the original tweet - the tweet that quoted the original	4 years ago
Mike Fährmann	bd0e1ca1a5	[imgur] build directory path for each file (closes #842)	4 years ago
Mike Fährmann	a8c2d997e8	[twitter] treat quoted tweets like retweets (#833) ... - filter them when 'retweets' is disabled - set 'author' to the creator of the quoted tweet like it was before the rewrite	4 years ago
Mike Fährmann	aed1c63e51	[twitter] improve search results (fixes #847) ... Adding 'tweet_search_mode=live' to the query parameters is the most important part here.	4 years ago
Mike Fährmann	0e714b9a0e	[pinterest] add 'section' extractor (#835)	4 years ago
Mike Fährmann	53cc498d9c	improve config lookup when there are multiple possible locations ... This specifically applies to all Mastodon extractors and all extractors with a 'basecategory', i.e. 'booru', 'foolslide', etc. Values inside those general config locations wouldn't be recognized when a value with the same was set on the 'extractor' level. For example 'extractor.mastodon.directory' should be used over 'extractor.directory' when both are set, but this was impossible with the previous implementation. (fixes #843)	4 years ago
Mike Fährmann	1b3870a4be	flush after writing JSON in DataJob() (#727) ... … and remove the dead handle_finalize() method, which is never called since DataJob() overrides run().	4 years ago
Mike Fährmann	d81a8e6544	[twitter] update tests	4 years ago
Mike Fährmann	d39eedd9bb	[twitter] improve handling of deleted tweets (fixes #838)	4 years ago
Mike Fährmann	1ae1df0d27	update '--write-pages' (#737) ... - fix infinite recursion for responses with multiple entries in 'history' - hide values of Set-Cookie headers - only write the response content by default (use '-o write-pages=all' to also include HTTP headers)	4 years ago
Mike Fährmann	7e8a747c56	improve output of '-K' for parent extractors 2 (#825) ... This is what bb882b8 was supposed to be, but I managed to not include those changes in the first commit …	4 years ago
Mike Fährmann	dc16f73965	[twitter] move '_guest_token()' into TwitterAPI class	4 years ago
Mike Fährmann	3561d1020a	[twitter] always provide an 'author' field (#831, #833) ... The idea was to have less metadata clutter for most Tweets were 'author' and 'user' are the same (non-retweets), and only provide a 'user' field. The original Tweet author could be gotten with {author[…]|user[…]}, but basically no one knows about that.	4 years ago
Mike Fährmann	7158bdd7c7	[weibo] improve extractor logic (#829)	4 years ago
Mike Fährmann	37d71f6e09	strip microseconds in text.parse_datetime()	4 years ago
Mike Fährmann	0371fd54a1	[artstation] add 'date' metadata field (#839)	4 years ago
Mike Fährmann	8c857052d7	[mastodon] ignore toots without media attachments	4 years ago
Mike Fährmann	de045d39b2	[mastodon] add 'date' metadata field (#839)	4 years ago
Mike Fährmann	d5d90a0450	[weibo] add 'date' field to 'status' objects (#829)	4 years ago
Mike Fährmann	5ba90f72ca	[pinterest] add support for sections (closes #835)	4 years ago
Mike Fährmann	c37a1c06c8	[twitter] add extractor for liked tweets (closes #837) ... You need to be logged in to get access to anyone's liked tweets, it seems.	4 years ago
Mike Fährmann	b94394104c	[twitter] don't download video previews (#833) ... when 'videos' is set to False	4 years ago
Mike Fährmann	bb882b8cdb	improve output of '-K' for parent extractors (#825)	4 years ago
Mike Fährmann	6db7ed90cb	release version 1.14.1	4 years ago
Mike Fährmann	4cf3d54718	[kissmanga] workaround for CAPTCHAs (fixes #818) ... Requesting the same page again when being redirected to a CAPTCHA lets us access that page without solving it.	4 years ago
Mike Fährmann	7daef6ee70	update extractor test results ... - certain posts on Instagram now return https://static.cdninstagram.com/rsrc.php/null.jpg for public users - MangaDex is deploying its new MangaDex@Home network similar to exhentai's Hentai@Home - realbooru has a new site layout, but the underlying booru API still works like before	4 years ago
Mike Fährmann	ffb6c5277a	[furaffinity] add 'artist_url' metadata field (closes #821)	4 years ago
Mike Fährmann	be04e44e2c	[reddit] catch JSON decode errors (#765)	4 years ago
Mike Fährmann	cf863f60b3	[redgifs] add 'user' and 'search' extractors (closes #724)	4 years ago
Mike Fährmann	998d1d3a5c	[webtoons] generalize and improve comic extraction (fixes #820)	4 years ago
Mike Fährmann	1489712325	resolve redirects after solving Cloudflare challanges	4 years ago
Mike Fährmann	b0b1feaa67	request 'transparent.gif' when solving Cloudflare challenges ... This currently also works without, but they might be using these to detect potential bots in the future.	4 years ago
Mike Fährmann	036a40943a	[twitter] don't cache results of 'user_by_screen_name()' ... A 'keyarg=1' argument to the memcache decorator would have worked as well, but keeping the user object in memory isn't useful for the vast majority of use cases and only wastes space. (closes #817)	4 years ago
Mike Fährmann	4442dfe7b8	[twitter] add 'reply_to' metadata to replies	4 years ago
Mike Fährmann	83b7bd0413	[nhentai] fix extraction (closes #819)	4 years ago
Mike Fährmann	d769bb4b80	[twitter] improve pagination	4 years ago
Mike Fährmann	5bc1097f9d	[twitter] metadata cleanup #2 ... - remove useless clutter by creating new tweet-data dicts instead of reusing the original Tweet objects - rename fields to how they were named before ('id_str' -> 'tweet_id', etc.) - only include 'author' if it would differ from 'user' - restore 'archive_fmt'	4 years ago
Mike Fährmann	1fcf938f9c	implement a general 'delete_items()' function	4 years ago
Mike Fährmann	c6c06c41f6	[deviantart] don't add journal text to description (#712)	4 years ago
Mike Fährmann	4aea5138dd	[sensescans] use https://	4 years ago
Mike Fährmann	3eed5f52d7	[twitter] small metadata cleanup ... - add 'date' field - remove 'entities' and 'extended_entities' - don't include 'focus_fields' from 'original_info'	4 years ago
Mike Fährmann	655c98cbef	[twitter] skip unavailable tweets	4 years ago
Mike Fährmann	41d03160ff	[deviantart] also search journals for sta.sh links (#712) ... when 'extra' is enabled	4 years ago
Mike Fährmann	2132e5461a	[twitter] restore TwitPic support	4 years ago
Mike Fährmann	bd0f21478a	[twitter] login using the mobile nojs login page	4 years ago
Mike Fährmann	a10f31dde5	[twitter] rewrite; use new interface (#740, #806) ... Everything except logging in with username & password and TwitPic embeds should be working again. Metadata per Tweet is massively different than before (mostly raw API responses - might need some cleaning up) and the default 'archive_fmt' changed.	4 years ago
Mike Fährmann	f1ef9082bb	release version 1.14.0	4 years ago
Mike Fährmann	3bad1579ee	update extractor test results	4 years ago
Mike Fährmann	864f4220d9	update output of 'oauth:…' (#616)	4 years ago
Mike Fährmann	0f459f340b	[instagram] fix and re-enable login with username&password ... This reverts commit 3e0848a482. (#756, #771, #797, #803) https://github.com/althonos/InsaLooter/issues/287#issuecomment-630456522	4 years ago
Mike Fährmann	3e0848a482	[instagram] disable login with username&password (#756)	4 years ago
Mike Fährmann	a32aea41e1	[instagram] update 'query_hash' values	4 years ago
Mike Fährmann	2bff8dd465	[hentainexus] fix flake8 issues (#787)	4 years ago
Mike Fährmann	a63682a9c0	[instagram] simplify code & complete tests (#743)	4 years ago
墨焓	a4e3d40672	hentainexus.py minor fix (#787) ... * rectify code of `join_title`, some minor fix. * + hentainexus self.data * fixed: call staticmethod join_title with data	4 years ago
Vrihub	62b65e59d0	Add instagram metadata: post_pageurl, post_tags (#743) ... * Add instagram metadata: post_pageurl, post_tags Add the following metadata for instagram: - post_pageurl: json string with url of the post page - post_tags: json array with instagram tags extracted from the post description * Oops: rename post_tags to tags for --write-tags This way, --write-tags will pick up the post tags. * Rename to post_url, improve regex * Add post_url and tags to tests * Remove duplicate tags and sort them * Bugfix: don't create empty tag lists * Metadata: add location * Metadata: add tagged_users for each media * Move self._find_tags() to base class * Make flake happy	4 years ago
Mike Fährmann	da22ea8ced	use %APPDATA%\gallery-dl for config/cache on Windows	4 years ago
Mike Fährmann	275cceeb6a	[redgifs] fix extraction (#724) ... … and prepare for more potential extractors	4 years ago
Mike Fährmann	45baa13615	update extractor test results ... - don't run Instagram tests on Travis anymore - replace Twitter test because timeline was made private - update Hiperdex domain to '.com' (again ...)	4 years ago
Mike Fährmann	dfcf2a2c91	write OAuth token to cache by default (#616)	4 years ago
Mike Fährmann	ddc253cf9a	implement a 'path-replace' option (#662, #755)	4 years ago
Mike Fährmann	15c3d29062	move dump_response() into a separate function (#737)	4 years ago
Mike Fährmann	a363da4b43	include redirects and headers in --write-pages dumps (#737)	4 years ago
Mike Fährmann	6bcdb264e0	[imgur] treat 't/unmuted' URLs as galleries	4 years ago
Mike Fährmann	b6cee3e45b	[imgur] fix extraction of animated images without 'mp4' entry	4 years ago
Mike Fährmann	bc53302ad6	extend 'path-restrict' option ... Allow its value to be a JSON object / Python dict that specifies a mapping from invalid/unwanted input characters to specific output characters. For example {"/": "-", "*": "+"} will transform "foo / ***bar***" into "foo - +++bar+++" (closes #662, #755)	4 years ago
Leonardo Taccari	bcac31b7c7	[webtoons] make archive_fmt unique (#779) ... close #778	4 years ago
Mike Fährmann	34929f673f	readd 'session' to base downloader class (fixes #768)	4 years ago
Mike Fährmann	e19f665a44	[danbooru] change default for 'ugoira' to 'false' ... Downloading the pre-rendered versions should be a better default than .zip files with individual frames.	4 years ago
Mike Fährmann	3201fe3521	add global SENTINEL object	4 years ago
Mike Fährmann	c8787647ed	add global WINDOWS bool	4 years ago
Mike Fährmann	6294e2c540	add 'text.ensure_http_scheme()'	4 years ago
Mike Fährmann	0378d079a5	[webtoons] fixes and simplifications (#593, #761) ... - fix episode listings for french comics - allow input URLs without explicit scheme - add 'lang'/'language' metadata - use str.format() instead of '+' to assemble URLs	4 years ago
Mike Fährmann	ab11b1c896	[imagechest] simplify code (#750)	4 years ago
Mike Fährmann	ece73b5b2a	make 'path' and 'keywords' available in logging messages ... Wrap all loggers used by job, extractor, downloader, and postprocessor objects into a (custom) LoggerAdapter that provides access to the underlying job, extractor, pathfmt, and kwdict objects and their properties. __init__() signatures for all downloader and postprocessor classes have been changed to take the current Job object as their first argument, instead of the current extractor or pathfmt. (#574, #575)	4 years ago
Mike Fährmann	846d3a2466	[sexcom] replace 404ed test	4 years ago
Mike Fährmann	9b4635917f	[gelbooru] simplify and fix pool extraction ... use 'pool:<pool id>' as search tag to get pool posts	4 years ago
Mike Fährmann	abbd8fbbd9	reset filenames on empty file extensions (#733)	4 years ago
Mike Fährmann	d17e9628b3	fix/improve Cloudflare bypass code (#728, #757) ... - support changing values for 'k' - use XML parser to get request parameters (some input fields are now embedded in an HTML comment)	4 years ago
Leonardo Taccari	39cd389679	[webtoons] Add a new extractor for webtoons.com (#761) ... The webtoons extractor can extract episode and entire comic (all episodes) from webtoons.com. All the logic of the extractors should be trivial except for a couple of kludges needed: - `ageGatePass' cookie is always set to avoid possible redirect and stop of extraction, especially in the comic extractor - The image URLs returned by the episode extractor could not be fetched directly and the `Referer:' HTTP header needs to be passed to fetch them Close #593.	4 years ago
Bepis	7b5711ee04	[imagechest] Add new extractor for ImageChest (#750) ... * [imagechest] Add new extractor for ImageChest * [imagechest] Fix flake8 compliance issues	4 years ago
Unit 193	b7ebf51229	Fix typo: defaut → default. (#754)	4 years ago
Mike Fährmann	f8661c6578	[downloader:ytdl] fix file extensions when merging into mkv	4 years ago
Mike Fährmann	a1e739b96c	reuse connection adapters from parent extractors	4 years ago
Mike Fährmann	f8f95e68a7	improve '--write-pages' (#737) ... - move code into its own function - add enumeration index to filenames - dump responses regardless of status code	4 years ago
Mike Fährmann	dba87ca99e	[downloader:ytdl] change 'forward-cookies' default to 'false' ... There are currently no situations where forwarding gallery-dl's cookies to youtube-dl is necessary, and it only causes problems when forcing youtube-dl for Twitter video downloads while logged in.	4 years ago
Mike Fährmann	09cc9dbec0	prevent flake8 errors from comments looking like type annotations	4 years ago
Mike Fährmann	2d6724180b	[hiperdex] update domain to hiperdex.info	4 years ago
Vrihub	4cc761c730	Implement --write-pages option (#736) ... * Implement --write-pages option * Fix long lines * Fix file mode to binary * Fix pattern for Windows compatibility	4 years ago
Mike Fährmann	f557cac074	[redgifs] add image extractor (#724)	4 years ago
Mike Fährmann	11d3247c85	use 'util.Formatter' when formatting logging output	4 years ago
Mike Fährmann	8b60bd6a91	mock 'time()' in cache tests ... instead of calling 'sleep()' to let time advance. This shortens the time needed to run those tests, and ensures consistent results. (Tests would randomly fail when using 'sleep()')	4 years ago
Mike Fährmann	42f29c3e11	improve and simplify attribute access in DownloadJob.initialize()	4 years ago
Mike Fährmann	65b1cb7acd	[deviantart] use private access tokens for Journals (fixes #738)	4 years ago
Mike Fährmann	0bf0146bfe	[reddit] don't send OAuth headers for file downloads (fixes #729)	4 years ago
Mike Fährmann	ba42ec321c	release version 1.13.6	4 years ago
Mike Fährmann	d6a480682f	update test results	4 years ago
Mike Fährmann	714566b6e3	fix/improve Cloudflare bypass code	4 years ago
Leonardo Taccari	b47cfc5ac9	[speakerdeck] Add a new extractor for speakerdeck.com (#726)	4 years ago
Mike Fährmann	90491ab606	[artstation] improve embed extraction (#720)	4 years ago