2030 Commits (e9a75e27d9298a5bacd7f8e62c516cd5077d46bb)

Author	SHA1	Message	Date
Mike Fährmann	d6a271d2c7	add 'response' objects to 'HttpError's	4 years ago
Mike Fährmann	72c5578a27	[hentainexus] improve/simplify code	4 years ago
Mike Fährmann	627d2141d3	[xhamster] fix extraction (closes #917)	4 years ago
Mike Fährmann	27e31f4a16	[myportfolio] raise 'NotFoundError' for deleted posts	4 years ago
Mike Fährmann	f317a57c5e	[simplyhentai] fix 'gallery_id' extraction	4 years ago
Mike Fährmann	daeef8a5e3	[vsco] handle missing 'description' fields	4 years ago
Mike Fährmann	26a967cbd4	[pinterest] match 'pinterest.co.uk' URLs (fixes #914)	4 years ago
Mike Fährmann	c5aaa1de77	[inkbunny] simplify metadata structure (#283) ... Just put everything at the top level, instead of having a separate 'post' object.	4 years ago
Mike Fährmann	b921fee24d	[inkbunny] fix submission order (#283) ... Getting detailed submission info via /api_submissions.php reordered the input submissions and sorted them by ID. InkbunnyAPI.detail() now sorts them back and ensures they are returned in their original order. This commit also removes the 'metadata' option and always requests submission descriptions.	4 years ago
Mike Fährmann	e50c75628c	[subscribestar] update 'date' parsing	4 years ago
Mike Fährmann	c4ed9f4faa	[inkbunny] add 'metadata' option (#283)	4 years ago
Mike Fährmann	493cadb1e7	[inkbunny] add 'orderby' option (#283)	4 years ago
Mike Fährmann	336e682a7a	[inkbunny] handle gallery/scraps URLs (#283)	4 years ago
Mike Fährmann	8dbf827649	[bobx] remove module	4 years ago
Mike Fährmann	8f64585ff2	[twitter] handle 429 responses without x-rate-limit-reset header	4 years ago
Mike Fährmann	d2e17e16bf	[inkbunny] update tests (#283)	4 years ago
Mike Fährmann	57f7d9b790	[inkbunny] improve error handling (#283)	4 years ago
Mike Fährmann	baf5d0e3c1	[gfycat] skip malformed gfycat responses (closes #902)	4 years ago
Mike Fährmann	453f3bc519	[blogger] improve error messages for missing posts/blogs (#903)	4 years ago
Mike Fährmann	87202b8d74	[inkbunny] add 'user' and 'post' extractors (#283)	4 years ago
Mike Fährmann	2ecf1efb16	update extractor test results ... - tumblr: remove deleted post - jaiminisbox: replace removed manga/chapters - smugmug: one inconsequential field got removed	4 years ago
Mike Fährmann	d5fcffcced	[subscribestar] add login capabilities (#852)	4 years ago
Mike Fährmann	ecaecc4064	[exhentai] add 'domain' option (#897)	4 years ago
Mike Fährmann	45c32213dc	[gfycat] retry 404'ed videos on redgifs (closes #874)	4 years ago
Mike Fährmann	cf44571fe0	[gfycat] add 'user' and 'search' extractors	4 years ago
Mike Fährmann	11b744d971	[mangakakalot] improve/fix chapter extraction	4 years ago
Mike Fährmann	2da71cb561	[twitter] raise proper exception if user doesn't exist (#891)	4 years ago
Leonardo Taccari	86e5a05e29	[twitter] add support for nitter.net URLs in pattern (#890) ... Please note that URLs are only "translated", all requests are still done always via the Twitter API.	4 years ago
Mike Fährmann	e17d4f44f6	[newgrounds] fix favorites extraction	4 years ago
Mike Fährmann	c51fbd72ba	update extractor test results	4 years ago
Mike Fährmann	9cd1bc6907	[mangakakalot] update URL patterns, fix flake8 errors (#876)	4 years ago
jakem72360	7dfdcc3fbf	[mangakakalot] Added extractors for MangaKakalot (#876)	4 years ago
Mike Fährmann	cb0132e441	[khinsider] add 'format' option (closes #840)	4 years ago
Mike Fährmann	d594977ca1	[artstation] add 'following' extractor (closes #888)	4 years ago
Mike Fährmann	3855d0dd3c	[twitter] add debug messages for all skipped Tweets (#867)	4 years ago
Mike Fährmann	27d163afb3	[imgur] support all '/t/...' URLs (closes #880) ... … instead of just '/t/unmuted/'	4 years ago
Mike Fährmann	f5c9f1d066	[subscribestar] use current date instead of hard-coded '2020' (#852)	4 years ago
Mike Fährmann	5a6e750704	[reddit] fix AttributeError when using 'recursion' (fixes #879)	4 years ago
Mike Fährmann	94a08f0bcb	[reddit] limit title length in default filenames (#873)	4 years ago
Mike Fährmann	3424fb96c3	[redgifs] support gifsdeliverynetwork.com URLs (#874)	4 years ago
Mike Fährmann	f1344fe552	[patreon] yield images and attachments before postfiles (#871) ... The reported filename of the 'postfile' entry of each post may differ from the corresponding entry in the list of images or attachments, and be outright "wrong".	4 years ago
Mike Fährmann	6e2af9a8d8	[twitter] improve error message formatting	4 years ago
Mike Fährmann	c28db7a6ea	[8muses] support 'comics.8muses.com' URLs	4 years ago
Mike Fährmann	d5bfb0b38c	set pseudo extension for Metadata messages (#865) ... This prevents pathfmt.filename from potentially being empty.	4 years ago
Mike Fährmann	821524e4ee	[subscribestar] add 'user' and 'post' extractors (#852)	4 years ago
Mike Fährmann	e62ebb4643	update CHANGELOG before building sdist and wheel packages	4 years ago
Mike Fährmann	f1ddbff0b5	[aryion] add 'recursive' option (fixes #832) ... This is enabled by default and will recursively go through all (sub)folders in an artist's gallery. The old method of using "Latest Updates" lists can be restored by disabling this option.	4 years ago
Mike Fährmann	699062b91f	Revert "[kissmanga] workaround for CAPTCHAs (#818)" ... This reverts commit 4cf3d54718.	4 years ago
Mike Fährmann	0cac14c3bd	update extractor test results	4 years ago
Mike Fährmann	5e5be67c26	[tumblr] prevent KeyErrors when using reblogs=same-blog ... (fixes #851)	4 years ago
Mike Fährmann	9da2bc67f8	[twitter] add option to filter media from quoted tweets (#854)	4 years ago
Mike Fährmann	56ab5fb8f4	[twitter] improve handling of quoted tweets (#854) ... Split each "quote" into two parts: - the original tweet - the tweet that quoted the original	4 years ago
Mike Fährmann	bd0e1ca1a5	[imgur] build directory path for each file (closes #842)	4 years ago
Mike Fährmann	a8c2d997e8	[twitter] treat quoted tweets like retweets (#833) ... - filter them when 'retweets' is disabled - set 'author' to the creator of the quoted tweet like it was before the rewrite	4 years ago
Mike Fährmann	aed1c63e51	[twitter] improve search results (fixes #847) ... Adding 'tweet_search_mode=live' to the query parameters is the most important part here.	4 years ago
Mike Fährmann	0e714b9a0e	[pinterest] add 'section' extractor (#835)	4 years ago
Mike Fährmann	53cc498d9c	improve config lookup when there are multiple possible locations ... This specifically applies to all Mastodon extractors and all extractors with a 'basecategory', i.e. 'booru', 'foolslide', etc. Values inside those general config locations wouldn't be recognized when a value with the same was set on the 'extractor' level. For example 'extractor.mastodon.directory' should be used over 'extractor.directory' when both are set, but this was impossible with the previous implementation. (fixes #843)	4 years ago
Mike Fährmann	d81a8e6544	[twitter] update tests	4 years ago
Mike Fährmann	d39eedd9bb	[twitter] improve handling of deleted tweets (fixes #838)	4 years ago
Mike Fährmann	1ae1df0d27	update '--write-pages' (#737) ... - fix infinite recursion for responses with multiple entries in 'history' - hide values of Set-Cookie headers - only write the response content by default (use '-o write-pages=all' to also include HTTP headers)	4 years ago
Mike Fährmann	dc16f73965	[twitter] move '_guest_token()' into TwitterAPI class	4 years ago
Mike Fährmann	3561d1020a	[twitter] always provide an 'author' field (#831, #833) ... The idea was to have less metadata clutter for most Tweets were 'author' and 'user' are the same (non-retweets), and only provide a 'user' field. The original Tweet author could be gotten with {author[…]|user[…]}, but basically no one knows about that.	4 years ago
Mike Fährmann	7158bdd7c7	[weibo] improve extractor logic (#829)	4 years ago
Mike Fährmann	0371fd54a1	[artstation] add 'date' metadata field (#839)	4 years ago
Mike Fährmann	8c857052d7	[mastodon] ignore toots without media attachments	4 years ago
Mike Fährmann	de045d39b2	[mastodon] add 'date' metadata field (#839)	4 years ago
Mike Fährmann	d5d90a0450	[weibo] add 'date' field to 'status' objects (#829)	4 years ago
Mike Fährmann	5ba90f72ca	[pinterest] add support for sections (closes #835)	4 years ago
Mike Fährmann	c37a1c06c8	[twitter] add extractor for liked tweets (closes #837) ... You need to be logged in to get access to anyone's liked tweets, it seems.	4 years ago
Mike Fährmann	b94394104c	[twitter] don't download video previews (#833) ... when 'videos' is set to False	4 years ago
Mike Fährmann	bb882b8cdb	improve output of '-K' for parent extractors (#825)	4 years ago
Mike Fährmann	4cf3d54718	[kissmanga] workaround for CAPTCHAs (fixes #818) ... Requesting the same page again when being redirected to a CAPTCHA lets us access that page without solving it.	4 years ago
Mike Fährmann	7daef6ee70	update extractor test results ... - certain posts on Instagram now return https://static.cdninstagram.com/rsrc.php/null.jpg for public users - MangaDex is deploying its new MangaDex@Home network similar to exhentai's Hentai@Home - realbooru has a new site layout, but the underlying booru API still works like before	4 years ago
Mike Fährmann	ffb6c5277a	[furaffinity] add 'artist_url' metadata field (closes #821)	4 years ago
Mike Fährmann	be04e44e2c	[reddit] catch JSON decode errors (#765)	4 years ago
Mike Fährmann	cf863f60b3	[redgifs] add 'user' and 'search' extractors (closes #724)	4 years ago
Mike Fährmann	998d1d3a5c	[webtoons] generalize and improve comic extraction (fixes #820)	4 years ago
Mike Fährmann	036a40943a	[twitter] don't cache results of 'user_by_screen_name()' ... A 'keyarg=1' argument to the memcache decorator would have worked as well, but keeping the user object in memory isn't useful for the vast majority of use cases and only wastes space. (closes #817)	4 years ago
Mike Fährmann	4442dfe7b8	[twitter] add 'reply_to' metadata to replies	4 years ago
Mike Fährmann	83b7bd0413	[nhentai] fix extraction (closes #819)	4 years ago
Mike Fährmann	d769bb4b80	[twitter] improve pagination	4 years ago
Mike Fährmann	5bc1097f9d	[twitter] metadata cleanup #2 ... - remove useless clutter by creating new tweet-data dicts instead of reusing the original Tweet objects - rename fields to how they were named before ('id_str' -> 'tweet_id', etc.) - only include 'author' if it would differ from 'user' - restore 'archive_fmt'	4 years ago
Mike Fährmann	c6c06c41f6	[deviantart] don't add journal text to description (#712)	4 years ago
Mike Fährmann	4aea5138dd	[sensescans] use https://	4 years ago
Mike Fährmann	3eed5f52d7	[twitter] small metadata cleanup ... - add 'date' field - remove 'entities' and 'extended_entities' - don't include 'focus_fields' from 'original_info'	4 years ago
Mike Fährmann	655c98cbef	[twitter] skip unavailable tweets	4 years ago
Mike Fährmann	41d03160ff	[deviantart] also search journals for sta.sh links (#712) ... when 'extra' is enabled	4 years ago
Mike Fährmann	2132e5461a	[twitter] restore TwitPic support	4 years ago
Mike Fährmann	bd0f21478a	[twitter] login using the mobile nojs login page	4 years ago
Mike Fährmann	a10f31dde5	[twitter] rewrite; use new interface (#740, #806) ... Everything except logging in with username & password and TwitPic embeds should be working again. Metadata per Tweet is massively different than before (mostly raw API responses - might need some cleaning up) and the default 'archive_fmt' changed.	4 years ago
Mike Fährmann	3bad1579ee	update extractor test results	4 years ago
Mike Fährmann	864f4220d9	update output of 'oauth:…' (#616)	4 years ago
Mike Fährmann	0f459f340b	[instagram] fix and re-enable login with username&password ... This reverts commit 3e0848a482. (#756, #771, #797, #803) https://github.com/althonos/InsaLooter/issues/287#issuecomment-630456522	4 years ago
Mike Fährmann	3e0848a482	[instagram] disable login with username&password (#756)	4 years ago
Mike Fährmann	a32aea41e1	[instagram] update 'query_hash' values	4 years ago
Mike Fährmann	2bff8dd465	[hentainexus] fix flake8 issues (#787)	4 years ago
Mike Fährmann	a63682a9c0	[instagram] simplify code & complete tests (#743)	4 years ago
墨焓	a4e3d40672	hentainexus.py minor fix (#787) ... * rectify code of `join_title`, some minor fix. * + hentainexus self.data * fixed: call staticmethod join_title with data	4 years ago
Vrihub	62b65e59d0	Add instagram metadata: post_pageurl, post_tags (#743) ... * Add instagram metadata: post_pageurl, post_tags Add the following metadata for instagram: - post_pageurl: json string with url of the post page - post_tags: json array with instagram tags extracted from the post description * Oops: rename post_tags to tags for --write-tags This way, --write-tags will pick up the post tags. * Rename to post_url, improve regex * Add post_url and tags to tests * Remove duplicate tags and sort them * Bugfix: don't create empty tag lists * Metadata: add location * Metadata: add tagged_users for each media * Move self._find_tags() to base class * Make flake happy	4 years ago
Mike Fährmann	275cceeb6a	[redgifs] fix extraction (#724) ... … and prepare for more potential extractors	4 years ago
Mike Fährmann	45baa13615	update extractor test results ... - don't run Instagram tests on Travis anymore - replace Twitter test because timeline was made private - update Hiperdex domain to '.com' (again ...)	4 years ago
Mike Fährmann	dfcf2a2c91	write OAuth token to cache by default (#616)	4 years ago
Mike Fährmann	15c3d29062	move dump_response() into a separate function (#737)	4 years ago
Mike Fährmann	a363da4b43	include redirects and headers in --write-pages dumps (#737)	4 years ago
Mike Fährmann	6bcdb264e0	[imgur] treat 't/unmuted' URLs as galleries	4 years ago
Mike Fährmann	b6cee3e45b	[imgur] fix extraction of animated images without 'mp4' entry	4 years ago
Leonardo Taccari	bcac31b7c7	[webtoons] make archive_fmt unique (#779) ... close #778	4 years ago
Mike Fährmann	e19f665a44	[danbooru] change default for 'ugoira' to 'false' ... Downloading the pre-rendered versions should be a better default than .zip files with individual frames.	4 years ago
Mike Fährmann	3201fe3521	add global SENTINEL object	4 years ago
Mike Fährmann	c8787647ed	add global WINDOWS bool	4 years ago
Mike Fährmann	6294e2c540	add 'text.ensure_http_scheme()'	4 years ago
Mike Fährmann	0378d079a5	[webtoons] fixes and simplifications (#593, #761) ... - fix episode listings for french comics - allow input URLs without explicit scheme - add 'lang'/'language' metadata - use str.format() instead of '+' to assemble URLs	4 years ago
Mike Fährmann	ab11b1c896	[imagechest] simplify code (#750)	4 years ago
Mike Fährmann	846d3a2466	[sexcom] replace 404ed test	4 years ago
Mike Fährmann	9b4635917f	[gelbooru] simplify and fix pool extraction ... use 'pool:<pool id>' as search tag to get pool posts	4 years ago
Leonardo Taccari	39cd389679	[webtoons] Add a new extractor for webtoons.com (#761) ... The webtoons extractor can extract episode and entire comic (all episodes) from webtoons.com. All the logic of the extractors should be trivial except for a couple of kludges needed: - `ageGatePass' cookie is always set to avoid possible redirect and stop of extraction, especially in the comic extractor - The image URLs returned by the episode extractor could not be fetched directly and the `Referer:' HTTP header needs to be passed to fetch them Close #593.	4 years ago
Bepis	7b5711ee04	[imagechest] Add new extractor for ImageChest (#750) ... * [imagechest] Add new extractor for ImageChest * [imagechest] Fix flake8 compliance issues	4 years ago
Mike Fährmann	a1e739b96c	reuse connection adapters from parent extractors	4 years ago
Mike Fährmann	f8f95e68a7	improve '--write-pages' (#737) ... - move code into its own function - add enumeration index to filenames - dump responses regardless of status code	4 years ago
Mike Fährmann	09cc9dbec0	prevent flake8 errors from comments looking like type annotations	4 years ago
Mike Fährmann	2d6724180b	[hiperdex] update domain to hiperdex.info	4 years ago
Vrihub	4cc761c730	Implement --write-pages option (#736) ... * Implement --write-pages option * Fix long lines * Fix file mode to binary * Fix pattern for Windows compatibility	4 years ago
Mike Fährmann	f557cac074	[redgifs] add image extractor (#724)	4 years ago
Mike Fährmann	65b1cb7acd	[deviantart] use private access tokens for Journals (fixes #738)	4 years ago
Mike Fährmann	0bf0146bfe	[reddit] don't send OAuth headers for file downloads (fixes #729)	4 years ago
Mike Fährmann	d6a480682f	update test results	4 years ago
Leonardo Taccari	b47cfc5ac9	[speakerdeck] Add a new extractor for speakerdeck.com (#726)	4 years ago
Mike Fährmann	90491ab606	[artstation] improve embed extraction (#720)	4 years ago
Mike Fährmann	999efec5cc	[deviantart] limit API wait times to 2**9=512 seconds (#721)	4 years ago
Mike Fährmann	504de79d8b	[vsco] fix extraction	4 years ago
Mike Fährmann	5e2974d699	[weibo] add 'videos' option	4 years ago
Mike Fährmann	9f638c2e01	[twitter] add 'replies' option (closes #705)	4 years ago
Mike Fährmann	fc3e54275b	[patreon] respect filters and sort order in query params (#711)	4 years ago
Mike Fährmann	46b9a4d8ff	[patreon] improve hash extraction (#693, #713) ... Instead of accessing a specific part of a download URL, potentially causing an exception if it doesn't exist, we're now searching through all parts for a potential MD5 hash without ever raising an exception.	4 years ago
Mike Fährmann	c56a751dae	[newgrounds] fix URLs produced by 'followng' extractors (#684)	4 years ago
Mike Fährmann	a4fd620a25	[hiperdex] revert domain back to hiperdex.com	4 years ago
Mike Fährmann	233b6f93a2	[patreon] recognize URLs with creator IDs (#711) ... e.g. https://www.patreon.com/user/posts?u=…	4 years ago
Mike Fährmann	38b6bd66b0	[500px] match 'web.500px.com' subdomains	4 years ago
Mike Fährmann	d3b3b30107	update test results	4 years ago
Mike Fährmann	5d7ca76885	retry Cloudflare challenges	4 years ago
Mike Fährmann	3eab07739f	[twitter] ensure videos have a 'filename' ... This usually gets set when invoking the 'ytdl' downloader, but when that fails, the error message would use 'None' as filename.	4 years ago
Mike Fährmann	c4371a6970	[twitter] add 'reply' metadata field (#705)	4 years ago
Mike Fährmann	12ff23b6cc	[mastodon] improve account searches (fixes #704) ... Searching for just the username ("@NAME") can produce multiple unrelated results, so we now search for username + mastodon instance ("@NAME@INSTANCE")	4 years ago
Mike Fährmann	400a0df661	[jaiminisbox] update decoding procedure (fixes #702)	4 years ago
Mike Fährmann	8fe858eb0e	improve parameter extraction when solving Cloudflare challenge	4 years ago
Mike Fährmann	fb98b567fa	[gelbooru] improve post ID extraction for pools	4 years ago
Mike Fährmann	d6facdee7b	[mastodon] add tests (#701)	4 years ago
Mike Fährmann	12eebb6f16	[xhamster] support xhamster.porncache.net domains (closes #700)	4 years ago
Mike Fährmann	e749402191	[mastodon] fix pagination (#701)	4 years ago
Mike Fährmann	921914141e	[imgbb] improve redirect handling	4 years ago
Mike Fährmann	6cc800aad4	[instagram] add 'post_id' and 'num' metadata fields (closes #698)	4 years ago
Mike Fährmann	a3de234e70	[hitomi] add extractor for tag searches (closes #697)	4 years ago
Mike Fährmann	456f6e8d05	[nozomi] move '_unpack()' method to global scope	4 years ago
Mike Fährmann	55ac408bdf	[hitomi] fix extraction of galleries without tags	4 years ago
Mike Fährmann	db6685eeae	[aryion] support downloading from folders (fixes #694)	4 years ago
Mike Fährmann	fa2952ac55	[furaffinity] add 'following' extractor (#515)	4 years ago
Mike Fährmann	9b194520db	[newgrounds] add 'following' extractor (closes #684)	4 years ago
Mike Fährmann	6386ee54e1	[deviantart] add extractor info to 'following' results	4 years ago
Mike Fährmann	d5273f9b0c	[hiperdex] update domain to hiperdex.net	4 years ago
Mike Fährmann	08674a91f3	[patreon] fix hash extraction from download URLs (closes #693) ... The old method was assuming every URL path ends with '/1'. For URLs where this is not the case, the segment containing the post ID was used as file hash.	4 years ago
Mike Fährmann	a6286bb551	[hiperdex] add 'artist' extractor (#606)	5 years ago
Mike Fährmann	291033720a	[hiperdex] fix manga extraction	5 years ago
Mike Fährmann	dfc0557807	[vsco] fix collection extraction	5 years ago
Mike Fährmann	fd438f0d78	update extractor test results	5 years ago
Mike Fährmann	bae1e8ed12	[deviantart] fix JPEG quality replacement pattern ... 'q_\d+' would sometimes also replace something in the 'token' query parameter, invalidating the URL.	5 years ago
Mike Fährmann	cf4cef3d63	[aryion] adjust 'date' to UTC time	5 years ago
Mike Fährmann	6c531be294	[aryion] fix malformed 'last-modified' headers (#390)	5 years ago
Mike Fährmann	dc65f7d8dc	[aryion] use generic download URLs (#390) ... i.e. /g4/data.php?id=… - get filename & extension from Content-Disposition header - handle all downloadable file types (docx, swf, etc)	5 years ago
Mike Fährmann	96b78bcf04	[aryion] include path in default directory format (#390)	5 years ago
Mike Fährmann	6143050980	[aryion] add gallery and post extractors (#390, #673)	5 years ago
Mike Fährmann	9e7dfc0cfc	[myportfolio] fix extraction of galleries without title	5 years ago
Mike Fährmann	88fca0a172	[mastodon] update OAuth credentials for pawoo.net (#665)	5 years ago
Mike Fährmann	4ae8a25567	[mastodon] use 'combine_dict()' to combine extractor info dicts	5 years ago
Mike Fährmann	220c06b86e	[mastodon] handle rate limits	5 years ago
Mike Fährmann	d02f7c1118	improve Extractor.wait() ... - allow 'until' to be a datetime object - do "time calculations" with UTC timestamps - set a default 'reason'	5 years ago
Mike Fährmann	5d7404ab58	[oauth] use the new name for 'DeviantartAPI' (fixes #670)	5 years ago
Mike Fährmann	762c758af4	[hiperdex] fix extraction	5 years ago
Mike Fährmann	f9a590f92b	[deviantart] apply HTTP request limits in more places ... "Request blocked" can also happen on sta.sh and for *any* HTTP request directed at deviantart.com	5 years ago
Mike Fährmann	2587296deb	[mastodon] add access tokens for mastodon.social and baraag.net ... (closes #665)	5 years ago
Mike Fährmann	ff7c0b7eff	[deviantart] handle "Request blocked" errors (#655) ... - add a 2 second wait time between requests to deviantart.com - catch 403 "Request blocked" errors and wait for 3 minutes until retrying	5 years ago
Mike Fährmann	c874684f05	[deviantart] retrieve *all* download URLs through OAuth API ... '/extended_fetch' as well as Deviation webpages now again contain Deviation UUIDs needed to grab Deviation info through the OAuth API, meaning cookies are no longer necessary to grab original files. The only instance were cookies are still needed are scraps marked as "mature", since those entries are hidden for public users. (#655, #657, #660)	5 years ago
Mike Fährmann	5c27b25a8f	[deviantart] improve sta.sh extraction ... Extract all sta.sh items in a single extractor run. Don't spawn a new StashExtractor for each individual sta.sh item to preserve the current requests.Session and its opened TCP connections.	5 years ago
Mike Fährmann	e2fc4eaa6f	[deviantart] detect stash folders (fixes #659)	5 years ago
Mike Fährmann	c034159701	[piczel] fix extraction for single images	5 years ago
Mike Fährmann	699036ea0c	[weibo] accept status URLs with non-numeric IDs (#664)	5 years ago
Mike Fährmann	fe96f99e4b	[hentainexus] reduce line length (flake8) & update test	5 years ago
墨焓	6f81cac8fa	Add metadata to hentainexus: circle, event, title_conventional. (#661)	5 years ago
Mike Fährmann	6f911aeb1c	[deviantart] add error message for cloudFront blocks (#655)	5 years ago
Mike Fährmann	7499d71d02	[simplyhentai] ignore certificate errors in video test	5 years ago
Mike Fährmann	4203dc0bdc	[mangapark] fix metadata extraction	5 years ago
Mike Fährmann	1b82d36ab2	[deviantart] handle decode errors for extended_fetch results (#655) ... This isn't going to solve the underlying problem, but it should at least provide the server response when those errors happen.	5 years ago
Mike Fährmann	09f2271528	[35photo] add 'tag' extractor	5 years ago
Mike Fährmann	77fda8190c	[35photo] simplify/remove tests for the 'genre' extractor ... There is still a nice genre overview page (https://35photo.pro/genre/) but the individual sub-pages don't list photos anymore	5 years ago
Mike Fährmann	fb846c9ee5	[instagram] reduce line lengths and make flake8 happy	5 years ago
Mike Fährmann	ad2efa8509	[e621] derive from Danbooru extractors (#651) ... - use extractor implementations from 'danbooru' - use "page": "b[ID]" to paginate over results instead of "tags": "id:<[ID]", avoiding infinite loops with certain post orders - bump User-Agent version	5 years ago
Mike Fährmann	9b39e1cd7e	[e621] fix bug in API rate limiting (#651)	5 years ago
Mike Fährmann	b607d0ad7f	[twitter] fix typo in 'x-twitter-auth-type' header (#625)	5 years ago
Mike Fährmann	2c3b9e1450	[nozomi] support multiple images per post (#646) ... This changes the default filename format as well as archive IDs, since those assumed that each post would only have one image.	5 years ago
Mike Fährmann	c606d0c854	[instagram] update pattern for user profile URLs ... Allow for query parameters and fragments, for example https://www.instagram.com/instagram/?hl=en	5 years ago
Mike Fährmann	2530db3f4d	[mangadex] transform 'date' timestamps to datetime objects	5 years ago
Mike Fährmann	ae2a33243b	[newgrounds] catch general Exceptions	5 years ago
Mike Fährmann	32e36d8f02	[sexcom] replace tests	5 years ago
Mike Fährmann	33b42dc847	[nozomi] sort search results (fixes #646)	5 years ago
Mike Fährmann	eaa60a438b	[piczel] fix extraction ... - manually filter by folder_id - extract data for single posts from embedded JSON, since the '/api/gallery/image/<id>' endpoint is no longer available	5 years ago
Mike Fährmann	5bcc7184c9	[danbooru][e621] increase page limits	5 years ago
Mike Fährmann	90d15e3682	[instagram] use 'itertools.chain()'	5 years ago
Leonardo Taccari	160328d21c	[instagram] Add support for user's saved medias (#644) ... * [instagram] Gracefully handle possible 'HttpErrorPage' in _extract_page() `HttpErrorPage' is returned in shared_data at least when not authenticated or when trying to fetch other users saved medias (i.e. `instagram.com/<user>/saved/'). Gracefully handle it by returning nothing. * [instagram] Add support for user's saved medias (Please note that this need the user to be authenticated and they can only see their saved media (not other users ones).) Close #643. * [instagram] Bump copyright year	5 years ago
Mike Fährmann	d3482ace7f	[furaffinity] extract more metadata ... - views - favorites - comments - rating - fa_category (since 'category' is already in use) - theme - species - gender - width - height	5 years ago
Mike Fährmann	fdd2dd5136	[kabeuchi] add 'user' extractor (closes #561)	5 years ago
Mike Fährmann	59edcdc822	[hitomi] restore metadata fields from before f33b13a ... ... and add a 'metadata' option to disable visiting the gallery page and extracting data from it if this is not needed.	5 years ago
Mike Fährmann	2d5703c493	[twitter] use a simpler data structure to store cookies in cache ... Use a dict with name-value pairs instead of an entire RequestsCookieJar object.	5 years ago
Mike Fährmann	87d4f83597	[newgrounds] make post extraction nonfatal	5 years ago
Mike Fährmann	823fbeaae6	[newgrounds] add 'favorite' extractor (#394)	5 years ago
Mike Fährmann	a45fbc38ea	[pixiv] implement 'avatar' option (#595, #623)	5 years ago
Mike Fährmann	a63a376ad2	[mangoxo] fix login	5 years ago
Mike Fährmann	ebc70e87ce	[e621] update to new interface / API endpoints (closes #635)	5 years ago
Mike Fährmann	d1cf7ccdb3	[instagram] add 'post_shortcode' metadata field (#525)	5 years ago
Mike Fährmann	32df8d06fe	[twitter] add 'bookmark' extractor (closes #625)	5 years ago
Mike Fährmann	3fb41c34c8	[bcy] reduce requests to '/item/detail/<id>' (#613) ... The former implementation would try to use the embedded data from '/item/detail/' pages for every post, even if that wasn't really necessary. This commit also fixes some issues with posts only visible to logged in users.	5 years ago
Mike Fährmann	f33b13aacf	[hitomi] simplify metadata extraction ... Use the data from https://ltn.hitomi.la/galleries/<id>.js for both image URLs and metadata and ignore any gallery or reader pages. This removes 'artist', 'characters', 'group', and 'parody' metadata fields since this information is, as for now, only available in gallery pages.	5 years ago
Mike Fährmann	ce5e2a58fe	[imgbb] update test results ... Image server domain changed from https://image.ibb.co/ to https://i.ibb.co/	5 years ago
Mike Fährmann	f117e32910	[danbooru] restore 'popular' functionality	5 years ago
Mike Fährmann	39b48d665b	[hiperdex] use proper name for 'chapter_minor'	5 years ago
Mike Fährmann	8fbbaa54ff	[bcy] fix partial image URLs (#613) ... Images from new posts can have incomplete/partial URLs (1) without any filename extension when fetching their data from '/apiv3/user/selfPosts', so now all data gets taken from '/item/detail/ID' pages. It is currently unknown how to get the non-watermarked original version of these images, or if that is possible at all. (2) Images with a watermark will have their 'filter' metadata field set to "watermark". For original images this field is an empty string "". Enabling the 'noop' option will, in addition to the watermarked version, yield the the '~noop.image' filter version (3), where 'filter' is set to "noop". (1) "https://img-bcy-qn.pstatp.com/banciyuan/3ccdff22479c4060aadc86718209b281" (2) "https://p1-bcy.byteimg.com/img/banciyuan/3ccdff22479c4060aadc86718209b281~tplv-banciyuan-logo-v3:wqnpnLLlhZLlpKfprZTnjotfCuWNiuasoeWFgyAtIEFDR-eIseWlveiAheekvuWMug==.image" (3) "https://p1-bcy.byteimg.com/img/banciyuan/3ccdff22479c4060aadc86718209b281~noop.image"	5 years ago
Mike Fährmann	86c00f9e66	[danbooru] move extractor logic from booru.py	5 years ago
Mike Fährmann	1d4a369ea2	update extractor test results	5 years ago
Mike Fährmann	7625912b31	[piczel] improve and update ... - fix tag names - fix a bug in _pagination() - parse datetime in 'created_at' as 'date' - rewrite main loop - replace user profile test	5 years ago
Mike Fährmann	913b8333cc	write DeviantArt refresh-tokens to cache (#616) ... Writing the token is currently disabled by default and must be enabled with 'extractor.oauth.cache'. 'extractor.deviantart.refresh-token' must be set to '"cache"' to use the cached token.	5 years ago
Mike Fährmann	2a4f227e08	warn about expired cookies	5 years ago
Mike Fährmann	4e361b3008	add tests for specific datetime values	5 years ago
Mike Fährmann	80ecb99089	[hitomi] fix extraction	5 years ago
Mike Fährmann	247c9e1416	[vsco] update gallery URL pattern	5 years ago
Mike Fährmann	19ae6f3fc4	update test results ... - twitter: Don't test the whole kwdict, only the actual content, since the keyword hash changes whenever that user changes his display name. - khinsider: Download host changed	5 years ago
Mike Fährmann	cc5079c844	[hiperdex] add chapter and manga extractors (closes #606)	5 years ago
Mike Fährmann	64bdec8430	[deviantart] check availability of intermediary URLs (fixes #609)	5 years ago
Mike Fährmann	5607dd3646	[hitomi] follow multiple redirects	5 years ago
Mike Fährmann	765b2a0527	[hentaihand] add extractors (closes #605)	5 years ago
Mike Fährmann	d94215d119	[tumblr] replace '-' with ' ' in tag searches (fixes #611) ... To search for tags with actual minus signs in them (there shouldn't be too many,) manually replace those with url-encoded minus characters ('-' -> '%2d') before inputting them into gallery-dl: https://s679874.tumblr.com/tagged/tag-with-minus -> https://s679874.tumblr.com/tagged/tag%2dwith%2dminus	5 years ago
Mike Fährmann	e6cd49e78b	update extractor test results	5 years ago
Mike Fährmann	5d9437b398	[vsco] skip "invalid" entities	5 years ago
Mike Fährmann	650f2b6d58	[furaffinity] accept sfw.furaffinity.net URLs (closes #608) ... Just as an alias for regular URLs with no extra content filtering.	5 years ago
Mike Fährmann	74e684e828	[twitter] change default value for 'videos' to 'true' ... Every other 'videos' option defaulted to 'true', except Twitter.	5 years ago
Mike Fährmann	c7cf9dd111	[furaffinity] support classic layout (#284)	5 years ago
Mike Fährmann	138135c190	[furaffinity] add extractors (#284)	5 years ago
Mike Fährmann	b9c574bd1d	[patreon] log skipped files (#590)	5 years ago
Mike Fährmann	80ea9104b8	[8kun] adjust URL pattern	5 years ago
Mike Fährmann	ce26070231	[pixiv] reduce calls to '/user/detail'	5 years ago
Mike Fährmann	da0d5f6092	[oauth] add 'port' option (#604)	5 years ago
Mike Fährmann	719b63d0ca	[bcy] add user and post extractors (#592)	5 years ago
Mike Fährmann	6426e3efc7	[khinsider] fix and improve metadata extraction	5 years ago