1875 Commits (055c32e0f77302f8f7263366463d115a7df0eaf0)

Author	SHA1	Message	Date
Mike Fährmann	fa2952ac55	[furaffinity] add 'following' extractor (#515)	4 years ago
Mike Fährmann	9b194520db	[newgrounds] add 'following' extractor (closes #684)	4 years ago
Mike Fährmann	6386ee54e1	[deviantart] add extractor info to 'following' results	4 years ago
Mike Fährmann	d5273f9b0c	[hiperdex] update domain to hiperdex.net	4 years ago
Mike Fährmann	08674a91f3	[patreon] fix hash extraction from download URLs (closes #693) ... The old method was assuming every URL path ends with '/1'. For URLs where this is not the case, the segment containing the post ID was used as file hash.	4 years ago
Mike Fährmann	a6286bb551	[hiperdex] add 'artist' extractor (#606)	5 years ago
Mike Fährmann	291033720a	[hiperdex] fix manga extraction	5 years ago
Mike Fährmann	dfc0557807	[vsco] fix collection extraction	5 years ago
Mike Fährmann	fd438f0d78	update extractor test results	5 years ago
Mike Fährmann	bae1e8ed12	[deviantart] fix JPEG quality replacement pattern ... 'q_\d+' would sometimes also replace something in the 'token' query parameter, invalidating the URL.	5 years ago
Mike Fährmann	cf4cef3d63	[aryion] adjust 'date' to UTC time	5 years ago
Mike Fährmann	6c531be294	[aryion] fix malformed 'last-modified' headers (#390)	5 years ago
Mike Fährmann	dc65f7d8dc	[aryion] use generic download URLs (#390) ... i.e. /g4/data.php?id=… - get filename & extension from Content-Disposition header - handle all downloadable file types (docx, swf, etc)	5 years ago
Mike Fährmann	96b78bcf04	[aryion] include path in default directory format (#390)	5 years ago
Mike Fährmann	6143050980	[aryion] add gallery and post extractors (#390, #673)	5 years ago
Mike Fährmann	9e7dfc0cfc	[myportfolio] fix extraction of galleries without title	5 years ago
Mike Fährmann	88fca0a172	[mastodon] update OAuth credentials for pawoo.net (#665)	5 years ago
Mike Fährmann	4ae8a25567	[mastodon] use 'combine_dict()' to combine extractor info dicts	5 years ago
Mike Fährmann	220c06b86e	[mastodon] handle rate limits	5 years ago
Mike Fährmann	d02f7c1118	improve Extractor.wait() ... - allow 'until' to be a datetime object - do "time calculations" with UTC timestamps - set a default 'reason'	5 years ago
Mike Fährmann	5d7404ab58	[oauth] use the new name for 'DeviantartAPI' (fixes #670)	5 years ago
Mike Fährmann	762c758af4	[hiperdex] fix extraction	5 years ago
Mike Fährmann	f9a590f92b	[deviantart] apply HTTP request limits in more places ... "Request blocked" can also happen on sta.sh and for *any* HTTP request directed at deviantart.com	5 years ago
Mike Fährmann	2587296deb	[mastodon] add access tokens for mastodon.social and baraag.net ... (closes #665)	5 years ago
Mike Fährmann	ff7c0b7eff	[deviantart] handle "Request blocked" errors (#655) ... - add a 2 second wait time between requests to deviantart.com - catch 403 "Request blocked" errors and wait for 3 minutes until retrying	5 years ago
Mike Fährmann	c874684f05	[deviantart] retrieve *all* download URLs through OAuth API ... '/extended_fetch' as well as Deviation webpages now again contain Deviation UUIDs needed to grab Deviation info through the OAuth API, meaning cookies are no longer necessary to grab original files. The only instance were cookies are still needed are scraps marked as "mature", since those entries are hidden for public users. (#655, #657, #660)	5 years ago
Mike Fährmann	5c27b25a8f	[deviantart] improve sta.sh extraction ... Extract all sta.sh items in a single extractor run. Don't spawn a new StashExtractor for each individual sta.sh item to preserve the current requests.Session and its opened TCP connections.	5 years ago
Mike Fährmann	e2fc4eaa6f	[deviantart] detect stash folders (fixes #659)	5 years ago
Mike Fährmann	c034159701	[piczel] fix extraction for single images	5 years ago
Mike Fährmann	699036ea0c	[weibo] accept status URLs with non-numeric IDs (#664)	5 years ago
Mike Fährmann	fe96f99e4b	[hentainexus] reduce line length (flake8) & update test	5 years ago
墨焓	6f81cac8fa	Add metadata to hentainexus: circle, event, title_conventional. (#661)	5 years ago
Mike Fährmann	6f911aeb1c	[deviantart] add error message for cloudFront blocks (#655)	5 years ago
Mike Fährmann	7499d71d02	[simplyhentai] ignore certificate errors in video test	5 years ago
Mike Fährmann	4203dc0bdc	[mangapark] fix metadata extraction	5 years ago
Mike Fährmann	1b82d36ab2	[deviantart] handle decode errors for extended_fetch results (#655) ... This isn't going to solve the underlying problem, but it should at least provide the server response when those errors happen.	5 years ago
Mike Fährmann	09f2271528	[35photo] add 'tag' extractor	5 years ago
Mike Fährmann	77fda8190c	[35photo] simplify/remove tests for the 'genre' extractor ... There is still a nice genre overview page (https://35photo.pro/genre/) but the individual sub-pages don't list photos anymore	5 years ago
Mike Fährmann	fb846c9ee5	[instagram] reduce line lengths and make flake8 happy	5 years ago
Mike Fährmann	ad2efa8509	[e621] derive from Danbooru extractors (#651) ... - use extractor implementations from 'danbooru' - use "page": "b[ID]" to paginate over results instead of "tags": "id:<[ID]", avoiding infinite loops with certain post orders - bump User-Agent version	5 years ago
Mike Fährmann	9b39e1cd7e	[e621] fix bug in API rate limiting (#651)	5 years ago
Mike Fährmann	b607d0ad7f	[twitter] fix typo in 'x-twitter-auth-type' header (#625)	5 years ago
Mike Fährmann	2c3b9e1450	[nozomi] support multiple images per post (#646) ... This changes the default filename format as well as archive IDs, since those assumed that each post would only have one image.	5 years ago
Mike Fährmann	c606d0c854	[instagram] update pattern for user profile URLs ... Allow for query parameters and fragments, for example https://www.instagram.com/instagram/?hl=en	5 years ago
Mike Fährmann	2530db3f4d	[mangadex] transform 'date' timestamps to datetime objects	5 years ago
Mike Fährmann	ae2a33243b	[newgrounds] catch general Exceptions	5 years ago
Mike Fährmann	32e36d8f02	[sexcom] replace tests	5 years ago
Mike Fährmann	33b42dc847	[nozomi] sort search results (fixes #646)	5 years ago
Mike Fährmann	eaa60a438b	[piczel] fix extraction ... - manually filter by folder_id - extract data for single posts from embedded JSON, since the '/api/gallery/image/<id>' endpoint is no longer available	5 years ago
Mike Fährmann	5bcc7184c9	[danbooru][e621] increase page limits	5 years ago
Mike Fährmann	90d15e3682	[instagram] use 'itertools.chain()'	5 years ago
Leonardo Taccari	160328d21c	[instagram] Add support for user's saved medias (#644) ... * [instagram] Gracefully handle possible 'HttpErrorPage' in _extract_page() `HttpErrorPage' is returned in shared_data at least when not authenticated or when trying to fetch other users saved medias (i.e. `instagram.com/<user>/saved/'). Gracefully handle it by returning nothing. * [instagram] Add support for user's saved medias (Please note that this need the user to be authenticated and they can only see their saved media (not other users ones).) Close #643. * [instagram] Bump copyright year	5 years ago
Mike Fährmann	d3482ace7f	[furaffinity] extract more metadata ... - views - favorites - comments - rating - fa_category (since 'category' is already in use) - theme - species - gender - width - height	5 years ago
Mike Fährmann	fdd2dd5136	[kabeuchi] add 'user' extractor (closes #561)	5 years ago
Mike Fährmann	59edcdc822	[hitomi] restore metadata fields from before f33b13a ... ... and add a 'metadata' option to disable visiting the gallery page and extracting data from it if this is not needed.	5 years ago
Mike Fährmann	2d5703c493	[twitter] use a simpler data structure to store cookies in cache ... Use a dict with name-value pairs instead of an entire RequestsCookieJar object.	5 years ago
Mike Fährmann	87d4f83597	[newgrounds] make post extraction nonfatal	5 years ago
Mike Fährmann	823fbeaae6	[newgrounds] add 'favorite' extractor (#394)	5 years ago
Mike Fährmann	a45fbc38ea	[pixiv] implement 'avatar' option (#595, #623)	5 years ago
Mike Fährmann	a63a376ad2	[mangoxo] fix login	5 years ago
Mike Fährmann	ebc70e87ce	[e621] update to new interface / API endpoints (closes #635)	5 years ago
Mike Fährmann	d1cf7ccdb3	[instagram] add 'post_shortcode' metadata field (#525)	5 years ago
Mike Fährmann	32df8d06fe	[twitter] add 'bookmark' extractor (closes #625)	5 years ago
Mike Fährmann	3fb41c34c8	[bcy] reduce requests to '/item/detail/<id>' (#613) ... The former implementation would try to use the embedded data from '/item/detail/' pages for every post, even if that wasn't really necessary. This commit also fixes some issues with posts only visible to logged in users.	5 years ago
Mike Fährmann	f33b13aacf	[hitomi] simplify metadata extraction ... Use the data from https://ltn.hitomi.la/galleries/<id>.js for both image URLs and metadata and ignore any gallery or reader pages. This removes 'artist', 'characters', 'group', and 'parody' metadata fields since this information is, as for now, only available in gallery pages.	5 years ago
Mike Fährmann	ce5e2a58fe	[imgbb] update test results ... Image server domain changed from https://image.ibb.co/ to https://i.ibb.co/	5 years ago
Mike Fährmann	f117e32910	[danbooru] restore 'popular' functionality	5 years ago
Mike Fährmann	39b48d665b	[hiperdex] use proper name for 'chapter_minor'	5 years ago
Mike Fährmann	8fbbaa54ff	[bcy] fix partial image URLs (#613) ... Images from new posts can have incomplete/partial URLs (1) without any filename extension when fetching their data from '/apiv3/user/selfPosts', so now all data gets taken from '/item/detail/ID' pages. It is currently unknown how to get the non-watermarked original version of these images, or if that is possible at all. (2) Images with a watermark will have their 'filter' metadata field set to "watermark". For original images this field is an empty string "". Enabling the 'noop' option will, in addition to the watermarked version, yield the the '~noop.image' filter version (3), where 'filter' is set to "noop". (1) "https://img-bcy-qn.pstatp.com/banciyuan/3ccdff22479c4060aadc86718209b281" (2) "https://p1-bcy.byteimg.com/img/banciyuan/3ccdff22479c4060aadc86718209b281~tplv-banciyuan-logo-v3:wqnpnLLlhZLlpKfprZTnjotfCuWNiuasoeWFgyAtIEFDR-eIseWlveiAheekvuWMug==.image" (3) "https://p1-bcy.byteimg.com/img/banciyuan/3ccdff22479c4060aadc86718209b281~noop.image"	5 years ago
Mike Fährmann	86c00f9e66	[danbooru] move extractor logic from booru.py	5 years ago
Mike Fährmann	1d4a369ea2	update extractor test results	5 years ago
Mike Fährmann	7625912b31	[piczel] improve and update ... - fix tag names - fix a bug in _pagination() - parse datetime in 'created_at' as 'date' - rewrite main loop - replace user profile test	5 years ago
Mike Fährmann	913b8333cc	write DeviantArt refresh-tokens to cache (#616) ... Writing the token is currently disabled by default and must be enabled with 'extractor.oauth.cache'. 'extractor.deviantart.refresh-token' must be set to '"cache"' to use the cached token.	5 years ago
Mike Fährmann	2a4f227e08	warn about expired cookies	5 years ago
Mike Fährmann	4e361b3008	add tests for specific datetime values	5 years ago
Mike Fährmann	80ecb99089	[hitomi] fix extraction	5 years ago
Mike Fährmann	247c9e1416	[vsco] update gallery URL pattern	5 years ago
Mike Fährmann	19ae6f3fc4	update test results ... - twitter: Don't test the whole kwdict, only the actual content, since the keyword hash changes whenever that user changes his display name. - khinsider: Download host changed	5 years ago
Mike Fährmann	cc5079c844	[hiperdex] add chapter and manga extractors (closes #606)	5 years ago
Mike Fährmann	64bdec8430	[deviantart] check availability of intermediary URLs (fixes #609)	5 years ago
Mike Fährmann	5607dd3646	[hitomi] follow multiple redirects	5 years ago
Mike Fährmann	765b2a0527	[hentaihand] add extractors (closes #605)	5 years ago
Mike Fährmann	d94215d119	[tumblr] replace '-' with ' ' in tag searches (fixes #611) ... To search for tags with actual minus signs in them (there shouldn't be too many,) manually replace those with url-encoded minus characters ('-' -> '%2d') before inputting them into gallery-dl: https://s679874.tumblr.com/tagged/tag-with-minus -> https://s679874.tumblr.com/tagged/tag%2dwith%2dminus	5 years ago
Mike Fährmann	e6cd49e78b	update extractor test results	5 years ago
Mike Fährmann	5d9437b398	[vsco] skip "invalid" entities	5 years ago
Mike Fährmann	650f2b6d58	[furaffinity] accept sfw.furaffinity.net URLs (closes #608) ... Just as an alias for regular URLs with no extra content filtering.	5 years ago
Mike Fährmann	74e684e828	[twitter] change default value for 'videos' to 'true' ... Every other 'videos' option defaulted to 'true', except Twitter.	5 years ago
Mike Fährmann	c7cf9dd111	[furaffinity] support classic layout (#284)	5 years ago
Mike Fährmann	138135c190	[furaffinity] add extractors (#284)	5 years ago
Mike Fährmann	b9c574bd1d	[patreon] log skipped files (#590)	5 years ago
Mike Fährmann	80ea9104b8	[8kun] adjust URL pattern	5 years ago
Mike Fährmann	ce26070231	[pixiv] reduce calls to '/user/detail'	5 years ago
Mike Fährmann	da0d5f6092	[oauth] add 'port' option (#604)	5 years ago
Mike Fährmann	719b63d0ca	[bcy] add user and post extractors (#592)	5 years ago
Mike Fährmann	6426e3efc7	[khinsider] fix and improve metadata extraction	5 years ago
Mike Fährmann	b7eb6cecbb	[pixiv] handle tags at the end of new bookmark URLs	5 years ago
Mike Fährmann	109f6c8685	[patreon] filter duplicate files per post (#590)	5 years ago
Mike Fährmann	b38cf59711	[sexcom] fix image URLs & parse 'date' fields	5 years ago
Mike Fährmann	1f4c9c5f9d	[8kun] add thread and board extractors (closes #582)	5 years ago
Mike Fährmann	facc5daa6d	[twitter] force old login page layout (fixes #584, fixes #598)	5 years ago
Mike Fährmann	d1de7dc296	[hitomi] implement workaround for "broken" redirects ... Some galleries redirect to a new "version" with different gallery id. This new version might not be available any more, but the /reader/ page for the original gallery id can still work.	5 years ago
Mike Fährmann	40fe062851	[pixiv] fix user id for bookmarks API calls (closes #596)	5 years ago
Mike Fährmann	91aaaf1a9e	[pixiv] add 'rating' metadata field (#595) ... A human-friendlier representation of 'x_restrict'	5 years ago
Mike Fährmann	dff33b260c	[reddit] add 'videos' option	5 years ago
Mike Fährmann	2ad43618cc	[piczel] fix extraction	5 years ago
Mike Fährmann	cf7a67d67f	[yaplog] remove module ... Yaplog! ended its service on 2020-01-31	5 years ago
Mike Fährmann	e0dd073ce0	[twitter] replace embedded tweet test ... the old one was deleted	5 years ago
Mike Fährmann	ec36df4851	[deviantart] fix video extraction from 'extended_fetch' results ... DeviantArt is now serving videos from wixmp servers (1), instead of the former film00.deviantart.com (2), even though those URLS are still functional. They seem to also have re-encoded those videos. The 10 MB 1080p video from (2) is now only available in 720p at ~20 MB (with a higher bitrate, but still …). Other videos are still available in 1080p, but not this one for some reason. (Changing the '720p' in (1) to '1080p' doesn't work.) (1) https://wixmp-ed30a86b8c4ca887773594c2.wixmp.com/v/mp4/9feaa2c9-1baf-4fc2-84f7-f3384b34cefe/d5gxnb5-282a2e9a-b552-40ff-8542-b3c5eed823f5.720p.a837d7cec12c41be8ca2ee53152cea3a.mp4 (2) https://film00.deviantart.net/4c1d/v/mp4/2012/279/d/1/_video____brushes_i_use_in_paint_tool_sai_by_chi_u-d5gxnb5.mp4	5 years ago
Mike Fährmann	48be2266ed	[deviantart] better error message for 'extended_fetch' (#585)	5 years ago
Mike Fährmann	71851a6241	[pixiv] update URLs of followed users to the new format	5 years ago
Mike Fährmann	d086f30b42	[reddit] restore archive keys for i.redd.it images	5 years ago
Mike Fährmann	56f1c96168	implement 'parent-directory' option (#551)	5 years ago
Mike Fährmann	ae07f92f7e	[reddit] rewrite extractor logic (closes #551) ... Handle images and videos hosted on Reddit "natively", allowing them to use reddit-specific metadata to build directory and file names.	5 years ago
Mike Fährmann	2852691d78	[paheal] replace test URL ... searching for 'k-on' doesn't yield any results anymore	5 years ago
Mike Fährmann	2a9be48511	improve util.load/save_cookiestxt() and add tests ... - take a file object as argument instead of an filename - accept whitespace before comments (" # comment") - map expiration "0" to None and not the number 0	5 years ago
Mike Fährmann	e35c2ea1a6	[weibo] use youtube-dl to download from m3u8 manifests	5 years ago
Mike Fährmann	6703b8a86b	[blogger] implement video extraction (closes #587)	5 years ago
Mike Fährmann	c1a6862863	implement functions to load/save cookies.txt files (closes #586) ... The methods of the standard libraries' MozillaCookieJar have several shortcomings (#HttpOnly_ cookies, 0 expiration timestamps, etc.) and require construction of an ultimately pointless CookieJar object.	5 years ago
Mike Fährmann	25d5ec4ff3	[twitter] add option to extract TwitPic embeds (#579)	5 years ago
Mike Fährmann	32d7195d08	[pinterest] improve detection of invalid pin.it links	5 years ago
Mike Fährmann	174117f827	allow multiple hashes for content tests	5 years ago
Alice	f498a9057f	[twitter] Fix stop before real end (#573) ... * [twitter] Fix stop before real end Fix for https://github.com/mikf/gallery-dl/issues/544. Makes sure that it really reached the end by checking that both "min_position" is null and "has_more_items" is false before stopping. * [twitter] Fix stop before real end (update)	5 years ago
Mike Fährmann	8bb32ee188	[hitomi] fix image URLs	5 years ago
Mike Fährmann	bd5ce9855c	allow GalleryExtractors to set URL-independent extensions	5 years ago
Mike Fährmann	af42c75152	[mangadex] revert domain to 'mangadex.org'	5 years ago
Mike Fährmann	e89413da22	update test results	5 years ago
Mike Fährmann	33a6e0ac6e	[hentaifoundry] extract more metadata (closes #565)	5 years ago
Mike Fährmann	5cac79c3d9	[erolord] remove extractor	5 years ago
Mike Fährmann	b9cbf932b4	[pixiv] update URL patterns (fixes #568) ... Pixiv now uses new URLs for - user profiles and illustration listings: - https://www.pixiv.net/en/users/<ID> - https://www.pixiv.net/en/users/<ID>/artworks - bookmarks: - https://www.pixiv.net/en/users/<ID>/bookmarks/artworks	5 years ago
Mike Fährmann	988cc2ec23	[mangadex] change domain to mangadex.cc (closes #559)	5 years ago
Mike Fährmann	f8e137d6b4	[deviantart] show warning about private deviations only once ... … per call to '_pagination()'	5 years ago
Mike Fährmann	939fec8ecd	[deviantart] match new search/popular URLs (closes #538)	5 years ago
Mike Fährmann	09cc88b715	[deviantart] match '/favourites/all' URLs (closes #555)	5 years ago
Mike Fährmann	3811fd8a25	fix time formatting for Python 3.4 and 3.5 ... 'datetime.time.isoformat()' only has an optional 'timespec' argument since Python 3.6.	5 years ago
Mike Fährmann	43ab9572b4	[twitter] handle API rate limits (#526)	5 years ago
Mike Fährmann	569747a78d	implement extractor.wait()	5 years ago
Mike Fährmann	5532e9c158	[twitter] handle quoted tweets (#526) ... … and categorize them as retweets	5 years ago
Mike Fährmann	0b4cb8e57a	[mangahere] send 'isAdult' cookie (fixes #556)	5 years ago
Mike Fährmann	1afb91363c	[imagefap] generalize URL patterns and add tests (#552)	5 years ago
Xope Totec	f701e9f33a	Handle beta.imagefap.com URLs (#552)	5 years ago
Mike Fährmann	ce54b8c04c	let extractors opt-out of cookie option usage ... useful to avoid sending unnecessary cookies when all authentication is done through OAuth tokens	5 years ago
Mike Fährmann	5ad92fc196	[newgrounds] fix tags metadata extraction	5 years ago
Mike Fährmann	82f7f4172a	update test results	5 years ago
Mike Fährmann	1f2a69f3c5	add '_extractor' information to redirect results	5 years ago
Mike Fährmann	a27f43dad1	[pixiv] wait and retry after rate limit error (closes #535)	5 years ago
Mike Fährmann	6b373cb7e2	[exhentai] restrict default directory name length (#545)	5 years ago
Mike Fährmann	b347bf68c7	[deviantart] add extractor for followed users (#515)	5 years ago
Mike Fährmann	c0f391a4e2	[pixiv] support listing followed users (#515)	5 years ago
Mike Fährmann	896896a490	[twitter] fix URLs forwarded to youtube-dl (closes #540) ... Since commit 3bba763 data["user"] is an entire dict object and no longer just the user nickname …	5 years ago
Mike Fährmann	1e2713b895	[artstation] fix search result pagination (closes #537)	5 years ago
Mike Fährmann	bf3df3d0b0	[directlink] send Referer headers (closes #536)	5 years ago
Mike Fährmann	9be7ff600e	[imagetwist] replace test image ... the old one expired, it seems	5 years ago
Mike Fährmann	66905b1664	[foolslide] add fallback for chapter data extraction	5 years ago
Mike Fährmann	48e42e73fb	[reddit] change default value for 'comments' to '0'	5 years ago
Mike Fährmann	9c0928457a	[reddit] fix errors with 't1_…' submissions	5 years ago
Mike Fährmann	bf658fd84b	[vsco] implement 'videos' option	5 years ago
Mike Fährmann	95c90722ee	[instagram] implement 'videos' option (closes #521)	5 years ago
Mike Fährmann	d0920e84e9	update test results	5 years ago
Mike Fährmann	8c11e81c9f	Merge commit '63e6993716db8d8bedfb7b0d445c7161493046b6'	5 years ago
Mike Fährmann	63e6993716	merge 'bypost' functionality into metadata postprocessor	5 years ago
Mike Fährmann	31a29835ff	[realbooru] simplify extractors and update tests (#514)	5 years ago
The Oddball	9a4ce20b8e	[realbooru] Add Realbooru extractor (#514)	5 years ago
Mike Fährmann	72b8fbfbad	[instagram] make post-page extraction nonfatal	5 years ago
Mike Fährmann	922b8a9595	[weibo] raise NotFoundError for unavailable/deleted statuses	5 years ago
Mike Fährmann	0cd157300e	[patreon] fix regex pattern for posts ... The previous one would match the first number in the URL slug as post ID, which would fail for posts with numbers in their title.	5 years ago
Mike Fährmann	fe19e233f3	[xvideos] improve ... - derive from GalleryExtractor - match '…-channels' URLs - "better" metadata structure	5 years ago
Mike Fährmann	d3e44e899d	raise NotFoundErrors for 404 responses in GalleryExtractors	5 years ago
Mike Fährmann	a4dd8b3dab	improve _check_cookies() ... Only loop over all cookies once instead of calling cookiejar._find() for each cookie name.	5 years ago
Mike Fährmann	76e60d10a6	[patreon] raise proper exception if creator/post doesn't exist	5 years ago
Mike Fährmann	9e63804347	[patreon] make retrieving user info nonfatal (#508) ... … and fall back to the included data if an error occurs.	5 years ago
Mike Fährmann	964dc57286	[vsco] improve image resolutions ... https://im.vsco.co/ URLs redirect to the appropriate CDN server and occasionally insert a '/1200x1600/' into the image path, limiting image dimensions. This commit constructs redirect targets out of the given im,vsco.co URLs without sending extra HTTP requests and without any "builtin" resolution restrictions.	5 years ago
Mike Fährmann	0629fe8fa4	[vsco] fix user profile extraction … again ... Given the pattern from last time, collections will also change in due time and use cursor-based pagination.	5 years ago
Mike Fährmann	ab17ea9632	[deviantart] only print warning if 'original' is enabled	5 years ago
Mike Fährmann	2188db6284	[gelbooru] fix non-API tag extraction	5 years ago
Mike Fährmann	c4702ec9b6	simplify some logging calls	5 years ago
Gio	c0b9ad678d	Separate metadata from handle_url into handle_metadata, commenting	5 years ago
Mike Fährmann	c9ef1b21c3	[patreon] get partial user info without /api/user/<id> (#507) ... It's a lot less data, but doesn't invoke any additional HTTP requests with potential Cloudflare CAPTCHAs.	5 years ago
Mike Fährmann	0ab9bb1721	[4chan] add extractor for entire boards (closes #510)	5 years ago
Gio	cfc70a97ab	Added an additional channel for downloading the metadata of an entire post or gallery.	5 years ago
Mike Fährmann	15f9bb3d14	add option to disable pyOpenSSL usage (#508) ... (pyOpenSSL is now disabled by default)	5 years ago
Mike Fährmann	c8e99e3b3b	[deviantart] fix crash on missing "token" field (#505)	5 years ago
Mike Fährmann	6ed2c7823c	[deviantart] disable original downloads if no cookies set ... For 'deviation' and 'scraps' extractors only, since original file downloads for those two will always fail with a 404 Not Found when not logged in.	5 years ago
Mike Fährmann	50deab5265	[deviantart] fix URL generation from /extended_fetch results ... (closes #505)	5 years ago
Mike Fährmann	1f209da4c0	[pixiv] match new search URLs (closes #507)	5 years ago
Mike Fährmann	e17907ee2a	change default value of 'cookies-update' to 'true'	5 years ago
Mike Fährmann	07dafad26d	[twitter] attempt to fix infinite loops (#499) ... (Hopefully this doesn't break anything else)	5 years ago
Mike Fährmann	71acbdabf4	[2chan] fix metadata extraction	5 years ago
Mike Fährmann	c0a1241648	[livedoor] force https:// for image URLs	5 years ago
Mike Fährmann	6e23c0da09	[imgur] add extractor for subreddit links (closes #500)	5 years ago
Mike Fährmann	372ffe95ee	[oauth] adjust Flickr redirect URI (fixes #503) ... Flickr now automatically forces https:// for all redirect URIs.	5 years ago
Mike Fährmann	004812258d	[hentaifox] fix extraction	5 years ago
Mike Fährmann	e2710702d4	fix Cloudflare bypss	5 years ago
Mike Fährmann	8759403f37	[plurk] add delay between comment requests	5 years ago
Mike Fährmann	a28552fd19	update test results ... - hbrowse: one tag got removed - mangoxo: gallery changed owner - photobucket: ?, but photo still downloads	5 years ago
Mike Fährmann	dcaa3d01bd	[imagefap] adapt to new image URL format	5 years ago
Mike Fährmann	e62c209ca0	[nijie] fix 'date' parsing	5 years ago
Mike Fährmann	3bba763ab9	[twitter] improve ... - update metadata structure - combine all user… entries into their own dict - let 'user' always specify the Timeline owner - add 'author' entry that specifies the original Tweet author - create directories per post (closes #491) - fix username issues with /i/web/ URLs	5 years ago
Mike Fährmann	db35c3b581	[directlink] separate filenames from paths ... With this, all default filename formats specify an '{extension}' and PathFormat.set_extension() reliably works for all files.	5 years ago
Mike Fährmann	41a3169c67	[foolfuuka] use '{extension}' in default filename format	5 years ago
Mike Fährmann	e9aed62c91	[imgur] unescape image titles	5 years ago
Mike Fährmann	2c332edaad	[plurk] fix comment pagination	5 years ago
Mike Fährmann	a3fa45bbb1	[behance] get images from 'media_collection' modules	5 years ago
Mike Fährmann	359c3bc1c5	[deviantart] revert to getting download URLs from OAuth API ... This commit (partially) reverts 27b5b24, 94eb7c6, and a437e78. Download URLs from the 'extended_fetch' endpoint are now only usable for logged in users, while those from the respective OAuth API endpoint are working again. Everything except scraps and direct deviation links should be fixed, and those two categories will work with exported cookies. (#488) TODO: - "native" login with --username and --password - better handling of internally stored cookies	5 years ago
Mike Fährmann	42b9633c7e	update test results	5 years ago
Mike Fährmann	b28bd1c73e	[bobx] set generated session cookie (closes #482) ... This reverts commit 490831f and also restores original image downloads by setting a randomly generated session cookie. No login required.	5 years ago
Mike Fährmann	ae09f87602	improve SharedConfigMixin config lookups	5 years ago
Mike Fährmann	f5604492c3	update interface of config functions	5 years ago
Mike Fährmann	4ca883c66f	[smugmug] replace test for custom URLs ... The old one (http://www.creativedogportraits.com/) is empty and/or no longer handled by SmugMug.	5 years ago
Mike Fährmann	d45fabb79d	match user profile handling on deviantart and newgrounds	5 years ago
Mike Fährmann	ea80dadd09	[deviantart] restore archive keys ... Commit 9fdc5e7 changed 'username' fields to have consistent capitalization, but that invalidated the archive keys of several extractors where 'username' was usually lowercase.	5 years ago
Mike Fährmann	ea094692c8	[vsco] fix collection extraction (#480)	5 years ago
Mike Fährmann	490831f84a	[bobx] "fix" image download URLs ... Access to original images got restricted to (paid) members only. All that's publicly accessible now are essentially preview pictures.	5 years ago
Mike Fährmann	978cb03f81	update misc test results ... - Livedoor now uses https:// for its image URLs - Instagram image URLs got simplified	5 years ago
Mike Fährmann	fca87974fe	[sexcom] fix video downloads by sending specific Referer headers	5 years ago
Mike Fährmann	edc080468d	[instagram] make 'video_url' fields optional (fixes #479) ... [ci skip]	5 years ago
Mike Fährmann	9fdc5e74cb	[deviantart] ensure consistent username capitalization (#455) ... The 'username' field was capitalized in a very inconsistent manner: Either all lowercase, or as given by the input URL, or with the "original" capitalization, depending on the extractor used among other things. Now usernames use their original capitalization for all extractors. ('UserName' instead of 'username' or 'uSeRnAmE')	5 years ago
Mike Fährmann	b1f0609de5	[newgrounds] rewrite (#394) ... - restructure extractor hierarchy - extract more metadata - extract videos without youtube-dl - be more resilient to errors TODO: - favorites - games, but that might be near impossible for non-flash titles	5 years ago
Mike Fährmann	3ece3976ae	[newgrounds] implement login support (#394)	5 years ago
Mike Fährmann	3a07c06865	[newgrounds] update ... - create directory per post - rename variables and methods	5 years ago
Mike Fährmann	5513b66eb0	[vsco] fix user profile extraction	5 years ago
Mike Fährmann	abfcb356fc	[flickr] support 3k, 4k, 5k, and 6k photo sizes (closes #472)	5 years ago
Mike Fährmann	521fcd2eb9	[imgbb] fix error in galleries without user info (closes #471)	5 years ago
Mike Fährmann	8061263d4c	[imgbb] improve pagination logic ... - avoid unnecessary API calls for small or empty galleries - combine duplicate code	5 years ago
Mike Fährmann	da6789b2b0	disable unique archive id checks for some tests ... - same image twice in a livedoor blog post - unreliable results for related pinterest items	5 years ago
Mike Fährmann	b0197098e6	[imgur] get title from webpage if missing in API response ... (closes #467)	5 years ago
Mike Fährmann	dd5d2b2eac	[deviantart] add user profile extractor (#377, #419)	5 years ago
Mike Fährmann	a437e78620	[deviantart] minimize cookie usage during scraps extraction ... (#445)	5 years ago
Mike Fährmann	1a197d2195	store the original cookiejar as Extractor._cookiejar	5 years ago
Mike Fährmann	de83ae4576	make 'method' argument of Extractor.request keyword-only	5 years ago
Mike Fährmann	4325695d74	[luscious] expand GraphQL queries	5 years ago
Mike Fährmann	94dbdbf506	[nijie] change default filename format ... … to be consistent with Pixiv filenames	5 years ago
Mike Fährmann	c18fadc221	[instagram] extract videos without youtube-dl (#391)	5 years ago
Mike Fährmann	f15eedb634	[sexcom] set Referer header for file downloads (closes #464)	5 years ago
Mike Fährmann	2a3bd4e3c7	rename extractor classes starting with a digit	5 years ago
Mike Fährmann	b3b9da6d74	[photobucket] replace test URL ... The other user deleted all of is images.	5 years ago
Mike Fährmann	64786363be	[4chan] simplify ... - remove 'chan.py' - slight adjustments to directory and filenames	5 years ago
Mike Fährmann	557e2c018b	[8chan] remove module	5 years ago
Mike Fährmann	e14782a948	[instagram] simplify graphql extraction for post pages	5 years ago
Mike Fährmann	c01ff78467	[twitter] extend 'videos' option to force extraction with ytdl ... (closes #459)	5 years ago
Mike Fährmann	f8ac67ce50	[hitomi] extend URL pattern + follow redirects	5 years ago
Mike Fährmann	e877ca97c3	[naver] adjust directory names and metadata structure	5 years ago
Mike Fährmann	702f2fbd1f	[issuu] add publication and user extractors (#413)	5 years ago
Mike Fährmann	8361d874d7	[hitomi] fix extraction	5 years ago
Mike Fährmann	5fa6ff04dd	[instagram] extract '__additionalDataLoaded' (#391) ... The '_sharedData' of Post pages is missing its 'graphql' part for logged in users. This data is now included in the parameters of a function call to '__additionalDataLoaded(...)' And, of course, video extraction with youtube-dl broke because of this change as well.	5 years ago
Mike Fährmann	87a87bff7e	[simplyhentai] fix image URLs	5 years ago
Mike Fährmann	4409d00141	embed error messages in StopExtraction exceptions	5 years ago
Mike Fährmann	d44f790e81	adjust output for HTTP status related errors	5 years ago
Mike Fährmann	109718a5e3	[blogger] add blog and post extractors (closes #364)	5 years ago
Mike Fährmann	49a6b1b6c0	[twitter] extract video stream info without youtube-dl (#452) ... This should allow video downloads when logged in without 'forward-cookies' disabled and from protected tweets. youtube-dl still gets used to download HLS playlists, but the data extraction part, which doesn't work with youtube-dl at the moment, now gets handled by gallery-dl itself.	5 years ago
Mike Fährmann	9f0dbf2a72	[twitter] raise proper exception for protected Tweets	5 years ago