3087 Commits (e9dc6ff26258a6560ba78f55a39dcbac2bdb00ab)

Author	SHA1	Message	Date
Mike Fährmann	a5de2244d4	[furaffinity] fix using 'category-tranfer' (#1274)	3 years ago
Mike Fährmann	cadfad4eea	[danbooru] add 'external' option (closes #1747)	3 years ago
Mike Fährmann	5b1c62bfa9	[furaffinity] add 'external' option (closes #1492)	3 years ago
Mike Fährmann	b5b1cf22b7	[downloader:http] reorder HTTP header sources ... so that any header can be overwritten by a user, except Range	3 years ago
Mike Fährmann	5d5ab669fa	[instagram] use custom User-Agent header for video downloads ... (#1682, #1623, #1580)	3 years ago
Mike Fährmann	7b029dfe85	[instagram] increase default delay between HTTP requests to 8s ... (closes #1732)	3 years ago
Mike Fährmann	5eca3781be	[kemonoparty] fix username extraction (#1750)	3 years ago
Mike Fährmann	fe970fc87f	[vk] prevent exception for empty/private profiles (fixes 1742)	3 years ago
Mike Fährmann	ac91a84543	[bbc] provide fallback URLs (#1706)	3 years ago
Mike Fährmann	a316e44f8e	[bbc] add 'width' option (#1706)	3 years ago
Mike Fährmann	c37c2818fb	[nsfwalbum] retry all requests when extracting image URLs ... (#1733, fixes #1271)	3 years ago
Mike Fährmann	220cfe244e	[deviantart] get original files for GIF previews (#1731)	3 years ago
Mike Fährmann	7a0da4f93f	[newgrounds] add 'format' option (closes #1729)	3 years ago
Mike Fährmann	223a4e79cd	[newgrounds] fix using 'category-tranfer' (#1274)	3 years ago
Mike Fährmann	4e95cef6d2	[nsfwalbum] retry backend requests (fixes #1733)	3 years ago
Mike Fährmann	6c11105587	[bbc] improve image dimensions (#1706) ... download the 1920xN versions instead of 976x549	3 years ago
Mike Fährmann	57c1a86082	[bbc] support multi-page gallery listings (closes #1730)	3 years ago
Mike Fährmann	2792ed6e4b	implement 'util.format_value()'	3 years ago
Mike Fährmann	552032d4fb	release version 1.18.2	3 years ago
Mike Fährmann	486474800f	[kemonoparty] skip duplicated patreon files (closes #1689) ... this behavior can be disabled with the 'patreon-skip-file' option	3 years ago
Mike Fährmann	da7297c0b9	[comicvine] add extractor (closes #1712)	3 years ago
Mike Fährmann	e4788fa663	[bbc] add 'gallery' and 'programme' extractors (closes #1706)	3 years ago
Mike Fährmann	13d4045a8a	add 'archive-prefix' option (#1711)	3 years ago
Mike Fährmann	c3b5c88b04	update extractor test results	3 years ago
Mike Fährmann	9e42cd58ea	replace ChainPredicate class with 'functools.partial'	3 years ago
Mike Fährmann	1b2f9050fb	rename all instances of 'kwds' to 'kwdict'	3 years ago
Mike Fährmann	3868ec02d1	[pururin] update domain and fix extraction	3 years ago
Mike Fährmann	b89a44090f	[naverwebtoon] fix comic extraction	3 years ago
Mike Fährmann	4048eabeab	allow 'extractor.modules' to be a string ... with comma-separated module names	3 years ago
Mike Fährmann	c8e678a5b4	[instagram] fix extraction of '/explore/tags/' posts ... (closes #1666)	3 years ago
Mike Fährmann	a6a51f207d	[moebooru] fix 'tags' ending with a '+' when logged in (#1702)	3 years ago
Mike Fährmann	f5b097165e	[ytdl] transfer YoutubeDL objects to downloader (#1680) ... allows specifying downloader-specific options per subcategory but overwrites all downloader.ytdl settings	3 years ago
Mike Fährmann	06e69ea79a	[ytdl] actually set options for YoutubeDL objects (#1680) ... I somehow managed to remove the options parameter for the YoutubeDL constructor in 9a849cdf without noticing ...	3 years ago
Mike Fährmann	fc19010808	[downloader:ytdl] fix 'outtmpl' setting for yt_dlp (#1680) ... yt_dlp supports multiple outtmpl settings for different file types and uses its 'outtmpl_dict' for that.	3 years ago
Mike Fährmann	dfe1f490e9	[mangadex] use custom User-Agent header (#1535)	3 years ago
Mike Fährmann	36a2aff363	[vk] improve metadata extraction and URL pattern (fixes #1691) ... - always fetch all user metadata - use 'user[name]' for directory names if available	3 years ago
Mike Fährmann	b9783403d9	add 'url-metadata' option (#1659, #1073)	3 years ago
Mike Fährmann	e622e004f0	[ytdl] improve module imports (#1680) ... Apply 'extractor.ytdl.module' for every URL, not just the first.	3 years ago
Mike Fährmann	e95f99882f	extend 'parent-metadata' functionality (#1687, #1651, #1364)	3 years ago
Mike Fährmann	193401ce3b	[ytdl] "fix" cookie transfer between session and ytdl (#1680) ... requests' CookieJar class is not quite compatible with the standard http.cookiejar.CookieJar used by youtube_dl	3 years ago
Mike Fährmann	9a849cdf61	[ytdl] allow setting 'module' for subcategories (#1680)	3 years ago
Mike Fährmann	dff0da60f9	[ytdl] add 'generic' option (#1680)	3 years ago
Mike Fährmann	d3da96142a	[ytdl] support cookies + username&password (#1680)	3 years ago
Mike Fährmann	36ac2197db	[ytdl] add extractor for sites supported by youtube-dl ... (#1680, #878) Can be used by prefixing any URL with 'ytdl:', or by setting 'extractor,ytdl.enabled' to 'true'.	3 years ago
Mike Fährmann	64240c8d42	[imagevenue] fix extraction ... (closes #1677)	3 years ago
Mike Fährmann	d287d2eb88	[kemonoparty] parse 'o' query parameters (#1674)	3 years ago
Mike Fährmann	8b036778e3	[kemonoparty] add 'max-posts' option (#1674)	3 years ago
Mike Fährmann	5612ca31c2	[hitomi] fix image URLs (closes #1679)	3 years ago
Mike Fährmann	8ecca3af58	[pixiv] add extractor for 'pixivision' articles (#1672)	3 years ago
Mike Fährmann	312a28e78a	[mastodon] add 'replies' option (#1669)	3 years ago
Mike Fährmann	513c491cea	[mastodon] reset 'params' after first pagination iteration ... otherwise query parameters in 'params' get specified twice the second time around - once from the 'links["next"]' URL and once from 'params' itself.	3 years ago
Mike Fährmann	a1f5b78039	[mastodon] add 'reblogs' option (#1669)	3 years ago
Mike Fährmann	317ecc8180	use HTML tables in docs/formatting.md	3 years ago
Mike Fährmann	5f1b13d1a5	release version 1.18.1	3 years ago
Mike Fährmann	21c2da454f	update extractor test results	3 years ago
Mike Fährmann	7f591c78cb	[mangafox] cleanup	3 years ago
FollieHiyuki	4763bc1e4e	Add MangaExtractor for mangafox (#1633)	3 years ago
Mike Fährmann	b519bf567c	[hiperdex] use domain from input URL	3 years ago
Mike Fährmann	93d356712c	[mastodon] implement 'text-posts' option (#1569) ... similar to Twitter's 'text-tweets'	3 years ago
Mike Fährmann	414bdc95a3	[twitter] set 'retweet_id' for original retweets (#1481)	3 years ago
Mike Fährmann	5323c1c73a	[twitter] ensure guest tokens are returned as string (#1665)	3 years ago
Mike Fährmann	9ee45f3617	[kemonoparty] warn about missing DDoS-GUARD cookies	3 years ago
Mike Fährmann	344aab3fb7	[seisoparty] warn about missing DDoS-GUARD cookies	3 years ago
Mike Fährmann	035562bd11	[twitter] remove old-style URLs from image fallback lists	3 years ago
Mike Fährmann	daf821b8b6	[seisoparty] use user names instead of IDs by default (#1635)	3 years ago
Mike Fährmann	e4db1bad14	[seisoparty] also extract files hosted on 'cdn-2' servers (#1635)	3 years ago
Mike Fährmann	267bbf5996	[mangasee] add 'chapter' and 'manga' extractors	3 years ago
Mike Fährmann	fad4918208	[deviantart] use UUIDs in internal folder/collection URLs	3 years ago
Mike Fährmann	64986f9435	fix depth counter in UrlJob ... regression from adf4d661 It would either stop at the first level (-g) or go infinitely deep (-G) Going down to for example level 3 with -ggg didn't work.	3 years ago
Mike Fährmann	0179581340	add 'T' format string conversion (#1646) ... to convert 'date'/datetime to timestamp	3 years ago
Mike Fährmann	f74cf52e2b	[seisoparty] add 'user' and 'post' extractors (#1635)	3 years ago
Mike Fährmann	759735fb02	[kemonoparty] fix 'username' extraction (fixes #1652) ... The site's <title> content changed from <title>NAME | Kemono</title> to <title> NAME | Kemono </title>	3 years ago
Mike Fährmann	befe635022	cache parsed Formatter functions	3 years ago
Mike Fährmann	a416e54765	[directlink] manually encode Referer URLs (fixes #1647) ... Trying to send a non-latin-1-encodable header raises an exception, so we encode the Referer value ourselves with 'errors=ignore'.	3 years ago
Mike Fährmann	8bdeb2a6dd	[webtoons] match arbitrary language codes (closes #1643)	3 years ago
Mike Fährmann	79b7ee2712	use 'functools.partial' in '_build_cleanfunc' when possible ... makes calls to the returned function a slight bit faster (~10%)	3 years ago
Mike Fährmann	ceaf7fd989	optimize 'base-directory' initialization and usage ... apply 'clean_path()' only once	3 years ago
Mike Fährmann	2ca011dfa8	add 'kwdict' argument to PathFormat.build_filename()	3 years ago
Mike Fährmann	fd00d47116	implement conditional directories (#1394) ... They work the same way as conditional filenames (84d2e640), e.g. "directory": { "score >= 20": ["high score"], "score >= 5" : ["mid score"], "" : ["{category}", "default"] }	3 years ago
Mike Fährmann	ee1064a2b2	release version 1.18.0	3 years ago
Mike Fährmann	4adc44df69	[furaffinity] improve metadata extraction (fixes #1630) ... Fetch 'title' and 'artist' metadata from a different location, since for posts with an empty title the <title> element is completely empty and does not contain the artist's name.	3 years ago
Mike Fährmann	e98fa01c44	[hitomi] update image URL code (fixes #1637)	3 years ago
Mike Fährmann	e9ab97396f	[kemonoparty] update default filenames and archive IDs (#1514) ... Add an enumeration index so that attachments and regular files with the same filename still get downloaded and not counted as duplicate files (even though for patreon posts they usually are) This invalidates all previously generated archive IDs. To keep using old names and IDs, set 'filename' to "{id}_{title}_{filename}.{extension}" and 'archive-format' to "{service}_{user}_{id}_{filename}.{extension}".	3 years ago
Mike Fährmann	fb4b4725ba	[hiperdex] match 'hiperdex2.com' URLs ... still doesn't properly work due to Cloudflare CAPTCHA and IUAM page	3 years ago
Mike Fährmann	95bc1139e0	[instagram] update query hashes	3 years ago
Mike Fährmann	23018a46f6	[instagram] fix login (fixes #1631)	3 years ago
Mike Fährmann	cac0110d8b	[redgifs] update API server address (fixes #1632) ... napi.redgifs.com -> api.redgifs.com	3 years ago
Mike Fährmann	0d2961ae81	[500px] remove last query hash entry ... forgot to include this in b56e2450	3 years ago
Mike Fährmann	7273cf8536	[pixiv] support fetching privately followed users (fixes #1628)	3 years ago
Mike Fährmann	e60962f7e5	[philomena] improve tag escapes handling (fixes #1629)	3 years ago
Mike Fährmann	d8908ca577	[unsplash] update collections URL pattern (fixes #1627)	3 years ago
Mike Fährmann	9ed13703cc	[sankaku] handle empty tags (fixes #1617)	3 years ago
Mike Fährmann	b56e245094	[500px] update GraphQL queries ... 500px changed its method from query hashes to sending the entire query string for every request.	3 years ago
Mike Fährmann	e9993451c3	remove Python version checks	3 years ago
Mike Fährmann	40eb8025cc	require an argument for --clear-cache	3 years ago
Mike Fährmann	a751afdfb3	[twitter] change some defaults ... - 'retweets' option: true -> false - 'quoted' option : true -> false i.e. disable downloading tweets from other user's timelines by default - search directory: '["{category}", "Search", "{search}"]' -> '["{category}", "{user[name]}"]' i.e. change it to the same as other twitter extractors (#1308)	3 years ago
Mike Fährmann	4e4ca3c330	[deviantart] pin API version (#1611) ... '/gallery/folders' in the newest version doesn't include subfolders. It probably only needs the right query parameter to do so, but that doesn't seem to be documented anywhere.	3 years ago
Mike Fährmann	d09bc5bd34	[subscribestar] improve attachment filenames (#1609)	3 years ago
Mike Fährmann	2986bf63bf	[mangafox] update URL pattern (fixes #1608) ... also accept non-numeric volume labels, e.g. vTBD	3 years ago
Mike Fährmann	53dab5c289	[mangadex] revert chapter handling (#1535) ... Spawn a new ChapterExtractor for each individual chapter instead of handling them directly with a MangaExtractor. Doing it that way broke too many features like --chapter-filter, --chapter-range, --zip, etc.	3 years ago
Mike Fährmann	def0148582	restructure code in PathFormat constructor	3 years ago
Mike Fährmann	84d2e64024	combine conditional filenames into `filename` option (#1394)	3 years ago
Mike Fährmann	1197ee2c20	[mangadex] add extractor for a user's followed feed (#1535)	3 years ago
Mike Fährmann	07c8adbd8b	[mangadex] implement login with username & password (#1535)	3 years ago
Mike Fährmann	b93cbe6720	[postprocessor:zip] run 'finalize()' regardless of 'filter' (#1460)	3 years ago
Mike Fährmann	3e332eaf53	[mangadex] update to API v5 (#1535)	3 years ago
Mike Fährmann	ff8d0d7280	[postprocessor:metadata] ensure reliable tags order (fixes #1598)	3 years ago
Mike Fährmann	04f4f9badb	[oauth] prevent exceptions when reporting errors (#1603)	3 years ago
Mike Fährmann	dfe1e09df0	add '-T/--terminate' command-line option (#1399)	3 years ago
Mike Fährmann	6ecb710a7f	add '-P/--postprocessor' command-line option	3 years ago
Mike Fährmann	83fc4c1098	update post processor config capabilities ... This change makes it possible to specify just the name of a post processor in the "postprocessors" list instead of a dict with all of its options. The options for it will then be taken from inside the "postprocessor" block similar to "extractor", "downloader", or "output" blocks. This makes it possible to for example override the default settings for --write-metadata by specifying a custom "metadata" block, or to set a custom post processor block ("cbz") and then use it by referencing just its name in "postprocessors" lists. { "postprocessor": { "metadata": { "name": "metadata", "event": "post", "filename": "{tweet_id|post_id|id}.json" }, "cbz": { "name" : "zip", "compression": "store", "extension" : "cbz" } } }	3 years ago
Mike Fährmann	4a747a31a3	[postprocessor:metadata] handle dicts in mode;tags (fixes #1598)	3 years ago
Mike Fährmann	a3bf878329	[idolcomplex] improve and fix pagination (#1601) ... always rely on the 'next-page-url' value and its query parameters	3 years ago
Mike Fährmann	3cbbefd4ed	support 'filter' option for post processors (#1460)	3 years ago
Mike Fährmann	4cf40434d7	initial support for conditional filenames (#1394)	3 years ago
Mike Fährmann	0abad8bc12	implement 'compile_expression()'	3 years ago
Mike Fährmann	e39c4633ba	[cyberdrop] b64decode -> a2b_base64	3 years ago
Mike Fährmann	407627ec86	[foolfuuka] support 'archive.wakarimasen.moe' (closes #1595)	3 years ago
Mike Fährmann	78f89d2e61	[idolcomplex] fix pagination (closes #1594)	3 years ago
Mike Fährmann	f14a36cfc9	release version 1.17.5	3 years ago
Mike Fährmann	52052a0e1a	[manganelo] update domain to 'manganato.com'	3 years ago
Mike Fährmann	c80b18a477	[weibo] extend 'retweets' option (closes #1542) ... Setting 'retweets' to "original" will use metadata from the original posts, and not from the retweeted ones.	3 years ago
Mike Fährmann	c0fa5058da	[kemonoparty] actually add a 'type' metadata field (#1556)	3 years ago
Mike Fährmann	bb50e85795	[postprocessor:ugoira] optimize writing ffconcat files ... collect all content in-memory first and write everything with a single 'write()'	3 years ago
Mike Fährmann	5c18db5fab	[postprocessor:ugoira] use 'concat' demuxer by default (#1550) ... 'image2' with nanasecond mtime timestamps doesn't work on Windows	3 years ago
Mike Fährmann	19a11fa11d	[postprocessor:ugoira] use frame extension from filenames (#1550)	3 years ago
Mike Fährmann	74d97e6c6b	[postprocessor:ugoira] add 'repeat-last-frame' option (#1550)	3 years ago
Mike Fährmann	be9547a5f3	[postprocessor:ugoira] use 'image2' demuxer (#1550) ... Ensures exact frame timecodes with no duplicate frames. Possible issues are the duration the last frame in an Ugoira with variable frame durations is shown and insufficient timestamp precision of the underlying file system (e.g. FAT32, ext3; works on ext4, tmpfs, NTFS).	3 years ago
thatfuckingbird	264beb8556	recognize v2.mangapark URLs (#1578) ... * recognize v2.mangapark URLs * update mangapark root url to use the v2 subdomain	3 years ago
thatfuckingbird	e6811c7450	[pixiv] implement 'max-posts' option (#1558) ... * implement max-rank for pixiv * rename to max-posts and make more generic	3 years ago
Mike Fährmann	8a909e478d	[imagebam] fix extraction of NSFW images (#1534)	3 years ago
Mike Fährmann	b5affc62aa	[twitter] rename 'text-only' to 'text-tweets' (#570)	3 years ago
Mike Fährmann	724ca61f36	[twitter] add 'text-only' option (#570)	3 years ago
Mike Fährmann	8fd8126117	fix ISO 639-1 code for Japanese ... "jp" -> "ja"	3 years ago
Mike Fährmann	2c60c7d798	[reactor] skip deleted/empty posts	3 years ago
Mike Fährmann	532ac79fb0	update extractor test results	3 years ago
Mike Fährmann	d7bc4a2b8b	[500px] update query hashes	3 years ago
Mike Fährmann	0f35aca728	[aryion] minor code updates	3 years ago
Mike Fährmann	2eb46452ad	[aryion] update 'needle' to not skip text posts (fixes #1568) ... on "Latest Updates" pages "class='thumb scrollthumb' href='/g4/view/" and "class='thumb' href='/g4/view/" both end with "thumb' href='/g4/view/"	3 years ago
Mike Fährmann	adf4d661b3	use '_extractor' info in UrlJobs	3 years ago
Mike Fährmann	4fc9668922	[imgur] update URL patterns (#1561)	3 years ago
Mike Fährmann	1eabfa5c7a	[pillowfort] implement login with username & password (#846)	3 years ago
Mike Fährmann	24dd10ac3c	[patreon] extract user defined 'tags' (#1539, closes #1540)	3 years ago
Mike Fährmann	a7e4917ee1	[pillowfort] add 'inline' option (#846) ... to support images present in a post's 'content', but not listed in 'media'. also separates the file hash present at the beginning of each 'filename' into its own field.	3 years ago
Mike Fährmann	efa6cc8ec3	[pillowfort] add 'external' option (#846) ... for links to external Twitter posts etc.	3 years ago
Mike Fährmann	394fbb5f56	[twitter] strip useless t.co links (#1532) ... The 'full_text' of Tweets with media content usually ends with a t.co link to itself. This commit removes those.	3 years ago
Mike Fährmann	3a7c3ff138	support XDG_CONFIG_HOME (closes #1545) ... This will only load either ${XDG_CONFIG_HOME}/gallery-dl/config.json or ${HOME}/.config/gallery-dl/config.json if XDG_CONFIG_HOME is empty, never both.	3 years ago
Mike Fährmann	41457dbb1b	[twitter] resolve t.co URLs in 'content' (#1532)	3 years ago
Mike Fährmann	2b5d80862e	[kemonoparty] add 'type' metadata field (#1556) ... 'file', 'attachment', or 'inline'	3 years ago
Mike Fährmann	17b0ccb071	[twitter] add missing retweet media entities (fixes #1555) ... from the original tweets	3 years ago