1054 Commits (4ffa94f634cbdd6d566defb2bcaf97b418e08c57)

Author	SHA1	Message	Date
Mike Fährmann	f9884e2338	[pixiv] update URL pattern ... add support for 'https://www.pixiv.net/user/<id>'	7 years ago
Mike Fährmann	85ed023c2e	[mangadex] remove the trailing ' - MangaDex' in a better way ... str.rstrip() works differently than assumed.	7 years ago
Mike Fährmann	9fb82e6b43	apply expand_path() to archive paths	7 years ago
Mike Fährmann	32bbd12f08	update extractor tests	7 years ago
Mike Fährmann	ca326bd275	[deviantart] fix folder and collection archive IDs ... {folder[index]} and {collection[index]} are both '0' when being delegated from Gallery- or FavoriteExtractors, as there is no way of knowing a folder's index when getting folder-information from the API.	7 years ago
Mike Fährmann	e32fe1cdf1	[pinterest] cast IDs to int ... ... and update test results. Image URLs changed from https://s-media-cache-ak0.pinimg.com/... to https://i.pinimg.com/...	7 years ago
Mike Fährmann	179ecee965	[turboimagehost] fix extraction	7 years ago
Mike Fährmann	1400868f53	[mangadex] general improvements ... - support >100 chapter entries per manga - custom archive ID format - detect non-existing chapters	7 years ago
Mike Fährmann	749fbbfa6c	[mangadex] add chapter- and manga-extractor	7 years ago
Mike Fährmann	b58449fd88	release version 1.3.0	7 years ago
Mike Fährmann	6e38cf5aab	[mangareader] use 'https://' ... The site now redirects from http://mangareader.net/ to https://mangareader.net/	7 years ago
Mike Fährmann	1d71123f91	[pixiv] update archive IDs and add metadata-fields ... (Pixiv bookmarks actually have their own IDs, comments and tags, independent of the bookmarked image, which makes creating an archive ID a lot easier)	7 years ago
Mike Fährmann	858fdbdb22	[tumblr] improve 'inline' extraction ... 'quote' posts store their HTML content in the 'source' field	7 years ago
Mike Fährmann	1d54a8e07d	fix logging output during downloads ... from: filename.ext[download][warning] ... to: filename.ext [download][warning] ...	7 years ago
Mike Fährmann	5008e105ee	update archive IDs ... ... to behave in a more straightforward way when dealing with bookmarks/favourites/etc. specific IDs are now grouped by their owner, album-id, ... to allow for duplicates when it would be expected.	7 years ago
Mike Fährmann	829ddf4ac1	[sankaku] general improvements ... - simplify regex - unquote search tags - increase default wait-time between HTTP requests - downloading several hundreds of images always resulted in '429 Too Many Requests' eventually - circumvent paging restrictions for unauthenticated users by only using the 'next' parameter - setting 'page' to a constant, low value (or simply omitting it) does the trick	7 years ago
Jad	49463f76bb	support multi-page URL (#79) ... * support multi-page URL * fix * all done. * fix, again	7 years ago
Mike Fährmann	19aefdfde3	[directlink] update test results	7 years ago
Mike Fährmann	74029c50bb	[directlink] unquote metadata fields	7 years ago
Mike Fährmann	2fad0b1f1b	add 'U' conversion for format strings to unquote their content ... (#74)	7 years ago
Mike Fährmann	8cdce21dcb	make archive keys user-configurable	7 years ago
Mike Fährmann	8f338347b6	[imagehosts] cleanup ... removed - chronos.to - unable to resolve hostname - coreimg.net - same - imgmaid.net - same - hosturimage.com - everything returns 404 - imageontime.org - redirects to some shady site - imgupload.yt - cloudflare error 522, host down - img4ever.net - read timeout	7 years ago
Mike Fährmann	edfd3d9fc9	[yeet] remove module ... - archive.yeet.net returns a 500 server error - yeet.net moved to yeet.rip, but the archive is gone	7 years ago
Mike Fährmann	e1e0668ca8	add option to set default replacement field value ... Missing or undefined keywords will now be replaced with the value set for 'keywords-default'. The default is Python's 'None', which is equivalent to setting this option to JSON's 'null'.	7 years ago
Mike Fährmann	ac3da8115e	[util] don't add text: URLs to list of downloaded URLs	7 years ago
Mike Fährmann	8704d850bf	add explicit proxy support (#76) ... - '--proxy' as command-line argument - 'extractor.*.proxy' as config option	7 years ago
Mike Fährmann	367b963d37	[pixiv] fix ugoira extraction ... again (#78) ... Some animations are not available for mobile devices, so we pretend to be a desktop browser when requesting the ugoira page.	7 years ago
Mike Fährmann	b79f1f2ca7	[pixiv] fix ugoira extraction (closes #78)	7 years ago
Mike Fährmann	731ffd4986	improve text.filename_from_url() performance ... - urlsplit() is faster than urlparse() - rpartition() is faster than rindex() + slicing - new version is 2.3 times as fast	7 years ago
Mike Fährmann	d122203be1	[mangastream] fix extraction	7 years ago
Mike Fährmann	8809b32aed	release version 1.2.0	7 years ago
Mike Fährmann	b50bdbf3d7	change config specifiers in input file format ... Instead of a dictionary/object, input file options are now specified by a 'key=value' pair starting with '-' for options only applying to the next URL or '-G' for Global options applying to all following URLs. See the docstring of parse_inputfile() for details. Example option specifiers: - filename = "{id}.{extension}" - extractor.pixiv.user.directory = ["Pixiv Users", "{user[id]}"] -spaces="are_optional" -G keywords = {"global": "option"}	7 years ago
Mike Fährmann	f970a8f13c	fix adding keys to download archive when using skip=false	7 years ago
Mike Fährmann	179bcdd349	adjust archive-ids	7 years ago
Mike Fährmann	be3ea4425d	test archive-id creation and uniqueness	7 years ago
Mike Fährmann	3cec533c28	Merge branch 'archive'	7 years ago
Mike Fährmann	20af86b2ea	add more extractor tests ... for mangastream, reddit and imgur	7 years ago
Mike Fährmann	b73b8b4f50	add OAuth unittests	7 years ago
Mike Fährmann	4d2fadfb6f	restore skip actions with download archive	7 years ago
Mike Fährmann	65773263fc	[util] implement OAuthSession.urlencode() (closes #75) ... - Python's own urllib.parse.urlencode() has no quote_via argument in Python 3.3 and 3.4, which is necessary to follow OAuth 1.0 quoting rules.	7 years ago
Mike Fährmann	7e0207bcf4	[imgur] strip trailing '?1' from 'ext'	7 years ago
Mike Fährmann	cf147dfee9	[hentai2read] fix manga extraction ... - site changed its HTML structure	7 years ago
Mike Fährmann	f5f2d29f56	[nijie] fix dojin extraction ... - correctly extract artist_id - set extension to "jpg" if it was empty and let filetype checks do the rest	7 years ago
Mike Fährmann	7f7c16ae37	add option to specify additional key-value pairs	7 years ago
Mike Fährmann	d38bf2f54c	[tumblr] recognize /image/... URLs ... xyz.tumblr.com/image/123 refers to the same images as xyz.tumblr.com/post/123.	7 years ago
Mike Fährmann	057668e17e	extend input-file format with per-URL config and comments ... - see docstring of parse_inputfile() for details - TODO: unittests, recursion (currently setting for example {"extractor": {"key": "value"}} will override the whole "extractor" branch instead of merging {"key": "value"} into the already existing dictionary)	7 years ago
Mike Fährmann	5b3c34aa96	use generic chapter-extractor in more modules	7 years ago
Mike Fährmann	347baf7ac5	improve util.parse_range() performance ... It is never going to actually matter, but using partition() instead of split() is twice as fast.	7 years ago
Mike Fährmann	7b5ba69951	[hentaihere] ensure consistent extraction results ... sometimes there is a random space before the next <a>	7 years ago
Mike Fährmann	377b78b3c9	[hentai2read] fix manga name extraction	7 years ago
Mike Fährmann	54c36a8a34	[subapics] add chapter- and manga-extractor (#70)	7 years ago
Mike Fährmann	2dd3aeeeae	[komikcast] add chapter- and manga-extractor (#70)	7 years ago
Mike Fährmann	7a412f5c32	implement generic manga-chapter extractor	7 years ago
Mike Fährmann	aa38eab2be	allow not-defined fields in format strings ... ... and replace them with "None", for now	7 years ago
Mike Fährmann	6a07e38366	implement extractor.add() and .add_module() ... ... as a public and non-hacky way to add (external) extractors to gallery-dl's pool and make them available for extractor.find()	7 years ago
Mike Fährmann	c0dd922c13	add '--download-archive' cmdline option ... … as well as a config file equivalent	7 years ago
Mike Fährmann	8c3b713362	rework DownloadJob.handle_url(); include archive functionality ... todo: "abort" and "exit" skip modes if download is skipped because of archive	7 years ago
Mike Fährmann	34873dbd90	set 'archive_fmt' values ... These are going to be used to create an unique id for each image.	7 years ago
Mike Fährmann	a34cebc253	[luscious] jump to first image if cover does not link to it	7 years ago
Mike Fährmann	84a52a9256	add DownloadArchive class	7 years ago
Mike Fährmann	915807dd77	log HTTP errors as warnings	7 years ago
Mike Fährmann	db7f04dd97	emit log messages on download failure ... and when retrying with fallback URLs	7 years ago
Mike Fährmann	d951f13e37	add config option for unsupported-URL file ... for consistency's sake	7 years ago
Mike Fährmann	619387cbb1	update extractor unittest results	7 years ago
Mike Fährmann	364e335440	smaller adjustments and improvements ... - requests and urllib3 version on 1 line - close input file after reading from it - use expand_path for unsupported-urls file - remove unnecessary logging from options.py	7 years ago
Mike Fährmann	c9a9664a65	change --write-log behaviour ... - log files now get truncated when opening them (mode "w" instead of "a") - log verbosity to file depends on -q/-v (same as logging to stderr)	7 years ago
Mike Fährmann	97f4f15ec0	add option to write logging output to a file ... - '--write-log FILE' as cmdline argument - 'output.logfile' as config file option	7 years ago
Mike Fährmann	f94e3706a8	use logging module for error messages during downloads	7 years ago
Mike Fährmann	db91cf871c	document message identifiers	7 years ago
Mike Fährmann	0dd48d644f	update test results ... nothing broke, but things got updated or changed	7 years ago
Mike Fährmann	1e93955170	[batoto] remove module ... Site officially shut down on 2018.01.18	7 years ago
Mike Fährmann	27fce6f600	fix UrlJob behavior	7 years ago
Mike Fährmann	76509a6d3c	[imgur] update test results	7 years ago
Mike Fährmann	9fccd7b783	[tumblr] provide fallback URLs (#64) ... Each image now produces 3 URLs: - amazonaws.com _raw (or _1280 for older images) - amazonaws.com _500 - media.tumblr.com (URL returned by API)	7 years ago
Mike Fährmann	b837420291	fix minor urllist issues	7 years ago
Mike Fährmann	9d69401391	initial support for multiple URLs per image	7 years ago
Mike Fährmann	6174a5c4ef	[download] adjust filename extension on filetype mismatch ... (closes #63)	7 years ago
Mike Fährmann	91ed147cef	[oauth] use custom key/secret values during oauth:…	7 years ago
Mike Fährmann	421a9740a3	[tumblr] add 'tumblr:' to force Tumblr extractor (#71)	7 years ago
Mike Fährmann	40d35c87bc	[paheal] add tag- and post-extractors (closes #69)	7 years ago
Mike Fährmann	cc0c2cca57	[reddit] add extractor for reddit-hosted images (closes #68)	7 years ago
Mike Fährmann	f10ffc0839	update extractor blacklist to also allow classes	7 years ago
Mike Fährmann	b6797032e3	release version 1.1.2	7 years ago
Mike Fährmann	35e09869d1	[mangapark] fix image URLs and use HTTPS	7 years ago
Mike Fährmann	9a049bdf51	[tumblr] add 'likes' extractor (#65)	7 years ago
Mike Fährmann	67d4462d26	[batoto] rudimentary Cloudflare bypass	7 years ago
Mike Fährmann	29d75fc3fa	[tumblr] add support for OAuth authentication (#65)	7 years ago
Mike Fährmann	4edb25346e	[slideshare] support mobile URLs (closes #67)	7 years ago
Mike Fährmann	e420a28bbc	fix cookie tests	7 years ago
Mike Fährmann	b33efc99a4	[idolcomplex] add support for idol.sankakucomplex.com	7 years ago
Mike Fährmann	75b2e84b6d	[tumblr] use s3.amazonaws.com for image URLs (#64)	7 years ago
Mike Fährmann	5b094328b5	[puremashiro] add chapter- and manga-extractor (closes #66) ... Also adds support for region subtags in language codes (e.g. en-us)	7 years ago
Mike Fährmann	974e73bdbb	[booru] smaller code adjustments	7 years ago
Mike Fährmann	03b8a548cb	[tumblr] change `reblogs` default value to `true` (#61)	7 years ago
Mike Fährmann	d235f68f59	[tumblr] add option to filter reblogged posts (#61) ... Reblogs are ignored by default, but can be included by setting 'extractor.tumblr.reblogs' to 'true'.	7 years ago
Mike Fährmann	a794fffc6d	[batoto] extend chapter-string regex (closes #60) ... Non-numeric chapter indices exist after all ...	7 years ago
Mike Fährmann	1219ebb7f5	[danbooru] use alternate subdomains; support safebooru	7 years ago
Mike Fährmann	9e8a84ab6c	[booru] rewrite using Mixin classes (#59) ... - improved code structure - improved URL patterns - better pagination to work around page limits on - Danbooru - e621 - 3dbooru	7 years ago
Mike Fährmann	0876541e43	[seiga] update tests	7 years ago
Mike Fährmann	1a70857a12	update extractor-unittest capabilities ... - "count" can now be a string defining a comparison in the form of '<operator> <value>', for example: '> 12' or '!= 1'. If its value is not a string, it is assumed to be a concrete integer as before. - "keyword" can now be a dictionary defining tests for individual keys. These tests can either be a type, a concrete value or a regex starting with "re:". Dictionaries can be stacked inside each other. Optional keys can be indicated with a "?" before its name. For example: "keyword:" { "image_id": int, "gallery_id", 123, "name": "re:pattern", "user": { "id": 321, }, "?optional": None, }	7 years ago