1103 Commits (7f899bd5d8b3fab882cc454dd659738e2f38670b)

Author	SHA1	Message	Date
Mike Fährmann	85ed023c2e	[mangadex] remove the trailing ' - MangaDex' in a better way ... str.rstrip() works differently than assumed.	7 years ago
Mike Fährmann	9fb82e6b43	apply expand_path() to archive paths	7 years ago
Mike Fährmann	32bbd12f08	update extractor tests	7 years ago
Mike Fährmann	ca326bd275	[deviantart] fix folder and collection archive IDs ... {folder[index]} and {collection[index]} are both '0' when being delegated from Gallery- or FavoriteExtractors, as there is no way of knowing a folder's index when getting folder-information from the API.	7 years ago
Mike Fährmann	e32fe1cdf1	[pinterest] cast IDs to int ... ... and update test results. Image URLs changed from https://s-media-cache-ak0.pinimg.com/... to https://i.pinimg.com/...	7 years ago
Mike Fährmann	179ecee965	[turboimagehost] fix extraction	7 years ago
Mike Fährmann	1400868f53	[mangadex] general improvements ... - support >100 chapter entries per manga - custom archive ID format - detect non-existing chapters	7 years ago
Mike Fährmann	749fbbfa6c	[mangadex] add chapter- and manga-extractor	7 years ago
Mike Fährmann	b58449fd88	release version 1.3.0	7 years ago
Mike Fährmann	6e38cf5aab	[mangareader] use 'https://' ... The site now redirects from http://mangareader.net/ to https://mangareader.net/	7 years ago
Mike Fährmann	1d71123f91	[pixiv] update archive IDs and add metadata-fields ... (Pixiv bookmarks actually have their own IDs, comments and tags, independent of the bookmarked image, which makes creating an archive ID a lot easier)	7 years ago
Mike Fährmann	858fdbdb22	[tumblr] improve 'inline' extraction ... 'quote' posts store their HTML content in the 'source' field	7 years ago
Mike Fährmann	1d54a8e07d	fix logging output during downloads ... from: filename.ext[download][warning] ... to: filename.ext [download][warning] ...	7 years ago
Mike Fährmann	5008e105ee	update archive IDs ... ... to behave in a more straightforward way when dealing with bookmarks/favourites/etc. specific IDs are now grouped by their owner, album-id, ... to allow for duplicates when it would be expected.	7 years ago
Mike Fährmann	829ddf4ac1	[sankaku] general improvements ... - simplify regex - unquote search tags - increase default wait-time between HTTP requests - downloading several hundreds of images always resulted in '429 Too Many Requests' eventually - circumvent paging restrictions for unauthenticated users by only using the 'next' parameter - setting 'page' to a constant, low value (or simply omitting it) does the trick	7 years ago
Jad	49463f76bb	support multi-page URL (#79) ... * support multi-page URL * fix * all done. * fix, again	7 years ago
Mike Fährmann	19aefdfde3	[directlink] update test results	7 years ago
Mike Fährmann	74029c50bb	[directlink] unquote metadata fields	7 years ago
Mike Fährmann	2fad0b1f1b	add 'U' conversion for format strings to unquote their content ... (#74)	7 years ago
Mike Fährmann	8cdce21dcb	make archive keys user-configurable	7 years ago
Mike Fährmann	8f338347b6	[imagehosts] cleanup ... removed - chronos.to - unable to resolve hostname - coreimg.net - same - imgmaid.net - same - hosturimage.com - everything returns 404 - imageontime.org - redirects to some shady site - imgupload.yt - cloudflare error 522, host down - img4ever.net - read timeout	7 years ago
Mike Fährmann	edfd3d9fc9	[yeet] remove module ... - archive.yeet.net returns a 500 server error - yeet.net moved to yeet.rip, but the archive is gone	7 years ago
Mike Fährmann	e1e0668ca8	add option to set default replacement field value ... Missing or undefined keywords will now be replaced with the value set for 'keywords-default'. The default is Python's 'None', which is equivalent to setting this option to JSON's 'null'.	7 years ago
Mike Fährmann	ac3da8115e	[util] don't add text: URLs to list of downloaded URLs	7 years ago
Mike Fährmann	8704d850bf	add explicit proxy support (#76) ... - '--proxy' as command-line argument - 'extractor.*.proxy' as config option	7 years ago
Mike Fährmann	367b963d37	[pixiv] fix ugoira extraction ... again (#78) ... Some animations are not available for mobile devices, so we pretend to be a desktop browser when requesting the ugoira page.	7 years ago
Mike Fährmann	b79f1f2ca7	[pixiv] fix ugoira extraction (closes #78)	7 years ago
Mike Fährmann	731ffd4986	improve text.filename_from_url() performance ... - urlsplit() is faster than urlparse() - rpartition() is faster than rindex() + slicing - new version is 2.3 times as fast	7 years ago
Mike Fährmann	d122203be1	[mangastream] fix extraction	7 years ago
Mike Fährmann	8809b32aed	release version 1.2.0	7 years ago
Mike Fährmann	b50bdbf3d7	change config specifiers in input file format ... Instead of a dictionary/object, input file options are now specified by a 'key=value' pair starting with '-' for options only applying to the next URL or '-G' for Global options applying to all following URLs. See the docstring of parse_inputfile() for details. Example option specifiers: - filename = "{id}.{extension}" - extractor.pixiv.user.directory = ["Pixiv Users", "{user[id]}"] -spaces="are_optional" -G keywords = {"global": "option"}	7 years ago
Mike Fährmann	f970a8f13c	fix adding keys to download archive when using skip=false	7 years ago
Mike Fährmann	179bcdd349	adjust archive-ids	7 years ago
Mike Fährmann	be3ea4425d	test archive-id creation and uniqueness	7 years ago
Mike Fährmann	3cec533c28	Merge branch 'archive'	7 years ago
Mike Fährmann	20af86b2ea	add more extractor tests ... for mangastream, reddit and imgur	7 years ago
Mike Fährmann	b73b8b4f50	add OAuth unittests	7 years ago
Mike Fährmann	4d2fadfb6f	restore skip actions with download archive	7 years ago
Mike Fährmann	65773263fc	[util] implement OAuthSession.urlencode() (closes #75) ... - Python's own urllib.parse.urlencode() has no quote_via argument in Python 3.3 and 3.4, which is necessary to follow OAuth 1.0 quoting rules.	7 years ago
Mike Fährmann	7e0207bcf4	[imgur] strip trailing '?1' from 'ext'	7 years ago
Mike Fährmann	cf147dfee9	[hentai2read] fix manga extraction ... - site changed its HTML structure	7 years ago
Mike Fährmann	f5f2d29f56	[nijie] fix dojin extraction ... - correctly extract artist_id - set extension to "jpg" if it was empty and let filetype checks do the rest	7 years ago
Mike Fährmann	7f7c16ae37	add option to specify additional key-value pairs	7 years ago
Mike Fährmann	d38bf2f54c	[tumblr] recognize /image/... URLs ... xyz.tumblr.com/image/123 refers to the same images as xyz.tumblr.com/post/123.	7 years ago
Mike Fährmann	057668e17e	extend input-file format with per-URL config and comments ... - see docstring of parse_inputfile() for details - TODO: unittests, recursion (currently setting for example {"extractor": {"key": "value"}} will override the whole "extractor" branch instead of merging {"key": "value"} into the already existing dictionary)	7 years ago
Mike Fährmann	5b3c34aa96	use generic chapter-extractor in more modules	7 years ago
Mike Fährmann	347baf7ac5	improve util.parse_range() performance ... It is never going to actually matter, but using partition() instead of split() is twice as fast.	7 years ago
Mike Fährmann	7b5ba69951	[hentaihere] ensure consistent extraction results ... sometimes there is a random space before the next <a>	7 years ago
Mike Fährmann	377b78b3c9	[hentai2read] fix manga name extraction	7 years ago
Mike Fährmann	54c36a8a34	[subapics] add chapter- and manga-extractor (#70)	7 years ago
Mike Fährmann	2dd3aeeeae	[komikcast] add chapter- and manga-extractor (#70)	7 years ago
Mike Fährmann	7a412f5c32	implement generic manga-chapter extractor	7 years ago
Mike Fährmann	aa38eab2be	allow not-defined fields in format strings ... ... and replace them with "None", for now	7 years ago
Mike Fährmann	6a07e38366	implement extractor.add() and .add_module() ... ... as a public and non-hacky way to add (external) extractors to gallery-dl's pool and make them available for extractor.find()	7 years ago
Mike Fährmann	c0dd922c13	add '--download-archive' cmdline option ... … as well as a config file equivalent	7 years ago
Mike Fährmann	8c3b713362	rework DownloadJob.handle_url(); include archive functionality ... todo: "abort" and "exit" skip modes if download is skipped because of archive	7 years ago
Mike Fährmann	34873dbd90	set 'archive_fmt' values ... These are going to be used to create an unique id for each image.	7 years ago
Mike Fährmann	a34cebc253	[luscious] jump to first image if cover does not link to it	7 years ago
Mike Fährmann	84a52a9256	add DownloadArchive class	7 years ago
Mike Fährmann	915807dd77	log HTTP errors as warnings	7 years ago
Mike Fährmann	db7f04dd97	emit log messages on download failure ... and when retrying with fallback URLs	7 years ago
Mike Fährmann	d951f13e37	add config option for unsupported-URL file ... for consistency's sake	7 years ago
Mike Fährmann	619387cbb1	update extractor unittest results	7 years ago
Mike Fährmann	364e335440	smaller adjustments and improvements ... - requests and urllib3 version on 1 line - close input file after reading from it - use expand_path for unsupported-urls file - remove unnecessary logging from options.py	7 years ago
Mike Fährmann	c9a9664a65	change --write-log behaviour ... - log files now get truncated when opening them (mode "w" instead of "a") - log verbosity to file depends on -q/-v (same as logging to stderr)	7 years ago
Mike Fährmann	97f4f15ec0	add option to write logging output to a file ... - '--write-log FILE' as cmdline argument - 'output.logfile' as config file option	7 years ago
Mike Fährmann	f94e3706a8	use logging module for error messages during downloads	7 years ago
Mike Fährmann	db91cf871c	document message identifiers	7 years ago
Mike Fährmann	0dd48d644f	update test results ... nothing broke, but things got updated or changed	7 years ago
Mike Fährmann	1e93955170	[batoto] remove module ... Site officially shut down on 2018.01.18	7 years ago
Mike Fährmann	27fce6f600	fix UrlJob behavior	7 years ago
Mike Fährmann	76509a6d3c	[imgur] update test results	7 years ago
Mike Fährmann	9fccd7b783	[tumblr] provide fallback URLs (#64) ... Each image now produces 3 URLs: - amazonaws.com _raw (or _1280 for older images) - amazonaws.com _500 - media.tumblr.com (URL returned by API)	7 years ago
Mike Fährmann	b837420291	fix minor urllist issues	7 years ago
Mike Fährmann	9d69401391	initial support for multiple URLs per image	7 years ago
Mike Fährmann	6174a5c4ef	[download] adjust filename extension on filetype mismatch ... (closes #63)	7 years ago
Mike Fährmann	91ed147cef	[oauth] use custom key/secret values during oauth:…	7 years ago
Mike Fährmann	421a9740a3	[tumblr] add 'tumblr:' to force Tumblr extractor (#71)	7 years ago
Mike Fährmann	40d35c87bc	[paheal] add tag- and post-extractors (closes #69)	7 years ago
Mike Fährmann	cc0c2cca57	[reddit] add extractor for reddit-hosted images (closes #68)	7 years ago
Mike Fährmann	f10ffc0839	update extractor blacklist to also allow classes	7 years ago
Mike Fährmann	b6797032e3	release version 1.1.2	7 years ago
Mike Fährmann	35e09869d1	[mangapark] fix image URLs and use HTTPS	7 years ago
Mike Fährmann	9a049bdf51	[tumblr] add 'likes' extractor (#65)	7 years ago
Mike Fährmann	67d4462d26	[batoto] rudimentary Cloudflare bypass	7 years ago
Mike Fährmann	29d75fc3fa	[tumblr] add support for OAuth authentication (#65)	7 years ago
Mike Fährmann	4edb25346e	[slideshare] support mobile URLs (closes #67)	7 years ago
Mike Fährmann	e420a28bbc	fix cookie tests	7 years ago
Mike Fährmann	b33efc99a4	[idolcomplex] add support for idol.sankakucomplex.com	7 years ago
Mike Fährmann	75b2e84b6d	[tumblr] use s3.amazonaws.com for image URLs (#64)	7 years ago
Mike Fährmann	5b094328b5	[puremashiro] add chapter- and manga-extractor (closes #66) ... Also adds support for region subtags in language codes (e.g. en-us)	7 years ago
Mike Fährmann	974e73bdbb	[booru] smaller code adjustments	7 years ago
Mike Fährmann	03b8a548cb	[tumblr] change `reblogs` default value to `true` (#61)	7 years ago
Mike Fährmann	d235f68f59	[tumblr] add option to filter reblogged posts (#61) ... Reblogs are ignored by default, but can be included by setting 'extractor.tumblr.reblogs' to 'true'.	7 years ago
Mike Fährmann	a794fffc6d	[batoto] extend chapter-string regex (closes #60) ... Non-numeric chapter indices exist after all ...	7 years ago
Mike Fährmann	1219ebb7f5	[danbooru] use alternate subdomains; support safebooru	7 years ago
Mike Fährmann	9e8a84ab6c	[booru] rewrite using Mixin classes (#59) ... - improved code structure - improved URL patterns - better pagination to work around page limits on - Danbooru - e621 - 3dbooru	7 years ago
Mike Fährmann	0876541e43	[seiga] update tests	7 years ago
Mike Fährmann	1a70857a12	update extractor-unittest capabilities ... - "count" can now be a string defining a comparison in the form of '<operator> <value>', for example: '> 12' or '!= 1'. If its value is not a string, it is assumed to be a concrete integer as before. - "keyword" can now be a dictionary defining tests for individual keys. These tests can either be a type, a concrete value or a regex starting with "re:". Dictionaries can be stacked inside each other. Optional keys can be indicated with a "?" before its name. For example: "keyword:" { "image_id": int, "gallery_id", 123, "name": "re:pattern", "user": { "id": 321, }, "?optional": None, }	7 years ago
Mike Fährmann	88bb0798fd	delay initialization of PathFormat objects ... This allows the DeviantArt group-check to be moved inside the Extractor.items() method which in turn allows for better exception handling. As a new general rule: Never raise exceptions during extractor initialization.	7 years ago
Mike Fährmann	c24e0e70a7	[pixiv] simplify main loop	7 years ago
Mike Fährmann	c1e331edbb	[mangapark] replace manga test	7 years ago
Mike Fährmann	5488643fac	add requests and urllib3 versions to debug output	7 years ago
Mike Fährmann	9d73ed4772	fix issue with using 'skip()' when a filter is present ... calling skip() skips over unfiltered items and does not apply the filter expression to them, which is not what should happen	7 years ago
Mike Fährmann	28cd78aae0	[kissmanga] extend chapter-string regex (closes #58)	7 years ago
Mike Fährmann	0ba618dd1a	release version 1.1.1	7 years ago
Mike Fährmann	a3e9b51bea	[imgbox] update test results ... Image URLs of older galleries have been updated to the new format. https://i.imgbox.com/qHhw7lpG.png --> https://images3.imgbox.com/6d/9a/qHhw7lpG_o.png	7 years ago
Mike Fährmann	d241a0fb60	[util] replace '/' with '\' in base-directory paths ... ... on Windows to have consistent path separators.	7 years ago
Mike Fährmann	d0886f411e	[gelbooru] re-enable API use (closes #56) ... Gelbooru's API allows access to all images and is not restricted to the first 20000. This also adds an option to select between API use and manual information extraction in case their API gets disabled again.	7 years ago
Mike Fährmann	8102aae311	[mangahere] support ".cc" TLD and mobile URLs	7 years ago
Mike Fährmann	676602056c	[reddit] unescape output URLs	7 years ago
Mike Fährmann	2eedbaaaf9	[deviantart] use cache to store new refresh_tokens ... The 'refresh_token' set in a user's config file gets used once to get a new 'access_token' and 'refresh_token', which is then stored in gallery-dl's cache and gets used the next time the 'access_token' needs to be refreshed. This means deleting the cache file invalidates the refresh_token- chain and requires the user to re-authenticate.	7 years ago
Mike Fährmann	fc7d165c97	[deviantart] add support for OAuth2 authentication ... Some user galleries [*] require you to be either logged in or authenticated via OAuth2 to access their deviations. [*] e.g. https://polinaegorussia.deviantart.com/gallery/ -------------- known issue: A deviantart 'refresh_token' can only be used once and gets updated whenever it is used to request a new 'access_token', so storing its initial value in a config file and reusing it again and again is not possible.	7 years ago
Mike Fährmann	91c2aed077	[nhentai] fix JSON extraction	7 years ago
Mike Fährmann	444008a14a	[khinsider] use urljoin() to complete page URLs	7 years ago
Mike Fährmann	263741d243	[luscious] update URL pattern (closes #55)	7 years ago
Mike Fährmann	0a9a07a6e1	[slideshare] improve metadata; flake8 ... - added 'views' and 'published' keywords - fixed longer titles and descriptions	7 years ago
Leonardo Taccari	a8d2dde8b2	[slideshare] Add a new extractor for slideshare.net (#54)	7 years ago
Mike Fährmann	19a6ae57b2	[sankaku] add pool extractor	7 years ago
Mike Fährmann	e52f0cc1ed	[sankaku] add post extractor	7 years ago
Mike Fährmann	595593a35e	[sankaku] rewrite ... - better code structure and extensibility - better metadata	7 years ago
Mike Fährmann	e96e1fea5d	release version 1.1.0	7 years ago
Mike Fährmann	a3924d2072	[sankaku] fix swf extraction (closes #52)	7 years ago
Mike Fährmann	ebe9b0a04c	another attempt at downloader retry behavior ... This commit changes the general behavior from 'Retry on every exception and abort on DownloadError' to 'Only retry on DownloadRetry exceptions and abort on every other one' The previous version would have retried on several states which would have no chance of ever succeeding (invalid URLs, etc.)	7 years ago
Mike Fährmann	291369eab2	various smaller changes/additions	7 years ago
Mike Fährmann	4fb6803fa6	add option to sleep before each download	7 years ago
Mike Fährmann	300346ecdf	[mangazuki] remove extractors ... This site has been in "rebuild"-mode for a fairly long time and the current extractor code isn't going to work for the new version either.	7 years ago
Mike Fährmann	d275b1d9a3	[khinsider] fix extraction ... ... again	7 years ago
Mike Fährmann	6b8e3003df	[hentai2read] ensure consistent extraction results	7 years ago
Mike Fährmann	a1980b16f3	[gelbooru] various improvements ... - better metadata for pools - map ratings to s/q/e like other boorus do - skip() support	7 years ago
Mike Fährmann	93482a1f88	implement 'util.advance()'	7 years ago
Mike Fährmann	0e5057b15d	remove deprecated options	7 years ago
Mike Fährmann	8f518e03f8	add options to set maximum download rate ... - -r/--limit-rate as cmdline option - downloader.http.rate as config option This implementation very roughly uses the idea of the token bucket algorithm [1] and mostly uses Wget's approach [2] as inspiration. [1] https://en.wikipedia.org/wiki/Token_bucket [2] http://git.savannah.gnu.org/cgit/wget.git/tree/src/retr.c?h=v1.19.2&id=ba6b44f6745b14dce414761a8e4b35d31b176bba#n111	7 years ago
Mike Fährmann	a718c6c6cd	implement 'util.parse_bytes()'	7 years ago
Mike Fährmann	038e3b3369	[kissmanga] handle "AreYouHuman" redirects (#51)	7 years ago
Mike Fährmann	2b9a783fc7	[khinsider] fix extraction	7 years ago
Mike Fährmann	3dc1169736	use own mapping before relying on the 'mimetypes' module	7 years ago
Mike Fährmann	214972bc9a	[gelbooru] use manual extraction ... ... to compensate for their disabled API. (https://gelbooru.com/index.php?page=forum&s=view&id=3875) This also adds an extractor for image-pools.	7 years ago
Mike Fährmann	55c64cad4b	[khinsider] fix filename extension and test-pattern	7 years ago
Mike Fährmann	c0bcf8e343	release version 1.0.2	7 years ago
Mike Fährmann	b14de6ffc2	[tumblr] small improvements ... - don't transform inline GIF URLs - set 'type' parameter for API calls if there is only one post type selected	7 years ago
Mike Fährmann	9296a26eae	[tumblr] add warning messages	7 years ago
Mike Fährmann	65c1c53eb8	[khinsider] fix extraction	7 years ago
Mike Fährmann	12de658937	[tumblr] add options to control extraction behavior (#48) ... - posts : list of post-types to inspect - inline : scan post bodies for inline images - external: follow external links	7 years ago
Mike Fährmann	077f8c12be	[tumblr] original video URLs + continuous offset	7 years ago
Mike Fährmann	8eb12ebeae	[tumblr] support more post/media types (#48) ... This adds support for audio and video posts (most videos are shared from youtube/instagram which isn't supported -> youtube-dl), as well as link posts and image-search inside of text posts. Most of this is just WIP and will need some sort of improvement and options to enable/disable different media types etc.	7 years ago
Mike Fährmann	6c9da67581	apply selection options (filter, range) when using '-j'	7 years ago
Mike Fährmann	b8cdd42cab	[senmanga] fix extraction (again) ... this is basically a re-revert of 2ace5c7	7 years ago
Mike Fährmann	e6814aebe2	add 'extractor.*.user-agent' config option	7 years ago
Mike Fährmann	6913eeaa40	[powermanga] replace manga extractor unit test ... My Hero Academia is gone	7 years ago