1657 Commits (f2000a69aa4f2de16c397c8ffa801a26d2e0aa2c)

Author	SHA1	Message	Date
Mike Fährmann	0a1863fce3	[pixiv] respect more query parameters for user URLs ... The API endpoint responsible for user illustrations does not provide sufficient filter capabilities* to match the actual website, so we are spinning our own filters. Respected parameters are 'type': illust, manga, ugoira 'tag' : any image tag (this was already supported) 'p' : the page to start on * - API can filter for illustrations and manga, but not for ugoira. - 'offset' is applied before filtering - no 'tag' filter	6 years ago
Mike Fährmann	f43d446692	[mangahere] extract chapter titles	6 years ago
Mike Fährmann	b8e53b8c6b	[pixiv] move query parsing out of constructor ... better exception handling, among other things	6 years ago
Mike Fährmann	909d105ae6	[pixiv] add extractor for illusts from followed users	6 years ago
Mike Fährmann	7f899bd5d8	Merge branch 'master' into 1.4-dev	6 years ago
Mike Fährmann	fe69d01083	[pixiv] add extractor for search results	6 years ago
Mike Fährmann	247f785af1	[pixiv] use App API ... Transitioning to the App API breaks favorites archive IDs (there is no longer any bookmark ID information), but the favorites API endpoint of the public API was gone anyways ...	6 years ago
Mike Fährmann	92fc199b07	[reddit] allow arbitrary subdomains	6 years ago
Mike Fährmann	4cea886177	[imgur] allow longer album hashes	6 years ago
Mike Fährmann	e1e23165a0	[pinterest] catch JSON decode errors	6 years ago
Mike Fährmann	789608c107	[imagebam] fix extraction for certain galleries	6 years ago
Mike Fährmann	7a58151566	fix util.parse_bytes invocations ... (should be text.parse_bytes)	6 years ago
Mike Fährmann	1c1e086d01	use common base class for OAuth1.0 based API interfaces	6 years ago
Mike Fährmann	f3483a2b7c	[smugmug] add OAuth support	6 years ago
Mike Fährmann	6a31ada9e3	re-implement OAuth1.0 code ... OAuth support for SmugMug needs some additional features (auth-rebuild on redirect, query parameters in URL, ...) and fixing this in the old code wouldn't work all that well.	6 years ago
Mike Fährmann	ec158776ed	[deviantart] add extractor for popular listings	6 years ago
Mike Fährmann	0e3883303f	[pixiv] implement AppAPI wrapper	6 years ago
Mike Fährmann	e2157f594e	[mangadex] fix manga extraction (closes #84) ... Chapter listings for manga now use https://mangadex.org/manga/<id>/_/chapters/2/ as URL instead of https://mangadex.org/manga/<id>/_//2/	6 years ago
Mike Fährmann	69a5e6ddb3	Merge branch 'master' into 1.4-dev	6 years ago
Mike Fährmann	82c50fa609	release version 1.3.5	6 years ago
Mike Fährmann	3ce5296313	[smugmug] code cleanup ... - combine User and Node extractors - (re)move miscellaneous helper functions - rename "Owner" to "User"	6 years ago
Mike Fährmann	42ed7667b8	[smugmug] support user- and general album URLs	6 years ago
Mike Fährmann	8bf3cdd82b	implement logging options ... Standard logging to stderr, logfiles, and unsupported URL files (which are now handled through the logging module) can now be configured by setting their respective option keys (log, logfile, unsupportedfile) to a dict and specifying the following options; - format: format string for logging messages available keys: see [1] default: "[{name}][{levelname}] {message}" - format-date: format string for {asctime} fields in logging messages available keys: see [2] default: "%Y-%m-%d %H:%M:%S" - level: the lowercase levelname until which the logger should activate; available levels are debug, info, warning, error, exception default: "info" - path: path of the file to be written to - mode: 'mode' argument when opening the specified file can be either "w" to truncate the file or "a" to append to it (see [3]) If 'output.log', '.logfile', or '.unsupportedfile' is a string, it will be interpreted, as it has been, as the filepath (or as format string for .log) [1] https://docs.python.org/3/library/logging.html#logrecord-attributes [2] https://docs.python.org/3/library/time.html#time.strftime [3] https://docs.python.org/3/library/functions.html#open	6 years ago
Mike Fährmann	2ea0d1da42	[smugmug] improve API code; use data expansions	6 years ago
Mike Fährmann	16e014baaa	[smugmug] added image and album extractor ... just some initial code that still requires a lot of work ... TODO: - folders - old-style albums (which are nearly all of them ...) - images from users - OAuth It could also happen that the API credentials used will become invalid whenever my 14 day trial period ends (7 days remaining), but that would just require users to supply their own.	6 years ago
Mike Fährmann	d96b3474e5	[puremashiro] remove module ... site has been unreachable for a couple of weeks and now the DNS record is gone as well	6 years ago
Mike Fährmann	b44a296404	[gomanga] remove module ... site has been unreachable for a couple of weeks and the cloudflare status page shows host errors	6 years ago
Mike Fährmann	95392554ee	use text.urljoin()	6 years ago
Mike Fährmann	2395d870dd	[pinterest] unquote board and user names, better errors	6 years ago
Mike Fährmann	8b79eaafea	[tumblr] log actual time of rate limit resets ... ... instead of the amount of seconds until a reset	7 years ago
Mike Fährmann	0f1e07f627	[pinterest] scrap OAuth implementation; code improvements ... OAuth authentication isn't needed anymore and other tools like Postman are better suited for this job anyway.	7 years ago
Mike Fährmann	55d4d23860	[pinterest] use Pinterest's "Web" API (#83) ... no access tokens, no user credentials of any kind ...	7 years ago
Mike Fährmann	2721417dd8	Merge branch 'master' into 1.4-dev	7 years ago
Mike Fährmann	c6d5154fc3	fix flake8 errors, ignore W504 ... pycodestyle 2.4.0 enforces some new style guidelines	7 years ago
Mike Fährmann	2d17a9e07f	improve extractor.request() ... - better retry behavior - exponential back-off - removed 'allow_empty' argument	7 years ago
Mike Fährmann	80521ae1f6	[deviantart] improve API error handling ... The previous implementation would retry requests with 4xx status codes in an infinite loop, which is especially a problem when querying non-existent users or groups. These are now properly handled with a NotFoundError exception.	7 years ago
Mike Fährmann	e54b43be08	[mangadex] add title info for chapter extractors	7 years ago
Mike Fährmann	f471161920	Merge branch 'master' into 1.4-dev	7 years ago
Mike Fährmann	a2020c736e	release version 1.3.4	7 years ago
Mike Fährmann	eb37fbf0e8	[hentaifoundry] improve extractor ... - use common base class - better pagination - respect '.../page/<num>' - implement skip() / --range support - get YII_CSRF_TOKEN from cookies	7 years ago
Mike Fährmann	80bead739d	[oauth] require custom client-* values for pinterest	7 years ago
Mike Fährmann	cc36f88586	rename safe_int to parse_int; move parse_* to text module	7 years ago
Mike Fährmann	ff643793bd	improve and document cloudflare bypass code	7 years ago
Mike Fährmann	10cc59f3b5	fix extractor names	7 years ago
Mike Fährmann	b1325d4d2c	fix extractor docstrings	7 years ago
Mike Fährmann	df7e18399e	[luscious] fix image order	7 years ago
Mike Fährmann	d10579edb5	[pinterest] improve PinterestAPI code; remove OAuth mentions ... on another note: access_tokens have been set to only allow for 10 requests per hour (from 200 yesterday)	7 years ago
Mike Fährmann	4bd182c107	[pinterest] implement `oauth:pinterest` (#83) ... Pinterest access tokens are rate limited at 200 requests per hour (or maybe per 2 or 3 hours?) so having just one access token for all users isn't going to work in the long run.	7 years ago
Mike Fährmann	9651f3fce0	[pinterest] improve error messages (#83)	7 years ago
Mike Fährmann	dbe250f7e5	[pinterest] update access_token (#83)	7 years ago
Mike Fährmann	dd49127408	[spectrumnexus] remove module ... Site stopped hosting manga scans (http://view.thespectrum.net/)	7 years ago
Mike Fährmann	5c487300ee	improve 'parse_query()' and add tests ... - another irrelevant micro-optimization ! - use urllib.parse.parse_qsl directly instead of parse_qs, which just packs the results of parse_qsl in a different data structure - reduced memory requirements since no additional dict and lists are created	7 years ago
Mike Fährmann	728c64a3fb	[tumblr] rename 'offset' to 'num and adjust formats ... Trying to somehow emulate Tumblr filenames is a bad idea ...	7 years ago
Mike Fährmann	4ffa94f634	remove 'shorten_path()' and 'shorten_filename()'	7 years ago
Mike Fährmann	27eab4e467	rewrite text tests and improve functions ... - test more edge cases - consistently return an empty string for invalid arguments - remove the ungreedy-flag in 'remove_html()'	7 years ago
Mike Fährmann	e3f2bd4087	add tests for 'text.clean_xml()' and improve it	7 years ago
Mike Fährmann	6d8b191ea7	improve 'parse_query()' and add tests ... - another irrelevant micro-optimization ! - use urllib.parse.parse_qsl directly instead of parse_qs, which just packs the results of parse_qsl in a different data structure - reduced memory requirements since no additional dict and lists are created	7 years ago
Mike Fährmann	51ea699083	add 'abort()' as function to filter expressions ... calling 'abort()' in a filter aborts the current extractor run in a cleaner way than using something like 1/0, which causes an error message to be printed	7 years ago
Mike Fährmann	6bd857a319	[tumblr] handle rate limits / 429 errors ... - wait for the hourly limit to reset - abort upon exceeding the daily limit (it doesn't seem useful to potentially wait for several hours)	7 years ago
Mike Fährmann	7073ab7707	[komikcast] update regex to only match manga pages ... The 'readerarea' section now includes some (shady) external Javascript file, which got matched as well.	7 years ago
Mike Fährmann	a1fa4b43b0	Revert "[tumblr] add option to sort photosets by upload order" ... This reverts commit 4a26ae32df.	7 years ago
Mike Fährmann	48a83a89e9	[loveisover] remove module ... archive.loveisover.me was shut down on 2018-03-29; https://www.archiveteam.org/index.php?title=4chan#archive.loveisover.me	7 years ago
Mike Fährmann	564e12ca8f	replace 'imgyt' with 'imxto' ... https://img.yt/ wasn't available for a couple of days, but has now re-emerged as https://imx.to/ with a new web-interface. Links to older images still work (see tests).	7 years ago
Mike Fährmann	1b80fa82a9	[imgur] update URL pattern and tests	7 years ago
Mike Fährmann	4a26ae32df	[tumblr] add option to sort photosets by upload order	7 years ago
Mike Fährmann	6b72be8ee6	[tumblr] add 'hash' keyword ... 'hash' is the middle part of the filename in a tumblr image URL. For example an image with '.../tumblr_p6tgemp1NZ1wgha4yo1_250.png' as its URL would have 'p6tgemp1NZ1wgha4yo1' as hash.	7 years ago
Mike Fährmann	ffc0c67701	release version 1.3.3	7 years ago
Mike Fährmann	d11fcf4804	smaller changes and fixes ... - fix the cloudflare challenge result if the last decimal places are zero (JS`s toFixed() removes trailing zeroes) - fix downloading of kissmanga chapter-pages hosted on blogspot (accessing blogspot with "kissmanga.com" as referrer yields a 401) - disable certificate validation for 'mangahere' tests - update flickr test result	7 years ago
Mike Fährmann	f6c95dccf9	[cloudflare] fix bypass procedure ... Cloudflare challenges, at least for kissmanga and readcomiconline, now use slightly different Javascript expressions. Instead of a single value per expression, they now have a numerator and a denominator of a fractional value, which in the end gets truncated to 10 decimal places.	7 years ago
Mike Fährmann	759ba26fb0	[luscious] proper image order for picture albums ... ... and (try) to start with the first image instead of somewhere in the middle of an album.	7 years ago
Mike Fährmann	68e9fbee16	[tumblr] check all 4 keys/secrets before using OAuth ... it was possible to cause a crash by setting api-key or -secret to null. (this commit also slightly improves the blog-cache implementation)	7 years ago
Mike Fährmann	4810d446bb	remove the obsolete safeprint() and error() functions ... - safeprint() was used to print values which might have caused a UnicodeEncodeError, but that is no longer necessary (0381ae5) - errors are now handled via logging output (f94e370)	7 years ago
Mike Fährmann	0381ae5318	replace error handlers for stdout and co. ... Python3.5 and lower throw an UnicodeEncodeError when trying to print not-encodable characters when not using 'utf-8' as encoding. Setting their error handlers to 'replace' should help.	7 years ago
Mike Fährmann	f8168c693e	[tumblr] avoid calls to '/blog/.../info' ... The same information returned by the 'blog/.../info' API endpoint is also included in the result of every 'blog/.../posts' call.	7 years ago
Mike Fährmann	64d7c85b55	[exhentai] improve metadata ... - add 'width', 'height' and 'size' (in bytes) for each image - change the former 'size' and 'size_units' into 'gallery_size'	7 years ago
Mike Fährmann	64b22e0fc1	[pawoo] update URL pattern ... adds support for 'https://pawoo.net/@.../media'	7 years ago
Mike Fährmann	7b562907c3	[nijie] add favorites extractor ... adds support for 'https://nijie.info/user_like_illust_view.php?id=...'	7 years ago
Mike Fährmann	445db75955	[nijie] improve extraction and metadata ... - add 'title' and 'description' - split 'artist_id' into 'user_id' and 'artist_id' - 'user_id' is the ID of the user from which the image entry originates from - 'artist_id' is the ID of the actual image artist - improve pagination and URL patterns	7 years ago
Mike Fährmann	a112e3f2a0	[nijie] add doujin extractor ... adds support for "https://nijie.info/members_dojin.php?id=<artist_id>"	7 years ago
Mike Fährmann	f39153b6e9	[nhentai] add extractor for search results	7 years ago
Mike Fährmann	52d41c41e7	[exhentai] add extractor for favorited galleries	7 years ago
Mike Fährmann	63cc2599c4	[exhentai] add extractor for search results	7 years ago
Mike Fährmann	d1c91a1f2b	[mangadex] fix manga-page extraction	7 years ago
Mike Fährmann	299ae24996	[test] add a few downloader tests	7 years ago
Mike Fährmann	dd314279fb	[test] add unit tests for extractor module functions	7 years ago
Mike Fährmann	a993d0ea90	release version 1.3.2	7 years ago
Mike Fährmann	e7525b1b0e	[artstation] add challenge extractor (#80)	7 years ago
Mike Fährmann	3f2dd6b6f8	avoid double path-separators ... (#74)	7 years ago
Mike Fährmann	f5c6a2d7f5	[nhentai] use API to get gallery info	7 years ago
Mike Fährmann	b2ba2b821d	[hitomi] fix image URLs and improve metadata ... - use '?a.hitomi.la' as subdomain depending in gallery-id - add 'characters', 'tags' and 'date' information - support multiple entires per metadata-value - rename 'num' to 'page'	7 years ago
Mike Fährmann	3905474805	[booru] call update_page() with correct dict (closes #82)	7 years ago
Mike Fährmann	44c267e362	[artstation] add search extractor (#80)	7 years ago
Mike Fährmann	40ca562d7b	[artstation] add album extractor (#80)	7 years ago
Mike Fährmann	7121eeae8b	check supportedsites.rst in release script	7 years ago
Mike Fährmann	c59f9b71f1	release version 1.3.1	7 years ago
Mike Fährmann	f367d5c281	[deviantart] move delay-increase after expect_error check ... [ci skip]	7 years ago
Mike Fährmann	557cb94f81	[deviantart] use proper exponential backoff on API errors ... ... and use separate API credentials for unit tests.	7 years ago
Mike Fährmann	723cc66bb1	[artstation] add user-, image- and likes-extractors	7 years ago
Mike Fährmann	b69cc94f0e	[util] implement bencode()	7 years ago
Mike Fährmann	4d74749496	[tests] rework filters for extractor tests ... CI incompatible tests will now only be skipped if tests are run in a CI environment.	7 years ago
Mike Fährmann	d6ef52897c	[imgchili] remove module ... All previously hosted images yield a 404 and the main page is just a logo.	7 years ago
Mike Fährmann	7847ab1d5a	[imagehosts] remove even more dead sites ... All removed sites either - reject all incoming connections or - display a message from their domain registrar	7 years ago
Mike Fährmann	5f37d40a3e	[komikcast] bypass cloudflare challenge	7 years ago
Mike Fährmann	f9884e2338	[pixiv] update URL pattern ... add support for 'https://www.pixiv.net/user/<id>'	7 years ago
Mike Fährmann	85ed023c2e	[mangadex] remove the trailing ' - MangaDex' in a better way ... str.rstrip() works differently than assumed.	7 years ago
Mike Fährmann	9fb82e6b43	apply expand_path() to archive paths	7 years ago
Mike Fährmann	32bbd12f08	update extractor tests	7 years ago
Mike Fährmann	ca326bd275	[deviantart] fix folder and collection archive IDs ... {folder[index]} and {collection[index]} are both '0' when being delegated from Gallery- or FavoriteExtractors, as there is no way of knowing a folder's index when getting folder-information from the API.	7 years ago
Mike Fährmann	e32fe1cdf1	[pinterest] cast IDs to int ... ... and update test results. Image URLs changed from https://s-media-cache-ak0.pinimg.com/... to https://i.pinimg.com/...	7 years ago
Mike Fährmann	179ecee965	[turboimagehost] fix extraction	7 years ago
Mike Fährmann	1400868f53	[mangadex] general improvements ... - support >100 chapter entries per manga - custom archive ID format - detect non-existing chapters	7 years ago
Mike Fährmann	749fbbfa6c	[mangadex] add chapter- and manga-extractor	7 years ago
Mike Fährmann	b58449fd88	release version 1.3.0	7 years ago
Mike Fährmann	6e38cf5aab	[mangareader] use 'https://' ... The site now redirects from http://mangareader.net/ to https://mangareader.net/	7 years ago
Mike Fährmann	1d71123f91	[pixiv] update archive IDs and add metadata-fields ... (Pixiv bookmarks actually have their own IDs, comments and tags, independent of the bookmarked image, which makes creating an archive ID a lot easier)	7 years ago
Mike Fährmann	858fdbdb22	[tumblr] improve 'inline' extraction ... 'quote' posts store their HTML content in the 'source' field	7 years ago
Mike Fährmann	1d54a8e07d	fix logging output during downloads ... from: filename.ext[download][warning] ... to: filename.ext [download][warning] ...	7 years ago
Mike Fährmann	5008e105ee	update archive IDs ... ... to behave in a more straightforward way when dealing with bookmarks/favourites/etc. specific IDs are now grouped by their owner, album-id, ... to allow for duplicates when it would be expected.	7 years ago
Mike Fährmann	829ddf4ac1	[sankaku] general improvements ... - simplify regex - unquote search tags - increase default wait-time between HTTP requests - downloading several hundreds of images always resulted in '429 Too Many Requests' eventually - circumvent paging restrictions for unauthenticated users by only using the 'next' parameter - setting 'page' to a constant, low value (or simply omitting it) does the trick	7 years ago
Jad	49463f76bb	support multi-page URL (#79) ... * support multi-page URL * fix * all done. * fix, again	7 years ago
Mike Fährmann	19aefdfde3	[directlink] update test results	7 years ago
Mike Fährmann	74029c50bb	[directlink] unquote metadata fields	7 years ago
Mike Fährmann	2fad0b1f1b	add 'U' conversion for format strings to unquote their content ... (#74)	7 years ago
Mike Fährmann	8cdce21dcb	make archive keys user-configurable	7 years ago
Mike Fährmann	8f338347b6	[imagehosts] cleanup ... removed - chronos.to - unable to resolve hostname - coreimg.net - same - imgmaid.net - same - hosturimage.com - everything returns 404 - imageontime.org - redirects to some shady site - imgupload.yt - cloudflare error 522, host down - img4ever.net - read timeout	7 years ago
Mike Fährmann	edfd3d9fc9	[yeet] remove module ... - archive.yeet.net returns a 500 server error - yeet.net moved to yeet.rip, but the archive is gone	7 years ago
Mike Fährmann	e1e0668ca8	add option to set default replacement field value ... Missing or undefined keywords will now be replaced with the value set for 'keywords-default'. The default is Python's 'None', which is equivalent to setting this option to JSON's 'null'.	7 years ago
Mike Fährmann	ac3da8115e	[util] don't add text: URLs to list of downloaded URLs	7 years ago
Mike Fährmann	8704d850bf	add explicit proxy support (#76) ... - '--proxy' as command-line argument - 'extractor.*.proxy' as config option	7 years ago
Mike Fährmann	367b963d37	[pixiv] fix ugoira extraction ... again (#78) ... Some animations are not available for mobile devices, so we pretend to be a desktop browser when requesting the ugoira page.	7 years ago
Mike Fährmann	b79f1f2ca7	[pixiv] fix ugoira extraction (closes #78)	7 years ago
Mike Fährmann	731ffd4986	improve text.filename_from_url() performance ... - urlsplit() is faster than urlparse() - rpartition() is faster than rindex() + slicing - new version is 2.3 times as fast	7 years ago
Mike Fährmann	d122203be1	[mangastream] fix extraction	7 years ago
Mike Fährmann	8809b32aed	release version 1.2.0	7 years ago
Mike Fährmann	b50bdbf3d7	change config specifiers in input file format ... Instead of a dictionary/object, input file options are now specified by a 'key=value' pair starting with '-' for options only applying to the next URL or '-G' for Global options applying to all following URLs. See the docstring of parse_inputfile() for details. Example option specifiers: - filename = "{id}.{extension}" - extractor.pixiv.user.directory = ["Pixiv Users", "{user[id]}"] -spaces="are_optional" -G keywords = {"global": "option"}	7 years ago
Mike Fährmann	f970a8f13c	fix adding keys to download archive when using skip=false	7 years ago
Mike Fährmann	179bcdd349	adjust archive-ids	7 years ago
Mike Fährmann	be3ea4425d	test archive-id creation and uniqueness	7 years ago
Mike Fährmann	3cec533c28	Merge branch 'archive'	7 years ago
Mike Fährmann	20af86b2ea	add more extractor tests ... for mangastream, reddit and imgur	7 years ago
Mike Fährmann	b73b8b4f50	add OAuth unittests	7 years ago
Mike Fährmann	4d2fadfb6f	restore skip actions with download archive	7 years ago
Mike Fährmann	65773263fc	[util] implement OAuthSession.urlencode() (closes #75) ... - Python's own urllib.parse.urlencode() has no quote_via argument in Python 3.3 and 3.4, which is necessary to follow OAuth 1.0 quoting rules.	7 years ago
Mike Fährmann	7e0207bcf4	[imgur] strip trailing '?1' from 'ext'	7 years ago
Mike Fährmann	cf147dfee9	[hentai2read] fix manga extraction ... - site changed its HTML structure	7 years ago
Mike Fährmann	f5f2d29f56	[nijie] fix dojin extraction ... - correctly extract artist_id - set extension to "jpg" if it was empty and let filetype checks do the rest	7 years ago
Mike Fährmann	7f7c16ae37	add option to specify additional key-value pairs	7 years ago
Mike Fährmann	d38bf2f54c	[tumblr] recognize /image/... URLs ... xyz.tumblr.com/image/123 refers to the same images as xyz.tumblr.com/post/123.	7 years ago
Mike Fährmann	057668e17e	extend input-file format with per-URL config and comments ... - see docstring of parse_inputfile() for details - TODO: unittests, recursion (currently setting for example {"extractor": {"key": "value"}} will override the whole "extractor" branch instead of merging {"key": "value"} into the already existing dictionary)	7 years ago
Mike Fährmann	5b3c34aa96	use generic chapter-extractor in more modules	7 years ago
Mike Fährmann	347baf7ac5	improve util.parse_range() performance ... It is never going to actually matter, but using partition() instead of split() is twice as fast.	7 years ago
Mike Fährmann	7b5ba69951	[hentaihere] ensure consistent extraction results ... sometimes there is a random space before the next <a>	7 years ago
Mike Fährmann	377b78b3c9	[hentai2read] fix manga name extraction	7 years ago
Mike Fährmann	54c36a8a34	[subapics] add chapter- and manga-extractor (#70)	7 years ago
Mike Fährmann	2dd3aeeeae	[komikcast] add chapter- and manga-extractor (#70)	7 years ago
Mike Fährmann	7a412f5c32	implement generic manga-chapter extractor	7 years ago
Mike Fährmann	aa38eab2be	allow not-defined fields in format strings ... ... and replace them with "None", for now	7 years ago
Mike Fährmann	6a07e38366	implement extractor.add() and .add_module() ... ... as a public and non-hacky way to add (external) extractors to gallery-dl's pool and make them available for extractor.find()	7 years ago
Mike Fährmann	c0dd922c13	add '--download-archive' cmdline option ... … as well as a config file equivalent	7 years ago
Mike Fährmann	8c3b713362	rework DownloadJob.handle_url(); include archive functionality ... todo: "abort" and "exit" skip modes if download is skipped because of archive	7 years ago
Mike Fährmann	34873dbd90	set 'archive_fmt' values ... These are going to be used to create an unique id for each image.	7 years ago
Mike Fährmann	a34cebc253	[luscious] jump to first image if cover does not link to it	7 years ago
Mike Fährmann	84a52a9256	add DownloadArchive class	7 years ago
Mike Fährmann	915807dd77	log HTTP errors as warnings	7 years ago
Mike Fährmann	db7f04dd97	emit log messages on download failure ... and when retrying with fallback URLs	7 years ago
Mike Fährmann	d951f13e37	add config option for unsupported-URL file ... for consistency's sake	7 years ago
Mike Fährmann	619387cbb1	update extractor unittest results	7 years ago
Mike Fährmann	364e335440	smaller adjustments and improvements ... - requests and urllib3 version on 1 line - close input file after reading from it - use expand_path for unsupported-urls file - remove unnecessary logging from options.py	7 years ago
Mike Fährmann	c9a9664a65	change --write-log behaviour ... - log files now get truncated when opening them (mode "w" instead of "a") - log verbosity to file depends on -q/-v (same as logging to stderr)	7 years ago
Mike Fährmann	97f4f15ec0	add option to write logging output to a file ... - '--write-log FILE' as cmdline argument - 'output.logfile' as config file option	7 years ago
Mike Fährmann	f94e3706a8	use logging module for error messages during downloads	7 years ago
Mike Fährmann	db91cf871c	document message identifiers	7 years ago
Mike Fährmann	0dd48d644f	update test results ... nothing broke, but things got updated or changed	7 years ago
Mike Fährmann	1e93955170	[batoto] remove module ... Site officially shut down on 2018.01.18	7 years ago
Mike Fährmann	27fce6f600	fix UrlJob behavior	7 years ago
Mike Fährmann	76509a6d3c	[imgur] update test results	7 years ago
Mike Fährmann	9fccd7b783	[tumblr] provide fallback URLs (#64) ... Each image now produces 3 URLs: - amazonaws.com _raw (or _1280 for older images) - amazonaws.com _500 - media.tumblr.com (URL returned by API)	7 years ago
Mike Fährmann	b837420291	fix minor urllist issues	7 years ago
Mike Fährmann	9d69401391	initial support for multiple URLs per image	7 years ago
Mike Fährmann	6174a5c4ef	[download] adjust filename extension on filetype mismatch ... (closes #63)	7 years ago
Mike Fährmann	91ed147cef	[oauth] use custom key/secret values during oauth:…	7 years ago
Mike Fährmann	421a9740a3	[tumblr] add 'tumblr:' to force Tumblr extractor (#71)	7 years ago
Mike Fährmann	40d35c87bc	[paheal] add tag- and post-extractors (closes #69)	7 years ago
Mike Fährmann	cc0c2cca57	[reddit] add extractor for reddit-hosted images (closes #68)	7 years ago
Mike Fährmann	f10ffc0839	update extractor blacklist to also allow classes	7 years ago
Mike Fährmann	b6797032e3	release version 1.1.2	7 years ago
Mike Fährmann	35e09869d1	[mangapark] fix image URLs and use HTTPS	7 years ago
Mike Fährmann	9a049bdf51	[tumblr] add 'likes' extractor (#65)	7 years ago
Mike Fährmann	67d4462d26	[batoto] rudimentary Cloudflare bypass	7 years ago
Mike Fährmann	29d75fc3fa	[tumblr] add support for OAuth authentication (#65)	7 years ago
Mike Fährmann	4edb25346e	[slideshare] support mobile URLs (closes #67)	7 years ago
Mike Fährmann	e420a28bbc	fix cookie tests	7 years ago
Mike Fährmann	b33efc99a4	[idolcomplex] add support for idol.sankakucomplex.com	7 years ago
Mike Fährmann	75b2e84b6d	[tumblr] use s3.amazonaws.com for image URLs (#64)	7 years ago
Mike Fährmann	5b094328b5	[puremashiro] add chapter- and manga-extractor (closes #66) ... Also adds support for region subtags in language codes (e.g. en-us)	7 years ago
Mike Fährmann	974e73bdbb	[booru] smaller code adjustments	7 years ago
Mike Fährmann	03b8a548cb	[tumblr] change `reblogs` default value to `true` (#61)	7 years ago
Mike Fährmann	d235f68f59	[tumblr] add option to filter reblogged posts (#61) ... Reblogs are ignored by default, but can be included by setting 'extractor.tumblr.reblogs' to 'true'.	7 years ago
Mike Fährmann	a794fffc6d	[batoto] extend chapter-string regex (closes #60) ... Non-numeric chapter indices exist after all ...	7 years ago
Mike Fährmann	1219ebb7f5	[danbooru] use alternate subdomains; support safebooru	7 years ago
Mike Fährmann	9e8a84ab6c	[booru] rewrite using Mixin classes (#59) ... - improved code structure - improved URL patterns - better pagination to work around page limits on - Danbooru - e621 - 3dbooru	7 years ago
Mike Fährmann	0876541e43	[seiga] update tests	7 years ago
Mike Fährmann	1a70857a12	update extractor-unittest capabilities ... - "count" can now be a string defining a comparison in the form of '<operator> <value>', for example: '> 12' or '!= 1'. If its value is not a string, it is assumed to be a concrete integer as before. - "keyword" can now be a dictionary defining tests for individual keys. These tests can either be a type, a concrete value or a regex starting with "re:". Dictionaries can be stacked inside each other. Optional keys can be indicated with a "?" before its name. For example: "keyword:" { "image_id": int, "gallery_id", 123, "name": "re:pattern", "user": { "id": 321, }, "?optional": None, }	7 years ago
Mike Fährmann	88bb0798fd	delay initialization of PathFormat objects ... This allows the DeviantArt group-check to be moved inside the Extractor.items() method which in turn allows for better exception handling. As a new general rule: Never raise exceptions during extractor initialization.	7 years ago
Mike Fährmann	c24e0e70a7	[pixiv] simplify main loop	7 years ago
Mike Fährmann	c1e331edbb	[mangapark] replace manga test	7 years ago
Mike Fährmann	5488643fac	add requests and urllib3 versions to debug output	7 years ago
Mike Fährmann	9d73ed4772	fix issue with using 'skip()' when a filter is present ... calling skip() skips over unfiltered items and does not apply the filter expression to them, which is not what should happen	7 years ago
Mike Fährmann	28cd78aae0	[kissmanga] extend chapter-string regex (closes #58)	7 years ago
Mike Fährmann	0ba618dd1a	release version 1.1.1	7 years ago
Mike Fährmann	a3e9b51bea	[imgbox] update test results ... Image URLs of older galleries have been updated to the new format. https://i.imgbox.com/qHhw7lpG.png --> https://images3.imgbox.com/6d/9a/qHhw7lpG_o.png	7 years ago
Mike Fährmann	d241a0fb60	[util] replace '/' with '\' in base-directory paths ... ... on Windows to have consistent path separators.	7 years ago
Mike Fährmann	d0886f411e	[gelbooru] re-enable API use (closes #56) ... Gelbooru's API allows access to all images and is not restricted to the first 20000. This also adds an option to select between API use and manual information extraction in case their API gets disabled again.	7 years ago
Mike Fährmann	8102aae311	[mangahere] support ".cc" TLD and mobile URLs	7 years ago
Mike Fährmann	676602056c	[reddit] unescape output URLs	7 years ago
Mike Fährmann	2eedbaaaf9	[deviantart] use cache to store new refresh_tokens ... The 'refresh_token' set in a user's config file gets used once to get a new 'access_token' and 'refresh_token', which is then stored in gallery-dl's cache and gets used the next time the 'access_token' needs to be refreshed. This means deleting the cache file invalidates the refresh_token- chain and requires the user to re-authenticate.	7 years ago
Mike Fährmann	fc7d165c97	[deviantart] add support for OAuth2 authentication ... Some user galleries [*] require you to be either logged in or authenticated via OAuth2 to access their deviations. [*] e.g. https://polinaegorussia.deviantart.com/gallery/ -------------- known issue: A deviantart 'refresh_token' can only be used once and gets updated whenever it is used to request a new 'access_token', so storing its initial value in a config file and reusing it again and again is not possible.	7 years ago
Mike Fährmann	91c2aed077	[nhentai] fix JSON extraction	7 years ago
Mike Fährmann	444008a14a	[khinsider] use urljoin() to complete page URLs	7 years ago
Mike Fährmann	263741d243	[luscious] update URL pattern (closes #55)	7 years ago
Mike Fährmann	0a9a07a6e1	[slideshare] improve metadata; flake8 ... - added 'views' and 'published' keywords - fixed longer titles and descriptions	7 years ago
Leonardo Taccari	a8d2dde8b2	[slideshare] Add a new extractor for slideshare.net (#54)	7 years ago
Mike Fährmann	19a6ae57b2	[sankaku] add pool extractor	7 years ago
Mike Fährmann	e52f0cc1ed	[sankaku] add post extractor	7 years ago
Mike Fährmann	595593a35e	[sankaku] rewrite ... - better code structure and extensibility - better metadata	7 years ago
Mike Fährmann	e96e1fea5d	release version 1.1.0	7 years ago
Mike Fährmann	a3924d2072	[sankaku] fix swf extraction (closes #52)	7 years ago
Mike Fährmann	ebe9b0a04c	another attempt at downloader retry behavior ... This commit changes the general behavior from 'Retry on every exception and abort on DownloadError' to 'Only retry on DownloadRetry exceptions and abort on every other one' The previous version would have retried on several states which would have no chance of ever succeeding (invalid URLs, etc.)	7 years ago
Mike Fährmann	291369eab2	various smaller changes/additions	7 years ago
Mike Fährmann	4fb6803fa6	add option to sleep before each download	7 years ago
Mike Fährmann	300346ecdf	[mangazuki] remove extractors ... This site has been in "rebuild"-mode for a fairly long time and the current extractor code isn't going to work for the new version either.	7 years ago
Mike Fährmann	d275b1d9a3	[khinsider] fix extraction ... ... again	7 years ago
Mike Fährmann	6b8e3003df	[hentai2read] ensure consistent extraction results	7 years ago
Mike Fährmann	a1980b16f3	[gelbooru] various improvements ... - better metadata for pools - map ratings to s/q/e like other boorus do - skip() support	7 years ago
Mike Fährmann	93482a1f88	implement 'util.advance()'	7 years ago
Mike Fährmann	0e5057b15d	remove deprecated options	7 years ago
Mike Fährmann	8f518e03f8	add options to set maximum download rate ... - -r/--limit-rate as cmdline option - downloader.http.rate as config option This implementation very roughly uses the idea of the token bucket algorithm [1] and mostly uses Wget's approach [2] as inspiration. [1] https://en.wikipedia.org/wiki/Token_bucket [2] http://git.savannah.gnu.org/cgit/wget.git/tree/src/retr.c?h=v1.19.2&id=ba6b44f6745b14dce414761a8e4b35d31b176bba#n111	7 years ago
Mike Fährmann	a718c6c6cd	implement 'util.parse_bytes()'	7 years ago
Mike Fährmann	038e3b3369	[kissmanga] handle "AreYouHuman" redirects (#51)	7 years ago
Mike Fährmann	2b9a783fc7	[khinsider] fix extraction	7 years ago
Mike Fährmann	3dc1169736	use own mapping before relying on the 'mimetypes' module	7 years ago
Mike Fährmann	214972bc9a	[gelbooru] use manual extraction ... ... to compensate for their disabled API. (https://gelbooru.com/index.php?page=forum&s=view&id=3875) This also adds an extractor for image-pools.	7 years ago
Mike Fährmann	55c64cad4b	[khinsider] fix filename extension and test-pattern	7 years ago
Mike Fährmann	c0bcf8e343	release version 1.0.2	7 years ago
Mike Fährmann	b14de6ffc2	[tumblr] small improvements ... - don't transform inline GIF URLs - set 'type' parameter for API calls if there is only one post type selected	7 years ago
Mike Fährmann	9296a26eae	[tumblr] add warning messages	7 years ago
Mike Fährmann	65c1c53eb8	[khinsider] fix extraction	7 years ago
Mike Fährmann	12de658937	[tumblr] add options to control extraction behavior (#48) ... - posts : list of post-types to inspect - inline : scan post bodies for inline images - external: follow external links	7 years ago
Mike Fährmann	077f8c12be	[tumblr] original video URLs + continuous offset	7 years ago
Mike Fährmann	8eb12ebeae	[tumblr] support more post/media types (#48) ... This adds support for audio and video posts (most videos are shared from youtube/instagram which isn't supported -> youtube-dl), as well as link posts and image-search inside of text posts. Most of this is just WIP and will need some sort of improvement and options to enable/disable different media types etc.	7 years ago
Mike Fährmann	6c9da67581	apply selection options (filter, range) when using '-j'	7 years ago
Mike Fährmann	b8cdd42cab	[senmanga] fix extraction (again) ... this is basically a re-revert of 2ace5c7	7 years ago
Mike Fährmann	e6814aebe2	add 'extractor.*.user-agent' config option	7 years ago
Mike Fährmann	6913eeaa40	[powermanga] replace manga extractor unit test ... My Hero Academia is gone	7 years ago
Mike Fährmann	7e0d9257a7	[hbrowse] fix manga extraction	7 years ago
Mike Fährmann	3c576d10c0	[seiga] better metadata + 'skip()' support	7 years ago
Mike Fährmann	f72318e593	[seiga] support more than 200 images ... Due to API restrictions and/or missing knowledge about and documentation of API usage, it was only possible to retrieve the latest 200 images of a niconico seiga user with said API. The new approach manually visits each HTML page and gets its information from there.	7 years ago
Mike Fährmann	baf8094868	improve Extractor.request()'s retry behavior	7 years ago
Mike Fährmann	7e7b64162b	[batoto] handle error 10031	7 years ago
Mike Fährmann	79bcaa8726	improve downloader retry behavior ... - only retry download on 5xx and 429 status codes - immediately fail on 4xx status codes	7 years ago
Mike Fährmann	5ee8ca0319	release version 1.0.1	7 years ago
Mike Fährmann	42e948584d	fix downloader error handling ... RequestException being a subclass of OSError caused all exceptions during file downloads to be ignored/re-raised.	7 years ago
Mike Fährmann	92027f67f9	use consistent names for URL constants ... root := <scheme>://<host> base_url := <root>/<common path>	7 years ago
Mike Fährmann	69cbc0619f	[mangastream] fix 'next-page' URLs (fixes #49)	7 years ago
Mike Fährmann	980fd3616d	[tumblr] use API v2 (#48)	7 years ago
Mike Fährmann	d6bed9f36f	[tumblr] prevent premature exit to get all images (fixes #48)	7 years ago
Mike Fährmann	305da540c3	[mangahere] fix metadata extraction	7 years ago
Mike Fährmann	2d0cfb33e1	[xvideos] add user profile extractor (#45)	7 years ago
Mike Fährmann	a393e6e538	[xvideos] add gallery extractor (#45)	7 years ago
Mike Fährmann	3a8a0c1f35	[imgbox] rewrite / fix extraction (closes #47)	7 years ago
Mike Fährmann	f97207a8e6	release version 1.0.0	7 years ago
Mike Fährmann	707b15b586	create missing directories for 'part-directory' ... also some code improvements regarding downloader config values	7 years ago
Mike Fährmann	035ef655f1	[imagefap] update unit tests ... old gallery/image has been deleted	7 years ago
Mike Fährmann	caf26412dd	add option to set alternate location of .part files (#29) ... Note: The path set for 'downloader.*.part-directory' needs to point to an already existing directory.	7 years ago
Mike Fährmann	ea8ca4cfa4	add 'util.expand_path()'	7 years ago
Mike Fährmann	9a41002b77	fix partial downloads for 'text:' URLs ... Using a filesize in bytes as offset into a Python string is not a good idea if said file contains non-ASCII characters.	7 years ago
Mike Fährmann	239d7afea7	[hosturimage] fix extraction of larger images	7 years ago
Mike Fährmann	27c026543f	re-enable download unit tests	7 years ago
Mike Fährmann	963670d73b	add options to control usage of .part files (#29) ... - '--no-part' command line option to disable them - 'downloader.http.part' and 'downloader.text.part' config options Disabling .part files restores the behaviour of the old downloader implementation.	7 years ago
Mike Fährmann	158e60ee89	[3dbooru] enable download continuation ... behoimi.org doesn't respect 'Range' headers and doesn't report 'Content-Length' for compressed content encodings.	7 years ago
Mike Fährmann	b0353aa02d	rewrite download modules (#29) ... - use '.part' files during file-download - implement continuation of incomplete downloads - check if file size matches the one reported by server	7 years ago
Mike Fährmann	c4fcdf2691	Revert "[senmanga] fix extraction and download" ... This reverts commit 2ace5c7b3c.	7 years ago
Mike Fährmann	81a7788b40	replace space characters in unit test URLs	7 years ago
Mike Fährmann	bf82181359	[jaiminisbox] fix extraction	7 years ago
Mike Fährmann	2e982f56af	use 'Content-Length' to determine incomplete downloads (#29)	7 years ago
Mike Fährmann	16783e327f	[common] fix UnboundLocalError in Extractor.request()	7 years ago
Mike Fährmann	2ace5c7b3c	[senmanga] fix extraction and download	7 years ago
Mike Fährmann	4d8387f93b	[pixiv] support mobile URLs (https://touch.pixiv.net/)	7 years ago
Mike Fährmann	ab2bf0b0dd	[deviantart] replace collection unittest	7 years ago
Mike Fährmann	289d6b65d2	[danbooru] extend and improve URL regex ... - add support for danbooru mirrors: - hijiribe.donmai.us - sonohara.donmai.us - todo: actually use these domains instead of redirecting everything to danbooru itself - improve handling of query string parameters	7 years ago
Mike Fährmann	5fa42336a2	[sankaku] add warning for unauthenticated users ... also improve URL pattern and add missing options to default config file	7 years ago
Mike Fährmann	6af921a952	[sankaku] rewrite/improve (fixes #44) ... - add wait-time between HTTP requests similar to exhentai - add 'wait-min' and 'wait-max' options - increase retry-count for HTTP requests to 10 - implement user authentication (non-authenticated users can only view images up to page 25) - implement 'skip()' functionality (only works up to page 50) - implement image-retrieval for pages >= 51 - fix issue with multiple tags	7 years ago
Mike Fährmann	9aecc67841	[common] explicitly handle HTTP status code 429	7 years ago
Mike Fährmann	d68a24aa70	[kissmanga] fix extraction ... site changed '\n' to '\r\n' for newlines	7 years ago
Mike Fährmann	864a63ed33	fix typo ... [skip ci]	7 years ago
Mike Fährmann	f3fbaa5c3e	[reddit] allow users to override the API User-Agent ... Only overriding the Client-ID is not enough if you want to follow Reddit's API access rules [1]. [1] https://github.com/reddit/reddit/wiki/API#rules	7 years ago
Mike Fährmann	31ea6001e8	[dynastyscans] improve metadata and filename formats	7 years ago
Mike Fährmann	2ef3c35c98	smaller textual changes ... - swapped doc for deviantart.mature and .original - updated gallery-dl.conf - "transferred" -> "delegated"	7 years ago
Mike Fährmann	68a0a7579c	fix/improve some regular expressions	7 years ago
Mike Fährmann	832b8b76ac	[util] extend global namespace for filter expressions	7 years ago
Mike Fährmann	393755ee94	[tumblr] update tests	7 years ago
Mike Fährmann	75d3a1f72f	[deviantart] always download original images ... Deviation-objects returned by the DeviantArt API don't always contain the URL and metadata of the original image ([1]). Getting this information requires an additional API call [2], which is indicated by the 'is_downloadable' and 'download_filesize' metadata within a deviation-object. [1] https://myria-moon.deviantart.com/art/Aime-Moi-part-en-vadrouille-261986576 [2] https://www.deviantart.com/developers/http/v1/20160316/deviation_download/bed6982b88949bdb08b52cd6763fcafd	7 years ago
Mike Fährmann	8e6a767109	[util] restructure formatter for better exception propagation	7 years ago
Mike Fährmann	0386503c80	fix (sub)category-transfer for DownloadJob instances (#41) ... ... and extend "parent" parameters to TestJob- and DataJob-classes as well.	7 years ago
Mike Fährmann	a1c8b21cfd	[senmanga] improve metadata	7 years ago
Mike Fährmann	8df023e144	[util:filter] re-enable builtins ... Trying to restrict access to Python's builtin functions (exec, print, __import__, ...) can easily be circumvented and is therefore completely pointless. This also adds 'safe_int()' and the 'datetime' module to the global namespace used when evaluating filter expressions.	7 years ago
Mike Fährmann	994b2fc1e7	[deviantart] replace 'author[urlname]' keyword ... author[urlname] has always only been the lowercase version of author[username], which can now be directly converted to lowercase using the 'l' conversion: '{author[username]!l}'	7 years ago
Mike Fährmann	633b376f35	improve/adjust default filename formats for manga sites	7 years ago
Mike Fährmann	41adb99e9c	[pawoo] fix extraction ... - changed access_token - use account-search instead of general search	7 years ago
Mike Fährmann	b319f4bab3	smaller code and text changes	7 years ago
Mike Fährmann	ad4580800c	[pixiv] add support for more URL patterns ... - https://www.pixiv.net/mypage.php#id=USERID - https://www.pixiv.net/#id=USERID	7 years ago
Mike Fährmann	82ea6c0cd3	adjust format strings with optional titles ... ... except for anything manga/comic related	7 years ago
Mike Fährmann	c1f0afe4c6	add custom string formatter class	7 years ago
Mike Fährmann	85a2b2ae59	[khinsider] fix extraction	7 years ago
Mike Fährmann	26a866e7d8	implement (sub)category-transfer between extractors (#41) ... ImageFap- and all Manga-Extractors will transfer their (sub)category values to other extractors instantiated by them, which will in turn allow those to use options set for their parents. Example: ImagefapGalleryExtractors will use options set under extractor.imagefap.user, if (and only if) they have been instantiated by a ImagefapUserExtractor; and options from extractor.imagefap.gallery otherwise.	7 years ago
Mike Fährmann	1ab4c7986f	[mangahere] fix extraction ... would switch to HTTPS, but there seem to be certificate issues	7 years ago
Mike Fährmann	8e14714c2b	[imgspice] fix extraction	7 years ago
Mike Fährmann	9c138dfc1f	[common] detect empty HTTP response bodies	7 years ago
Mike Fährmann	c51616f8d8	[foolslide] fix minor chapter number	7 years ago
H R X N	77bf923c56	Update imgur.py to include 'title' of single image (#40) ... Add {title} keyword.. Images on Imgur don't necessarily have a title, but I think most of them do, and since this should not break anything else..	7 years ago
Mike Fährmann	a85f06d2d1	[foolslide] restructure; convert suitable values to int	7 years ago
Mike Fährmann	deb2e803ba	simplify MangaExtractor class	7 years ago
Mike Fährmann	9fc1d0c901	implement and use 'util.safe_int()' ... same as Python's 'int()', except it doesn't raise any exceptions and accepts a default value	7 years ago
Mike Fährmann	8a97bd0433	rename '--images' and '--chapters' ... ... to '--range' and '--chapter-range' to be consistent with '--filter' and '--chapter-filter'	7 years ago
Mike Fährmann	8963da8fd8	[spectrumnexus] extract manga metadata	7 years ago
Mike Fährmann	a3e40734d1	[mangareader] extract manga metadata	7 years ago
Mike Fährmann	9196005a4d	[mangazuki] extract manga metadata	7 years ago
Mike Fährmann	543ba245eb	[deviantart] update test results ... thumbnail URLs changed from //tXX.… to //t00.…	7 years ago
Mike Fährmann	b7a54a51d0	[mangapark] extract manga metadata + code improvements	7 years ago
Mike Fährmann	d39b8779af	[mangahere] extract manga metadata	7 years ago
Mike Fährmann	c265cc074a	[hbrowse] fix syntax for Python3.3 and 3.4	7 years ago
Mike Fährmann	a9e7145651	[hbrowse] extract hmanga metadata & general maintenance	7 years ago
Mike Fährmann	92c8a6cb01	[hentai2read] extract hmanga metadata	7 years ago
Mike Fährmann	de174b40d6	[hentaihere] extract hmanga metadata	7 years ago
Mike Fährmann	04cc1ffe34	[kissmanga] extract manga metadata	7 years ago
Mike Fährmann	885bd4cbe2	[readcomiconline] extract comic metadata	7 years ago
Mike Fährmann	cebf800a7f	[foolfuuka] add support for more sites (#18) ... - https://arch.b4k.co - https://archive.whatisthisimnotgoodwithcomputers.com - https://archive.yeet.net Notes: - The name "whatisthisimnotgoodwithcomputers" is way too long ... - archive.yeet.net is out of date and also blocked by 4chan servers - newest threads are 2 weeks old - using "https://archive.yeet.net" as Referer header results in "403 Forbidden" when accessing 4chan	7 years ago
Mike Fährmann	84d4450410	[fallenangels] extract manga metadata	7 years ago
Mike Fährmann	f32b1a0292	[imgyt] fix extraction	7 years ago
Mike Fährmann	4ad903b797	[warosu] fix extraction	7 years ago
Mike Fährmann	b84f48dfa5	[batoto] extract manga metadata	7 years ago
Mike Fährmann	4ceb176c6b	[foolslide] extract manga metadata ... enables chapter filtering for - https://kobato.hologfx.com/ - https://jaiminisbox.com/ - https://reader.kireicake.com/ - https://powermanga.org/ - https://reader.seaotterscans.com/ - http://sensescans.com/ - http://www.slide.world-three.org/	7 years ago
Mike Fährmann	24e5f154a4	[deviantart] update test results ... API responses now contain proper https:// URLs and their image download server is now "orig00.deviantart.net" for all images.	7 years ago
Mike Fährmann	0dedbe759c	enable '--chapter-filter' ... The same filter infrastructure that can be applied to image URLS now also works for manga chapters and other delegated URLs. TODO: actually provide any metadata (currently supported is only deviantart and imagefap).	7 years ago
Mike Fährmann	31cd5b1c1d	[luscious] detect high-load responses	7 years ago
Mike Fährmann	470bbe9d8c	fix smaller stuff ... - change filename option in example config file - adapt default filename format for mangafox - remove unnecessary newline [skip ci]	7 years ago
Mike Fährmann	6f30cf4c64	change keyword names to valid Python identifiers ... This commit mostly replaces all minus-signs ('-') in keyword names with underscores ('_') to allow them to be used in filter-expressions. For example 'gallery-id' got renamed to 'gallery_id'. (It is theoretically possible to access any variable, regardless of its name, with 'locals()["NAME"]', but that seems a bit too convoluted if just 'NAME' could be enough)	7 years ago
Mike Fährmann	81877bb5f6	add '-K' as shortcut for '--list-keywords'	7 years ago
Mike Fährmann	54c0715135	allow users to set their own API access_tokens/client_ids	7 years ago
Mike Fährmann	49c7e70c10	[acidimg] add image extractor	7 years ago
Mike Fährmann	5704c709fa	apply filter before range	7 years ago
Mike Fährmann	9b21d3f13c	add '--filter' command-line option ... This allows for image filtering via Python expressions by the same metadata that is also used to build filenames (--list-keywords). The usually shunned eval() function is used to evaluate filter-expressions, but it seemed quite appropriate in this case and shouldn't introduce any new security issues, as any attacker that could do > gallery-dl --filter "delete-everything()" ... could as well do > python -c "delete-everything()"	7 years ago
Mike Fährmann	268cfa3cfe	filter duplicate URLs (#36) ... Duplicate URLs might occur if, for example, an artist adds another image to his gallery while an extractor is running and images are being downloaded on sites like pixiv/nijie/hentaifoundry. The next image on the next page will have already been downloaded and will cause a premature end if '--abort-on-skip' is being used.	7 years ago
Mike Fährmann	00420ff202	[booru] consistent order for "popular" results	7 years ago
Mike Fährmann	83cf1e1d6d	[sankaku] unescape image URLs	7 years ago
Mike Fährmann	f98e3e8002	[luscious] fix tag extraction	7 years ago
Mike Fährmann	b8862ff15e	add 'downloader.http.verify' option ... (also: change the default 'timeout' from None to 30)	7 years ago
Mike Fährmann	65997d835b	replace popular/ranking tests with older ones ... Metadata of several year old lists shouldn't change as much as it would for newer ones, which makes metadata-comparisons of the output of build_testresult_db.oy easier.	7 years ago
Mike Fährmann	be30fb2f98	add common config category for boorus and foolslide	7 years ago
Mike Fährmann	c0755a4d5e	[exhentai] revert login-method to its old version (#37) ... Additional cookies don't seem to help and have to be manually set anyway. The older method is more likely to succeed, so I'd rather use this one.	7 years ago
Mike Fährmann	3ee39ffd93	[exhentai] update login procedure (#37) ... This new version behaves pretty much exactly like a browser would and caches all cookies sent to it and not just "ipb_member_id" and "ipb_pass_hash".	7 years ago
Mike Fährmann	88a386977e	[booru] add "popular" extractors for more sites ... - konachan.com - behoimi.org - e621.net	7 years ago
Mike Fährmann	07214f4007	[booru] place subcategories into base classes	7 years ago
Mike Fährmann	60a888a1e4	[foolfuuka] add common config category ... All FoolFuuka based 4chan-archive extractors can now be configured using their own config keys (extractor.<category>) as well as a common shared one (extractor.foolfuuka).	7 years ago
Mike Fährmann	47bcf53ec1	implement support for additional unit test result types ... - "pattern" matches all resulting URLs against the given regex - "count" allows to specify the amount of returned URLs	7 years ago
Mike Fährmann	2d0dfe9d56	[exhenai] init headers before login and detect sadpanda ... - also debug-logs html after failed login - #37	7 years ago
Mike Fährmann	c7ec103e15	[batoto] fix extraction of chapter URLs	7 years ago
Mike Fährmann	18e6ed1c7e	[booru] add extractors for "Popular" images	7 years ago
Mike Fährmann	f7cdfd4c25	add a simplified version of 'parse_qs' ... This version only returns a dict of plain string to string key-value pairs and ignores multiple values for the same query variable.	7 years ago
Mike Fährmann	3b21e0703c	[deviantart] allow distinction between users and groups (#26) ... This is done by prepending "group-" to an extractor's subcategory if the URL belongs to a group ("folder" becomes "group-folder" and so on). This changes the configuration-path being used and is also reflected in the output of '--list-keywords'.	7 years ago
Mike Fährmann	e61a3a56d1	[hentai2read] fix and update keywords ... Added the "author" keyword and changed the name of a few others to be consistent with other manga/chapter extractors.	7 years ago
Mike Fährmann	c45770331a	use 'str.partition()' ... The (r)partition method is always faster then split() or any other method that has been replaced in this commit.	7 years ago
Mike Fährmann	017a72f448	[pixiv] improve input validation	7 years ago
Mike Fährmann	dcf42c5e89	[pixiv] add extractor for ranking lists	7 years ago
Mike Fährmann	4ea82ea556	[warosu] add thread extractor	7 years ago
Mike Fährmann	6078ec5908	restructure the output of --help ... Using argument groups is a definite improvement over how things looked previously, but general group membership of individual items might be a thing to reconsider.	7 years ago
Mike Fährmann	9aa95fba8c	[deviantart] adapt download URLs to use https ... Even though DeviantArt is "completely switching over to HTTPS"[1], every URL contained in an API response is still using HTTP [1] https://danlev.deviantart.com/journal/DeviantArt-Is-Switching-To-HTTPS-697996906	7 years ago
Mike Fährmann	d70c66c516	fix "text:" downloader	7 years ago
Mike Fährmann	f7de048980	add additional debug output	7 years ago
Mike Fährmann	9bf9d64ad8	update unittests for util.py	7 years ago
Mike Fährmann	02e89700fc	[foolfuuka] ensure sorted posts	7 years ago
Mike Fährmann	8bcf88bff7	[flickr] fix extraction ... This issue was only noticeable with older Python versions, as these don't exhibit a consistent ordering of dict keys.	7 years ago
Mike Fährmann	e3bfb8325a	fix circular dependency ... - util.py imported config.py and vice versa - Python < 3.5 doesn't like this	7 years ago
Mike Fährmann	004456d5d5	properly update the config-dictionary ... When using 2 or more config files, the values of the second would improperly overwrite nested dictionaries of the first one. The new method properly combines these nested dictionaries as well.	7 years ago
Mike Fährmann	ae2d61e5b3	handle format string exceptions separately	7 years ago
Mike Fährmann	3c9f190757	extend output of --list-keywords	7 years ago
Mike Fährmann	cfa479fab5	update error message for unspecified exceptions ... - ask user to report unexpected errors, which usually indicate extractor failure - handle OSErrors separately (permissions, disk full, etc) - revert 30eef52	7 years ago
Mike Fährmann	7e936e9c06	[luscious] simplify and remove dead code	7 years ago
Mike Fährmann	d74a635e41	[util] update 'default' values and improve test coverage ... for 'code_to_language()' and 'language_to_code()'	7 years ago
Mike Fährmann	0245a0ba5f	fix extraction and update test results ... - fixes for hbrowse, imgyt, imgcandy, hosturimage - test updates for deviantart, gfycat	7 years ago
Mike Fährmann	abd7c559cd	[yonkouprod] remove module ... Every manga chapter on this site has been removed.	7 years ago
Mike Fährmann	da7219ba74	[kisscomic] remove module ... Image links on this site are dead.	7 years ago
Mike Fährmann	852e7acd31	[twitter] ignore "Promoted Tweets"	7 years ago
Mike Fährmann	915a0137de	improve 'extractor.request' ... - add 'fatal' argument - improve internal logic and flow - raise known exception on error - update exception hierarchy	7 years ago
rachmadani haryono	dcd573806e	chg: dev: fix error (#32) ... * fix: dev: error * fix: dev: AttributeError when getting artist * fix: dev: typo on luscious parser	7 years ago
Mike Fährmann	c4713404c8	[directlink] improve URL pattern	7 years ago
Mike Fährmann	d443822fdb	[luacious] get correct image URLs (fixes #33) ... Instead of using thumbnail URLs and modifying them the extractor now goes through every single image-page and gets its download URL from there.	7 years ago
Mike Fährmann	6950708e52	[hentaicdn] use HTTPS	7 years ago
Mike Fährmann	4f1e6c109f	[deviantart] remove 'invalid escape sequence' warning ... - use r"\w" or "\\w" instead of "\w"	7 years ago
Mike Fährmann	c864be479e	[directlink] update URL pattern & PEP 8 ... - combine some file extensions - don't match '.je' - line length < 80	7 years ago
H R X N	45f9d64c23	Update directlink.py with additional file exts. (#30) ... Add WebP, still not that common, but it's increasing. Add 3rd JPEG variant (https://en.wikipedia.org/wiki/JPEG#JPEG_filename_extensions) Never seen JFIF in the wild, would probably be overkill. Extend Ogg formats (https://en.wikipedia.org/wiki/Ogg; https://wiki.xiph.org/MIME_Types_and_File_Extensions)	7 years ago
Mike Fährmann	4357966a70	[kissmanga] make URL pattern case-insensitive (fixes 28)	7 years ago
Mike Fährmann	493bd235cf	workaround for missing 'assert_called_once' method ... this method was introduced in Python 3.6, but calling it still works (i.e. it doesn't cause the test to fail) on Python 3.3/3.4	7 years ago
Mike Fährmann	7aa9fa796a	code cleanup and fixes	7 years ago
Mike Fährmann	f08af03845	Merge branch 'cookies'	7 years ago
Mike Fährmann	55f048d02b	ignore case of cookiejar magic strings	7 years ago
Mike Fährmann	de68cf84a8	release version 0.9.1	7 years ago
Mike Fährmann	f53bf1a323	[thebarchive] add thread extractor	7 years ago
Mike Fährmann	b8cf434bb0	[rebeccablacktech] add thread extractor	7 years ago
Mike Fährmann	808f67ba7d	use 'cookiedomain' for cookies set by object-config-values ... otherwise these cookies would not be picked up by the _check_cookies() method.	7 years ago
Mike Fährmann	390eeded4c	[mangazuki] support 'raws.…' subdomain	7 years ago
Mike Fährmann	4a60f6068a	[mangazuki] add manga extractor	7 years ago
Mike Fährmann	394241cd6f	[2chan] fix extraction	7 years ago
Mike Fährmann	a13eb6010f	[fallenangels] fix extraction of chapter URLs	7 years ago
Mike Fährmann	1cb1d2e0a3	[mangazuki] add chapter extractor	7 years ago
Mike Fährmann	2f2e363c97	[imgur] use /a/<key>/all as album-url	7 years ago
Mike Fährmann	1cec03c9c6	[imgur] fix extraction of large albums	7 years ago
Mike Fährmann	0610ae5000	skip login if cookies are present	7 years ago
Mike Fährmann	f105782435	[fireden] add thread extractor	7 years ago
Mike Fährmann	c93f7d7496	[archiveofsins] add thread extractor	7 years ago
Mike Fährmann	96e13604da	[archivedmoe] add thread extractor	7 years ago
Mike Fährmann	30d3a5f9b2	support redirects on 4chan archives	7 years ago
Mike Fährmann	98464d1f1b	[loveisover] add thread extractor	7 years ago
Mike Fährmann	47692f28da	[2chan] add thread extractor	7 years ago
Mike Fährmann	3460dc8950	update gallery-dl.conf	7 years ago
Mike Fährmann	9be8f7e106	[deviantart] add "extractor.deviantart.flat" option ... Setting this to 'false' downloads images into individual subdirectories for each gallery-folder or favourite-collection, otherwise it is just creating a flat list of images.	7 years ago
Mike Fährmann	d075627fd9	[deviantart] support group galleries (#26) ... For groups the 'GalleryExtractor' collects all gallery-folder URLs and defers its work to the 'FolderExtractor'.	7 years ago
Mike Fährmann	b37a62501b	[pixiv] unquote tags	7 years ago
Mike Fährmann	fbd7dcdfdb	[desuarchive] add thread extractor	7 years ago
Mike Fährmann	af9bd17b19	[deviantart] adjust default paths ... - user.deviantart.com/(gallery|favourites|journal)/ images go into * <user>/ * <user>/Favourites/ * <user>/Journal/ (having an extra "Gallery" folder for a user's gallery-images seems a bit too much if these are all you want to download, which is probably the default use-case) - single "deviations" (user.deviantart.com/(art|journal)/name-123) go into their owner's directory: * <user>/ (putting them into their own directory seems weird in practice)	7 years ago
Mike Fährmann	eb64fb267c	[nyafuu] add thread extractor (#18)	7 years ago
Mike Fährmann	726c6f01ae	allow 'cookies' config option to be a dictionary	7 years ago
Mike Fährmann	4877ef6314	[deviantart] support '?catpath=/' URLs (#26) ... They previously weren't supported for galleries and journals. This also increases the 'limit' parameter for API calls to its respective maximum.	7 years ago
Mike Fährmann	8c16cbe7ea	fix tests	7 years ago
Mike Fährmann	a6f689e01a	[deviantart] add gallery-folder extractor (#26) ... The code for this and the available metadata is probably going to change again. This extractor is very similar to the favorite- extractor, so they might be "combined" or something like that.	7 years ago
Mike Fährmann	474e9c1aec	[4plebs] add thread extractor (#18)	7 years ago
Mike Fährmann	a804a42e23	add '--cookies' command-line option	7 years ago
Mike Fährmann	dcc1d3b2ea	[hentaifoundry] fix infinite loop for multiple of 25 images	7 years ago
Mike Fährmann	34e6e1099e	[batoto] adapt to https chapter URLs	7 years ago
Mike Fährmann	85696d0b3b	[reddit] fix issue with datetime errors	7 years ago
Mike Fährmann	80c2e03aaa	[reddit] allow 'date-min/max' to be human readable dates ... If the date-min/max config value is a string, try parsing it using datetime.strptime [1] with 'date-format' as format string [2] (default: "%Y-%m-%dT%H:%M:%S") Example: get all submissions posted in 2016 $ gallery-dl reddit.com/r/... \ -o date-format=%Y \ -o date-min=\"2016\" \ -o date-max=\"2017\" [1] https://docs.python.org/3/library/datetime.html#datetime.datetime.strptime [2] https://docs.python.org/3/library/datetime.html#strftime-strptime-behavior	7 years ago
Mike Fährmann	58e95a7487	share extractor and downloader sessions ... There was never any "good" reason for the strict separation between extractors and downloaders. This change allows for reduced resource usage (probably unnoticeable) and less lines of code at the "cost" of tighter coupling.	7 years ago
Mike Fährmann	4414aefe97	small fix for aes_cbc_decrypt_text	7 years ago
Mike Fährmann	21064146c1	fix test	7 years ago
Mike Fährmann	f3d0373120	[reddit] add ability to filter by submission id ... 'extractor.reddit.id-min' and '….id-max' specify the lowest and highest submission-/post-id to consider, similar to 'date-min' and 'date-max'	7 years ago
Mike Fährmann	06c4cae05b	extend the output of '--list-extractors' ... It now includes category and subcategory values for each extractor class.	7 years ago
Mike Fährmann	1dac76fd1c	update extractor docstrings	7 years ago
Mike Fährmann	e217e23e29	release version 0.9.0	7 years ago
Mike Fährmann	92a11528d1	smaller changes	7 years ago
Mike Fährmann	44d98e562b	[pixiv] support pixiv.me URLs (#23)	7 years ago
Mike Fährmann	b373fe0eea	[pixiv] support shortened URLs and other variants (#23)	7 years ago
Mike Fährmann	c951d6276c	[imagetwist] use https	7 years ago
Mike Fährmann	d3b04076f7	add .netrc support (#22) ... Use the '--netrc' cmdline option or set the 'netrc' config option to 'true' to enable the use of .netrc authentication data. The 'machine' names for the .netrc info are the lowercase extractor names (or categories): batoto, exhentai, nijie, pixiv, seiga.	7 years ago
Mike Fährmann	e1d82af5e0	small fixes	7 years ago
Mike Fährmann	719d45f89e	[flickr] allow the use of Flickr's specifiers for format selection ... - renamed the 'width-max' option to 'size-max' - filter by both width and height	7 years ago
Mike Fährmann	b4c438c9ad	[oauth] add the 'extractor.oauth.browser' option ... enables/disables the use of webbrowser.open() during OAuth authorization	7 years ago
Mike Fährmann	2633337833	[kissmanga] update regex (fixes #20)	7 years ago
Mike Fährmann	fac6c02224	[downloader] fix extension from content-type	7 years ago
Mike Fährmann	e68af4febe	[flickr] add 'width-max' option (#16) ... This option allows for simple format selection by specifying a maximum image width.	7 years ago
Mike Fährmann	2993206c4b	smaller fixes and "security" measures ... - move the OAuthSession class into util.py - block special extractors for reddit and recursive - ignore 'only matching' tests for testresults script	7 years ago
Mike Fährmann	8d5e92f641	resolve cyclic dependency between oauth and flickr	7 years ago
Mike Fährmann	d60781de7b	[oauth] workaround for ctrl+c on Windows	7 years ago
Mike Fährmann	9759fe8c6b	allow 'only_matching' tests	7 years ago
Mike Fährmann	56bec79e6a	[reddit] add ability to load more comments (#15) ... The 'extractor.reddit.morecomments' option enables the use of the '/api/morechildren' API endpoint (1) to load even more comments than the usual submission-request provides. Possible values are the booleans 'true' and 'false' (default). Note: this feature comes at the cost of 1 extra API call towards the rate limit for every 100 extra comments. (1) https://www.reddit.com/dev/api/#GET_api_morechildren	7 years ago
Mike Fährmann	05ed95e5b0	[flickr] add search extractor	7 years ago
Mike Fährmann	5f55c854b9	[flickr] replace getPublic... API call with regular ones	7 years ago
Mike Fährmann	9a620784f9	[flickr] add support for user authentication (#16) ... Call '$ gallery-dl oauth:flickr' to get an access_token and access_token_secret for your account.	7 years ago
Mike Fährmann	d5a70f2580	add simple progress indicator for multiple URLs (#19) ... The output can be configured via the 'output.progress' config value. Possible values: - true: Show the default progress indicator "[{current}/{total}] {url}" (default) - false: Never show the progress indicator - <string>: Show the progress indicator using this as a custom format string(1). Possible replacement keys are: - current: current URL index - total : total number of URLs - url : current URL (1) https://docs.python.org/3/library/string.html#formatstrings	7 years ago
Mike Fährmann	3ee77a0902	[oauth] print URL if webbrowser.open fails	7 years ago
Mike Fährmann	090e11b35d	[reddit] enable user authentication with OAuth2 (#15) ... Call '$ gallery-dl oauth:reddit' to get a refresh_token for your account.	7 years ago
Mike Fährmann	e682e06518	[flickr] add group extractor (#16)	7 years ago
Mike Fährmann	8fd66ef0b3	[flickr] add gallery extractor (#16)	7 years ago
Mike Fährmann	8456b84a12	fix tests and small stuff	7 years ago
Mike Fährmann	fbfc8d0f78	[reddit] ignore Authorization errors for subreddits ... - also made the limit for retrieved comments customizable via the 'extractor.reddit.comments' config value - default is 500; 0 ignores comments completely	7 years ago
Mike Fährmann	e365f1d799	[pixiv] rewrite ... - same functionality, better(?) code quality, easier to extend - added test for the user-tag functionality - removed the 'artist-id', 'artist-name' and 'artist-nick' keywords, which can be replaced with 'user[id]', 'user[name]' and 'user[account]' respectively	7 years ago
aiasdfd	338f79147f	[pixiv] support tag for user downloads (#17) ... [pixiv] support tag for user downloads	7 years ago
Mike Fährmann	5f05543f23	[reddit] support filtering by timestamp (#15) ... - Added the 'extractor.reddit.date-min' and '….date-max' config options. These values should be UTC timestamps. - All submissions not posted in date-min <= T <= date-max will be ignored. - Fixed the limit parameter for submission comments by setting it to its apparent max value (500).	7 years ago
Mike Fährmann	4e80e0c884	[flickr] add user extractor (#16)	7 years ago
Mike Fährmann	b81d068a6d	[flickr] add favorites extractor (#16)	7 years ago
Mike Fährmann	c921b4f32a	code cleanup and fixing tests	7 years ago
Mike Fährmann	72f1c6f87a	[flickr] add support for flic.kr/p/... URLs ... Example: https://flic.kr/p/FPVo9U	7 years ago
Mike Fährmann	93e5d8cba3	[flickr] add album extractor	7 years ago
Mike Fährmann	659c65dbb0	[flickr] add image extractor	7 years ago
Mike Fährmann	b6fffa9e26	[directlink] update filename format and metadata	7 years ago
Mike Fährmann	c184e47ee3	put common directory- and filename formats in base classes	7 years ago
Mike Fährmann	bce51e90a5	[reddit] support sorting options and sub-options (#15) ... Example: https://www.reddit.com/r/<subreddit>/top/?sort=top&t=month (the 'sort=top' parameter is irrelevant and can be omitted)	7 years ago
Mike Fährmann	5f45ce2930	[gfycat] add "format" config key to select a video format ... Possible values: - one of "mp4" (default), "webm", "gif", "webp", "mjpg" If the selected format is not available, "mp4", "webm" and "gif" (in that order) will be tried instead, until an available format is found.	7 years ago
Mike Fährmann	011659ced5	[imgur] add "mp4" config key to decide between GIF and MP4 ... possible values: - false : always use GIF - true : use MP4 if "prefer_video" flag is set, GIF otherwise (default) - "always": always use MP4	7 years ago
Mike Fährmann	48ccee2505	[gfycat] add image extractor	7 years ago
Mike Fährmann	25bcdc8aa9	add `--write-unsupported` option (#15)	7 years ago
Mike Fährmann	bf452a8516	[imgur] choose .mp4 over .gif if available	7 years ago
Mike Fährmann	f79320e35b	fix tests	7 years ago
Mike Fährmann	67791e1b36	[imgur] improve and add image extractor	7 years ago
Mike Fährmann	99b72130ee	[reddit] enable recursion (#15) ... reddit extractors now recursively visit other submissions/posts linked to in the initial set of submissions. This behaviour can be configured via the 'extractor.reddit.recursion' key in the configuration file or by `-o recursion=<value>`. Example: {"extractor": { "reddit": { "recursion": <value> }}} Possible values: * -1 - infinite recursion (don't do this) * 0 - recursion is disabled (default) * 1 and higher - maximum recursion level	7 years ago
Mike Fährmann	ae686c4c08	run queue items immediately	7 years ago
Mike Fährmann	691c4dd709	support direct image links	7 years ago
Mike Fährmann	d2dceb35b7	implement context-manager to blacklist extractors	7 years ago
Mike Fährmann	30eef527d8	update output logic on error ... [ci skip]	7 years ago
Mike Fährmann	e425243b1e	[reddit] some small fixes ... - filter or complete some URLs - remove the 'nofollow:' scheme before printing URLs - (#15)	7 years ago
Mike Fährmann	a22892f494	[reddit] add subreddit- and submission-extractor ... - these extractors scan submissions and their comments for (external) URLs and defer them to other extractors - (#15)	7 years ago
Mike Fährmann	398506da45	update release script	7 years ago
Mike Fährmann	8db3a2fea8	release version 0.8.4	7 years ago
Mike Fährmann	832a4a8ee9	[fallenangels] add manga extractor	7 years ago
Mike Fährmann	f226417420	simplify code by using a MangaExtractor base class	7 years ago
Mike Fährmann	2974d782a3	[yomanga] remove module ... site has been shut down	7 years ago
Mike Fährmann	cbb4323f66	add setup.cfg to configure flake8	7 years ago
Mike Fährmann	232fe2dd08	improve the test extractor	7 years ago
Mike Fährmann	b0131ea402	[fallenangels] support this site's Vietnamese version ... - https://truyen.fascans.com/	7 years ago
Mike Fährmann	a90c6acc9c	code cleanup + fixes	7 years ago
Mike Fährmann	4c88c0d496	rework the output format for --list-keywords	7 years ago
Mike Fährmann	b6b214f7e9	[deviantart] fix headers for custom-style journals ... example: http://shimoda7.deviantart.com/journal/Temporary-absence-231936282	7 years ago
Mike Fährmann	e9a2738257	[deviantart] support images on top of journal entries ... example: http://raxnae.deviantart.com/art/Kami-s-Journal-679482236	7 years ago
Mike Fährmann	92597f46d4	[deviantart] add title to journals	7 years ago
Mike Fährmann	107d29ad8a	improve handling of text:... URLs ... - don't require // after the colon - open output files in text mode	7 years ago
Mike Fährmann	677c8ced11	[deviantart] add "journal" extractor ... (#14)	7 years ago
Mike Fährmann	e5f79ae839	[deviantart] add support for all media types ... - this includes - images - videos - flash-animations - journals - also renamed some of the extractors - User -> Gallery - Image -> Deviation	7 years ago
Mike Fährmann	9f1c83297f	[pinterest] allow URLs with any TLD	7 years ago
Mike Fährmann	b3b92ac243	[deviantart] support "All" favorites and add "mature" option ... - since there is apparently no actual way to get the "All" favorites listing via API, corresponding URLs (.../favourites/?catpath=/) will be handled by yielding all deviations from all favorite collections of that user - the "mature" config key works on a per extractor basis (like "username" or "password"). values can be the strings "true" or "false", or the booleans true or false. - (#14)	7 years ago
Mike Fährmann	7376ad7f3d	[deviantart] turn the "Mature Content Filter" off ... (#14)	7 years ago
Mike Fährmann	ef90a2de2f	implement the "exit" option for the "skip" config-key	7 years ago
Mike Fährmann	cfbf79d788	[pixiv] fix login	7 years ago
Mike Fährmann	85a46ed700	[booru] fix issue with multiple tags	7 years ago
Mike Fährmann	fc9223c072	add '--abort-on-skip' option and ability to control skip behavior ... the 'skip' config option controls skipping behavior: true - skip download if file already exist (default) false - download and overwrite files even if it exists "abort" - abort extractor run if a download would be skipped (same as '--abort-on-skip')	7 years ago
Mike Fährmann	7c8f61a116	release version 0.8.3	7 years ago
Mike Fährmann	d948ba1322	[readcomics] remove module ... - site has been unavailable for two weeks - (#12)	7 years ago
Mike Fährmann	a610b35a0d	[mangashare] remove module ... this site has been unavailable for at least two months	7 years ago
Mike Fährmann	4e8587bad4	[pixiv] add support for https://i.pximg.net URLs	7 years ago
Mike Fährmann	e41efbd2d9	[kissmanga] fix edge-case	8 years ago
Mike Fährmann	ffd72424bf	[kissmanga] another attempt at getting the AES key	8 years ago
Mike Fährmann	af56887a47	[exhentai] fall back to e-hentai if no username is given	8 years ago
Mike Fährmann	48a5b11204	fix error if no file extension is found	8 years ago
Mike Fährmann	701c016b97	add '-q/--quiet' option	8 years ago
Mike Fährmann	4b967fa189	implement and use extractor.config() method	8 years ago
Mike Fährmann	f0aa35ac84	add '--ignore-config' option	8 years ago
Mike Fährmann	82ab1fca07	[seiga] reduce cache maxage to one week	8 years ago
Mike Fährmann	ec48d25afc	[pawoo] fix extraction results	8 years ago
Mike Fährmann	244ab75cad	[kissmanga] update AES key retrieval	8 years ago
Chen John L	a5485a46cb	fixed the module for pixhost	8 years ago
Mike Fährmann	13dc5d72bc	update some extractors to use https	8 years ago
Mike Fährmann	342371086b	[pawoo] add extractors for accounts and statuses ... https://pawoo.net is a Mastodon[1] instance hosted by Pixiv [1] https://github.com/tootsuite/mastodon	8 years ago
Mike Fährmann	5af35ea150	add -v/--verbose option and reduce error verbosity ... (#12)	8 years ago
Mike Fährmann	0770de0ea1	[deviantart:image] add support for sta.sh URLs	8 years ago
Mike Fährmann	f4aa452bd1	update unit test results	8 years ago
Mike Fährmann	71e08dc9c4	[tumblr] keyword consistency	8 years ago
Mike Fährmann	b43cd88101	add '-j/--dump-json' option ... this outputs the extractor-results in JSON format rather then downloading files	8 years ago
Mike Fährmann	c9a5650cf8	add manga extractors to all foolslide-based modules	8 years ago
Mike Fährmann	bd95fea82c	update unit test results	8 years ago
Mike Fährmann	0456efaa5a	[hentaifoundry] update unit tests	8 years ago
Mike Fährmann	af82467627	release version 0.8.2	8 years ago
Mike Fährmann	fd3f84d913	[kissmanga] get AES key from site itself	8 years ago
Mike Fährmann	fa1cb6f74c	release version 0.8.1	8 years ago
Mike Fährmann	fece09d326	[fallenangels] update to new domain and site-layout	8 years ago
Mike Fährmann	cf79a47b59	update unit tests	8 years ago
Mike Fährmann	e23e4b1100	[seaotterscans] add extractor	8 years ago
Mike Fährmann	616464f8f8	[kireicake] add extractor	8 years ago
Mike Fährmann	8d5f26d530	small fixes to the output module	8 years ago
Mike Fährmann	f537ad5f2f	[kissmanga] re-enable module	8 years ago
Mike Fährmann	c9738ea80b	add an aes implementation	8 years ago
Mike Fährmann	b603b592cf	[exhentai] accept "e-hentai.org" URLs (#11)	8 years ago
Mike Fährmann	11ce871718	[deviantart] add favourites-extractor	8 years ago
Mike Fährmann	d430732502	[deviantart] update extractors ... - use base class for all extractors - use API for single images - add support for more API endpoints - add additional keywords	8 years ago
Mike Fährmann	298d7c45f7	[nijie] support multi-page image listings	8 years ago
Mike Fährmann	91ad8ac2b3	[mangastream] adapt to domain-change to readms.net	8 years ago
Mike Fährmann	e9b445a0f8	update release script	8 years ago
Mike Fährmann	43d8b66e70	release version 0.8.0	8 years ago
Mike Fährmann	841fd50242	move code into util.py	8 years ago
Mike Fährmann	e3212dd98f	fix some smaller stuff ... - remove support for old windows config paths - catch exception if cache-database can't be opened - fix username/password settings for unit tests - rename variable 'max_tries' to 'retries'	8 years ago
Mike Fährmann	e4b3077168	improve config module ... - speed improvements, especially in the 'interpolate' function - 'interpolate' now prioritizes base-level values if they exist - "username" is chosen before "extractor.<category>.username" - -u/--username & co can now override config-file values	8 years ago
Mike Fährmann	6cbb078392	[batoto] fix cache time for login sessions ... from 1 year to 1 week ...	8 years ago
Mike Fährmann	e2b5cd9918	change config-path for 'retries' and 'timeout'	8 years ago
Mike Fährmann	6fd9e66223	[whentai] remove module ... apparently they changed their system and now you can't download the original images without an account with VIP status.	8 years ago
Mike Fährmann	cdecd0b37b	add notification when solving cloudflare challenge	8 years ago
Mike Fährmann	dce5bce74e	[kissmanga] temporarily disable module ... ... until I've figured out how to decode their image urls	8 years ago
Mike Fährmann	f2ef49563b	fix argument order for python33 and 34	8 years ago
Mike Fährmann	11d5c6f717	move option parsing to seperate module	8 years ago
Mike Fährmann	0b5076815d	always delete incompletely downloaded files	8 years ago
Mike Fährmann	4e7661ab01	[imgtrex] re-add extractor	8 years ago
Mike Fährmann	0257d3e7ac	[mangamint] remove extractors - site is down	8 years ago
Mike Fährmann	1d46be545c	add login notifications	8 years ago
Mike Fährmann	ed94d9b92d	fix/improve various things	8 years ago
Mike Fährmann	abfe7456d6	add '-R/--retries' and '--http-timeout' options ... (#10)	8 years ago
Mike Fährmann	22910f9562	improve error handling of http file downloads ... (#10)	8 years ago
Mike Fährmann	80df2b3527	add custom argparse action	8 years ago
Mike Fährmann	619c74159a	[seiga] fix file extension and xml parsing ... - The file extension of the first image had been used for all further images - API responses can contain invalid characters, which cause the XML parser to fail (http://seiga.nicovideo.jp/user/illust/26377934 contains several \x08 characters)	8 years ago
Mike Fährmann	8816dab549	add 'dev' suffix for non-release versions ... (#9)	8 years ago
Mike Fährmann	66eb3d3488	fix tests	8 years ago
Mike Fährmann	f622411be5	[deviantart] implement 'skip' method	8 years ago
Mike Fährmann	27ae152f57	use logging to report errors	8 years ago
Mike Fährmann	dfe2c2dced	[batoto] update test results	8 years ago
Mike Fährmann	43e3bb24ae	[imagefap] don't rely on image-count ... (fixes #9)	8 years ago
Mike Fährmann	0cfe51dc78	add '--config-yaml' option ... (#8)	8 years ago
Mike Fährmann	379125746a	[deviantart] improve API error handling ... (#7)	8 years ago
Mike Fährmann	f782282f97	add logger objects to extractors	8 years ago
Mike Fährmann	0b214eeb9d	bump version to 0.7.0	8 years ago
Mike Fährmann	2c2de778f2	[deviantart] adjust error message	8 years ago
Mike Fährmann	c24aeb5cbc	[whentai] update test results	8 years ago
Mike Fährmann	f36d685ce1	[deviantart] always request access token before api calls ... (#7)	8 years ago
Mike Fährmann	ff92674379	[booru] simple skip functionality ... (#6)	8 years ago
Mike Fährmann	7a9d66fbce	implement basic way to tell extractors to skip ahead	8 years ago
Mike Fährmann	872b8aed97	[4chan] update test	8 years ago