911 Commits (7442d2940c0220c553de3ffd9ab23ae226f31625)

Author	SHA1	Message	Date
Mike Fährmann	5c487300ee	improve 'parse_query()' and add tests ... - another irrelevant micro-optimization ! - use urllib.parse.parse_qsl directly instead of parse_qs, which just packs the results of parse_qsl in a different data structure - reduced memory requirements since no additional dict and lists are created	7 years ago
Mike Fährmann	728c64a3fb	[tumblr] rename 'offset' to 'num and adjust formats ... Trying to somehow emulate Tumblr filenames is a bad idea ...	7 years ago
Mike Fährmann	6bd857a319	[tumblr] handle rate limits / 429 errors ... - wait for the hourly limit to reset - abort upon exceeding the daily limit (it doesn't seem useful to potentially wait for several hours)	7 years ago
Mike Fährmann	7073ab7707	[komikcast] update regex to only match manga pages ... The 'readerarea' section now includes some (shady) external Javascript file, which got matched as well.	7 years ago
Mike Fährmann	a1fa4b43b0	Revert "[tumblr] add option to sort photosets by upload order" ... This reverts commit 4a26ae32df.	7 years ago
Mike Fährmann	48a83a89e9	[loveisover] remove module ... archive.loveisover.me was shut down on 2018-03-29; https://www.archiveteam.org/index.php?title=4chan#archive.loveisover.me	7 years ago
Mike Fährmann	564e12ca8f	replace 'imgyt' with 'imxto' ... https://img.yt/ wasn't available for a couple of days, but has now re-emerged as https://imx.to/ with a new web-interface. Links to older images still work (see tests).	7 years ago
Mike Fährmann	1b80fa82a9	[imgur] update URL pattern and tests	7 years ago
Mike Fährmann	4a26ae32df	[tumblr] add option to sort photosets by upload order	7 years ago
Mike Fährmann	6b72be8ee6	[tumblr] add 'hash' keyword ... 'hash' is the middle part of the filename in a tumblr image URL. For example an image with '.../tumblr_p6tgemp1NZ1wgha4yo1_250.png' as its URL would have 'p6tgemp1NZ1wgha4yo1' as hash.	7 years ago
Mike Fährmann	d11fcf4804	smaller changes and fixes ... - fix the cloudflare challenge result if the last decimal places are zero (JS`s toFixed() removes trailing zeroes) - fix downloading of kissmanga chapter-pages hosted on blogspot (accessing blogspot with "kissmanga.com" as referrer yields a 401) - disable certificate validation for 'mangahere' tests - update flickr test result	7 years ago
Mike Fährmann	759ba26fb0	[luscious] proper image order for picture albums ... ... and (try) to start with the first image instead of somewhere in the middle of an album.	7 years ago
Mike Fährmann	68e9fbee16	[tumblr] check all 4 keys/secrets before using OAuth ... it was possible to cause a crash by setting api-key or -secret to null. (this commit also slightly improves the blog-cache implementation)	7 years ago
Mike Fährmann	f8168c693e	[tumblr] avoid calls to '/blog/.../info' ... The same information returned by the 'blog/.../info' API endpoint is also included in the result of every 'blog/.../posts' call.	7 years ago
Mike Fährmann	64d7c85b55	[exhentai] improve metadata ... - add 'width', 'height' and 'size' (in bytes) for each image - change the former 'size' and 'size_units' into 'gallery_size'	7 years ago
Mike Fährmann	64b22e0fc1	[pawoo] update URL pattern ... adds support for 'https://pawoo.net/@.../media'	7 years ago
Mike Fährmann	7b562907c3	[nijie] add favorites extractor ... adds support for 'https://nijie.info/user_like_illust_view.php?id=...'	7 years ago
Mike Fährmann	445db75955	[nijie] improve extraction and metadata ... - add 'title' and 'description' - split 'artist_id' into 'user_id' and 'artist_id' - 'user_id' is the ID of the user from which the image entry originates from - 'artist_id' is the ID of the actual image artist - improve pagination and URL patterns	7 years ago
Mike Fährmann	a112e3f2a0	[nijie] add doujin extractor ... adds support for "https://nijie.info/members_dojin.php?id=<artist_id>"	7 years ago
Mike Fährmann	f39153b6e9	[nhentai] add extractor for search results	7 years ago
Mike Fährmann	52d41c41e7	[exhentai] add extractor for favorited galleries	7 years ago
Mike Fährmann	63cc2599c4	[exhentai] add extractor for search results	7 years ago
Mike Fährmann	d1c91a1f2b	[mangadex] fix manga-page extraction	7 years ago
Mike Fährmann	299ae24996	[test] add a few downloader tests	7 years ago
Mike Fährmann	dd314279fb	[test] add unit tests for extractor module functions	7 years ago
Mike Fährmann	e7525b1b0e	[artstation] add challenge extractor (#80)	7 years ago
Mike Fährmann	f5c6a2d7f5	[nhentai] use API to get gallery info	7 years ago
Mike Fährmann	b2ba2b821d	[hitomi] fix image URLs and improve metadata ... - use '?a.hitomi.la' as subdomain depending in gallery-id - add 'characters', 'tags' and 'date' information - support multiple entires per metadata-value - rename 'num' to 'page'	7 years ago
Mike Fährmann	3905474805	[booru] call update_page() with correct dict (closes #82)	7 years ago
Mike Fährmann	44c267e362	[artstation] add search extractor (#80)	7 years ago
Mike Fährmann	40ca562d7b	[artstation] add album extractor (#80)	7 years ago
Mike Fährmann	f367d5c281	[deviantart] move delay-increase after expect_error check ... [ci skip]	7 years ago
Mike Fährmann	557cb94f81	[deviantart] use proper exponential backoff on API errors ... ... and use separate API credentials for unit tests.	7 years ago
Mike Fährmann	723cc66bb1	[artstation] add user-, image- and likes-extractors	7 years ago
Mike Fährmann	4d74749496	[tests] rework filters for extractor tests ... CI incompatible tests will now only be skipped if tests are run in a CI environment.	7 years ago
Mike Fährmann	d6ef52897c	[imgchili] remove module ... All previously hosted images yield a 404 and the main page is just a logo.	7 years ago
Mike Fährmann	7847ab1d5a	[imagehosts] remove even more dead sites ... All removed sites either - reject all incoming connections or - display a message from their domain registrar	7 years ago
Mike Fährmann	5f37d40a3e	[komikcast] bypass cloudflare challenge	7 years ago
Mike Fährmann	f9884e2338	[pixiv] update URL pattern ... add support for 'https://www.pixiv.net/user/<id>'	7 years ago
Mike Fährmann	85ed023c2e	[mangadex] remove the trailing ' - MangaDex' in a better way ... str.rstrip() works differently than assumed.	7 years ago
Mike Fährmann	32bbd12f08	update extractor tests	7 years ago
Mike Fährmann	ca326bd275	[deviantart] fix folder and collection archive IDs ... {folder[index]} and {collection[index]} are both '0' when being delegated from Gallery- or FavoriteExtractors, as there is no way of knowing a folder's index when getting folder-information from the API.	7 years ago
Mike Fährmann	e32fe1cdf1	[pinterest] cast IDs to int ... ... and update test results. Image URLs changed from https://s-media-cache-ak0.pinimg.com/... to https://i.pinimg.com/...	7 years ago
Mike Fährmann	179ecee965	[turboimagehost] fix extraction	7 years ago
Mike Fährmann	1400868f53	[mangadex] general improvements ... - support >100 chapter entries per manga - custom archive ID format - detect non-existing chapters	7 years ago
Mike Fährmann	749fbbfa6c	[mangadex] add chapter- and manga-extractor	7 years ago
Mike Fährmann	6e38cf5aab	[mangareader] use 'https://' ... The site now redirects from http://mangareader.net/ to https://mangareader.net/	7 years ago
Mike Fährmann	1d71123f91	[pixiv] update archive IDs and add metadata-fields ... (Pixiv bookmarks actually have their own IDs, comments and tags, independent of the bookmarked image, which makes creating an archive ID a lot easier)	7 years ago
Mike Fährmann	858fdbdb22	[tumblr] improve 'inline' extraction ... 'quote' posts store their HTML content in the 'source' field	7 years ago
Mike Fährmann	5008e105ee	update archive IDs ... ... to behave in a more straightforward way when dealing with bookmarks/favourites/etc. specific IDs are now grouped by their owner, album-id, ... to allow for duplicates when it would be expected.	7 years ago
Mike Fährmann	829ddf4ac1	[sankaku] general improvements ... - simplify regex - unquote search tags - increase default wait-time between HTTP requests - downloading several hundreds of images always resulted in '429 Too Many Requests' eventually - circumvent paging restrictions for unauthenticated users by only using the 'next' parameter - setting 'page' to a constant, low value (or simply omitting it) does the trick	7 years ago
Jad	49463f76bb	support multi-page URL (#79) ... * support multi-page URL * fix * all done. * fix, again	7 years ago
Mike Fährmann	19aefdfde3	[directlink] update test results	7 years ago
Mike Fährmann	74029c50bb	[directlink] unquote metadata fields	7 years ago
Mike Fährmann	8f338347b6	[imagehosts] cleanup ... removed - chronos.to - unable to resolve hostname - coreimg.net - same - imgmaid.net - same - hosturimage.com - everything returns 404 - imageontime.org - redirects to some shady site - imgupload.yt - cloudflare error 522, host down - img4ever.net - read timeout	7 years ago
Mike Fährmann	edfd3d9fc9	[yeet] remove module ... - archive.yeet.net returns a 500 server error - yeet.net moved to yeet.rip, but the archive is gone	7 years ago
Mike Fährmann	8704d850bf	add explicit proxy support (#76) ... - '--proxy' as command-line argument - 'extractor.*.proxy' as config option	7 years ago
Mike Fährmann	367b963d37	[pixiv] fix ugoira extraction ... again (#78) ... Some animations are not available for mobile devices, so we pretend to be a desktop browser when requesting the ugoira page.	7 years ago
Mike Fährmann	b79f1f2ca7	[pixiv] fix ugoira extraction (closes #78)	7 years ago
Mike Fährmann	d122203be1	[mangastream] fix extraction	7 years ago
Mike Fährmann	179bcdd349	adjust archive-ids	7 years ago
Mike Fährmann	3cec533c28	Merge branch 'archive'	7 years ago
Mike Fährmann	20af86b2ea	add more extractor tests ... for mangastream, reddit and imgur	7 years ago
Mike Fährmann	7e0207bcf4	[imgur] strip trailing '?1' from 'ext'	7 years ago
Mike Fährmann	cf147dfee9	[hentai2read] fix manga extraction ... - site changed its HTML structure	7 years ago
Mike Fährmann	f5f2d29f56	[nijie] fix dojin extraction ... - correctly extract artist_id - set extension to "jpg" if it was empty and let filetype checks do the rest	7 years ago
Mike Fährmann	d38bf2f54c	[tumblr] recognize /image/... URLs ... xyz.tumblr.com/image/123 refers to the same images as xyz.tumblr.com/post/123.	7 years ago
Mike Fährmann	5b3c34aa96	use generic chapter-extractor in more modules	7 years ago
Mike Fährmann	7b5ba69951	[hentaihere] ensure consistent extraction results ... sometimes there is a random space before the next <a>	7 years ago
Mike Fährmann	377b78b3c9	[hentai2read] fix manga name extraction	7 years ago
Mike Fährmann	54c36a8a34	[subapics] add chapter- and manga-extractor (#70)	7 years ago
Mike Fährmann	2dd3aeeeae	[komikcast] add chapter- and manga-extractor (#70)	7 years ago
Mike Fährmann	7a412f5c32	implement generic manga-chapter extractor	7 years ago
Mike Fährmann	6a07e38366	implement extractor.add() and .add_module() ... ... as a public and non-hacky way to add (external) extractors to gallery-dl's pool and make them available for extractor.find()	7 years ago
Mike Fährmann	34873dbd90	set 'archive_fmt' values ... These are going to be used to create an unique id for each image.	7 years ago
Mike Fährmann	a34cebc253	[luscious] jump to first image if cover does not link to it	7 years ago
Mike Fährmann	84a52a9256	add DownloadArchive class	7 years ago
Mike Fährmann	619387cbb1	update extractor unittest results	7 years ago
Mike Fährmann	db91cf871c	document message identifiers	7 years ago
Mike Fährmann	0dd48d644f	update test results ... nothing broke, but things got updated or changed	7 years ago
Mike Fährmann	1e93955170	[batoto] remove module ... Site officially shut down on 2018.01.18	7 years ago
Mike Fährmann	76509a6d3c	[imgur] update test results	7 years ago
Mike Fährmann	9fccd7b783	[tumblr] provide fallback URLs (#64) ... Each image now produces 3 URLs: - amazonaws.com _raw (or _1280 for older images) - amazonaws.com _500 - media.tumblr.com (URL returned by API)	7 years ago
Mike Fährmann	9d69401391	initial support for multiple URLs per image	7 years ago
Mike Fährmann	91ed147cef	[oauth] use custom key/secret values during oauth:…	7 years ago
Mike Fährmann	421a9740a3	[tumblr] add 'tumblr:' to force Tumblr extractor (#71)	7 years ago
Mike Fährmann	40d35c87bc	[paheal] add tag- and post-extractors (closes #69)	7 years ago
Mike Fährmann	cc0c2cca57	[reddit] add extractor for reddit-hosted images (closes #68)	7 years ago
Mike Fährmann	f10ffc0839	update extractor blacklist to also allow classes	7 years ago
Mike Fährmann	35e09869d1	[mangapark] fix image URLs and use HTTPS	7 years ago
Mike Fährmann	9a049bdf51	[tumblr] add 'likes' extractor (#65)	7 years ago
Mike Fährmann	67d4462d26	[batoto] rudimentary Cloudflare bypass	7 years ago
Mike Fährmann	29d75fc3fa	[tumblr] add support for OAuth authentication (#65)	7 years ago
Mike Fährmann	4edb25346e	[slideshare] support mobile URLs (closes #67)	7 years ago
Mike Fährmann	e420a28bbc	fix cookie tests	7 years ago
Mike Fährmann	b33efc99a4	[idolcomplex] add support for idol.sankakucomplex.com	7 years ago
Mike Fährmann	75b2e84b6d	[tumblr] use s3.amazonaws.com for image URLs (#64)	7 years ago
Mike Fährmann	5b094328b5	[puremashiro] add chapter- and manga-extractor (closes #66) ... Also adds support for region subtags in language codes (e.g. en-us)	7 years ago
Mike Fährmann	974e73bdbb	[booru] smaller code adjustments	7 years ago
Mike Fährmann	03b8a548cb	[tumblr] change `reblogs` default value to `true` (#61)	7 years ago
Mike Fährmann	d235f68f59	[tumblr] add option to filter reblogged posts (#61) ... Reblogs are ignored by default, but can be included by setting 'extractor.tumblr.reblogs' to 'true'.	7 years ago
Mike Fährmann	a794fffc6d	[batoto] extend chapter-string regex (closes #60) ... Non-numeric chapter indices exist after all ...	7 years ago
Mike Fährmann	1219ebb7f5	[danbooru] use alternate subdomains; support safebooru	7 years ago
Mike Fährmann	9e8a84ab6c	[booru] rewrite using Mixin classes (#59) ... - improved code structure - improved URL patterns - better pagination to work around page limits on - Danbooru - e621 - 3dbooru	7 years ago
Mike Fährmann	0876541e43	[seiga] update tests	7 years ago
Mike Fährmann	88bb0798fd	delay initialization of PathFormat objects ... This allows the DeviantArt group-check to be moved inside the Extractor.items() method which in turn allows for better exception handling. As a new general rule: Never raise exceptions during extractor initialization.	7 years ago
Mike Fährmann	c24e0e70a7	[pixiv] simplify main loop	7 years ago
Mike Fährmann	c1e331edbb	[mangapark] replace manga test	7 years ago
Mike Fährmann	28cd78aae0	[kissmanga] extend chapter-string regex (closes #58)	7 years ago
Mike Fährmann	a3e9b51bea	[imgbox] update test results ... Image URLs of older galleries have been updated to the new format. https://i.imgbox.com/qHhw7lpG.png --> https://images3.imgbox.com/6d/9a/qHhw7lpG_o.png	7 years ago
Mike Fährmann	d0886f411e	[gelbooru] re-enable API use (closes #56) ... Gelbooru's API allows access to all images and is not restricted to the first 20000. This also adds an option to select between API use and manual information extraction in case their API gets disabled again.	7 years ago
Mike Fährmann	8102aae311	[mangahere] support ".cc" TLD and mobile URLs	7 years ago
Mike Fährmann	676602056c	[reddit] unescape output URLs	7 years ago
Mike Fährmann	2eedbaaaf9	[deviantart] use cache to store new refresh_tokens ... The 'refresh_token' set in a user's config file gets used once to get a new 'access_token' and 'refresh_token', which is then stored in gallery-dl's cache and gets used the next time the 'access_token' needs to be refreshed. This means deleting the cache file invalidates the refresh_token- chain and requires the user to re-authenticate.	7 years ago
Mike Fährmann	fc7d165c97	[deviantart] add support for OAuth2 authentication ... Some user galleries [*] require you to be either logged in or authenticated via OAuth2 to access their deviations. [*] e.g. https://polinaegorussia.deviantart.com/gallery/ -------------- known issue: A deviantart 'refresh_token' can only be used once and gets updated whenever it is used to request a new 'access_token', so storing its initial value in a config file and reusing it again and again is not possible.	7 years ago
Mike Fährmann	91c2aed077	[nhentai] fix JSON extraction	7 years ago
Mike Fährmann	444008a14a	[khinsider] use urljoin() to complete page URLs	7 years ago
Mike Fährmann	263741d243	[luscious] update URL pattern (closes #55)	7 years ago
Mike Fährmann	0a9a07a6e1	[slideshare] improve metadata; flake8 ... - added 'views' and 'published' keywords - fixed longer titles and descriptions	7 years ago
Leonardo Taccari	a8d2dde8b2	[slideshare] Add a new extractor for slideshare.net (#54)	7 years ago
Mike Fährmann	19a6ae57b2	[sankaku] add pool extractor	7 years ago
Mike Fährmann	e52f0cc1ed	[sankaku] add post extractor	7 years ago
Mike Fährmann	595593a35e	[sankaku] rewrite ... - better code structure and extensibility - better metadata	7 years ago
Mike Fährmann	a3924d2072	[sankaku] fix swf extraction (closes #52)	7 years ago
Mike Fährmann	291369eab2	various smaller changes/additions	7 years ago
Mike Fährmann	300346ecdf	[mangazuki] remove extractors ... This site has been in "rebuild"-mode for a fairly long time and the current extractor code isn't going to work for the new version either.	7 years ago
Mike Fährmann	d275b1d9a3	[khinsider] fix extraction ... ... again	7 years ago
Mike Fährmann	6b8e3003df	[hentai2read] ensure consistent extraction results	7 years ago
Mike Fährmann	a1980b16f3	[gelbooru] various improvements ... - better metadata for pools - map ratings to s/q/e like other boorus do - skip() support	7 years ago
Mike Fährmann	93482a1f88	implement 'util.advance()'	7 years ago
Mike Fährmann	038e3b3369	[kissmanga] handle "AreYouHuman" redirects (#51)	7 years ago
Mike Fährmann	2b9a783fc7	[khinsider] fix extraction	7 years ago
Mike Fährmann	214972bc9a	[gelbooru] use manual extraction ... ... to compensate for their disabled API. (https://gelbooru.com/index.php?page=forum&s=view&id=3875) This also adds an extractor for image-pools.	7 years ago
Mike Fährmann	55c64cad4b	[khinsider] fix filename extension and test-pattern	7 years ago
Mike Fährmann	b14de6ffc2	[tumblr] small improvements ... - don't transform inline GIF URLs - set 'type' parameter for API calls if there is only one post type selected	7 years ago
Mike Fährmann	9296a26eae	[tumblr] add warning messages	7 years ago
Mike Fährmann	65c1c53eb8	[khinsider] fix extraction	7 years ago
Mike Fährmann	12de658937	[tumblr] add options to control extraction behavior (#48) ... - posts : list of post-types to inspect - inline : scan post bodies for inline images - external: follow external links	7 years ago
Mike Fährmann	077f8c12be	[tumblr] original video URLs + continuous offset	7 years ago
Mike Fährmann	8eb12ebeae	[tumblr] support more post/media types (#48) ... This adds support for audio and video posts (most videos are shared from youtube/instagram which isn't supported -> youtube-dl), as well as link posts and image-search inside of text posts. Most of this is just WIP and will need some sort of improvement and options to enable/disable different media types etc.	7 years ago
Mike Fährmann	b8cdd42cab	[senmanga] fix extraction (again) ... this is basically a re-revert of 2ace5c7	7 years ago
Mike Fährmann	e6814aebe2	add 'extractor.*.user-agent' config option	7 years ago
Mike Fährmann	6913eeaa40	[powermanga] replace manga extractor unit test ... My Hero Academia is gone	7 years ago
Mike Fährmann	7e0d9257a7	[hbrowse] fix manga extraction	7 years ago
Mike Fährmann	3c576d10c0	[seiga] better metadata + 'skip()' support	7 years ago
Mike Fährmann	f72318e593	[seiga] support more than 200 images ... Due to API restrictions and/or missing knowledge about and documentation of API usage, it was only possible to retrieve the latest 200 images of a niconico seiga user with said API. The new approach manually visits each HTML page and gets its information from there.	7 years ago
Mike Fährmann	baf8094868	improve Extractor.request()'s retry behavior	7 years ago
Mike Fährmann	7e7b64162b	[batoto] handle error 10031	7 years ago
Mike Fährmann	92027f67f9	use consistent names for URL constants ... root := <scheme>://<host> base_url := <root>/<common path>	7 years ago
Mike Fährmann	69cbc0619f	[mangastream] fix 'next-page' URLs (fixes #49)	7 years ago