a8d2dde8b2
[slideshare] Add a new extractor for slideshare.net ( #54 )
7 years ago
19a6ae57b2
[sankaku] add pool extractor
7 years ago
e52f0cc1ed
[sankaku] add post extractor
7 years ago
595593a35e
[sankaku] rewrite
...
- better code structure and extensibility
- better metadata
7 years ago
e96e1fea5d
release version 1.1.0
7 years ago
a3924d2072
[sankaku] fix swf extraction ( closes #52 )
7 years ago
ebe9b0a04c
another attempt at downloader retry behavior
...
This commit changes the general behavior from
'Retry on every exception and abort on DownloadError' to
'Only retry on DownloadRetry exceptions and abort on every other one'
The previous version would have retried on several states which
would have no chance of ever succeeding (invalid URLs, etc.)
7 years ago
291369eab2
various smaller changes/additions
7 years ago
4fb6803fa6
add option to sleep before each download
7 years ago
300346ecdf
[mangazuki] remove extractors
...
This site has been in "rebuild"-mode for a fairly long time and the
current extractor code isn't going to work for the new version either.
7 years ago
d275b1d9a3
[khinsider] fix extraction
...
... again
7 years ago
6b8e3003df
[hentai2read] ensure consistent extraction results
7 years ago
a1980b16f3
[gelbooru] various improvements
...
- better metadata for pools
- map ratings to s/q/e like other boorus do
- skip() support
7 years ago
93482a1f88
implement 'util.advance()'
7 years ago
0e5057b15d
remove deprecated options
7 years ago
8f518e03f8
add options to set maximum download rate
...
- -r/--limit-rate as cmdline option
- downloader.http.rate as config option
This implementation very roughly uses the idea of the token bucket
algorithm [1] and mostly uses Wget's approach [2] as inspiration.
[1] https://en.wikipedia.org/wiki/Token_bucket
[2] http://git.savannah.gnu.org/cgit/wget.git/tree/src/retr.c?h=v1.19.2&id=ba6b44f6745b14dce414761a8e4b35d31b176bba#n111
7 years ago
a718c6c6cd
implement 'util.parse_bytes()'
7 years ago
038e3b3369
[kissmanga] handle "AreYouHuman" redirects ( #51 )
7 years ago
2b9a783fc7
[khinsider] fix extraction
7 years ago
3dc1169736
use own mapping before relying on the 'mimetypes' module
7 years ago
214972bc9a
[gelbooru] use manual extraction
...
... to compensate for their disabled API.
(https://gelbooru.com/index.php?page=forum&s=view&id=3875 )
This also adds an extractor for image-pools.
7 years ago
55c64cad4b
[khinsider] fix filename extension and test-pattern
7 years ago
c0bcf8e343
release version 1.0.2
7 years ago
28bf25f37d
update CHANGELOG
7 years ago
b14de6ffc2
[tumblr] small improvements
...
- don't transform inline GIF URLs
- set 'type' parameter for API calls if there is only
one post type selected
7 years ago
9296a26eae
[tumblr] add warning messages
7 years ago
65c1c53eb8
[khinsider] fix extraction
7 years ago
12de658937
[tumblr] add options to control extraction behavior ( #48 )
...
- posts : list of post-types to inspect
- inline : scan post bodies for inline images
- external: follow external links
7 years ago
077f8c12be
[tumblr] original video URLs + continuous offset
7 years ago
8eb12ebeae
[tumblr] support more post/media types ( #48 )
...
This adds support for audio and video posts (most videos are shared
from youtube/instagram which isn't supported -> youtube-dl),
as well as link posts and image-search inside of text posts.
Most of this is just WIP and will need some sort of improvement
and options to enable/disable different media types etc.
7 years ago
6c9da67581
apply selection options (filter, range) when using '-j'
7 years ago
b8cdd42cab
[senmanga] fix extraction (again)
...
this is basically a re-revert of 2ace5c7
7 years ago
e6814aebe2
add 'extractor.*.user-agent' config option
7 years ago
6913eeaa40
[powermanga] replace manga extractor unit test
...
My Hero Academia is gone
7 years ago
7e0d9257a7
[hbrowse] fix manga extraction
7 years ago
3c576d10c0
[seiga] better metadata + 'skip()' support
7 years ago
f72318e593
[seiga] support more than 200 images
...
Due to API restrictions and/or missing knowledge about and
documentation of API usage, it was only possible to retrieve the
latest 200 images of a niconico seiga user with said API.
The new approach manually visits each HTML page and gets its
information from there.
7 years ago
baf8094868
improve Extractor.request()'s retry behavior
7 years ago
2457b71633
skip tests on 5xx status codes
7 years ago
7e7b64162b
[batoto] handle error 10031
7 years ago
79bcaa8726
improve downloader retry behavior
...
- only retry download on 5xx and 429 status codes
- immediately fail on 4xx status codes
7 years ago
5ee8ca0319
release version 1.0.1
7 years ago
2c1adda784
update release.sh script
...
- update CHANGELOG on new releases
- change issue references to actual links
- replace "Unreleased" with new version and date
- fix filenames of old Windows executables
[no ci]
7 years ago
e913e5ec77
add a CHANGELOG
...
This is basically just a copy&paste from the Releases page, but it
has the benefits of (1) better visibility and (2) "forcing" me to
write a changelog section before releasing a new version and not
several days after.
7 years ago
42e948584d
fix downloader error handling
...
RequestException being a subclass of OSError caused all exceptions
during file downloads to be ignored/re-raised.
7 years ago
92027f67f9
use consistent names for URL constants
...
root := <scheme>://<host>
base_url := <root>/<common path>
7 years ago
69cbc0619f
[mangastream] fix 'next-page' URLs ( fixes #49 )
7 years ago
980fd3616d
[tumblr] use API v2 ( #48 )
7 years ago
d6bed9f36f
[tumblr] prevent premature exit to get all images ( fixes #48 )
7 years ago
305da540c3
[mangahere] fix metadata extraction
7 years ago
Leonardo Taccari