e5735361ed
[exhentai] add 'metadata' option ( #1325 )
...
to select between gallery metadata from 'api' or 'html'
4 years ago
8f095a0980
[exhentai] extract more metadata from gallery pages ( #1325 )
4 years ago
9b2e5f72d6
[exhentai] update image URL parsing ( #1094 )
4 years ago
3ebb174f2c
add missing extractor info when spawning new ones ( fixes #1051 )
...
Not having this information causes the blacklist/whitelist logic to
trigger and prevents things from functioning as intended when using
default settings.
Fixes issues for 8muses, deviantart, exhentai, and mangoxo.
4 years ago
da87a5fb7e
[exhentai] fix accessing config before main constructor
...
bug introduced with 055c32e0
4 years ago
a0d916ed41
[exhentai] update wait time before original image download ( #978 )
...
depend on 'wait-max', don't use a hard-coded value
4 years ago
0f55b8e80a
[exhentai] fix type check from dbbbb21 ( #940 )
...
'bool' is a subclass of 'int', and therefore
'isinstance(self.limits, int)' also returns True when
'self.limits' has a boolean value
4 years ago
dbbbb21180
[exhentai] add ability to specify custom image limit ( #940 )
4 years ago
cd9de613a2
[exhentai] adjust image limit costs ( #940 )
...
Each original file costs 10 points per 10^6 bytes,
not 10 per 2^20 == 1048576 bytes.
4 years ago
ecaecc4064
[exhentai] add 'domain' option ( #897 )
4 years ago
6b373cb7e2
[exhentai] restrict default directory name length ( #545 )
5 years ago
4409d00141
embed error messages in StopExtraction exceptions
5 years ago
1848788970
update test results etc
5 years ago
9ae58a6b3e
[exhentai] update image limit checks
...
- adjust cost of original images
- delay limit initialization until gallery and first image page have
been requested and all cookies are available
5 years ago
fa60109e97
[exhentai] don't use e-hentai.org for exhentai URLs
5 years ago
beb4fab2e6
[exhentai] improve limit and error handling ( #360 )
...
- check image limit before opening the first gallery or image page
- prevent any further exhentai extractors from running after the image
limit has been reached
5 years ago
81b35ed3cb
[exhentai] catch more error states ( #356 , #360 )
...
- warn on MPV-enabled galleries
- catch parsing errors for gallery pages and image info
- write page content to debug output
5 years ago
6ce22f606b
[exhentai] update login procedure and tests
...
Logging in now follows the natural login flow that also happens in a
browser more closely and collects more cookies than just ipb_member_id
and ipb_pass_hash.
Test URLs have been updated and now point to the e-hentai.org domain.
5 years ago
dc73d02d87
[exhentai] always use e-hentai.org as domain + set nw cookie
5 years ago
fdec59f8e2
replace extractor.request() 'expect' argument
...
with
- 'fatal': allow 4xx status codes
- 'notfound': raise NotFoundError on 404
5 years ago
1c36e65e9b
[exhentai] choose site version depending on input URL ( #278 )
...
Use e-hentai.org as root and cookiedomain if the input URL is from
e-hentai (or g.e-hentai), use exhentai.org otherwise.
5 years ago
1f7fa9dc8e
[exhentai] update data extraction code
...
- parse 'date' to datetime object
- use 'text.extract_from()'
5 years ago
5398bfbd69
[exhentai] fix search and favorite extraction
...
removes basically all metadata, but that can be compensated for with the
right search query. writing "parsers" for all 4 possible views that have
been introduced in the latest changes is too much of a hassle ...
6 years ago
a2af2d2965
adjust cache maxage values
6 years ago
5530871b5a
change results of text.nameext_from_url()
...
Instead of getting a complete 'filename' from an URL and splitting that
into 'name' and 'extension', the new approach gets rid of the complete
version and renames 'name' to 'filename'. (Using anything other than
{extension} for a filename extension doesn't really work anyway)
Example: "https://example.org/path/filename.ext "
before:
- filename : filename.ext
- name : filename
- extension: ext
now:
- filename : filename
- extension: ext
6 years ago
61741d7333
provide type information for Queue messages
...
Child extractors are now directly constructed with Extractor.from_url()
if the extractor class is known beforehand, instead of using
extractor.find() and searching through all possible extractor classes.
6 years ago
2e516a1e3e
store the full original URL in Extractor.url
6 years ago
4b1880fa5e
propagate 'match' to base extractor constructor
6 years ago
6284731107
simplify extractor constants
...
- single strings for URL patterns
- tuples instead of lists for 'directory_fmt' and 'test'
- single-tuple tests where applicable
6 years ago
dd358b4564
improve cookie handling during logins
6 years ago
134487ffb0
[exhentai] stop extraction if image limit is exceeded ( #141 )
...
can be turned off with the `exhentai.limits' option
6 years ago
e868fb4393
[exhentai] improve gallery extraction
...
- match image page URLs and extract galleries from that point onward
- add a few more metadata entries: 'parent', 'visible', 'cost'
6 years ago
2ffc105887
[exhentai] extract tag metadata
6 years ago
2801a0d997
[exhentai] skip "Content Warning" page when not logged in
...
(closes #97 )
6 years ago
b8c97d2295
use 'extractor.request()' for more HTTP requests
6 years ago
017188d268
improve extractor.request()
...
Replace the 'fatal' parameter with 'expect', which is a list/range
of HTTP status codes >= 400 that should also be accepted.
6 years ago
7a58151566
fix util.parse_bytes invocations
...
(should be text.parse_bytes)
6 years ago
cc36f88586
rename safe_int to parse_int; move parse_* to text module
7 years ago
64d7c85b55
[exhentai] improve metadata
...
- add 'width', 'height' and 'size' (in bytes) for each image
- change the former 'size' and 'size_units' into 'gallery_size'
7 years ago
52d41c41e7
[exhentai] add extractor for favorited galleries
7 years ago
63cc2599c4
[exhentai] add extractor for search results
7 years ago
34873dbd90
set 'archive_fmt' values
...
These are going to be used to create an unique id for each image.
7 years ago
e6814aebe2
add 'extractor.*.user-agent' config option
7 years ago
9fc1d0c901
implement and use 'util.safe_int()'
...
same as Python's 'int()', except it doesn't raise any exceptions and
accepts a default value
7 years ago
6f30cf4c64
change keyword names to valid Python identifiers
...
This commit mostly replaces all minus-signs ('-') in keyword names with
underscores ('_') to allow them to be used in filter-expressions. For
example 'gallery-id' got renamed to 'gallery_id'.
(It is theoretically possible to access any variable, regardless of its
name, with 'locals()["NAME"]', but that seems a bit too convoluted if
just 'NAME' could be enough)
7 years ago
c0755a4d5e
[exhentai] revert login-method to its old version ( #37 )
...
Additional cookies don't seem to help and have to be manually set
anyway. The older method is more likely to succeed, so I'd rather
use this one.
7 years ago
3ee39ffd93
[exhentai] update login procedure ( #37 )
...
This new version behaves pretty much exactly like a browser would and
caches all cookies sent to it and not just "ipb_member_id" and
"ipb_pass_hash".
7 years ago
2d0dfe9d56
[exhenai] init headers before login and detect sadpanda
...
- also debug-logs html after failed login
- #37
7 years ago
915a0137de
improve 'extractor.request'
...
- add 'fatal' argument
- improve internal logic and flow
- raise known exception on error
- update exception hierarchy
7 years ago
7aa9fa796a
code cleanup and fixes
7 years ago
808f67ba7d
use 'cookiedomain' for cookies set by object-config-values
...
otherwise these cookies would not be picked up by the
_check_cookies() method.
7 years ago
0610ae5000
skip login if cookies are present
7 years ago
58e95a7487
share extractor and downloader sessions
...
There was never any "good" reason for the strict separation
between extractors and downloaders. This change allows for
reduced resource usage (probably unnoticeable) and less lines
of code at the "cost" of tighter coupling.
7 years ago
1dac76fd1c
update extractor docstrings
7 years ago
d3b04076f7
add .netrc support ( #22 )
...
Use the '--netrc' cmdline option or set the 'netrc' config option
to 'true' to enable the use of .netrc authentication data.
The 'machine' names for the .netrc info are the lowercase extractor
names (or categories): batoto, exhentai, nijie, pixiv, seiga.
7 years ago
af56887a47
[exhentai] fall back to e-hentai if no username is given
8 years ago
4b967fa189
implement and use extractor.config() method
8 years ago
b603b592cf
[exhentai] accept "e-hentai.org" URLs ( #11 )
8 years ago
841fd50242
move code into util.py
8 years ago
1d46be545c
add login notifications
8 years ago
e87e6fbc67
change some config keys
...
directory_fmt -> directory
filename_fmt -> filename
download-original -> original
8 years ago
0a6487afe8
[exhentai] fix detection of invalid gallery keys
8 years ago
94e10f249a
code adjustments according to pep8 nr2
8 years ago
4a8d74973c
adjust login methods to a specific style
8 years ago
a849d8f2f7
add a few more tests
8 years ago
ff2a65d5c1
[exhentai] raise proper exception for 'unavailable' galleries
8 years ago
492cb38391
[exhentai] use image-count as stop signal
8 years ago
607f50effb
[exhentai] retry failed api calls
8 years ago
12c99293b6
allow extension by Content-Type for exhentai, seiga, senmanga
8 years ago
56d810c896
update keyword hashes for tests
8 years ago
19c2d4ff6f
remove explicit (sub)category keywords
8 years ago
e3d156078c
[exhentai] rewrite
8 years ago
49a05c32ed
add missing tests
8 years ago
a017cd9e63
[exhentai] metadata consistency
8 years ago
d7e168799d
consistent extractor naming scheme + docstrings
8 years ago
49359d375c
[exhentai] unescape title
8 years ago
95097fac00
[exhentai] transition to https
8 years ago
0e20be5a0a
[exhentai] provide username/password auth
8 years ago
4d56b76aa8
update all other extractors
9 years ago
e9547ebdf3
[exhentai] configurable wait-times
9 years ago
9c63027d2c
[exhentai] fix 'imgkey' handling
9 years ago
4440b40234
[exhentai] better language keywords
9 years ago
c2f0720184
code cleanup to use nameext_from_url
9 years ago
3560935318
[exhentai] fix module dicstring
9 years ago
353ac1e00b
[exhentai] use text.extract_all
9 years ago
b47f3b9d78
[exhentai] reenable extractor
9 years ago
4950cb0ab9
use 'safe_request' to handle connection errors
10 years ago
7588974306
fixed various bugs
...
- forgot "self." before "name_fmt"
- image keys where off by one
10 years ago
deef91eddc
initial commit
10 years ago
Mike Fährmann