4409d00141
embed error messages in StopExtraction exceptions
5 years ago
22bac14452
[pixiv] match '/artworks/' URLs
5 years ago
f9285f99e6
[pixiv] fix authentication
5 years ago
32447d0d24
[pixiv] simplify default filename format
...
(#366 )
5 years ago
ad3ac02fbc
[pixiv] update metadata entries ( #366 )
...
- change 'num' to a simple enumerating integer
- change default filename format
- provide content of the old 'num' field as 'suffix'
- add 'filename' for ugoira
5 years ago
64b2935d8e
[pixiv] provide 'filename' and change default filename format
...
to '{filename}.{extension}' (closes #366 )
5 years ago
fdec59f8e2
replace extractor.request() 'expect' argument
...
with
- 'fatal': allow 4xx status codes
- 'notfound': raise NotFoundError on 404
5 years ago
b0e85a42e3
apply workaround from 4736912 in parse_datetime() itself
5 years ago
4736912d4e
[pixiv] work around strptime limitations in Python < 3.7
...
"%z" doesn't allow a colon separator in older Python versions:
- "+0900" is OK
- "+09:00" raises an exception
5 years ago
80fdb11508
[pixiv] add 'date' metadata field ( closes #248 )
5 years ago
a2af2d2965
adjust cache maxage values
6 years ago
4b1880fa5e
propagate 'match' to base extractor constructor
6 years ago
6284731107
simplify extractor constants
...
- single strings for URL patterns
- tuples instead of lists for 'directory_fmt' and 'test'
- single-tuple tests where applicable
6 years ago
34bab080ae
rewrite URL patterns to use only 1 per extractor
6 years ago
7471933d5f
use extractor.request for all other API calls
...
- deviantart
- pawoo
- pixiv
- reddit
6 years ago
f6bf66f72c
[pixiv] create directory for each "work" item ( #136 )
6 years ago
8faf03ed84
[pixiv] use refresh_token based authentication
...
The first login will still use username and password, but everything
afterwards will use the refresh_token obtained from that.
This will prevent pixiv from sending a "New login to pixiv" email every
time a new access_token is requested.
6 years ago
f4df6c2396
[pixiv] remove 'type' and 'page' query parameter handling
...
The "new and improved" /member_illust.php and /bookmark.php listings
don't quite work with how things were.
6 years ago
b8c97d2295
use 'extractor.request()' for more HTTP requests
6 years ago
91340d9d27
[pixiv] fix ugoira test
6 years ago
eb7a1f3b98
[pixiv] rework ugoira handling
...
Frame information now gets attached to the ZIP file's keyword dict
instead of being written to a separate text file.
6 years ago
a62bd81e9b
[pixiv] fix filter for 'type=all'
6 years ago
1ff626db97
[pixiv] improve bookmark extraction
...
- combine 'favorite' and 'bookmark' extractors
- it is now one extractor class, but its subcategory still
distinguishes between your own bookmarks ('bookmark') and other
user's bookmarks ('favorite') like before
- allow filtering by bookmark tags and public/private bookmarks
- fix pagination for bookmark results
6 years ago
0a1863fce3
[pixiv] respect more query parameters for user URLs
...
The API endpoint responsible for user illustrations does not
provide sufficient filter capabilities* to match the actual
website, so we are spinning our own filters.
Respected parameters are
'type': illust, manga, ugoira
'tag' : any image tag (this was already supported)
'p' : the page to start on
*
- API can filter for illustrations and manga, but not for ugoira.
- 'offset' is applied before filtering
- no 'tag' filter
6 years ago
b8e53b8c6b
[pixiv] move query parsing out of constructor
...
better exception handling, among other things
6 years ago
909d105ae6
[pixiv] add extractor for illusts from followed users
6 years ago
fe69d01083
[pixiv] add extractor for search results
6 years ago
247f785af1
[pixiv] use App API
...
Transitioning to the App API breaks favorites archive IDs (there is
no longer any bookmark ID information), but the favorites API endpoint
of the public API was gone anyways ...
6 years ago
0e3883303f
[pixiv] implement AppAPI wrapper
6 years ago
f9884e2338
[pixiv] update URL pattern
...
add support for 'https://www.pixiv.net/user/ <id>'
7 years ago
1d71123f91
[pixiv] update archive IDs and add metadata-fields
...
(Pixiv bookmarks actually have their own IDs, comments and tags,
independent of the bookmarked image, which makes creating an
archive ID a lot easier)
7 years ago
367b963d37
[pixiv] fix ugoira extraction ... again ( #78 )
...
Some animations are not available for mobile devices, so we
pretend to be a desktop browser when requesting the ugoira page.
7 years ago
b79f1f2ca7
[pixiv] fix ugoira extraction ( closes #78 )
7 years ago
34873dbd90
set 'archive_fmt' values
...
These are going to be used to create an unique id for each image.
7 years ago
c24e0e70a7
[pixiv] simplify main loop
7 years ago
4d8387f93b
[pixiv] support mobile URLs ( https://touch.pixiv.net/ )
7 years ago
b319f4bab3
smaller code and text changes
7 years ago
ad4580800c
[pixiv] add support for more URL patterns
...
- https://www.pixiv.net/mypage.php#id=USERID
- https://www.pixiv.net/#id=USERID
7 years ago
0dedbe759c
enable '--chapter-filter'
...
The same filter infrastructure that can be applied to image URLS now
also works for manga chapters and other delegated URLs.
TODO: actually provide any metadata (currently supported is only
deviantart and imagefap).
7 years ago
47bcf53ec1
implement support for additional unit test result types
...
- "pattern" matches all resulting URLs against the given regex
- "count" allows to specify the amount of returned URLs
7 years ago
f7cdfd4c25
add a simplified version of 'parse_qs'
...
This version only returns a dict of plain string to string key-value
pairs and ignores multiple values for the same query variable.
7 years ago
e61a3a56d1
[hentai2read] fix and update keywords
...
Added the "author" keyword and changed the name of a few others to be
consistent with other manga/chapter extractors.
7 years ago
c45770331a
use 'str.partition()'
...
The (r)partition method is always faster then split() or any other
method that has been replaced in this commit.
7 years ago
017a72f448
[pixiv] improve input validation
7 years ago
dcf42c5e89
[pixiv] add extractor for ranking lists
7 years ago
7aa9fa796a
code cleanup and fixes
7 years ago
f08af03845
Merge branch 'cookies'
7 years ago
b37a62501b
[pixiv] unquote tags
7 years ago
58e95a7487
share extractor and downloader sessions
...
There was never any "good" reason for the strict separation
between extractors and downloaders. This change allows for
reduced resource usage (probably unnoticeable) and less lines
of code at the "cost" of tighter coupling.
7 years ago
44d98e562b
[pixiv] support pixiv.me URLs ( #23 )
7 years ago
Mike Fährmann