c8db2a87e9
fix create_test_data.py script
5 years ago
1b82d36ab2
[deviantart] handle decode errors for extended_fetch results ( #655 )
...
This isn't going to solve the underlying problem, but it should at
least provide the server response when those errors happen.
5 years ago
88ebbadc58
remove dashes from subcategory names in supportedsites.rst
5 years ago
09f2271528
[35photo] add 'tag' extractor
5 years ago
77fda8190c
[35photo] simplify/remove tests for the 'genre' extractor
...
There is still a nice genre overview page (https://35photo.pro/genre/ )
but the individual sub-pages don't list photos anymore
5 years ago
4bc161ca0f
prevent crash when sys.stdout and co. are None ( #653 )
5 years ago
d47d0f757c
[travis] allow 'results' and 'snap' tests to fail
5 years ago
ce73796eaa
[travis] add flake8 job
5 years ago
fb846c9ee5
[instagram] reduce line lengths and make flake8 happy
5 years ago
ad2efa8509
[e621] derive from Danbooru extractors ( #651 )
...
- use extractor implementations from 'danbooru'
- use "page": "b[ID]" to paginate over results instead of
"tags": "id:<[ID]", avoiding infinite loops with certain
post orders
- bump User-Agent version
5 years ago
9b39e1cd7e
[e621] fix bug in API rate limiting ( #651 )
5 years ago
b607d0ad7f
[twitter] fix typo in 'x-twitter-auth-type' header ( #625 )
5 years ago
9159cb8fb3
remove trailing dots and spaces from directory names ( #647 )
5 years ago
2c3b9e1450
[nozomi] support multiple images per post ( #646 )
...
This changes the default filename format as well as archive IDs,
since those assumed that each post would only have one image.
5 years ago
c606d0c854
[instagram] update pattern for user profile URLs
...
Allow for query parameters and fragments,
for example https://www.instagram.com/instagram/?hl=en
5 years ago
2530db3f4d
[mangadex] transform 'date' timestamps to datetime objects
5 years ago
ae2a33243b
[newgrounds] catch general Exceptions
5 years ago
32e36d8f02
[sexcom] replace tests
5 years ago
33b42dc847
[nozomi] sort search results ( fixes #646 )
5 years ago
eaa60a438b
[piczel] fix extraction
...
- manually filter by folder_id
- extract data for single posts from embedded JSON, since the
'/api/gallery/image/<id>' endpoint is no longer available
5 years ago
5bcc7184c9
[danbooru][e621] increase page limits
5 years ago
90d15e3682
[instagram] use 'itertools.chain()'
5 years ago
160328d21c
[instagram] Add support for user's saved medias ( #644 )
...
* [instagram] Gracefully handle possible 'HttpErrorPage' in _extract_page()
`HttpErrorPage' is returned in shared_data at least when not authenticated or
when trying to fetch other users saved medias
(i.e. `instagram.com/<user>/saved/').
Gracefully handle it by returning nothing.
* [instagram] Add support for user's saved medias
(Please note that this need the user to be authenticated and they can
only see their saved media (not other users ones).)
Close #643 .
* [instagram] Bump copyright year
5 years ago
e0b0e8d62a
release version 1.13.2
5 years ago
5b676ea59d
[e621] document username & password support ( #640 )
5 years ago
1b3ba86110
improve lists in man pages
5 years ago
d3482ace7f
[furaffinity] extract more metadata
...
- views
- favorites
- comments
- rating
- fa_category (since 'category' is already in use)
- theme
- species
- gender
- width
- height
5 years ago
f6c5edb76b
pre-compile regex pattern for remove_html() and split_html()
5 years ago
fdd2dd5136
[kabeuchi] add 'user' extractor ( closes #561 )
5 years ago
59edcdc822
[hitomi] restore metadata fields from before f33b13a
...
... and add a 'metadata' option to disable
visiting the gallery page and extracting data from it
if this is not needed.
5 years ago
2d5703c493
[twitter] use a simpler data structure to store cookies in cache
...
Use a dict with name-value pairs instead of an entire
RequestsCookieJar object.
5 years ago
87d4f83597
[newgrounds] make post extraction nonfatal
5 years ago
823fbeaae6
[newgrounds] add 'favorite' extractor ( #394 )
5 years ago
a45fbc38ea
[pixiv] implement 'avatar' option ( #595 , #623 )
5 years ago
a63a376ad2
[mangoxo] fix login
5 years ago
ebc70e87ce
[e621] update to new interface / API endpoints ( closes #635 )
5 years ago
d1cf7ccdb3
[instagram] add 'post_shortcode' metadata field ( #525 )
5 years ago
402025c3c3
fix some build issues
...
- use 'os.name' to decide between Windows/Linux build
- don't check Windows executable version number,
since Wine fails to run the executable and causes release.sh to stop
5 years ago
32df8d06fe
[twitter] add 'bookmark' extractor ( closes #625 )
5 years ago
3fb41c34c8
[bcy] reduce requests to '/item/detail/<id>' ( #613 )
...
The former implementation would try to use the embedded data from
'/item/detail/' pages for every post, even if that wasn't really
necessary.
This commit also fixes some issues with posts only visible to
logged in users.
5 years ago
f33b13aacf
[hitomi] simplify metadata extraction
...
Use the data from https://ltn.hitomi.la/galleries/ <id>.js for both
image URLs and metadata and ignore any gallery or reader pages.
This removes 'artist', 'characters', 'group', and 'parody' metadata
fields since this information is, as for now, only available in
gallery pages.
5 years ago
115fd2c6f2
"fix" incomplete MIME types ( #632 )
...
e-/exhentai's original image downloads currently send
incomplete/invalid Content-Type headers, "jpg" instead
of "image/jpg" etc, since the last update.
(https://forums.e-hentai.org/index.php?showtopic=236113 )
This change prepends any Content-Type value missing a
media type specification with "image/", transforming it
into a valid MIME type.
(A global solution to a local problem, but it shouldn't
cause any issues anywhere else)
5 years ago
72122eb9b3
release version 1.13.1
5 years ago
adcd7cb24a
[downloader:http] add another MIME type for '.rar' files ( #628 )
5 years ago
ce5e2a58fe
[imgbb] update test results
...
Image server domain changed from
https://image.ibb.co/ to https://i.ibb.co/
5 years ago
f117e32910
[danbooru] restore 'popular' functionality
5 years ago
39b48d665b
[hiperdex] use proper name for 'chapter_minor'
5 years ago
8fbbaa54ff
[bcy] fix partial image URLs ( #613 )
...
Images from new posts can have incomplete/partial URLs (1)
without any filename extension when fetching their data from
'/apiv3/user/selfPosts', so now all data gets taken from
'/item/detail/ID' pages.
It is currently unknown how to get the non-watermarked original version
of these images, or if that is possible at all. (2)
Images with a watermark will have their 'filter' metadata field set to
"watermark". For original images this field is an empty string "".
Enabling the 'noop' option will, in addition to the watermarked version,
yield the the '~noop.image' filter version (3),
where 'filter' is set to "noop".
(1) "https://img-bcy-qn.pstatp.com/banciyuan/3ccdff22479c4060aadc86718209b281 "
(2) "https://p1-bcy.byteimg.com/img/banciyuan/3ccdff22479c4060aadc86718209b281~tplv-banciyuan-logo-v3:wqnpnLLlhZLlpKfprZTnjotfCuWNiuasoeWFgyAtIEFDR-eIseWlveiAheekvuWMug==.image "
(3) "https://p1-bcy.byteimg.com/img/banciyuan/3ccdff22479c4060aadc86718209b281~noop.image "
5 years ago
86c00f9e66
[danbooru] move extractor logic from booru.py
5 years ago
1d4a369ea2
update extractor test results
5 years ago
Mike Fährmann