df2b3c6888
restore OAuth2 authentication error messages
5 years ago
6779512fc7
[nozomi] add post and tag extractors ( #388 )
5 years ago
6abe5f5bbb
[patreon] fix pagination ( #444 )
...
The Patreon-provided URLs for the next set of posts aren't
always complete, i.e. they can be missing their scheme and
the subsequent double slash: "www.patreon.com/…"
5 years ago
ff1e4a86aa
release version 1.10.6
5 years ago
d4ffd6c952
[yaplog] improve metadata extraction ( #443 )
...
- provide a fallback if there is no numerical image ID
- add a 'filename' field
- convert 'date' to an actual datetime object
5 years ago
15af2f8464
[hitomi] fallback to /reader/ page if main page returns 404
...
Some galleries return a 404: Not Found error when trying to access
them through the main gallery URL, but their content is still
available on the respective /reader/ page.
5 years ago
8af59a4bba
fix & update docs
...
- update Requests links
- add example for --exec
- set '-dev' version
5 years ago
dc6ad81e2e
[yaplog] prevent crash on empty posts ( #443 )
5 years ago
94eb7c6cad
[deviantart] fix sta.sh extraction (436)
5 years ago
1032cfa34b
[downloader:http] extend mimetype map with archive formats
5 years ago
27b5b2497e
[deviantart] fix download URLs ( #436 )
...
... except for sta.sh content.
Instead of using the old '/api/v1/oauth2/deviation/download' endpoint,
which started delivering URLs to 404 pages a while ago,
it is also possible to get a download URL from the relatively new
'/_napi/da-browse/shared_api/deviation/extended_fetch' endpoint
used by DeviantArt's Eclipse interface.
The current strategy is therefore:
- Iterate over deviations using the OAuth2 API
- Fetch original download URLs with the new NAPI/Shared API
5 years ago
93aac8dfea
[yaplog] fix incomplete image URLs ( #443 )
5 years ago
a782b009b8
[yaplog] match blog names with '-' ( #443 )
5 years ago
cf5e716b9d
[hitomi] fix image URLs
5 years ago
ad81c07204
[postprocessor] match logger names of downloader modules
...
The logger name for a postprocessor object got changed to
"postprocessor.<module-name>" instead of just
"postprocessor"
5 years ago
03bc8adfc7
[postprocessor:exec] run after file moved to target location
...
(#421 )
5 years ago
35958bebd4
[postprocessor:exec] fix filename quoting on Windows ( #421 )
5 years ago
b06c372e4d
[postprocessor:exec] improve; add command-line option ( #421 )
5 years ago
5a54efa025
[xhamster] unescape 'title' and 'description'
5 years ago
1b9bf4fc6e
[behance] fix 'tags' extraction
5 years ago
bb97e87989
[komikcast] ignore banner image
5 years ago
0ff90a3f7d
[gfycat] include title in default filenames ( closes #434 )
5 years ago
fabdc3b0c6
release version 1.10.5
5 years ago
de4e2029d1
[nsfwalbum] update test album
...
the old one is no longer available
5 years ago
1faec285d1
[nijie] further improvements ( closes #423 )
...
- provide a 'user_name' metadata field
- usually the same as 'artist_id', except for favorite downloads
- extract the whole description text and properly escape HTML entities
- fixed an issue with titles or tags containing double quotes
5 years ago
6d0a533d68
[reddit] respect 'comments:0' for single submissions ( #429 )
5 years ago
803d8f814e
[oauth] update scope for reddit tokens ( #428 )
...
'/user/<username>/...' requires the 'history' scope to be accessible
(https://www.reddit.com/dev/api/#GET_user_{username}_{where} )
5 years ago
46ba173ded
[reddit] fix documentation inconsistencies ( closes #429 )
...
- Require 'reddit.comments' to be a number and convert it to an
integer to be extra sure
- Link to the README's OAuth section were appropriate
5 years ago
20eb6c401f
[nijie] improvements and fixes ( #423 )
...
- ignore unavailable image pages
- more metadata fields: artist_name, date, tags
- rename 'index' to 'num'
- improved code structure
5 years ago
d1ea08c67d
[weibo] fixes and improvements
...
- ignore unavailable videos (fixes #427 )
- handle empty 'geo' fields
- consistent metadata fields for images and videos
5 years ago
38d97f3da6
[deviantart] add debug message about API credentials ( #424 )
5 years ago
80c2104fb5
[deviantart] fix 429 handling if 'fatal' is False ( closes #424 )
5 years ago
913460240d
[reddit] fix 'extractor.blacklist()' arguments
...
The second argument must support 'append()'.
5 years ago
22bac14452
[pixiv] match '/artworks/' URLs
5 years ago
66cac207ac
[twitter] match and use 'i/web' status URLs
5 years ago
946f2751e2
[reddit] add 'user' extractor ( closes #350 )
5 years ago
c14abb9fb8
[reddit] improve URL parameter handling for subreddit links
5 years ago
ee8b654464
[instagram] implement 'highlights' option ( closes #329 )
5 years ago
f63c3097a9
[instagram] rework some code paths
...
- combine fetching an HTML page and extracting its 'shared_data'
- move 'shared_data' and field access info out of '_extract_page()'
- introduce a '_request_graphql()' method
5 years ago
4330133114
[imgur] add 'favorite' extractor ( closes #420 )
...
… and use a newer site-internal API endpoint for user posts
5 years ago
ee5e20221f
[imgth] fix image URLs
5 years ago
b63b126808
[hentaicafe] extend URL pattern
5 years ago
d780f0357e
[imgur] add user extractor
5 years ago
11ea689013
[simplyhentai] fix image and video URLs
5 years ago
15632a1570
[tsumino] fix extraction
5 years ago
d92802fd37
[luscious] fix detection of unavailable galleries
5 years ago
f99da2b866
[imgbb] detect invalid album and user profile links
...
and update test results, since the old album got deleted
5 years ago
01bc7adadc
[deviantart] improve journal detection ( #419 )
...
Some journal-like posts are not reported to be journals (isJournal
is set to False), even though they have a textContent field.
https://www.deviantart.com/gliitchlord/art/brashstrokes-812942668
5 years ago
776e9e073f
close archive on job completion ( #417 )
5 years ago
5ac9732adc
call 'sys.exit()' on Ctrl+c
5 years ago
Mike Fährmann