5a98bcec3a
[deviantart] improve folder name matching ( fixes #1451 )
3 years ago
457abf0e71
[deviantart] fix pagination for Eclipse results ( fixes #1444 )
...
- don't crash on missing keys
- use fallback for invalid 'nextOffset' values
3 years ago
214ecf62ce
[deviantart] fix arguments for search/popular results ( #1408 )
4 years ago
139fb84108
[deviantart] fix username for 'watch' results ( #794 )
...
before it'd use "/" as username
4 years ago
91c2e15da9
[deviantart] add support for posts from watched users ( #794 )
4 years ago
03c20d8c8e
[deviantart] update 'watch' URL pattern ( #794 )
4 years ago
dc23cfd684
[deviantart] use fallback for /intermediary/ URLs
...
instead of checking availability with HEAD requests
4 years ago
b0438c8f99
Revert "[deviantart] extend 'extra' option"
...
This reverts commit
5ad2b9c82b5c32a7bf5883f465faca#1387 , #1356 )
4 years ago
69ca4e29f1
[deviantart] add 'watch' extractor ( #794 )
4 years ago
83f465faca
[deviantart] refactor 'extra' ( #1356 )
...
- change its expected type to string
- let users specify a list of sources (stash, posts) or 'all'
4 years ago
5c32a7bf58
[deviantart] allow selecting source for 'extra' ( #1356 )
...
Setting 'extra' to "stash" or "deviations" will only download embedded
sta.sh content or deviations. 'true' still downloads both.
4 years ago
5ad2b9c82b
[deviantart] extend 'extra' option
...
also download from embedded DeviantArt posts
4 years ago
23be48427c
[deviantart] fix 'folders' option ( closes #1302 )
...
don't assume parent folders are listed before their children
4 years ago
c6cc86d7d0
[deviantart] update parameters for '/browse/popular'
...
- limit results to 50 when also querying metadata (fixes #1267 )
- remove deprecated 'category_path' parameter
4 years ago
c26de0929d
[deviantart] provide 'extension' for original file downloads
...
(#1272 )
4 years ago
193dca2ce1
update extractor test results
4 years ago
e2d4ca4955
[deviantart] improve '--range' for favorites ( closes #1226 )
4 years ago
91db8df1c7
[deviantart] add 'index_base36' metadata field ( closes #1099 )
...
This is the same ID as found in 'filename' without the 'd' in front,
which is just 'index' encoded in base36.
4 years ago
968d3e8465
remove '&' from URL patterns
...
'/?&#' -> '/?#' and '?&#' -> '?#'
According to https://www.ietf.org/rfc/rfc3986.txt , URLs are
"organized hierarchically" by using "the slash ("/"), question
mark ("?"), and number sign ("#") characters to delimit components"
4 years ago
3ebb174f2c
add missing extractor info when spawning new ones ( fixes #1051 )
...
Not having this information causes the blacklist/whitelist logic to
trigger and prevents things from functioning as intended when using
default settings.
Fixes issues for 8muses, deviantart, exhentai, and mangoxo.
4 years ago
136df52d1f
[deviantart] support watchers-only/paid deviations ( #995 )
4 years ago
f6fd449b59
reduce wait time growth rate from exponential to linear
...
Waiting for 2**N seconds after each error grows too fast.
Simply waiting N seconds seems far more reasonable.
4 years ago
c6c06c41f6
[deviantart] don't add journal text to description ( #712 )
4 years ago
41d03160ff
[deviantart] also search journals for sta.sh links ( #712 )
...
when 'extra' is enabled
4 years ago
dfcf2a2c91
write OAuth token to cache by default ( #616 )
4 years ago
6294e2c540
add 'text.ensure_http_scheme()'
4 years ago
65b1cb7acd
[deviantart] use private access tokens for Journals ( fixes #738 )
4 years ago
999efec5cc
[deviantart] limit API wait times to 2**9=512 seconds ( #721 )
4 years ago
6386ee54e1
[deviantart] add extractor info to 'following' results
4 years ago
bae1e8ed12
[deviantart] fix JPEG quality replacement pattern
...
'q_\d+' would sometimes also replace something in the 'token' query
parameter, invalidating the URL.
5 years ago
d02f7c1118
improve Extractor.wait()
...
- allow 'until' to be a datetime object
- do "time calculations" with UTC timestamps
- set a default 'reason'
5 years ago
f9a590f92b
[deviantart] apply HTTP request limits in more places
...
"Request blocked" can also happen on sta.sh and for *any* HTTP
request directed at deviantart.com
5 years ago
ff7c0b7eff
[deviantart] handle "Request blocked" errors ( #655 )
...
- add a 2 second wait time between requests to deviantart.com
- catch 403 "Request blocked" errors and wait for 3 minutes until
retrying
5 years ago
c874684f05
[deviantart] retrieve *all* download URLs through OAuth API
...
'/extended_fetch' as well as Deviation webpages now again contain
Deviation UUIDs needed to grab Deviation info through the OAuth API,
meaning cookies are no longer necessary to grab original files.
The only instance were cookies are still needed are scraps marked as
"mature", since those entries are hidden for public users.
(#655 , #657 , #660 )
5 years ago
5c27b25a8f
[deviantart] improve sta.sh extraction
...
Extract all sta.sh items in a single extractor run.
Don't spawn a new StashExtractor for each individual sta.sh item to
preserve the current requests.Session and its opened TCP connections.
5 years ago
e2fc4eaa6f
[deviantart] detect stash folders ( fixes #659 )
5 years ago
6f911aeb1c
[deviantart] add error message for cloudFront blocks ( #655 )
5 years ago
1b82d36ab2
[deviantart] handle decode errors for extended_fetch results ( #655 )
...
This isn't going to solve the underlying problem, but it should at
least provide the server response when those errors happen.
5 years ago
913b8333cc
write DeviantArt refresh-tokens to cache ( #616 )
...
Writing the token is currently disabled by default and must be
enabled with 'extractor.oauth.cache'.
'extractor.deviantart.refresh-token' must be set to '"cache"'
to use the cached token.
5 years ago
64bdec8430
[deviantart] check availability of intermediary URLs ( fixes #609 )
5 years ago
ec36df4851
[deviantart] fix video extraction from 'extended_fetch' results
...
DeviantArt is now serving videos from wixmp servers (1), instead of
the former film00.deviantart.com (2), even though those URLS are still
functional.
They seem to also have re-encoded those videos. The 10 MB 1080p video
from (2) is now only available in 720p at ~20 MB (with a higher
bitrate, but still …). Other videos are still available in 1080p, but
not this one for some reason.
(Changing the '720p' in (1) to '1080p' doesn't work.)
(1) https://wixmp-ed30a86b8c4ca887773594c2.wixmp.com/v/mp4/9feaa2c9-1baf-4fc2-84f7-f3384b34cefe/d5gxnb5-282a2e9a-b552-40ff-8542-b3c5eed823f5.720p.a837d7cec12c41be8ca2ee53152cea3a.mp4
(2) https://film00.deviantart.net/4c1d/v/mp4/2012/279/d/1/_video____brushes_i_use_in_paint_tool_sai_by_chi_u-d5gxnb5.mp4
5 years ago
48be2266ed
[deviantart] better error message for 'extended_fetch' ( #585 )
5 years ago
f8e137d6b4
[deviantart] show warning about private deviations only once
...
… per call to '_pagination()'
5 years ago
939fec8ecd
[deviantart] match new search/popular URLs ( closes #538 )
5 years ago
09cc88b715
[deviantart] match '/favourites/all' URLs ( closes #555 )
5 years ago
ce54b8c04c
let extractors opt-out of cookie option usage
...
useful to avoid sending unnecessary cookies when all authentication
is done through OAuth tokens
5 years ago
b347bf68c7
[deviantart] add extractor for followed users ( #515 )
5 years ago
ab17ea9632
[deviantart] only print warning if 'original' is enabled
5 years ago
c8e99e3b3b
[deviantart] fix crash on missing "token" field ( #505 )
5 years ago
6ed2c7823c
[deviantart] disable original downloads if no cookies set
...
For 'deviation' and 'scraps' extractors only, since original file
downloads for those two will always fail with a 404 Not Found
when not logged in.
5 years ago
Mike Fährmann