c489aecb3e
[deviantart] add support for fxdeviantart.com URLs
...
fxdeviantart.com is a service that fixes embeds on Discord, similar to
fxtwitter.com
2 years ago
6b03506655
[deviantart] allow searching when not logged in
2 years ago
3fa456d989
[deviantart] remove mature scraps warning ( #3691 )
...
warn about private deviations
when paginating over eclipse results
2 years ago
51301e0c31
replace remaining time.sleep() calls
...
with Extractor.sleep() or request_interval
2 years ago
6ed4309aba
[deviantart] add 'gallery-search' extractor ( #1695 )
2 years ago
e1df7f73b1
[deviantart] add 'search' extractor
...
(#538 , #1264 , #2954 , #2970 , #3577 )
Requires login to fetch any results, since the API endpoint raises an
error for not logged in requests.
TODO: parse HTML search results
2 years ago
b4899c266f
merge #3656 : [deviantart] fix crash when handling deleted deviations in status updates
2 years ago
725baedad3
[deviantart] use '/collections/all' endpoint for favorites
...
(#3666 ,#3668)
2 years ago
f4e211356d
[deviantart] slight refactor
2 years ago
a74114ef7a
[deviantart] fix crash when handling deleted deviations
...
in status updates
2 years ago
7e2fd2e573
merge #3560 : [deviantart] add support for /deviation/ and fav.me URLs
2 years ago
c90b4ea8d9
[deviantart] add support for fav.me URLs
2 years ago
674c719646
[deviantart] refactor base36 conversion
2 years ago
293abb8921
[deviantart] add support for /deviation/ URLs
2 years ago
597b89245e
[deviantart] misc improvements to status extractor
...
- relax regex pattern
- handle invalid 'items' field
- add a test for shared sta.sh item
Co-authored-by: Mike Fährmann <mike_faehrmann@web.de>
2 years ago
013733c9e9
[deviantart] fix index fields for embedded/shared images
2 years ago
c4aeca7a5a
[deviantart] improve handling of statuses
...
- recursively yield statuses
- ignore items with missing or unexpected field(s)
2 years ago
3b32671fbd
[deviantart] add extractor for status updates
...
extract user status updates using the '/user/statuses/' endpoint
2 years ago
77df8d3116
[deviantart] implement username&password login for scraps ( #1029 )
...
re-login when getting prematurely logged out by dA
is missing at the moment
2 years ago
8d7585534e
Merge pull request #3367 from the-blank-x/deviantart-view
...
[deviantart] add /view URL support
2 years ago
6614d94b08
[deviantart] add /view URL support
2 years ago
bf3fd5951a
Merge branch 'master' into da-extra-stash
2 years ago
cd931e1139
update extractor test results
2 years ago
13d825731e
[deviantart] fix test for sta.sh URL extraction
...
Without the 'count' assertion, the test would be essentially useless.
2 years ago
6356c9be96
[deviantart] extract sta.sh URLs from 'text_content'
2 years ago
b0cb4a1b9c
replace 'text.extract()' with 'text.extr()' where possible
2 years ago
2a1cb403ee
Revert "[Deviantart] [ #1776 ] Remove the "you need session cookies to download mature scraps" warning ( #1777 )"
...
This reverts commit 1f02878351
2 years ago
14717f3fc9
[deviantart] add 'group' option ( #3018 )
...
disabling this option allows to better download from deleted accounts
2 years ago
a12ce2bb41
[deviantart] fix 'deviation' extraction ( #2981 )
2 years ago
6c76b5f90c
[deviantart] fix extraction ( #2981 , #2983 )
...
send a 'csrf_token' with every Eclipse API request
2 years ago
43ec315a7f
[deviantart] use public access token for journals ( #2702 )
...
and retry with a private token if needed
2 years ago
c6a9bab019
update extractor test results
2 years ago
4f7fe9b4be
[deviantart] fix folder listings with 'pagination: manual'
...
(#2488 )
2 years ago
97e8a15295
[deviantart] implement 'pagination' option ( #2488 )
2 years ago
1f9a0e2fd8
update extractor test results
2 years ago
b5236656d5
[deviantart] notify users about registering an oauth application
...
if they get repeated 429 errors and are using default API credentials
3 years ago
d441888bfb
[deviantart] adjust API endpoints
...
Start all endpoints with a forward slash '/'
to be consistent with other API interfaces.
3 years ago
8f0cf0bf71
[deviantart] use '/browse/newest' for most-recent searches
...
(#2096 )
3 years ago
8bea02c38c
[deviantart] fix 'index' values for stashed deviations
3 years ago
02a247f4e5
[deviantart] full resolution for non-downloadable images ( #293 )
...
Many thanks to @Ironchest337 for discovering this method
and providing a well-documented implementation.
3 years ago
a7ddb5f5fa
[deviantart] update 'search' argument handling ( fixes #1911 )
...
- use 'alltime' by default
- support newer 'order' values (most-recent, this-week, etc)
3 years ago
a23f5d45d7
[deviantart] fix bug with fetching premium content ( #1879 )
...
When a user has both 'watchers' and 'paid' folders and one of them is
inaccessible, the other one could get handled as inaccessible as well.
3 years ago
ada36c2044
[deviantart] update default archive_fmt for single deviations
...
(#1874 )
use the same as gallery downloads
3 years ago
44f572c27f
[deviantart] implement a 'auto-unwatch' option ( #1466 , #1757 )
3 years ago
47a780942c
update extractor test results
3 years ago
96fec14ef7
[deviantart] rename 'watch' option to 'auto-watch'
...
(#1466 , #1757 )
Similar reason as in e05a96db
3 years ago
e75f2de9da
[deviantart] add 'comments' option ( #1800 )
3 years ago
6ce16c6d31
[deviantart] add 'tag' extractor ( closes #1803 )
3 years ago
ecc8da4704
[deviantart] implement a 'watch' option ( #1466 , #1757 )
3 years ago
a4f249c22e
[deviantart] prevent exception on empty videos ( fixes #1796 )
3 years ago
1f02878351
[Deviantart] [ #1776 ] Remove the "you need session cookies to download mature scraps" warning ( #1777 )
3 years ago
bd08ee2859
remove most 'yield Message.Version' statements
...
only leave them in oauth.py as noop results
3 years ago
220cfe244e
[deviantart] get original files for GIF previews ( #1731 )
3 years ago
fad4918208
[deviantart] use UUIDs in internal folder/collection URLs
3 years ago
4e4ca3c330
[deviantart] pin API version ( #1611 )
...
'/gallery/folders' in the newest version doesn't include subfolders.
It probably only needs the right query parameter to do so, but that
doesn't seem to be documented anywhere.
3 years ago
14f983eab6
[deviantart] use default ID when 'client-id' is None
3 years ago
5a98bcec3a
[deviantart] improve folder name matching ( fixes #1451 )
3 years ago
457abf0e71
[deviantart] fix pagination for Eclipse results ( fixes #1444 )
...
- don't crash on missing keys
- use fallback for invalid 'nextOffset' values
3 years ago
214ecf62ce
[deviantart] fix arguments for search/popular results ( #1408 )
4 years ago
139fb84108
[deviantart] fix username for 'watch' results ( #794 )
...
before it'd use "/" as username
4 years ago
91c2e15da9
[deviantart] add support for posts from watched users ( #794 )
4 years ago
03c20d8c8e
[deviantart] update 'watch' URL pattern ( #794 )
4 years ago
dc23cfd684
[deviantart] use fallback for /intermediary/ URLs
...
instead of checking availability with HEAD requests
4 years ago
b0438c8f99
Revert "[deviantart] extend 'extra' option"
...
This reverts commit
5ad2b9c82b5c32a7bf5883f465faca#1387 , #1356 )
4 years ago
69ca4e29f1
[deviantart] add 'watch' extractor ( #794 )
4 years ago
83f465faca
[deviantart] refactor 'extra' ( #1356 )
...
- change its expected type to string
- let users specify a list of sources (stash, posts) or 'all'
4 years ago
5c32a7bf58
[deviantart] allow selecting source for 'extra' ( #1356 )
...
Setting 'extra' to "stash" or "deviations" will only download embedded
sta.sh content or deviations. 'true' still downloads both.
4 years ago
5ad2b9c82b
[deviantart] extend 'extra' option
...
also download from embedded DeviantArt posts
4 years ago
23be48427c
[deviantart] fix 'folders' option ( closes #1302 )
...
don't assume parent folders are listed before their children
4 years ago
c6cc86d7d0
[deviantart] update parameters for '/browse/popular'
...
- limit results to 50 when also querying metadata (fixes #1267 )
- remove deprecated 'category_path' parameter
4 years ago
c26de0929d
[deviantart] provide 'extension' for original file downloads
...
(#1272 )
4 years ago
193dca2ce1
update extractor test results
4 years ago
e2d4ca4955
[deviantart] improve '--range' for favorites ( closes #1226 )
4 years ago
91db8df1c7
[deviantart] add 'index_base36' metadata field ( closes #1099 )
...
This is the same ID as found in 'filename' without the 'd' in front,
which is just 'index' encoded in base36.
4 years ago
968d3e8465
remove '&' from URL patterns
...
'/?&#' -> '/?#' and '?&#' -> '?#'
According to https://www.ietf.org/rfc/rfc3986.txt , URLs are
"organized hierarchically" by using "the slash ("/"), question
mark ("?"), and number sign ("#") characters to delimit components"
4 years ago
3ebb174f2c
add missing extractor info when spawning new ones ( fixes #1051 )
...
Not having this information causes the blacklist/whitelist logic to
trigger and prevents things from functioning as intended when using
default settings.
Fixes issues for 8muses, deviantart, exhentai, and mangoxo.
4 years ago
136df52d1f
[deviantart] support watchers-only/paid deviations ( #995 )
4 years ago
f6fd449b59
reduce wait time growth rate from exponential to linear
...
Waiting for 2**N seconds after each error grows too fast.
Simply waiting N seconds seems far more reasonable.
4 years ago
c6c06c41f6
[deviantart] don't add journal text to description ( #712 )
4 years ago
41d03160ff
[deviantart] also search journals for sta.sh links ( #712 )
...
when 'extra' is enabled
4 years ago
dfcf2a2c91
write OAuth token to cache by default ( #616 )
4 years ago
6294e2c540
add 'text.ensure_http_scheme()'
4 years ago
65b1cb7acd
[deviantart] use private access tokens for Journals ( fixes #738 )
4 years ago
999efec5cc
[deviantart] limit API wait times to 2**9=512 seconds ( #721 )
4 years ago
6386ee54e1
[deviantart] add extractor info to 'following' results
4 years ago
bae1e8ed12
[deviantart] fix JPEG quality replacement pattern
...
'q_\d+' would sometimes also replace something in the 'token' query
parameter, invalidating the URL.
5 years ago
d02f7c1118
improve Extractor.wait()
...
- allow 'until' to be a datetime object
- do "time calculations" with UTC timestamps
- set a default 'reason'
5 years ago
f9a590f92b
[deviantart] apply HTTP request limits in more places
...
"Request blocked" can also happen on sta.sh and for *any* HTTP
request directed at deviantart.com
5 years ago
ff7c0b7eff
[deviantart] handle "Request blocked" errors ( #655 )
...
- add a 2 second wait time between requests to deviantart.com
- catch 403 "Request blocked" errors and wait for 3 minutes until
retrying
5 years ago
c874684f05
[deviantart] retrieve *all* download URLs through OAuth API
...
'/extended_fetch' as well as Deviation webpages now again contain
Deviation UUIDs needed to grab Deviation info through the OAuth API,
meaning cookies are no longer necessary to grab original files.
The only instance were cookies are still needed are scraps marked as
"mature", since those entries are hidden for public users.
(#655 , #657 , #660 )
5 years ago
5c27b25a8f
[deviantart] improve sta.sh extraction
...
Extract all sta.sh items in a single extractor run.
Don't spawn a new StashExtractor for each individual sta.sh item to
preserve the current requests.Session and its opened TCP connections.
5 years ago
e2fc4eaa6f
[deviantart] detect stash folders ( fixes #659 )
5 years ago
6f911aeb1c
[deviantart] add error message for cloudFront blocks ( #655 )
5 years ago
1b82d36ab2
[deviantart] handle decode errors for extended_fetch results ( #655 )
...
This isn't going to solve the underlying problem, but it should at
least provide the server response when those errors happen.
5 years ago
913b8333cc
write DeviantArt refresh-tokens to cache ( #616 )
...
Writing the token is currently disabled by default and must be
enabled with 'extractor.oauth.cache'.
'extractor.deviantart.refresh-token' must be set to '"cache"'
to use the cached token.
5 years ago
64bdec8430
[deviantart] check availability of intermediary URLs ( fixes #609 )
5 years ago
ec36df4851
[deviantart] fix video extraction from 'extended_fetch' results
...
DeviantArt is now serving videos from wixmp servers (1), instead of
the former film00.deviantart.com (2), even though those URLS are still
functional.
They seem to also have re-encoded those videos. The 10 MB 1080p video
from (2) is now only available in 720p at ~20 MB (with a higher
bitrate, but still …). Other videos are still available in 1080p, but
not this one for some reason.
(Changing the '720p' in (1) to '1080p' doesn't work.)
(1) https://wixmp-ed30a86b8c4ca887773594c2.wixmp.com/v/mp4/9feaa2c9-1baf-4fc2-84f7-f3384b34cefe/d5gxnb5-282a2e9a-b552-40ff-8542-b3c5eed823f5.720p.a837d7cec12c41be8ca2ee53152cea3a.mp4
(2) https://film00.deviantart.net/4c1d/v/mp4/2012/279/d/1/_video____brushes_i_use_in_paint_tool_sai_by_chi_u-d5gxnb5.mp4
5 years ago
48be2266ed
[deviantart] better error message for 'extended_fetch' ( #585 )
5 years ago
f8e137d6b4
[deviantart] show warning about private deviations only once
...
… per call to '_pagination()'
5 years ago
939fec8ecd
[deviantart] match new search/popular URLs ( closes #538 )
5 years ago
ClosedPort22