dedea3b4db
[deviantart] fix journal creation ( #400 )
5 years ago
efb64ad031
[deviantart] generate filenames ( #392 , #400 )
5 years ago
49f6d7176d
[deviantart] restore filenames ( #392 )
...
<title>_by_<user>_<id> --> <title>_by_<user>-<id>
5 years ago
63daa68d67
[deviantart] improvements ( #392 )
...
- consistent 'filename' entries, at least as far as possible
- GIFs and SWFs don't have a <title>_by_<artist>_<id> anywhere in
their metadata
- Generating <id> (from 'deviationid'?) might be something that needs
to be figured out, so we can build those filenames ourselves
- better code structure etc.
- tests for videos, archives, and flash animations
5 years ago
30d6e284b0
[deviantart] use NAPI for artworks and scraps ( #392 )
...
TODO:
- journal downloads
- test for all media types
5 years ago
423f68f585
[deviantart] fix scraps extraction ( closes #376 )
5 years ago
f4bc75e854
fix rate limit handling for OAuth APIs ( #368 )
5 years ago
3957d27d79
[deviantart] add 'quality' option ( #369 )
5 years ago
5d968412ca
[deviantart] case-insensitive folder name matching ( fixes #343 )
5 years ago
fdec59f8e2
replace extractor.request() 'expect' argument
...
with
- 'fatal': allow 4xx status codes
- 'notfound': raise NotFoundError on 404
5 years ago
7856e5e7dc
]deviantart] "fix" scraps extraction
5 years ago
76ae9957c2
[deviantart] force legacy version for single deviations
...
Let's see how long this works ...
DeviantArt is rolling out a new version of their website, including a
new internal and potentially usable API (rewrite incoming, yay).
The issue with the new layout is that it doesn't include the "old"
UUIDs for single deviations, i.e. mapping a numeric deviation ID to its
UUID counterpart is impossible with the new layout.
5 years ago
258e8b2060
[deviantart] small code improvements
5 years ago
f5961ac968
[deviantart] download deviations with no 'content' field
...
Some deviations (possibly only from sta.sh sources) are downloadable
(i.e. 'is_downloadable' is true and /deviation/download/ works), but
have no 'content' or similar in their JSON representation.
(fixes #307 )
5 years ago
e05a96db5e
[deviantart] rename 'stash' to 'extra' ( #302 )
...
'stash' is already used as a name for the StashExtractor and therefore
expected to be a dictionary.
5 years ago
c23bf263fe
[deviantart] rename 'external' to 'stash' ( #302 )
...
restrict extracted URLs to ones from https://sta.sh/ ...
5 years ago
2fb85178da
[deviantart] add 'external' option ( #302 )
...
If a description is available, this will extract URLs from the
description text and try to find Extractors for them.
5 years ago
f85e42cffc
[deviantart] fix --range for deviation & stash extractor
5 years ago
f1893b2b5b
[deviantart] add 'folders' option ( #276 )
5 years ago
f837ea98cb
[deviantart] don't call 'extend()' on folders ( fixes #271 )
5 years ago
51e0e92429
[deviantart] fix GIF downloads ( #242 )
...
The "original" download URL for GIF animations is only a preview version
of the original file.
5 years ago
9544683d56
[deviantart] provide 'date' metadata ( #232 )
5 years ago
d6ddb74cde
update test results
...
- deviantart: 'index' is now an integer
- flickr: image file with lower quality
- paheal: image server name changed
- rule34: post got deleted
6 years ago
9587aea98f
[deviantart] don't rewrite URLs for newer deviations
...
The '/intermediary/' trick stopped working for recently posted
deviations, but it still appears to be functional for older ones.
6 years ago
5ec55ec4fc
[deviantart] improve URLs for non-downloadable deviations
6 years ago
c7a6b0ed90
[deviantart] add 'metadata' option ( #189 )
6 years ago
a2af2d2965
adjust cache maxage values
6 years ago
13e0f2a78f
[deviantart] add 'scraps' extractor ( closes #168 )
6 years ago
c7b8421333
[deviantart] don't match 'www' as a potential username
6 years ago
61741d7333
provide type information for Queue messages
...
Child extractors are now directly constructed with Extractor.from_url()
if the extractor class is known beforehand, instead of using
extractor.find() and searching through all possible extractor classes.
6 years ago
2e516a1e3e
store the full original URL in Extractor.url
6 years ago
4b1880fa5e
propagate 'match' to base extractor constructor
6 years ago
6284731107
simplify extractor constants
...
- single strings for URL patterns
- tuples instead of lists for 'directory_fmt' and 'test'
- single-tuple tests where applicable
6 years ago
fa7fa2f8ff
[deviantart1 update tests]
6 years ago
6c71e9cf5d
[deviantart] add separate 'sta.sh' extractor ( #113 )
...
- supports multiple stashed deviations per page
- explicitly mentions sta.sh support on supportedsites.rst
6 years ago
7471933d5f
use extractor.request for all other API calls
...
- deviantart
- pawoo
- pixiv
- reddit
6 years ago
7e2d6bcd62
[deviantart] fix original image downloads
6 years ago
d1f3d32eec
[fallenangels] unescape chapter titles
6 years ago
2221cf97ff
implement 'update()' for caches
6 years ago
d8492df51b
[deviantart] extend functionality of 'original' option
6 years ago
1532d1b690
fix 'range' tests and update a few test results
6 years ago
e066f35118
update extractor tests
6 years ago
0232d80cec
[deviantart] convert 'published_time' to int ( fixes #108 )
...
The 'published_time' field (a timestamp) changed from integer to string
and caused journal creation to fail.
6 years ago
a493fed376
[deviantart] fix journal creation if no 'username' is set
6 years ago
4313c95bc9
improve error message for OAuth2 authentication
6 years ago
5f27cfeff6
[deviantart] remove `prefer-public` option
...
All API requests now always use a public token and only switch to
a private token for pagination results if `refresh-token` is set
and less deviations than requested were returned.
6 years ago
212130b048
[deviantart] improve public-private token switching
...
- rename option to `prefer-public`
- now also works for galleries with less than 24 items
6 years ago
886d662582
[deviantart] add option to minimize refresh-token usage
...
Always trying with a public token first and repeating the API request
with a private token if deviations are missing doesn't quite work for
galleries and folders with less than 25 items, so its an option and
not the default.
6 years ago
d98e47817d
[deviantart] reduce refresh-token usage
...
Instead of using a refresh-token-based access-token for every API
request, they are now only used for paginated results.
API requests to get a user's profile and the original download URL
now always use a public access-token.
6 years ago
ff436692bf
["deviantart] add 'journals' option
6 years ago
Mike Fährmann