Mike Fährmann
3b21e0703c
[deviantart] allow distinction between users and groups ( #26 )
...
This is done by prepending "group-" to an extractor's subcategory
if the URL belongs to a group ("folder" becomes "group-folder" and
so on). This changes the configuration-path being used and is also
reflected in the output of '--list-keywords'.
7 years ago
Mike Fährmann
9aa95fba8c
[deviantart] adapt download URLs to use https
...
Even though DeviantArt is "completely switching over to HTTPS"[1],
every URL contained in an API response is still using HTTP
[1] https://danlev.deviantart.com/journal/DeviantArt-Is-Switching-To-HTTPS-697996906
7 years ago
Mike Fährmann
02e89700fc
[foolfuuka] ensure sorted posts
7 years ago
Mike Fährmann
004456d5d5
properly update the config-dictionary
...
When using 2 or more config files, the values of the second would
improperly overwrite nested dictionaries of the first one.
The new method properly combines these nested dictionaries as well.
7 years ago
Mike Fährmann
0245a0ba5f
fix extraction and update test results
...
- fixes for hbrowse, imgyt, imgcandy, hosturimage
- test updates for deviantart, gfycat
7 years ago
Mike Fährmann
915a0137de
improve 'extractor.request'
...
- add 'fatal' argument
- improve internal logic and flow
- raise known exception on error
- update exception hierarchy
7 years ago
Mike Fährmann
4f1e6c109f
[deviantart] remove 'invalid escape sequence' warning
...
- use r"\w" or "\\w" instead of "\w"
7 years ago
Mike Fährmann
3460dc8950
update gallery-dl.conf
7 years ago
Mike Fährmann
9be8f7e106
[deviantart] add "extractor.deviantart.flat" option
...
Setting this to 'false' downloads images into individual subdirectories
for each gallery-folder or favourite-collection, otherwise it is just
creating a flat list of images.
7 years ago
Mike Fährmann
d075627fd9
[deviantart] support group galleries ( #26 )
...
For groups the 'GalleryExtractor' collects all gallery-folder URLs
and defers its work to the 'FolderExtractor'.
7 years ago
Mike Fährmann
af9bd17b19
[deviantart] adjust default paths
...
- user.deviantart.com/(gallery|favourites|journal)/ images go into
* <user>/
* <user>/Favourites/
* <user>/Journal/
(having an extra "Gallery" folder for a user's gallery-images seems
a bit too much if these are all you want to download, which is
probably the default use-case)
- single "deviations" (user.deviantart.com/(art|journal)/name-123) go
into their owner's directory:
* <user>/
(putting them into their own directory seems weird in practice)
7 years ago
Mike Fährmann
4877ef6314
[deviantart] support '?catpath=/' URLs ( #26 )
...
They previously weren't supported for galleries and journals.
This also increases the 'limit' parameter for API calls to its
respective maximum.
7 years ago
Mike Fährmann
a6f689e01a
[deviantart] add gallery-folder extractor ( #26 )
...
The code for this and the available metadata is probably going
to change again. This extractor is very similar to the favorite-
extractor, so they might be "combined" or something like that.
7 years ago
Mike Fährmann
1dac76fd1c
update extractor docstrings
7 years ago
Mike Fährmann
e1d82af5e0
small fixes
7 years ago
Mike Fährmann
c921b4f32a
code cleanup and fixing tests
7 years ago
Mike Fährmann
bce51e90a5
[reddit] support sorting options and sub-options ( #15 )
...
Example:
https://www.reddit.com/r/ <subreddit>/top/?sort=top&t=month
(the 'sort=top' parameter is irrelevant and can be omitted)
7 years ago
Mike Fährmann
cbb4323f66
add setup.cfg to configure flake8
7 years ago
Mike Fährmann
b6b214f7e9
[deviantart] fix headers for custom-style journals
...
example: http://shimoda7.deviantart.com/journal/Temporary-absence-231936282
7 years ago
Mike Fährmann
e9a2738257
[deviantart] support images on top of journal entries
...
example: http://raxnae.deviantart.com/art/Kami-s-Journal-679482236
7 years ago
Mike Fährmann
92597f46d4
[deviantart] add title to journals
7 years ago
Mike Fährmann
107d29ad8a
improve handling of text:... URLs
...
- don't require // after the colon
- open output files in text mode
7 years ago
Mike Fährmann
677c8ced11
[deviantart] add "journal" extractor
...
(#14 )
7 years ago
Mike Fährmann
e5f79ae839
[deviantart] add support for all media types
...
- this includes
- images
- videos
- flash-animations
- journals
- also renamed some of the extractors
- User -> Gallery
- Image -> Deviation
7 years ago
Mike Fährmann
9f1c83297f
[pinterest] allow URLs with any TLD
7 years ago
Mike Fährmann
b3b92ac243
[deviantart] support "All" favorites and add "mature" option
...
- since there is apparently no actual way to get the "All" favorites
listing via API, corresponding URLs (.../favourites/?catpath=/) will
be handled by yielding all deviations from all favorite collections of
that user
- the "mature" config key works on a per extractor basis (like "username"
or "password"). values can be the strings "true" or "false", or the
booleans true or false.
- (#14 )
7 years ago
Mike Fährmann
7376ad7f3d
[deviantart] turn the "Mature Content Filter" off
...
(#14 )
7 years ago
Mike Fährmann
af56887a47
[exhentai] fall back to e-hentai if no username is given
8 years ago
Mike Fährmann
13dc5d72bc
update some extractors to use https
8 years ago
Mike Fährmann
0770de0ea1
[deviantart:image] add support for sta.sh URLs
8 years ago
Mike Fährmann
11ce871718
[deviantart] add favourites-extractor
8 years ago
Mike Fährmann
d430732502
[deviantart] update extractors
...
- use base class for all extractors
- use API for single images
- add support for more API endpoints
- add additional keywords
8 years ago
Mike Fährmann
f622411be5
[deviantart] implement 'skip' method
8 years ago
Mike Fährmann
379125746a
[deviantart] improve API error handling
...
(#7 )
8 years ago
Mike Fährmann
2c2de778f2
[deviantart] adjust error message
8 years ago
Mike Fährmann
f36d685ce1
[deviantart] always request access token before api calls
...
(#7 )
8 years ago
Mike Fährmann
38aa496ecb
[deviantart] restore tests + set api version
8 years ago
Mike Fährmann
39b1a18f12
[deviantart] fix issue with missing 'content'-key
8 years ago
Mike Fährmann
94e10f249a
code adjustments according to pep8 nr2
8 years ago
Mike Fährmann
72d3ca0bf9
[deviantart] use API for user-extractor
8 years ago
Mike Fährmann
58ce0b55e3
[deviantart] make sure to pick the highest quality image
8 years ago
Mike Fährmann
56d810c896
update keyword hashes for tests
8 years ago
Mike Fährmann
19c2d4ff6f
remove explicit (sub)category keywords
8 years ago
Mike Fährmann
d7e168799d
consistent extractor naming scheme + docstrings
8 years ago
Mike Fährmann
2449532132
add more extractor test-cases
9 years ago
Mike Fährmann
5589bc2ed7
[deviantart] fix non-terminating loop
9 years ago
Mike Fährmann
995d8578bf
[deviantart] fix dates in image-extractor
9 years ago
Mike Fährmann
7d965a2897
[deviantart] add single image extractor
9 years ago
Mike Fährmann
e4a661fd6b
[deviantart] fix issue with small images
9 years ago
Mike Fährmann
4d56b76aa8
update all other extractors
9 years ago
Mike Fährmann
c2f0720184
code cleanup to use nameext_from_url
9 years ago
Mike Fährmann
7d7c1afe20
[deviantart] adjust constructor paramters
9 years ago
Mike Fährmann
ae535ba0c1
add extractor 'deviantart'
9 years ago