035b850e82
update postprocessor entries in example config
...
- use whitelists
- add ugoira example (#299 )
5 years ago
2fb85178da
[deviantart] add 'external' option ( #302 )
...
If a description is available, this will extract URLs from the
description text and try to find Extractors for them.
5 years ago
096009367b
[xhamster] add gallery & user extractor ( #281 )
5 years ago
d33f5a7423
[wallhaven] rewrite
...
- use API
- remove login support, add 'api-key' option
- remove support for "alpha" subdomain - alpha.wallhaven.cc used numeric
IDs that can't be translated to the new ID system
- support direct links to wallpapers
5 years ago
f1893b2b5b
[deviantart] add 'folders' option ( #276 )
5 years ago
c849574def
[keenspot] add comic extractor ( #223 )
...
Doesn't work for
- http://brawlinthefamily.keenspot.com/
- http://flipside.keenspot.com/
- http://lastblood.keenspot.com/
- http://mysticrevolution.keenspot.com/
- http://porcelain.keenspot.com/
- http://twokinds.keenspot.com/
yet, because of custom layouts.
5 years ago
bcd1801aa8
[sankakucomplex] add 'tag' extractor ( #258 )
5 years ago
74c2415138
[sankakucomplex] move article extractor to its own module ( #258 )
5 years ago
4465a3ea68
[kissmanga][readcomiconline] add 'captcha' option ( #279 )
...
to configure how to handle CAPTCHA page redirects:
- either interactively wait for the user to solve the CAPTCHA
- or raise StopExtraction like before
5 years ago
1e3e15c4f3
[sankaku] add article extractor ( #258 )
5 years ago
e82cadac61
[patreon] add extractors ( #226 )
5 years ago
4891f4a328
[hentainexus] add search extractor ( #256 )
5 years ago
9890bfdf23
[flickr] improve code and metadata
...
- simplify pagination
- add more metadata and slightly change its structure
- convert suitable values to int or list
- move keys from ["photo"] to the base level
- proper video support (#246 )
- rename method and variable names to better fit with other extractors
5 years ago
ba8eb1ffec
[hentainexus] add gallery extractor ( #256 )
5 years ago
0318c610dc
[sexcom] add extractor for search results ( #147 )
5 years ago
a247c94c34
[sexcom] add pin and board extractors ( #147 )
5 years ago
70be494161
[plurk] add a 'comments' options ( #212 )
6 years ago
0b2ff406f6
[plurk] add timeline- and post-extractors ( #212 )
6 years ago
e7cd5510d5
[pixnet] add extractors ( closes #177 )
...
for:
- users/blogs: http://albertayu773.pixnet.net/
- folders: https://albertayu773.pixnet.net/album/folder/1405768
- sets : https://albertayu773.pixnet.net/album/set/15078995
- photos : https://albertayu773.pixnet.net/album/photo/159443828
6 years ago
d9b94a585d
[mangoxo] add login support ( #184 )
...
A very recent change: It is now only possible to see more
than the first 5 images of an album if you are logged in.
6 years ago
e730fc9045
[twitter] add login support ( #214 )
6 years ago
35919a9bb8
[livedoor] add blog- and post-extractors ( #190 )
6 years ago
2ff043edfa
[yaplog] add user- and post-extractors ( #190 )
6 years ago
790f15a56f
[photobucket] use HTTPS
6 years ago
6da665f32e
[mangoxo] add album- and channel-extractors ( closes #184 )
6 years ago
c70b21248d
[wikiart] add extractors ( #179 )
...
for
- artists: https://www.wikiart.org/en/thomas-cole
- artist-listings: https://www.wikiart.org/en/artists-by-century/12
- artwork-listings: https://www.wikiart.org/en/paintings-by-media/grisaille
6 years ago
bc2020e86c
release version 1.8.1
6 years ago
369eb66125
consistently use '*' for rst lists
6 years ago
c7a6b0ed90
[deviantart] add 'metadata' option ( #189 )
6 years ago
8d96a8ce4c
[500px] add user-, gallery-, and image-extractors ( #185 )
6 years ago
6277a739e4
[35photo] add user-, genre-, and image-extractors ( #162 )
6 years ago
12482553bd
update links to youtube-dl
6 years ago
6f57d44ec2
[seaotterscans] remove extractor
...
http://seaotterscans.com/ now redirects to their MangaDex profile
6 years ago
25aaf55514
[smugmug] improve format selection ( closes #183 )
...
- use original image if available
- support video formats
- remove user info for ImageExtractor (it is no longer possible to get
image owner information for a single image)
6 years ago
976ccb267f
[myportfolio] combine gallery and user extractors
...
An URL alone isn't good enough to distinguish between a gallery or a
gallery-listing, so the new extractor decides what to do based on the
page's content.
6 years ago
9c0e2f294b
[shopify] add generic collection and product extractors ( #175 )
...
with fashionnova.com as a default domain
6 years ago
13e0f2a78f
[deviantart] add 'scraps' extractor ( closes #168 )
6 years ago
280531c8ff
[pururin] add gallery extractor ( closes #174 )
6 years ago
3159dd79d5
[seiga] use HTTPS
6 years ago
ea6fbfdd39
simplify build_supportedsites.py
6 years ago
22d3a2fcc8
[artstation] add extractor for artwork listings ( #80 )
...
like https://www.artstation.com/artwork?sorting=latest
or https://www.artstation.com/artwork?sorting=picks
6 years ago
937a802b49
[dynastyscans] add extractors for images and image searches
...
(closes #163 )
6 years ago
19860655a3
[weibo] add 'user' and 'status' extractors
6 years ago
ae353ed3b0
provide "extractor" and "job" keys for logging output
...
This allows for stuff like "{extractor.url}" and "{extractor.category}"
in logging format strings.
Accessing 'extractor' and 'job' in any way will return "None" if those
fields aren't defined, i.e. in general logging messages.
6 years ago
53c2fd4664
add mastodon/foolslide/foolfuuka examples to example config
6 years ago
d36ec51e5a
[tsumino] add extractor for search results ( #161 )
6 years ago
bfbbac4495
[tsumino] add login capabilities ( #161 )
6 years ago
6126615698
update URLs for supportedsites.rst
6 years ago
80a75a1ecf
[tsumino] add gallery extractor ( #161 )
6 years ago
0c32dc5858
[hentaifox] add extractor for search results ( #160 )
6 years ago
0156189468
[hentaifox] add chapter extractor ( #160 )
6 years ago
e4171d6baf
[luscious] add login capabilities ( closes #159 )
6 years ago
134487ffb0
[exhentai] stop extraction if image limit is exceeded ( #141 )
...
can be turned off with the `exhentai.limits' option
6 years ago
b50b30f1c9
[photobucket] download subalbums ( #117 )
6 years ago
277b52101a
add 'category-transfer' option
...
[ci skip]
6 years ago
4b441c162e
release version 1.6.3
6 years ago
8aba2bdebf
[postprocessor:metadata] add 'tags' and 'custom' modes ( #135 )
6 years ago
e53cdfd6a8
update build_supportedsites.py
6 years ago
1e4d351ad3
[danbooru] add authentication support ( closes #151 )
...
... via HTTP Basic Auth with username and "password".
The password value in this case is not the account password itself,
but the"api_key" found in your user profile.
6 years ago
06cbf5f9c4
implement 'chapter-reverse' option ( #149 )
...
Setting it to `true` will start with the latest chapter instead of the
first one.
6 years ago
e95b24f056
[reactor] add wait-min & -max options ( #148 )
6 years ago
a36f52a730
[joyreactor] add extractor for search results ( #148 )
6 years ago
6c71e9cf5d
[deviantart] add separate 'sta.sh' extractor ( #113 )
...
- supports multiple stashed deviations per page
- explicitly mentions sta.sh support on supportedsites.rst
6 years ago
28f9539551
[tumblr] change default values for post types and inline media
6 years ago
5be95034ba
[tumblr] add option to download avatars ( #137 )
6 years ago
be52069cbc
update CHANGELOG and docs/supportedsites
6 years ago
c47482b110
smaller changes, missing docs, etc.
...
- make 'netrc' extractor-specific
- rename 'downloader.enable' to 'enabled'
- document 'downloader.ytdl.format'
- consistent newlines in configuration.rst
6 years ago
8ef84a6823
add option to enable/disable specific downloader modules
...
... and write URLs with no (active) downloader to unsupported-file
6 years ago
2fa28a2609
update default user-agent string ( closes #122 )
6 years ago
655549df7c
[downloader:ytdl] add several options
...
The "default" downloader options (rate, retries, timeout, verify) are
mapped to corresponding youtube-dl options.
downloader.ytdl.logging tells the downloader to pass youtube-dl's output
to a Logger object.
downloader.ytdl.raw-options allows to pass arbitrary options to the
YoutubeDL constructor.
6 years ago
6ed629f2b6
allow specifying number of skips before abort/exit ( closes #115 )
...
In addition to 'abort' and 'exit', it is now possible to specify
'abort:N' and 'exit:N' (where N is any integer) as value for 'skip'
to abort/exit after consecutively skipping N downloads.
6 years ago
d8492df51b
[deviantart] extend functionality of 'original' option
6 years ago
392a081657
update 'docs/configuration.rst'
6 years ago
f8b3b00249
[twitter] add experimental 'videos' option ( #99 )
...
Enabling this option will detect videos in tweets and output them as
"unsupported" URLs, so that these can then be downloaded with youtube-dl
There are a lot of improvements to be made to the current
implementation, but it works and does what it is supposed to, even if
inefficient as can be ...
6 years ago
2cf3f53839
[yuki] add thread extractor ( closes #111 )
6 years ago
17611bfec0
update build_supportedsites.py script
6 years ago
09d2f3e5e7
[postprocessor:ugoira] improve libx264 detection
6 years ago
c402cc4047
[hentaifoundry] add 'popular' and 'recent' extractors
...
for "Popular Pictures" and "Recent Pictures" listings
6 years ago
a5fc311dfa
[hentaifoundry] add 'favorite' extractor
6 years ago
55f5c87160
[postprocessor:ugoira] add 'libx264-prevent-odd' option
...
A rather crude workaround for "width/height not divisible by 2"
errors when using libx264.
6 years ago
006f75b538
[hentaifoundry] rewrite + more metadata
...
- extract width, height, artist per image
- improve pattern regex
- better extensibility for other listings
6 years ago
6ea9a78588
[wallhaven] add login capabilities
...
Being logged in is required to access NSFW wallpapers.
6 years ago
c9290d8212
[wallhaven] add wallpaper and search extractors
...
todo:
- login support to gain access to NSFW wallpapers
- extractors for tag-, similar-, latest-listings
- skip() support
6 years ago
22ab509a70
[bobx] rename "model" to "idol" extractor
6 years ago
69fd61ea86
[bobx] add gallery and model extractors
6 years ago
7742cf8601
[tumblr] change 'reblogs' option ( #103 )
...
- rename "deleted" to "same-blog"
- change test for deleted original post to test if
original post owner has the same UUID (full blog name) as the one
being downloaded from
- add 'blog[uuid]' metadata to allow comparison with
'reblogged_from_uuid'
6 years ago
f1695567e8
adjust values in template config file
...
(#104 )
6 years ago
a666ddd16b
[tumblr] extend 'reblogs' functionality ( #103 )
...
Setting 'reblogs' to "deleted" will check if the parent post of a
reblog has been deleted and download its media content if that is the
case, otherwise it will be skipped.
This is a rather costly operation (1 API request per reblogged post)
and should therefore be used with care.
6 years ago
c9b8e6aefc
[reddit] fix submission-ID parsing ( #104 )
...
Uppercase characters caused a ValueError exception
6 years ago
3ecea4cf36
[hentaicafe] add chapter and manga extractors ( #101 )
6 years ago
75862715ac
[behance] add user extractor
6 years ago
6ecb36d88c
[postprocessor:ugoira] add 'ffmpeg-output' option
6 years ago
ab2bfaeb46
[ngomik] add replacement for 'subapics'
...
http://subapics.com/ got discontinued and replaced by http://ngomik.in/ .
ngomik.in is still displaying a link to the "old site" showing a big
"Account Suspended" sign.
6 years ago
e9dd2eff1d
[twitter] add extractor for media-tweet timelines ( #96 )
...
For example "https://twitter.com/PicturesEarth/media ".
They are different from normal timelines in that they do not contain
any (re)tweets from other users and feature all media the user ever
posted, including responses to other tweets.
6 years ago
9b1c39032c
[twitter] changes and improvements
...
- rename User- to TimelineExtractor
- rename 'userid' to 'user_id' to conform to the other ..._id values
- adjust archive_fmt to deal with retweets
- emulate browser behavior for API calls
6 years ago
10365394d7
[twitter] add support for user-timelines ( closes #96 )
...
also adds a 'retweets' option to filter retweeted content
6 years ago
63fa0b2006
[pinterest] add extractors for related pins
...
Related pins can not be accessed by adding a "#related" fragment
to the end of a Pinterest URL, for example:
- https://www.pinterest.com/pin/858146903966145189/#related
- https://www.pinterest.com/g1952849/test-/#related
There are no explicit real URLs for related pins,
using an option to enable them results in "clunky" code,
and a custom "related:<URL>" scheme doesn't feel right either.
6 years ago
3c90df6635
[piczel] add user, folder and image extractors
6 years ago
b164231bca
[sankaku] increase default values for 'wait-min/-max'
6 years ago
df082e923c
[behance] add gallery extractor ( #95 )
6 years ago
5f27cfeff6
[deviantart] remove `prefer-public` option
...
All API requests now always use a public token and only switch to
a private token for pagination results if `refresh-token` is set
and less deviations than requested were returned.
6 years ago
212130b048
[deviantart] improve public-private token switching
...
- rename option to `prefer-public`
- now also works for galleries with less than 24 items
6 years ago
886d662582
[deviantart] add option to minimize refresh-token usage
...
Always trying with a public token first and repeating the API request
with a private token if deviations are missing doesn't quite work for
galleries and folders with less than 25 items, so its an option and
not the default.
6 years ago
54a0d72dc8
[postprocessor:ugoira] improve frame rate handling
...
By default FFmpeg assumes a 25 FPS input frame rate, leading to dropped
frames if the source requires a higher frame rate than that.
This commit adds a `framerate` option (default "auto"), which allows to
automatically assign a (more or less) fitting frame rate based on
delays between ugoira frames and avoids dropped frames.
6 years ago
84854fcad7
[myportfolio] add user and gallery extractors ( #95 )
6 years ago
c9f70e0a19
[paheal] use HTTPS
6 years ago
ff436692bf
["deviantart] add 'journals' option
6 years ago
00032b828c
[deviantart] add 'wait-min' option
6 years ago
269dc2bbd5
[sankaku] add 'tags' option ( #94 )
6 years ago
1d43cbbf52
[gelbooru] tag-splitting for non-api mode
6 years ago
c20c0a4820
[safebooru] add pool extractor
6 years ago
4a57509392
generalize tag-splitting option ( #92 )
...
- extend functionality to other booru sites:
- http://behoimi.org/
- https://konachan.com/
- https://e621.net/
- https://rule34.xxx/
- https://safebooru.org/
- https://yande.re/
6 years ago
188e956c4e
[imagefap] use HTTPS + update test results
6 years ago
87853538b4
[yandere] add option to split tags by type ( #92 )
6 years ago
4eb94aca17
[postprocessor:ugoira] pass '-f' if not present
6 years ago
0c1c4557dd
[postprocessor:ugoira] add option for two-pass encoding
6 years ago
eb7a1f3b98
[pixiv] rework ugoira handling
...
Frame information now gets attached to the ZIP file's keyword dict
instead of being written to a separate text file.
6 years ago
b84e71da91
add postprocessor documentation to configuration.rst
6 years ago
6ac403c5d3
add postprocessor config example
6 years ago
cdcc3427a0
[simplyhentai] add video extractor ( #89 )
...
All videos hosted on their own servers seem be to dead,
but myhentai.tv embeds, which are most of the videos, work fine.
6 years ago
12797e3b1f
update configuration.rst
...
... again
- some more 'Path' references
- fixed some inconsistencies and errors
- added note about logging config for files
6 years ago
c43f02245f
update configuration.rst
...
- fix default values for 'log' and 'unsupportedfile'
[ci skip]
6 years ago
dacda69c9e
update configuration.rst
...
- document logging options
- add a section for "custom types"
[ci skip]
6 years ago
55b0913412
[simplyhentai] add gallery extractor ( #89 )
6 years ago
53f36176fd
update configuration.rst
...
- update the API Tokens & IDs section
- mention redirect URIs for deviantart
- include api-secret for tumblr
- add instructions for smugmug
- [ci skip]
6 years ago
ecdc3475b8
[pixhost] support .to TLDs
6 years ago
909d105ae6
[pixiv] add extractor for illusts from followed users
6 years ago
f3483a2b7c
[smugmug] add OAuth support
6 years ago
69a5e6ddb3
Merge branch 'master' into 1.4-dev
6 years ago
82c50fa609
release version 1.3.5
6 years ago
8bf3cdd82b
implement logging options
...
Standard logging to stderr, logfiles, and unsupported URL files (which
are now handled through the logging module) can now be configured by
setting their respective option keys (log, logfile, unsupportedfile)
to a dict and specifying the following options;
- format:
format string for logging messages
available keys: see [1]
default: "[{name}][{levelname}] {message}"
- format-date:
format string for {asctime} fields in logging messages
available keys: see [2]
default: "%Y-%m-%d %H:%M:%S"
- level:
the lowercase levelname until which the logger should activate;
available levels are debug, info, warning, error, exception
default: "info"
- path:
path of the file to be written to
- mode:
'mode' argument when opening the specified file
can be either "w" to truncate the file or "a" to append to it (see [3])
If 'output.log', '.logfile', or '.unsupportedfile' is a string, it will
be interpreted, as it has been, as the filepath
(or as format string for .log)
[1] https://docs.python.org/3/library/logging.html#logrecord-attributes
[2] https://docs.python.org/3/library/time.html#time.strftime
[3] https://docs.python.org/3/library/functions.html#open
6 years ago
16e014baaa
[smugmug] added image and album extractor
...
just some initial code that still requires a lot of work ...
TODO:
- folders
- old-style albums (which are nearly all of them ...)
- images from users
- OAuth
It could also happen that the API credentials used will become invalid
whenever my 14 day trial period ends (7 days remaining), but that
would just require users to supply their own.
6 years ago
d96b3474e5
[puremashiro] remove module
...
site has been unreachable for a couple of weeks
and now the DNS record is gone as well
6 years ago
b44a296404
[gomanga] remove module
...
site has been unreachable for a couple of weeks
and the cloudflare status page shows host errors
6 years ago
0f1e07f627
[pinterest] scrap OAuth implementation; code improvements
...
OAuth authentication isn't needed anymore and other tools
like Postman are better suited for this job anyway.
6 years ago
d10579edb5
[pinterest] improve PinterestAPI code; remove OAuth mentions
...
on another note: access_tokens have been set to only allow for
10 requests per hour (from 200 yesterday)
7 years ago
4bd182c107
[pinterest] implement `oauth:pinterest` ( #83 )
...
Pinterest access tokens are rate limited at 200 requests per
hour (or maybe per 2 or 3 hours?) so having just one access token
for all users isn't going to work in the long run.
7 years ago
dd49127408
[spectrumnexus] remove module
...
Site stopped hosting manga scans (http://view.thespectrum.net/ )
7 years ago
a1fa4b43b0
Revert "[tumblr] add option to sort photosets by upload order"
...
This reverts commit 4a26ae32df
7 years ago
564e12ca8f
replace 'imgyt' with 'imxto'
...
https://img.yt/ wasn't available for a couple of days, but has now
re-emerged as https://imx.to/ with a new web-interface.
Links to older images still work (see tests).
7 years ago
4a26ae32df
[tumblr] add option to sort photosets by upload order
7 years ago
7b562907c3
[nijie] add favorites extractor
...
adds support for 'https://nijie.info/user_like_illust_view.php?id= ...'
7 years ago
f39153b6e9
[nhentai] add extractor for search results
7 years ago
52d41c41e7
[exhentai] add extractor for favorited galleries
7 years ago
a993d0ea90
release version 1.3.2
7 years ago
e7525b1b0e
[artstation] add challenge extractor ( #80 )
7 years ago
b25ae30a95
reorder options in gallery-dl.conf
7 years ago
40ca562d7b
[artstation] add album extractor ( #80 )
7 years ago
241eda8c3d
Update supportedsites.rst ( #81 )
...
Add ArtStation to Supported Sites.
7 years ago
f0e3861ec7
update configuration.rst
...
- move `base-directory` entry to extractor options
(it should've been there since v1.0.0 / commit e3bfb83
7 years ago
723cc66bb1
[artstation] add user-, image- and likes-extractors
7 years ago
d6ef52897c
[imgchili] remove module
...
All previously hosted images yield a 404
and the main page is just a logo.
7 years ago
7847ab1d5a
[imagehosts] remove even more dead sites
...
All removed sites either
- reject all incoming connections or
- display a message from their domain registrar
7 years ago
74c8f83056
update gallery-dl.conf and add gallery-dl-example.conf
...
- gallery-dl.conf contains all config options set to their default
values, with the exception of 'filename', 'directory', and
'cache.file'
- gallery-dl-example.conf contains a potential real-world configuration
file and demonstrates some more involved options
7 years ago
1400868f53
[mangadex] general improvements
...
- support >100 chapter entries per manga
- custom archive ID format
- detect non-existing chapters
7 years ago
749fbbfa6c
[mangadex] add chapter- and manga-extractor
7 years ago
6e38cf5aab
[mangareader] use 'https://'
...
The site now redirects from http://mangareader.net/
to https://mangareader.net/
7 years ago
8cdce21dcb
make archive keys user-configurable
7 years ago
8f338347b6
[imagehosts] cleanup
...
removed
- chronos.to - unable to resolve hostname
- coreimg.net - same
- imgmaid.net - same
- hosturimage.com - everything returns 404
- imageontime.org - redirects to some shady site
- imgupload.yt - cloudflare error 522, host down
- img4ever.net - read timeout
7 years ago
e1e0668ca8
add option to set default replacement field value
...
Missing or undefined keywords will now be replaced with the value
set for 'keywords-default'. The default is Python's 'None', which
is equivalent to setting this option to JSON's 'null'.
7 years ago
8704d850bf
add explicit proxy support ( #76 )
...
- '--proxy' as command-line argument
- 'extractor.*.proxy' as config option
7 years ago
3cec533c28
Merge branch 'archive'
7 years ago
7f7c16ae37
add option to specify additional key-value pairs
7 years ago
54c36a8a34
[subapics] add chapter- and manga-extractor ( #70 )
7 years ago
2dd3aeeeae
[komikcast] add chapter- and manga-extractor ( #70 )
7 years ago
c0dd922c13
add '--download-archive' cmdline option
...
… as well as a config file equivalent
7 years ago
d951f13e37
add config option for unsupported-URL file
...
for consistency's sake
7 years ago
97f4f15ec0
add option to write logging output to a file
...
- '--write-log FILE' as cmdline argument
- 'output.logfile' as config file option
7 years ago
1e93955170
[batoto] remove module
...
Site officially shut down on 2018.01.18
7 years ago
40d35c87bc
[paheal] add tag- and post-extractors ( closes #69 )
7 years ago
cc0c2cca57
[reddit] add extractor for reddit-hosted images ( closes #68 )
7 years ago
35e09869d1
[mangapark] fix image URLs and use HTTPS
7 years ago
29d75fc3fa
[tumblr] add support for OAuth authentication ( #65 )
7 years ago
b33efc99a4
[idolcomplex] add support for idol.sankakucomplex.com
7 years ago
5b094328b5
[puremashiro] add chapter- and manga-extractor ( closes #66 )
...
Also adds support for region subtags in language codes (e.g. en-us)
7 years ago
03b8a548cb
[tumblr] change `reblogs` default value to `true` ( #61 )
7 years ago
d235f68f59
[tumblr] add option to filter reblogged posts ( #61 )
...
Reblogs are ignored by default, but can be included by setting
'extractor.tumblr.reblogs' to 'true'.
7 years ago
d0886f411e
[gelbooru] re-enable API use ( closes #56 )
...
Gelbooru's API allows access to all images and is not restricted
to the first 20000.
This also adds an option to select between API use and manual
information extraction in case their API gets disabled again.
7 years ago
fc7d165c97
[deviantart] add support for OAuth2 authentication
...
Some user galleries [*] require you to be either logged in or
authenticated via OAuth2 to access their deviations.
[*] e.g. https://polinaegorussia.deviantart.com/gallery/
--------------
known issue:
A deviantart 'refresh_token' can only be used once and gets updated
whenever it is used to request a new 'access_token', so storing its
initial value in a config file and reusing it again and again is not
possible.
7 years ago
0a9a07a6e1
[slideshare] improve metadata; flake8
...
- added 'views' and 'published' keywords
- fixed longer titles and descriptions
7 years ago
19a6ae57b2
[sankaku] add pool extractor
7 years ago
4fb6803fa6
add option to sleep before each download
7 years ago
300346ecdf
[mangazuki] remove extractors
...
This site has been in "rebuild"-mode for a fairly long time and the
current extractor code isn't going to work for the new version either.
7 years ago
8f518e03f8
add options to set maximum download rate
...
- -r/--limit-rate as cmdline option
- downloader.http.rate as config option
This implementation very roughly uses the idea of the token bucket
algorithm [1] and mostly uses Wget's approach [2] as inspiration.
[1] https://en.wikipedia.org/wiki/Token_bucket
[2] http://git.savannah.gnu.org/cgit/wget.git/tree/src/retr.c?h=v1.19.2&id=ba6b44f6745b14dce414761a8e4b35d31b176bba#n111
7 years ago
214972bc9a
[gelbooru] use manual extraction
...
... to compensate for their disabled API.
(https://gelbooru.com/index.php?page=forum&s=view&id=3875 )
This also adds an extractor for image-pools.
7 years ago
12de658937
[tumblr] add options to control extraction behavior ( #48 )
...
- posts : list of post-types to inspect
- inline : scan post bodies for inline images
- external: follow external links
7 years ago
e6814aebe2
add 'extractor.*.user-agent' config option
7 years ago
2d0cfb33e1
[xvideos] add user profile extractor ( #45 )
7 years ago
707b15b586
create missing directories for 'part-directory'
...
also some code improvements regarding downloader config values
7 years ago
caf26412dd
add option to set alternate location of .part files ( #29 )
...
Note: The path set for 'downloader.*.part-directory' needs to point to an
already existing directory.
7 years ago
963670d73b
add options to control usage of .part files ( #29 )
...
- '--no-part' command line option to disable them
- 'downloader.http.part' and 'downloader.text.part' config options
Disabling .part files restores the behaviour of the old downloader
implementation.
7 years ago
5fa42336a2
[sankaku] add warning for unauthenticated users
...
also improve URL pattern and add missing options to default config file
7 years ago
6af921a952
[sankaku] rewrite/improve ( fixes #44 )
...
- add wait-time between HTTP requests similar to exhentai
- add 'wait-min' and 'wait-max' options
- increase retry-count for HTTP requests to 10
- implement user authentication (non-authenticated users can only view
images up to page 25)
- implement 'skip()' functionality (only works up to page 50)
- implement image-retrieval for pages >= 51
- fix issue with multiple tags
7 years ago
abfd2924a9
update configuration.rst
...
- add instructions on how to register your own applications
- combine descriptions of a few config keys
7 years ago
f3fbaa5c3e
[reddit] allow users to override the API User-Agent
...
Only overriding the Client-ID is not enough if you want to follow
Reddit's API access rules [1].
[1] https://github.com/reddit/reddit/wiki/API#rules
7 years ago
2ef3c35c98
smaller textual changes
...
- swapped doc for deviantart.mature and .original
- updated gallery-dl.conf
- "transferred" -> "delegated"
7 years ago
75d3a1f72f
[deviantart] always download original images
...
Deviation-objects returned by the DeviantArt API don't always contain
the URL and metadata of the original image ([1]). Getting this
information requires an additional API call [2], which is indicated by
the 'is_downloadable' and 'download_filesize' metadata within a
deviation-object.
[1] https://myria-moon.deviantart.com/art/Aime-Moi-part-en-vadrouille-261986576
[2] https://www.deviantart.com/developers/http/v1/20160316/deviation_download/bed6982b88949bdb08b52cd6763fcafd
7 years ago
cebf800a7f
[foolfuuka] add support for more sites ( #18 )
...
- https://arch.b4k.co
- https://archive.whatisthisimnotgoodwithcomputers.com
- https://archive.yeet.net
Notes:
- The name "whatisthisimnotgoodwithcomputers" is way too long ...
- archive.yeet.net is out of date and also blocked by 4chan servers
- newest threads are 2 weeks old
- using "https://archive.yeet.net " as Referer header results in
"403 Forbidden" when accessing 4chan
7 years ago
470bbe9d8c
fix smaller stuff
...
- change filename option in example config file
- adapt default filename format for mangafox
- remove unnecessary newline
[skip ci]
7 years ago
81877bb5f6
add '-K' as shortcut for '--list-keywords'
7 years ago
54c0715135
allow users to set their own API access_tokens/client_ids
7 years ago
49c7e70c10
[acidimg] add image extractor
7 years ago
b8862ff15e
add 'downloader.http.verify' option
...
(also: change the default 'timeout' from None to 30)
7 years ago
18e6ed1c7e
[booru] add extractors for "Popular" images
7 years ago
4ea82ea556
[warosu] add thread extractor
7 years ago
3c9f190757
extend output of --list-keywords
7 years ago
abd7c559cd
[yonkouprod] remove module
...
Every manga chapter on this site has been removed.
7 years ago
f08af03845
Merge branch 'cookies'
7 years ago
d7cb3c668a
update supportedsites.rst
7 years ago
00a23fe208
update configuration.rst
7 years ago
4a60f6068a
[mangazuki] add manga extractor
7 years ago
1cb1d2e0a3
[mangazuki] add chapter extractor
7 years ago
74a2b5af06
update supportedsites.rst
7 years ago
3460dc8950
update gallery-dl.conf
7 years ago
9be8f7e106
[deviantart] add "extractor.deviantart.flat" option
...
Setting this to 'false' downloads images into individual subdirectories
for each gallery-folder or favourite-collection, otherwise it is just
creating a flat list of images.
7 years ago
9edbd6ff01
update configuration.rst
7 years ago
06c4cae05b
extend the output of '--list-extractors'
...
It now includes category and subcategory values for
each extractor class.
7 years ago
1dac76fd1c
update extractor docstrings
7 years ago
b757687606
Update configuration.rst ( #24 )
...
Fixed minor typo.
7 years ago
598e3a8d88
update configuration.rst
7 years ago
86561d7c8b
update configuration.rst
7 years ago
d3b04076f7
add .netrc support ( #22 )
...
Use the '--netrc' cmdline option or set the 'netrc' config option
to 'true' to enable the use of .netrc authentication data.
The 'machine' names for the .netrc info are the lowercase extractor
names (or categories): batoto, exhentai, nijie, pixiv, seiga.
7 years ago
fbe8c519e2
update README
...
- mention and link to docs/configuration.rst
- add OAuth section
7 years ago
ccb7973b57
update configuration.rst
7 years ago
b206b831ee
Update configuration.rst
...
Fix heading.
7 years ago
48b444e276
add a list of all configuration options
...
(the format might be subject to change; putting everything in
one single table is even worse)
7 years ago
fb1904dd59
add 'Authentication' column to supportedsites.rst
7 years ago
5f55c854b9
[flickr] replace getPublic... API call with regular ones
7 years ago
d5a70f2580
add simple progress indicator for multiple URLs ( #19 )
...
The output can be configured via the 'output.progress'
config value.
Possible values:
- true: Show the default progress indicator
"[{current}/{total}] {url}" (default)
- false: Never show the progress indicator
- <string>: Show the progress indicator using this
as a custom format string(1).
Possible replacement keys are:
- current: current URL index
- total : total number of URLs
- url : current URL
(1) https://docs.python.org/3/library/string.html#formatstrings
7 years ago
090e11b35d
[reddit] enable user authentication with OAuth2 ( #15 )
...
Call '$ gallery-dl oauth:reddit' to get a refresh_token
for your account.
7 years ago
4e80e0c884
[flickr] add user extractor ( #16 )
7 years ago
c921b4f32a
code cleanup and fixing tests
7 years ago
67791e1b36
[imgur] improve and add image extractor
7 years ago
832a4a8ee9
[fallenangels] add manga extractor
7 years ago
2974d782a3
[yomanga] remove module
...
site has been shut down
7 years ago
677c8ced11
[deviantart] add "journal" extractor
...
(#14 )
7 years ago
1e416e5227
update README
7 years ago
Mike Fährmann