Mike Fährmann
0f1e07f627
[pinterest] scrap OAuth implementation; code improvements
...
OAuth authentication isn't needed anymore and other tools
like Postman are better suited for this job anyway.
6 years ago
Mike Fährmann
d10579edb5
[pinterest] improve PinterestAPI code; remove OAuth mentions
...
on another note: access_tokens have been set to only allow for
10 requests per hour (from 200 yesterday)
7 years ago
Mike Fährmann
4bd182c107
[pinterest] implement `oauth:pinterest` ( #83 )
...
Pinterest access tokens are rate limited at 200 requests per
hour (or maybe per 2 or 3 hours?) so having just one access token
for all users isn't going to work in the long run.
7 years ago
Mike Fährmann
dd49127408
[spectrumnexus] remove module
...
Site stopped hosting manga scans (http://view.thespectrum.net/ )
7 years ago
Mike Fährmann
a1fa4b43b0
Revert "[tumblr] add option to sort photosets by upload order"
...
This reverts commit 4a26ae32df
.
7 years ago
Mike Fährmann
564e12ca8f
replace 'imgyt' with 'imxto'
...
https://img.yt/ wasn't available for a couple of days, but has now
re-emerged as https://imx.to/ with a new web-interface.
Links to older images still work (see tests).
7 years ago
Mike Fährmann
4a26ae32df
[tumblr] add option to sort photosets by upload order
7 years ago
Mike Fährmann
7b562907c3
[nijie] add favorites extractor
...
adds support for 'https://nijie.info/user_like_illust_view.php?id= ...'
7 years ago
Mike Fährmann
f39153b6e9
[nhentai] add extractor for search results
7 years ago
Mike Fährmann
52d41c41e7
[exhentai] add extractor for favorited galleries
7 years ago
Mike Fährmann
a993d0ea90
release version 1.3.2
7 years ago
Mike Fährmann
e7525b1b0e
[artstation] add challenge extractor ( #80 )
7 years ago
Mike Fährmann
b25ae30a95
reorder options in gallery-dl.conf
7 years ago
Mike Fährmann
40ca562d7b
[artstation] add album extractor ( #80 )
7 years ago
HRXN
241eda8c3d
Update supportedsites.rst ( #81 )
...
Add ArtStation to Supported Sites.
7 years ago
Mike Fährmann
f0e3861ec7
update configuration.rst
...
- move `base-directory` entry to extractor options
(it should've been there since v1.0.0 / commit e3bfb83
)
- rename "General" to "Miscellaneous" and move it down a bit, since it
has only two options left
- mention the ability to use a list for tumblr.posts
- replace any appearance of bato.to
7 years ago
Mike Fährmann
723cc66bb1
[artstation] add user-, image- and likes-extractors
7 years ago
Mike Fährmann
d6ef52897c
[imgchili] remove module
...
All previously hosted images yield a 404
and the main page is just a logo.
7 years ago
Mike Fährmann
7847ab1d5a
[imagehosts] remove even more dead sites
...
All removed sites either
- reject all incoming connections or
- display a message from their domain registrar
7 years ago
Mike Fährmann
74c8f83056
update gallery-dl.conf and add gallery-dl-example.conf
...
- gallery-dl.conf contains all config options set to their default
values, with the exception of 'filename', 'directory', and
'cache.file'
- gallery-dl-example.conf contains a potential real-world configuration
file and demonstrates some more involved options
7 years ago
Mike Fährmann
1400868f53
[mangadex] general improvements
...
- support >100 chapter entries per manga
- custom archive ID format
- detect non-existing chapters
7 years ago
Mike Fährmann
749fbbfa6c
[mangadex] add chapter- and manga-extractor
7 years ago
Mike Fährmann
6e38cf5aab
[mangareader] use 'https://'
...
The site now redirects from http://mangareader.net/
to https://mangareader.net/
7 years ago
Mike Fährmann
8cdce21dcb
make archive keys user-configurable
7 years ago
Mike Fährmann
8f338347b6
[imagehosts] cleanup
...
removed
- chronos.to - unable to resolve hostname
- coreimg.net - same
- imgmaid.net - same
- hosturimage.com - everything returns 404
- imageontime.org - redirects to some shady site
- imgupload.yt - cloudflare error 522, host down
- img4ever.net - read timeout
7 years ago
Mike Fährmann
e1e0668ca8
add option to set default replacement field value
...
Missing or undefined keywords will now be replaced with the value
set for 'keywords-default'. The default is Python's 'None', which
is equivalent to setting this option to JSON's 'null'.
7 years ago
Mike Fährmann
8704d850bf
add explicit proxy support ( #76 )
...
- '--proxy' as command-line argument
- 'extractor.*.proxy' as config option
7 years ago
Mike Fährmann
3cec533c28
Merge branch 'archive'
7 years ago
Mike Fährmann
7f7c16ae37
add option to specify additional key-value pairs
7 years ago
Mike Fährmann
54c36a8a34
[subapics] add chapter- and manga-extractor ( #70 )
7 years ago
Mike Fährmann
2dd3aeeeae
[komikcast] add chapter- and manga-extractor ( #70 )
7 years ago
Mike Fährmann
c0dd922c13
add '--download-archive' cmdline option
...
… as well as a config file equivalent
7 years ago
Mike Fährmann
d951f13e37
add config option for unsupported-URL file
...
for consistency's sake
7 years ago
Mike Fährmann
97f4f15ec0
add option to write logging output to a file
...
- '--write-log FILE' as cmdline argument
- 'output.logfile' as config file option
7 years ago
Mike Fährmann
1e93955170
[batoto] remove module
...
Site officially shut down on 2018.01.18
7 years ago
Mike Fährmann
40d35c87bc
[paheal] add tag- and post-extractors ( closes #69 )
7 years ago
Mike Fährmann
cc0c2cca57
[reddit] add extractor for reddit-hosted images ( closes #68 )
7 years ago
Mike Fährmann
35e09869d1
[mangapark] fix image URLs and use HTTPS
7 years ago
Mike Fährmann
29d75fc3fa
[tumblr] add support for OAuth authentication ( #65 )
7 years ago
Mike Fährmann
b33efc99a4
[idolcomplex] add support for idol.sankakucomplex.com
7 years ago
Mike Fährmann
5b094328b5
[puremashiro] add chapter- and manga-extractor ( closes #66 )
...
Also adds support for region subtags in language codes (e.g. en-us)
7 years ago
Mike Fährmann
03b8a548cb
[tumblr] change `reblogs` default value to `true` ( #61 )
7 years ago
Mike Fährmann
d235f68f59
[tumblr] add option to filter reblogged posts ( #61 )
...
Reblogs are ignored by default, but can be included by setting
'extractor.tumblr.reblogs' to 'true'.
7 years ago
Mike Fährmann
d0886f411e
[gelbooru] re-enable API use ( closes #56 )
...
Gelbooru's API allows access to all images and is not restricted
to the first 20000.
This also adds an option to select between API use and manual
information extraction in case their API gets disabled again.
7 years ago
Mike Fährmann
fc7d165c97
[deviantart] add support for OAuth2 authentication
...
Some user galleries [*] require you to be either logged in or
authenticated via OAuth2 to access their deviations.
[*] e.g. https://polinaegorussia.deviantart.com/gallery/
--------------
known issue:
A deviantart 'refresh_token' can only be used once and gets updated
whenever it is used to request a new 'access_token', so storing its
initial value in a config file and reusing it again and again is not
possible.
7 years ago
Mike Fährmann
0a9a07a6e1
[slideshare] improve metadata; flake8
...
- added 'views' and 'published' keywords
- fixed longer titles and descriptions
7 years ago
Mike Fährmann
19a6ae57b2
[sankaku] add pool extractor
7 years ago
Mike Fährmann
4fb6803fa6
add option to sleep before each download
7 years ago
Mike Fährmann
300346ecdf
[mangazuki] remove extractors
...
This site has been in "rebuild"-mode for a fairly long time and the
current extractor code isn't going to work for the new version either.
7 years ago
Mike Fährmann
8f518e03f8
add options to set maximum download rate
...
- -r/--limit-rate as cmdline option
- downloader.http.rate as config option
This implementation very roughly uses the idea of the token bucket
algorithm [1] and mostly uses Wget's approach [2] as inspiration.
[1] https://en.wikipedia.org/wiki/Token_bucket
[2] http://git.savannah.gnu.org/cgit/wget.git/tree/src/retr.c?h=v1.19.2&id=ba6b44f6745b14dce414761a8e4b35d31b176bba#n111
7 years ago
Mike Fährmann
214972bc9a
[gelbooru] use manual extraction
...
... to compensate for their disabled API.
(https://gelbooru.com/index.php?page=forum&s=view&id=3875 )
This also adds an extractor for image-pools.
7 years ago
Mike Fährmann
12de658937
[tumblr] add options to control extraction behavior ( #48 )
...
- posts : list of post-types to inspect
- inline : scan post bodies for inline images
- external: follow external links
7 years ago
Mike Fährmann
e6814aebe2
add 'extractor.*.user-agent' config option
7 years ago
Mike Fährmann
2d0cfb33e1
[xvideos] add user profile extractor ( #45 )
7 years ago
Mike Fährmann
707b15b586
create missing directories for 'part-directory'
...
also some code improvements regarding downloader config values
7 years ago
Mike Fährmann
caf26412dd
add option to set alternate location of .part files ( #29 )
...
Note: The path set for 'downloader.*.part-directory' needs to point to an
already existing directory.
7 years ago
Mike Fährmann
963670d73b
add options to control usage of .part files ( #29 )
...
- '--no-part' command line option to disable them
- 'downloader.http.part' and 'downloader.text.part' config options
Disabling .part files restores the behaviour of the old downloader
implementation.
7 years ago
Mike Fährmann
5fa42336a2
[sankaku] add warning for unauthenticated users
...
also improve URL pattern and add missing options to default config file
7 years ago
Mike Fährmann
6af921a952
[sankaku] rewrite/improve ( fixes #44 )
...
- add wait-time between HTTP requests similar to exhentai
- add 'wait-min' and 'wait-max' options
- increase retry-count for HTTP requests to 10
- implement user authentication (non-authenticated users can only view
images up to page 25)
- implement 'skip()' functionality (only works up to page 50)
- implement image-retrieval for pages >= 51
- fix issue with multiple tags
7 years ago
Mike Fährmann
abfd2924a9
update configuration.rst
...
- add instructions on how to register your own applications
- combine descriptions of a few config keys
7 years ago
Mike Fährmann
f3fbaa5c3e
[reddit] allow users to override the API User-Agent
...
Only overriding the Client-ID is not enough if you want to follow
Reddit's API access rules [1].
[1] https://github.com/reddit/reddit/wiki/API#rules
7 years ago
Mike Fährmann
2ef3c35c98
smaller textual changes
...
- swapped doc for deviantart.mature and .original
- updated gallery-dl.conf
- "transferred" -> "delegated"
7 years ago
Mike Fährmann
75d3a1f72f
[deviantart] always download original images
...
Deviation-objects returned by the DeviantArt API don't always contain
the URL and metadata of the original image ([1]). Getting this
information requires an additional API call [2], which is indicated by
the 'is_downloadable' and 'download_filesize' metadata within a
deviation-object.
[1] https://myria-moon.deviantart.com/art/Aime-Moi-part-en-vadrouille-261986576
[2] https://www.deviantart.com/developers/http/v1/20160316/deviation_download/bed6982b88949bdb08b52cd6763fcafd
7 years ago
Mike Fährmann
cebf800a7f
[foolfuuka] add support for more sites ( #18 )
...
- https://arch.b4k.co
- https://archive.whatisthisimnotgoodwithcomputers.com
- https://archive.yeet.net
Notes:
- The name "whatisthisimnotgoodwithcomputers" is way too long ...
- archive.yeet.net is out of date and also blocked by 4chan servers
- newest threads are 2 weeks old
- using "https://archive.yeet.net " as Referer header results in
"403 Forbidden" when accessing 4chan
7 years ago
Mike Fährmann
470bbe9d8c
fix smaller stuff
...
- change filename option in example config file
- adapt default filename format for mangafox
- remove unnecessary newline
[skip ci]
7 years ago
Mike Fährmann
81877bb5f6
add '-K' as shortcut for '--list-keywords'
7 years ago
Mike Fährmann
54c0715135
allow users to set their own API access_tokens/client_ids
7 years ago
Mike Fährmann
49c7e70c10
[acidimg] add image extractor
7 years ago
Mike Fährmann
b8862ff15e
add 'downloader.http.verify' option
...
(also: change the default 'timeout' from None to 30)
7 years ago
Mike Fährmann
18e6ed1c7e
[booru] add extractors for "Popular" images
7 years ago
Mike Fährmann
4ea82ea556
[warosu] add thread extractor
7 years ago
Mike Fährmann
3c9f190757
extend output of --list-keywords
7 years ago
Mike Fährmann
abd7c559cd
[yonkouprod] remove module
...
Every manga chapter on this site has been removed.
7 years ago
Mike Fährmann
f08af03845
Merge branch 'cookies'
7 years ago
Mike Fährmann
d7cb3c668a
update supportedsites.rst
7 years ago
Mike Fährmann
00a23fe208
update configuration.rst
7 years ago
Mike Fährmann
4a60f6068a
[mangazuki] add manga extractor
7 years ago
Mike Fährmann
1cb1d2e0a3
[mangazuki] add chapter extractor
7 years ago
Mike Fährmann
74a2b5af06
update supportedsites.rst
7 years ago
Mike Fährmann
3460dc8950
update gallery-dl.conf
7 years ago
Mike Fährmann
9be8f7e106
[deviantart] add "extractor.deviantart.flat" option
...
Setting this to 'false' downloads images into individual subdirectories
for each gallery-folder or favourite-collection, otherwise it is just
creating a flat list of images.
7 years ago
Mike Fährmann
9edbd6ff01
update configuration.rst
7 years ago
Mike Fährmann
06c4cae05b
extend the output of '--list-extractors'
...
It now includes category and subcategory values for
each extractor class.
7 years ago
Mike Fährmann
1dac76fd1c
update extractor docstrings
7 years ago
H R X N
b757687606
Update configuration.rst ( #24 )
...
Fixed minor typo.
7 years ago
Mike Fährmann
598e3a8d88
update configuration.rst
7 years ago
Mike Fährmann
86561d7c8b
update configuration.rst
7 years ago
Mike Fährmann
d3b04076f7
add .netrc support ( #22 )
...
Use the '--netrc' cmdline option or set the 'netrc' config option
to 'true' to enable the use of .netrc authentication data.
The 'machine' names for the .netrc info are the lowercase extractor
names (or categories): batoto, exhentai, nijie, pixiv, seiga.
7 years ago
Mike Fährmann
fbe8c519e2
update README
...
- mention and link to docs/configuration.rst
- add OAuth section
7 years ago
Mike Fährmann
ccb7973b57
update configuration.rst
7 years ago
H R X N
b206b831ee
Update configuration.rst
...
Fix heading.
7 years ago
Mike Fährmann
48b444e276
add a list of all configuration options
...
(the format might be subject to change; putting everything in
one single table is even worse)
7 years ago
Mike Fährmann
fb1904dd59
add 'Authentication' column to supportedsites.rst
7 years ago
Mike Fährmann
5f55c854b9
[flickr] replace getPublic... API call with regular ones
7 years ago
Mike Fährmann
d5a70f2580
add simple progress indicator for multiple URLs ( #19 )
...
The output can be configured via the 'output.progress'
config value.
Possible values:
- true: Show the default progress indicator
"[{current}/{total}] {url}" (default)
- false: Never show the progress indicator
- <string>: Show the progress indicator using this
as a custom format string(1).
Possible replacement keys are:
- current: current URL index
- total : total number of URLs
- url : current URL
(1) https://docs.python.org/3/library/string.html#formatstrings
7 years ago
Mike Fährmann
090e11b35d
[reddit] enable user authentication with OAuth2 ( #15 )
...
Call '$ gallery-dl oauth:reddit' to get a refresh_token
for your account.
7 years ago
Mike Fährmann
4e80e0c884
[flickr] add user extractor ( #16 )
7 years ago
Mike Fährmann
c921b4f32a
code cleanup and fixing tests
7 years ago
Mike Fährmann
67791e1b36
[imgur] improve and add image extractor
7 years ago
Mike Fährmann
832a4a8ee9
[fallenangels] add manga extractor
7 years ago
Mike Fährmann
2974d782a3
[yomanga] remove module
...
site has been shut down
7 years ago
Mike Fährmann
677c8ced11
[deviantart] add "journal" extractor
...
(#14 )
7 years ago
Mike Fährmann
1e416e5227
update README
7 years ago