1c1e086d01
use common base class for OAuth1.0 based API interfaces
6 years ago
6a31ada9e3
re-implement OAuth1.0 code
...
OAuth support for SmugMug needs some additional features
(auth-rebuild on redirect, query parameters in URL, ...)
and fixing this in the old code wouldn't work all that well.
6 years ago
69a5e6ddb3
Merge branch 'master' into 1.4-dev
6 years ago
8b79eaafea
[tumblr] log actual time of rate limit resets
...
... instead of the amount of seconds until a reset
6 years ago
f471161920
Merge branch 'master' into 1.4-dev
7 years ago
b1325d4d2c
fix extractor docstrings
7 years ago
728c64a3fb
[tumblr] rename 'offset' to 'num and adjust formats
...
Trying to somehow emulate Tumblr filenames is a bad idea ...
7 years ago
6bd857a319
[tumblr] handle rate limits / 429 errors
...
- wait for the hourly limit to reset
- abort upon exceeding the daily limit (it doesn't seem useful to
potentially wait for several hours)
7 years ago
a1fa4b43b0
Revert "[tumblr] add option to sort photosets by upload order"
...
This reverts commit 4a26ae32df
7 years ago
4a26ae32df
[tumblr] add option to sort photosets by upload order
7 years ago
6b72be8ee6
[tumblr] add 'hash' keyword
...
'hash' is the middle part of the filename in a tumblr image URL.
For example an image with '.../tumblr_p6tgemp1NZ1wgha4yo1_250.png' as
its URL would have 'p6tgemp1NZ1wgha4yo1' as hash.
7 years ago
68e9fbee16
[tumblr] check all 4 keys/secrets before using OAuth
...
it was possible to cause a crash by setting api-key or -secret to null.
(this commit also slightly improves the blog-cache implementation)
7 years ago
f8168c693e
[tumblr] avoid calls to '/blog/.../info'
...
The same information returned by the 'blog/.../info' API endpoint
is also included in the result of every 'blog/.../posts' call.
7 years ago
858fdbdb22
[tumblr] improve 'inline' extraction
...
'quote' posts store their HTML content in the 'source' field
7 years ago
5008e105ee
update archive IDs
...
... to behave in a more straightforward way when dealing with
bookmarks/favourites/etc.
specific IDs are now grouped by their owner, album-id, ... to
allow for duplicates when it would be expected.
7 years ago
3cec533c28
Merge branch 'archive'
7 years ago
d38bf2f54c
[tumblr] recognize /image/... URLs
...
xyz.tumblr.com/image/123 refers to the same images
as xyz.tumblr.com/post/123.
7 years ago
34873dbd90
set 'archive_fmt' values
...
These are going to be used to create an unique id for each image.
7 years ago
9fccd7b783
[tumblr] provide fallback URLs ( #64 )
...
Each image now produces 3 URLs:
- amazonaws.com _raw (or _1280 for older images)
- amazonaws.com _500
- media.tumblr.com (URL returned by API)
7 years ago
421a9740a3
[tumblr] add 'tumblr:' to force Tumblr extractor ( #71 )
7 years ago
9a049bdf51
[tumblr] add 'likes' extractor ( #65 )
7 years ago
29d75fc3fa
[tumblr] add support for OAuth authentication ( #65 )
7 years ago
75b2e84b6d
[tumblr] use s3.amazonaws.com for image URLs ( #64 )
7 years ago
03b8a548cb
[tumblr] change `reblogs` default value to `true` ( #61 )
7 years ago
d235f68f59
[tumblr] add option to filter reblogged posts ( #61 )
...
Reblogs are ignored by default, but can be included by setting
'extractor.tumblr.reblogs' to 'true'.
7 years ago
b14de6ffc2
[tumblr] small improvements
...
- don't transform inline GIF URLs
- set 'type' parameter for API calls if there is only
one post type selected
7 years ago
9296a26eae
[tumblr] add warning messages
7 years ago
12de658937
[tumblr] add options to control extraction behavior ( #48 )
...
- posts : list of post-types to inspect
- inline : scan post bodies for inline images
- external: follow external links
7 years ago
077f8c12be
[tumblr] original video URLs + continuous offset
7 years ago
8eb12ebeae
[tumblr] support more post/media types ( #48 )
...
This adds support for audio and video posts (most videos are shared
from youtube/instagram which isn't supported -> youtube-dl),
as well as link posts and image-search inside of text posts.
Most of this is just WIP and will need some sort of improvement
and options to enable/disable different media types etc.
7 years ago
980fd3616d
[tumblr] use API v2 ( #48 )
7 years ago
d6bed9f36f
[tumblr] prevent premature exit to get all images ( fixes #48 )
7 years ago
81a7788b40
replace space characters in unit test URLs
7 years ago
393755ee94
[tumblr] update tests
7 years ago
6f30cf4c64
change keyword names to valid Python identifiers
...
This commit mostly replaces all minus-signs ('-') in keyword names with
underscores ('_') to allow them to be used in filter-expressions. For
example 'gallery-id' got renamed to 'gallery_id'.
(It is theoretically possible to access any variable, regardless of its
name, with 'locals()["NAME"]', but that seems a bit too convoluted if
just 'NAME' could be enough)
7 years ago
80c2e03aaa
[reddit] allow 'date-min/max' to be human readable dates
...
If the date-min/max config value is a string, try parsing it using
datetime.strptime [1] with 'date-format' as format string [2]
(default: "%Y-%m-%dT%H:%M:%S")
Example: get all submissions posted in 2016
$ gallery-dl reddit.com/r/... \
-o date-format=%Y \
-o date-min=\"2016\" \
-o date-max=\"2017\"
[1] https://docs.python.org/3/library/datetime.html#datetime.datetime.strptime
[2] https://docs.python.org/3/library/datetime.html#strftime-strptime-behavior
7 years ago
71e08dc9c4
[tumblr] keyword consistency
8 years ago
bd95fea82c
update unit test results
8 years ago
94e10f249a
code adjustments according to pep8 nr2
8 years ago
0211ec4114
update some tests
8 years ago
8d106a447c
[tumblr] delete more useless keywords
8 years ago
56d810c896
update keyword hashes for tests
8 years ago
19c2d4ff6f
remove explicit (sub)category keywords
8 years ago
85ff3d160e
[tumblr] fix json parsing + metadata consistency
8 years ago
d7e168799d
consistent extractor naming scheme + docstrings
8 years ago
808cf69556
update a few tests
8 years ago
6f7d42b974
update tests
8 years ago
81096f7790
[tumblr] fix json parsing
9 years ago
f974ea73db
[tumblr] add tag-extractor
9 years ago
58a0029bb2
[tumblr] add post-extractor
9 years ago
Mike Fährmann