Mike Fährmann
fd858eed7b
[twitter] add 'user_likes' metadata field for liked tweets
...
i.e. the 'screen_name' of the user whose liked tweets get extracted.
Ideally this would replace 'user' or at least be in the same format,
but that would break backwards compatibility or be impossible/too
complicated thanks to API result differences.
(#1421 )
4 years ago
Mike Fährmann
8d124a3766
[twitter] rename variables
4 years ago
Mike Fährmann
105f3c9666
[twitter] add extractor for direct image links ( closes #1417 )
4 years ago
Mike Fährmann
ebd142e2a8
[twitter] don't use youtube-dl for cards when videos are disabled
...
(#1416 )
4 years ago
Mike Fährmann
ccfa5a8694
[twitter] better error message when logging in with 2FA ( #1409 )
4 years ago
Mike Fährmann
2846235669
[twitter] allow specifying a custom format for user results
...
(#1337 )
4 years ago
Mike Fährmann
3378b39719
[twitter] implement 'users' option ( #1337 )
4 years ago
Mike Fährmann
5d69e437d0
[twitter] add option to download all media from a conversation
...
(fixes #1319 )
4 years ago
Mike Fährmann
de0656941b
[twitter] add extractor for followed users ( #1337 )
...
https://twitter.com/USER/following or
https://twitter.com/id:USERID/following
4 years ago
Mike Fährmann
5542a11c46
[twitter] update GraphQL endpoints
4 years ago
Mike Fährmann
24e8e398e0
[twitter] skip login if 'auth_token' cookie is present
4 years ago
Mike Fährmann
95e5911895
[twitter] match '/i/user/ID' URLs
4 years ago
Mike Fährmann
069b113cbf
[twitter] improve and fix retry after hitting rate limit
...
- replace recursive call with infinite loop
- fix function arguments for recursive call
4 years ago
Mike Fährmann
780b6adb91
rename 'generate_csrf_token()' to just 'generate_token()'
...
and add a 'size' argument
4 years ago
Mike Fährmann
25074aec47
[twitter] fetch media from pinned tweets ( #1203 )
4 years ago
Mike Fährmann
2475176d99
[twitter] fetch tweets from 'homeConversation' entries
...
When logged in, some entries returned by Twitter's API are so called
'homeConversation's (they would be regular tweet entries otherwise.)
Those weren't picked up before and resulted in missing files compared
to accessing a timeline as guest.
('/media' timelines and search results were not affected)
4 years ago
Mike Fährmann
3af9350648
[twitter] update API calls
...
- use 'https://twitter.com/i/api ' for all requests
except '/guest/activate.json'
- update (default) URL parameters
- update GraphQL endpoints
4 years ago
Mike Fährmann
b656b829db
[twitter] fix login with username & password
...
It is no longer possible to get an 'authenticity_token' from Twitter's
Javascript-free login form, which got disabled few days ago.
Generating a random 16 byte hex string client-side and sending that as
a cookie alongside the regular login form works just as well.
4 years ago
Mike Fährmann
a00b60fbe7
[twitter] update 'x-csrf-token' header ( fixes #1170 )
...
Twitter started using a bigger (80 instead of 16 bytes) CSRf token for
logged in users, and expects those to be used as 'x-csrf-token' header
when send via 'ct0' cookie.
Generating an 80 byte token ourselves doesn't work, and Twitter will
still insist on using its own.
4 years ago
Mike Fährmann
63e61a0932
[twitter] update image URL format ( #1145 )
...
use
'/<name>?format=<fmt>&name=<size>'
instead of the potentially deprecated
'/<name>.<fmt>:<size>'
but keep all of them as fallback URLs
4 years ago
Mike Fährmann
ddfb4fd07a
[twitter] use ' https://twitter.com/i/api/ ' for logged in users
...
Doesn't seem to make a difference from what I can tell,
i.e. downloaded files are the same, but the website does it.
4 years ago
Mike Fährmann
de0c57886d
[twitter] add 'list-members' extractor ( closes #1096 )
4 years ago
Mike Fährmann
41d4968866
[twitter] add 'list' extractor ( #1096 )
4 years ago
Mike Fährmann
5d10520f4c
[twitter] update GraphQL endpoint & fix width/height entries
4 years ago
Mike Fährmann
968d3e8465
remove '&' from URL patterns
...
'/?&#' -> '/?#' and '?&#' -> '?#'
According to https://www.ietf.org/rfc/rfc3986.txt , URLs are
"organized hierarchically" by using "the slash ("/"), question
mark ("?"), and number sign ("#") characters to delimit components"
4 years ago
Mike Fährmann
1686dc1757
[twitter] support media from Cards ( #1005 , #937 )
...
Can be enabled with 'extractor.twitter.cards', but for now disabled by
default because cards can redirect to rather large videos from YouTube
or Twitch.
4 years ago
Mike Fährmann
a3ca2f6080
update fallback URL handling
...
remove Message.Urllist and use a '_fallback' field inside a kwdict
4 years ago
Mike Fährmann
1b1cf01d0d
add a general 'generate_csrf_token()' function
4 years ago
Mike Fährmann
844502cad5
update extractor test results
4 years ago
Mike Fährmann
430b6d6e2e
[twitter] extend 'retweets' option ( closes #1026 )
...
Setting 'retweets' to '"original"' will use metadata from the
original retweeted Tweets, and not from the Retweet entry.
4 years ago
Mike Fährmann
aeb0d32333
[twitter] improve twitpic extraction ( fixes #1019 )
...
- ignore twitpic.com/photos/… URLs
- ignore empty image URLs
4 years ago
Mike Fährmann
2b8d57f0ab
[twitter] support '/intent/user?user_id=…' URLs ( #980 )
4 years ago
Mike Fährmann
a3b473bd2f
[twitter] support specifying users by ID ( #980 )
...
by using 'id:…' as their screen name, i.e.
https://www.twitter.com/id:2976459548/media
instead of
https://twitter.com/supernaturepics/media
The user ID can, for example, be obtained from the output of
$ gallery-dl -j --range 1 https://twitter.com/ <screen-name>
4 years ago
Mike Fährmann
8f64585ff2
[twitter] handle 429 responses without x-rate-limit-reset header
4 years ago
Mike Fährmann
2da71cb561
[twitter] raise proper exception if user doesn't exist ( #891 )
4 years ago
Leonardo Taccari
86e5a05e29
[twitter] add support for nitter.net URLs in pattern ( #890 )
...
Please note that URLs are only "translated", all requests are still
done always via the Twitter API.
4 years ago
Mike Fährmann
3855d0dd3c
[twitter] add debug messages for all skipped Tweets ( #867 )
4 years ago
Mike Fährmann
6e2af9a8d8
[twitter] improve error message formatting
4 years ago
Mike Fährmann
9da2bc67f8
[twitter] add option to filter media from quoted tweets ( #854 )
4 years ago
Mike Fährmann
56ab5fb8f4
[twitter] improve handling of quoted tweets ( #854 )
...
Split each "quote" into two parts:
- the original tweet
- the tweet that quoted the original
4 years ago
Mike Fährmann
a8c2d997e8
[twitter] treat quoted tweets like retweets ( #833 )
...
- filter them when 'retweets' is disabled
- set 'author' to the creator of the quoted tweet
like it was before the rewrite
4 years ago
Mike Fährmann
aed1c63e51
[twitter] improve search results ( fixes #847 )
...
Adding 'tweet_search_mode=live' to the query parameters
is the most important part here.
4 years ago
Mike Fährmann
d81a8e6544
[twitter] update tests
4 years ago
Mike Fährmann
d39eedd9bb
[twitter] improve handling of deleted tweets ( fixes #838 )
4 years ago
Mike Fährmann
dc16f73965
[twitter] move '_guest_token()' into TwitterAPI class
4 years ago
Mike Fährmann
3561d1020a
[twitter] always provide an 'author' field ( #831 , #833 )
...
The idea was to have less metadata clutter for most Tweets were
'author' and 'user' are the same (non-retweets), and only provide
a 'user' field.
The original Tweet author could be gotten with
{author[…]|user[…]}, but basically no one knows about that.
4 years ago
Mike Fährmann
c37a1c06c8
[twitter] add extractor for liked tweets ( closes #837 )
...
You need to be logged in to get access to anyone's liked tweets,
it seems.
4 years ago
Mike Fährmann
b94394104c
[twitter] don't download video previews ( #833 )
...
when 'videos' is set to False
4 years ago
Mike Fährmann
036a40943a
[twitter] don't cache results of 'user_by_screen_name()'
...
A 'keyarg=1' argument to the memcache decorator would have worked as
well, but keeping the user object in memory isn't useful for the vast
majority of use cases and only wastes space.
(closes #817 )
4 years ago
Mike Fährmann
4442dfe7b8
[twitter] add 'reply_to' metadata to replies
4 years ago