Mike Fährmann
a453335a9f
remove test results in extractor modules
...
and add generic example URLs
1 year ago
Mike Fährmann
4c0b3d5dc5
[twitter] fix crash when 'sortIndex' is None ( #4499 )
1 year ago
Mike Fährmann
06aaedded5
[twitter] extract 'source' metadata ( #4459 )
1 year ago
Mike Fährmann
e0829ff0fd
[twitter] add 'date_original' metadata for retweets ( #4337 , #4443 )
1 year ago
Mike Fährmann
2b88ad19e9
[twitter] accept 'x.com' URLs ( #4452 )
1 year ago
Mike Fährmann
089d1a4f67
[twitter] fix 'TweetWithVisibilityResults' ( #4369 )
1 year ago
Mike Fährmann
fb3f0453db
[twitter] improve error messages for single Tweets ( #4369 )
...
also fixes '"quoted": false' not having any effect
1 year ago
Mike Fährmann
7fbc304ae9
[twitter] fix crash on private user ( #4349 )
1 year ago
Mike Fährmann
a383eca7f6
decouple extractor initialization
...
Introduce an 'initialize()' function that does the actual init
(session, cookies, config options) and can called separately from
the constructor __init__().
This allows, for example, to adjust config access inside a Job
before most of it already happened when calling 'extractor.find()'.
1 year ago
Mike Fährmann
d97b8c2fba
consistent cookie-related names
...
- rename every cookie variable or method to 'cookies_*'
- simplify '.session.cookies' to just '.cookies'
- more consistent 'login()' structure
1 year ago
Mike Fährmann
90231f2d5a
[twitter] add 'tweet-endpoint' option ( #4307 )
...
use the newer TweetResultByRestId only for guests by default
1 year ago
Mike Fährmann
20ed647f6f
[twitter] add 'user' extractor and 'include' option ( #4275 )
1 year ago
Mike Fährmann
86be197d11
[twitter] remove '/search/adaptive.json'
1 year ago
Mike Fährmann
0b08e2e8a8
merge #4287 : [twitter] Fix following extractor not getting all users
1 year ago
Mike Fährmann
f6553ffd2f
[twitter] simplify '_pagination_users'
...
- remove 'stop' variable
- call 'cursor.startswith()' only once
1 year ago
Mike Fährmann
a27dbe8c82
[twitter] use 'TweetResultByRestId' endpoint ( #4250 )
...
allows accessing single Tweets without login
1 year ago
Mike Fährmann
d3d639a159
[twitter] don't treat missing 'TimelineAddEntries' as fatal ( #4278 )
1 year ago
ActuallyKit
c321c773f2
make the code less ugly
1 year ago
ActuallyKit
a437a34bcf
fix lint i guess?
1 year ago
ActuallyKit
6cbc434b54
Fix users pagination
1 year ago
Mike Fährmann
1bf9f52c99
[twitter] add 'ratelimit' option ( #4251 )
1 year ago
Mike Fährmann
f86fdf64a6
[twitter] use GraphQL search by default ( #4264 )
1 year ago
Mike Fährmann
c1cce4a80b
[twitter] extend 'conversations' option ( #4211 )
1 year ago
Mike Fährmann
54cf1fa3e7
[twitter] use GraphQL search endpoint ( #3942 )
...
for guest users; selectable with 'search-endpoint' option.
adapted from 9c7b888ffa
1 year ago
Mike Fährmann
864a654b25
[twitter] update query hashes
1 year ago
Mike Fährmann
45cc7cee1a
[twitter] better error message for guest searches ( #3942 )
1 year ago
Mike Fährmann
271f23d971
[twitter] extract 'conversation_id' metadata ( #3839 )
1 year ago
Mike Fährmann
d0184fddcf
[twitter] optimize '_extract_twitpic()'
...
- use findall instead of finditer
- store URLs in a dict to discard duplicates
1 year ago
Mike Fährmann
3dc862c7fc
merge #3796 : [twitter] extract TwitPic URLs in text ( #3792 )
1 year ago
Mike Fährmann
1d505b39f8
[twitter] support 'profile-conversation' entries ( #3938 )
1 year ago
Mike Fährmann
f500b45b5e
[twitter] improve 480bc34e
...
only check for double user assignment where necessary
1 year ago
Mike Fährmann
480bc34e54
[twitter] do not overwrite previously assigned users ( #3922 )
1 year ago
Mike Fährmann
f5a59c4170
[twitter] add 'date_bookmarked' metadata ( #3816 )
1 year ago
Mike Fährmann
1c1f6fdc80
[twitter] fix regression from 160335ad
...
Tweets from 'homeConversation' or 'conversationthread' entries do not
contain a 'sortIndex' field. Accessing it raises a KeyError and would
erroneously get them labeled as 'deleted'.
1 year ago
Mike Fährmann
160335ad44
[twitter] add 'date_liked' metadata for liked Tweets ( #3816 )
1 year ago
Mike Fährmann
6d850ce629
[twitter] calculate 'date' from Tweet IDs
...
20 times faster than parsing 'created_at'
1 year ago
Mike Fährmann
dbe06cdba1
[twitter] warn about 'withheld' Tweets and users ( #3864 )
1 year ago
Mike Fährmann
3cc1dd1572
[twitter] update query hashes
1 year ago
Mike Fährmann
3846ce0de5
[twitter] update to bookmark timeline v2 ( #3859 )
1 year ago
Mike Fährmann
e6cb92864a
[twitter] allow setting custom features per API endpoint
1 year ago
Amer Jazaerli
bebbff6578
fix: graphql_timeline_v2_bookmark_timeline cannot be null
...
twitter: 400 Bad Request (The following features cannot be null: graphql_timeline_v2_bookmark_timeline)
1 year ago
Mike Fährmann
197882cf12
[twitter] add 'hashtag' extractor ( #3783 )
2 years ago
ClosedPort22
d4fb4ff47f
[twitter] extract TwitPic URLs in text ( #3792 )
...
also ignore previously seen URLs
2 years ago
Mike Fährmann
2bb937014f
[twitter] fall back to legacy /media endpoint when not logged in
2 years ago
Mike Fährmann
b68094d326
[twitter] support 'note_tweet's
2 years ago
Mike Fährmann
3dcabc97ed
[twitter] update API endpoints and parameters
2 years ago
Mike Fährmann
9037128315
[twitter] fix some 'original' retweets not downloading ( #3744 )
2 years ago
Mike Fährmann
dd884b02ee
replace json.loads with direct calls to JSONDecoder.decode
2 years ago
Mike Fährmann
1ae48a54f8
[twitter] add 'transform' option
2 years ago
ClosedPort22
ab58c375b4
[twitter] fix search ( #3536 )
...
- partially revert 18fe4b334d
- properly search for cursor when processing 'replaceEntry'
2 years ago