When logged in, some entries returned by Twitter's API are so called
'homeConversation's (they would be regular tweet entries otherwise.)
Those weren't picked up before and resulted in missing files compared
to accessing a timeline as guest.
('/media' timelines and search results were not affected)
It is no longer possible to get an 'authenticity_token' from Twitter's
Javascript-free login form, which got disabled few days ago.
Generating a random 16 byte hex string client-side and sending that as
a cookie alongside the regular login form works just as well.
Before it would use http:// for all requests and
get a redirect to a https:// version if those are supported.
Now the redirect only happens once during the first request.
Split the functionality of the old 'user' extractor into separate
'posts' and 'highlights' extractors, which respond to virtual URLs
('/<user>/posts' and '/<user>/highlights')
Twitter started using a bigger (80 instead of 16 bytes) CSRf token for
logged in users, and expects those to be used as 'x-csrf-token' header
when send via 'ct0' cookie.
Generating an 80 byte token ourselves doesn't work, and Twitter will
still insist on using its own.
(#1113, #1122, #1128, #1130, #1149)
Rely on the results of GraphQL queries instead of requesting data
for each post separately via '/p/<shortcode>/?__a=1'.
This might result in some missing metadata, and there might be some
issues for '/channel/' and '/saved/' URLs, but at least downloading
from the regular post listings should work without issues and without
getting users blocked/banned.
TODO: reimplement support for stories
- add support for sakugabooru.com (closes#1136)
- add support for lolibooru.moe (closes#1050)
This allows users to dynamically add support for moebooru/myimouto
based sites by adding an entry to their config file
(like for foolslide, foolfuuka, etc)
For example:
{
"extractor": {
"moebooru": {
"new-site-1": {"root": "https://site1.net"},
"new-site-2": {"root": "https://www.site2.moe"}
}
}
}
Mike Fährmann