Mike Fährmann
d39eedd9bb
[twitter] improve handling of deleted tweets ( fixes #838 )
4 years ago
Mike Fährmann
1ae1df0d27
update '--write-pages' ( #737 )
...
- fix infinite recursion for responses with multiple entries in
'history'
- hide values of Set-Cookie headers
- only write the response content by default
(use '-o write-pages=all' to also include HTTP headers)
4 years ago
Mike Fährmann
7e8a747c56
improve output of '-K' for parent extractors 2 ( #825 )
...
This is what bb882b8
was supposed to be, but I managed to
not include those changes in the first commit …
4 years ago
Mike Fährmann
dc16f73965
[twitter] move '_guest_token()' into TwitterAPI class
4 years ago
Mike Fährmann
3561d1020a
[twitter] always provide an 'author' field ( #831 , #833 )
...
The idea was to have less metadata clutter for most Tweets were
'author' and 'user' are the same (non-retweets), and only provide
a 'user' field.
The original Tweet author could be gotten with
{author[…]|user[…]}, but basically no one knows about that.
4 years ago
Mike Fährmann
7158bdd7c7
[weibo] improve extractor logic ( #829 )
4 years ago
Mike Fährmann
37d71f6e09
strip microseconds in text.parse_datetime()
4 years ago
Mike Fährmann
0371fd54a1
[artstation] add 'date' metadata field ( #839 )
4 years ago
Mike Fährmann
8c857052d7
[mastodon] ignore toots without media attachments
4 years ago
Mike Fährmann
de045d39b2
[mastodon] add 'date' metadata field ( #839 )
4 years ago
Mike Fährmann
d5d90a0450
[weibo] add 'date' field to 'status' objects ( #829 )
4 years ago
Mike Fährmann
5ba90f72ca
[pinterest] add support for sections ( closes #835 )
4 years ago
Mike Fährmann
c37a1c06c8
[twitter] add extractor for liked tweets ( closes #837 )
...
You need to be logged in to get access to anyone's liked tweets,
it seems.
4 years ago
Mike Fährmann
b94394104c
[twitter] don't download video previews ( #833 )
...
when 'videos' is set to False
4 years ago
Mike Fährmann
bb882b8cdb
improve output of '-K' for parent extractors ( #825 )
4 years ago
Mike Fährmann
6db7ed90cb
release version 1.14.1
4 years ago
Mike Fährmann
087e3184dc
use a non-twitter URL when testing snap creation
4 years ago
Mike Fährmann
c184cce876
update configuration.rst
...
- fix anonymous links
- update description of 'extractor.twitter.videos'
- document 'extractor.redgifs.format' (#724 )
4 years ago
Mike Fährmann
4cf3d54718
[kissmanga] workaround for CAPTCHAs ( fixes #818 )
...
Requesting the same page again when being redirected to a CAPTCHA
lets us access that page without solving it.
4 years ago
Mike Fährmann
7daef6ee70
update extractor test results
...
- certain posts on Instagram now return
https://static.cdninstagram.com/rsrc.php/null.jpg
for public users
- MangaDex is deploying its new MangaDex@Home network similar to
exhentai's Hentai@Home
- realbooru has a new site layout, but the underlying booru API still
works like before
4 years ago
Mike Fährmann
ffb6c5277a
[furaffinity] add 'artist_url' metadata field ( closes #821 )
4 years ago
Mike Fährmann
be04e44e2c
[reddit] catch JSON decode errors ( #765 )
4 years ago
Mike Fährmann
cf863f60b3
[redgifs] add 'user' and 'search' extractors ( closes #724 )
4 years ago
Mike Fährmann
998d1d3a5c
[webtoons] generalize and improve comic extraction ( fixes #820 )
4 years ago
Mike Fährmann
1489712325
resolve redirects after solving Cloudflare challanges
4 years ago
Mike Fährmann
b0b1feaa67
request 'transparent.gif' when solving Cloudflare challenges
...
This currently also works without, but they might be using these to
detect potential bots in the future.
4 years ago
Mike Fährmann
036a40943a
[twitter] don't cache results of 'user_by_screen_name()'
...
A 'keyarg=1' argument to the memcache decorator would have worked as
well, but keeping the user object in memory isn't useful for the vast
majority of use cases and only wastes space.
(closes #817 )
4 years ago
Mike Fährmann
4442dfe7b8
[twitter] add 'reply_to' metadata to replies
4 years ago
Mike Fährmann
83b7bd0413
[nhentai] fix extraction ( closes #819 )
4 years ago
Mike Fährmann
d769bb4b80
[twitter] improve pagination
4 years ago
Mike Fährmann
5bc1097f9d
[twitter] metadata cleanup #2
...
- remove useless clutter by creating new tweet-data dicts instead of
reusing the original Tweet objects
- rename fields to how they were named before
('id_str' -> 'tweet_id', etc.)
- only include 'author' if it would differ from 'user'
- restore 'archive_fmt'
4 years ago
Mike Fährmann
1fcf938f9c
implement a general 'delete_items()' function
4 years ago
Mike Fährmann
c6c06c41f6
[deviantart] don't add journal text to description ( #712 )
4 years ago
Mike Fährmann
4aea5138dd
[sensescans] use https://
4 years ago
Mike Fährmann
3eed5f52d7
[twitter] small metadata cleanup
...
- add 'date' field
- remove 'entities' and 'extended_entities'
- don't include 'focus_fields' from 'original_info'
4 years ago
Mike Fährmann
655c98cbef
[twitter] skip unavailable tweets
4 years ago
Mike Fährmann
41d03160ff
[deviantart] also search journals for sta.sh links ( #712 )
...
when 'extra' is enabled
4 years ago
Mike Fährmann
0138e9c1d8
[twitter] remove 'content' option
...
The text content of each tweet is always available as 'full_text'
4 years ago
Mike Fährmann
2132e5461a
[twitter] restore TwitPic support
4 years ago
Mike Fährmann
bd0f21478a
[twitter] login using the mobile nojs login page
4 years ago
Mike Fährmann
a10f31dde5
[twitter] rewrite; use new interface ( #740 , #806 )
...
Everything except logging in with username & password and TwitPic
embeds should be working again.
Metadata per Tweet is massively different than before (mostly raw API
responses - might need some cleaning up) and the default 'archive_fmt'
changed.
4 years ago
Mike Fährmann
f1ef9082bb
release version 1.14.0
4 years ago
Mike Fährmann
3bad1579ee
update extractor test results
4 years ago
Mike Fährmann
864f4220d9
update output of 'oauth:…' ( #616 )
4 years ago
Mike Fährmann
0f459f340b
[instagram] fix and re-enable login with username&password
...
This reverts commit 3e0848a482
.
(#756 , #771 , #797 , #803 )
https://github.com/althonos/InsaLooter/issues/287#issuecomment-630456522
4 years ago
Mike Fährmann
c4d06a8331
add section about cookies to README.rst
4 years ago
Mike Fährmann
3e0848a482
[instagram] disable login with username&password ( #756 )
4 years ago
Mike Fährmann
a32aea41e1
[instagram] update 'query_hash' values
4 years ago
Mike Fährmann
2bff8dd465
[hentainexus] fix flake8 issues ( #787 )
4 years ago
Mike Fährmann
a63682a9c0
[instagram] simplify code & complete tests ( #743 )
4 years ago