bd0e1ca1a5
[imgur] build directory path for each file ( closes #842 )
4 years ago
a8c2d997e8
[twitter] treat quoted tweets like retweets ( #833 )
...
- filter them when 'retweets' is disabled
- set 'author' to the creator of the quoted tweet
like it was before the rewrite
4 years ago
aed1c63e51
[twitter] improve search results ( fixes #847 )
...
Adding 'tweet_search_mode=live' to the query parameters
is the most important part here.
4 years ago
0e714b9a0e
[pinterest] add 'section' extractor ( #835 )
4 years ago
53cc498d9c
improve config lookup when there are multiple possible locations
...
This specifically applies to all Mastodon extractors and all
extractors with a 'basecategory', i.e. 'booru', 'foolslide', etc.
Values inside those general config locations wouldn't be recognized
when a value with the same was set on the 'extractor' level.
For example 'extractor.mastodon.directory' should be used over
'extractor.directory' when both are set, but this was impossible
with the previous implementation.
(fixes #843 )
4 years ago
d81a8e6544
[twitter] update tests
4 years ago
d39eedd9bb
[twitter] improve handling of deleted tweets ( fixes #838 )
4 years ago
1ae1df0d27
update '--write-pages' ( #737 )
...
- fix infinite recursion for responses with multiple entries in
'history'
- hide values of Set-Cookie headers
- only write the response content by default
(use '-o write-pages=all' to also include HTTP headers)
4 years ago
dc16f73965
[twitter] move '_guest_token()' into TwitterAPI class
4 years ago
3561d1020a
[twitter] always provide an 'author' field ( #831 , #833 )
...
The idea was to have less metadata clutter for most Tweets were
'author' and 'user' are the same (non-retweets), and only provide
a 'user' field.
The original Tweet author could be gotten with
{author[…]|user[…]}, but basically no one knows about that.
4 years ago
7158bdd7c7
[weibo] improve extractor logic ( #829 )
4 years ago
0371fd54a1
[artstation] add 'date' metadata field ( #839 )
4 years ago
8c857052d7
[mastodon] ignore toots without media attachments
4 years ago
de045d39b2
[mastodon] add 'date' metadata field ( #839 )
4 years ago
d5d90a0450
[weibo] add 'date' field to 'status' objects ( #829 )
4 years ago
5ba90f72ca
[pinterest] add support for sections ( closes #835 )
4 years ago
c37a1c06c8
[twitter] add extractor for liked tweets ( closes #837 )
...
You need to be logged in to get access to anyone's liked tweets,
it seems.
4 years ago
b94394104c
[twitter] don't download video previews ( #833 )
...
when 'videos' is set to False
4 years ago
bb882b8cdb
improve output of '-K' for parent extractors ( #825 )
4 years ago
4cf3d54718
[kissmanga] workaround for CAPTCHAs ( fixes #818 )
...
Requesting the same page again when being redirected to a CAPTCHA
lets us access that page without solving it.
4 years ago
7daef6ee70
update extractor test results
...
- certain posts on Instagram now return
https://static.cdninstagram.com/rsrc.php/null.jpg
for public users
- MangaDex is deploying its new MangaDex@Home network similar to
exhentai's Hentai@Home
- realbooru has a new site layout, but the underlying booru API still
works like before
4 years ago
ffb6c5277a
[furaffinity] add 'artist_url' metadata field ( closes #821 )
4 years ago
be04e44e2c
[reddit] catch JSON decode errors ( #765 )
4 years ago
cf863f60b3
[redgifs] add 'user' and 'search' extractors ( closes #724 )
4 years ago
998d1d3a5c
[webtoons] generalize and improve comic extraction ( fixes #820 )
4 years ago
036a40943a
[twitter] don't cache results of 'user_by_screen_name()'
...
A 'keyarg=1' argument to the memcache decorator would have worked as
well, but keeping the user object in memory isn't useful for the vast
majority of use cases and only wastes space.
(closes #817 )
4 years ago
4442dfe7b8
[twitter] add 'reply_to' metadata to replies
4 years ago
83b7bd0413
[nhentai] fix extraction ( closes #819 )
4 years ago
d769bb4b80
[twitter] improve pagination
4 years ago
5bc1097f9d
[twitter] metadata cleanup #2
...
- remove useless clutter by creating new tweet-data dicts instead of
reusing the original Tweet objects
- rename fields to how they were named before
('id_str' -> 'tweet_id', etc.)
- only include 'author' if it would differ from 'user'
- restore 'archive_fmt'
4 years ago
c6c06c41f6
[deviantart] don't add journal text to description ( #712 )
4 years ago
4aea5138dd
[sensescans] use https://
4 years ago
3eed5f52d7
[twitter] small metadata cleanup
...
- add 'date' field
- remove 'entities' and 'extended_entities'
- don't include 'focus_fields' from 'original_info'
4 years ago
655c98cbef
[twitter] skip unavailable tweets
4 years ago
41d03160ff
[deviantart] also search journals for sta.sh links ( #712 )
...
when 'extra' is enabled
4 years ago
2132e5461a
[twitter] restore TwitPic support
4 years ago
bd0f21478a
[twitter] login using the mobile nojs login page
4 years ago
a10f31dde5
[twitter] rewrite; use new interface ( #740 , #806 )
...
Everything except logging in with username & password and TwitPic
embeds should be working again.
Metadata per Tweet is massively different than before (mostly raw API
responses - might need some cleaning up) and the default 'archive_fmt'
changed.
4 years ago
3bad1579ee
update extractor test results
4 years ago
864f4220d9
update output of 'oauth:…' ( #616 )
4 years ago
0f459f340b
[instagram] fix and re-enable login with username&password
...
This reverts commit 3e0848a482#756 , #771 , #797 , #803 )
https://github.com/althonos/InsaLooter/issues/287#issuecomment-630456522
4 years ago
3e0848a482
[instagram] disable login with username&password ( #756 )
4 years ago
a32aea41e1
[instagram] update 'query_hash' values
4 years ago
2bff8dd465
[hentainexus] fix flake8 issues ( #787 )
4 years ago
a63682a9c0
[instagram] simplify code & complete tests ( #743 )
4 years ago
a4e3d40672
hentainexus.py minor fix ( #787 )
...
* rectify code of `join_title`, some minor fix.
* + hentainexus self.data
* fixed: call staticmethod join_title with data
4 years ago
62b65e59d0
Add instagram metadata: post_pageurl, post_tags ( #743 )
...
* Add instagram metadata: post_pageurl, post_tags
Add the following metadata for instagram:
- post_pageurl: json string with url of the post page
- post_tags: json array with instagram tags extracted from the post description
* Oops: rename post_tags to tags for --write-tags
This way, --write-tags will pick up the post tags.
* Rename to post_url, improve regex
* Add post_url and tags to tests
* Remove duplicate tags and sort them
* Bugfix: don't create empty tag lists
* Metadata: add location
* Metadata: add tagged_users for each media
* Move self._find_tags() to base class
* Make flake happy
4 years ago
275cceeb6a
[redgifs] fix extraction ( #724 )
...
… and prepare for more potential extractors
4 years ago
45baa13615
update extractor test results
...
- don't run Instagram tests on Travis anymore
- replace Twitter test because timeline was made private
- update Hiperdex domain to '.com' (again ...)
4 years ago
dfcf2a2c91
write OAuth token to cache by default ( #616 )
4 years ago
15c3d29062
move dump_response() into a separate function ( #737 )
4 years ago
a363da4b43
include redirects and headers in --write-pages dumps ( #737 )
4 years ago
6bcdb264e0
[imgur] treat 't/unmuted' URLs as galleries
4 years ago
b6cee3e45b
[imgur] fix extraction of animated images without 'mp4' entry
4 years ago
bcac31b7c7
[webtoons] make archive_fmt unique ( #779 )
...
close #778
4 years ago
e19f665a44
[danbooru] change default for 'ugoira' to 'false'
...
Downloading the pre-rendered versions should be a better default
than .zip files with individual frames.
4 years ago
3201fe3521
add global SENTINEL object
4 years ago
c8787647ed
add global WINDOWS bool
4 years ago
6294e2c540
add 'text.ensure_http_scheme()'
4 years ago
0378d079a5
[webtoons] fixes and simplifications ( #593 , #761 )
...
- fix episode listings for french comics
- allow input URLs without explicit scheme
- add 'lang'/'language' metadata
- use str.format() instead of '+' to assemble URLs
4 years ago
ab11b1c896
[imagechest] simplify code ( #750 )
4 years ago
846d3a2466
[sexcom] replace 404ed test
4 years ago
9b4635917f
[gelbooru] simplify and fix pool extraction
...
use 'pool:<pool id>' as search tag to get pool posts
4 years ago
39cd389679
[webtoons] Add a new extractor for webtoons.com ( #761 )
...
The webtoons extractor can extract episode and entire comic (all
episodes) from webtoons.com.
All the logic of the extractors should be trivial except for a couple
of kludges needed:
- `ageGatePass' cookie is always set to avoid possible redirect and stop of
extraction, especially in the comic extractor
- The image URLs returned by the episode extractor could not be fetched
directly and the `Referer:' HTTP header needs to be passed to fetch them
Close #593 .
4 years ago
7b5711ee04
[imagechest] Add new extractor for ImageChest ( #750 )
...
* [imagechest] Add new extractor for ImageChest
* [imagechest] Fix flake8 compliance issues
4 years ago
a1e739b96c
reuse connection adapters from parent extractors
4 years ago
f8f95e68a7
improve '--write-pages' ( #737 )
...
- move code into its own function
- add enumeration index to filenames
- dump responses regardless of status code
4 years ago
09cc9dbec0
prevent flake8 errors from comments looking like type annotations
4 years ago
2d6724180b
[hiperdex] update domain to hiperdex.info
4 years ago
4cc761c730
Implement --write-pages option ( #736 )
...
* Implement --write-pages option
* Fix long lines
* Fix file mode to binary
* Fix pattern for Windows compatibility
4 years ago
f557cac074
[redgifs] add image extractor ( #724 )
4 years ago
65b1cb7acd
[deviantart] use private access tokens for Journals ( fixes #738 )
4 years ago
0bf0146bfe
[reddit] don't send OAuth headers for file downloads ( fixes #729 )
4 years ago
d6a480682f
update test results
4 years ago
b47cfc5ac9
[speakerdeck] Add a new extractor for speakerdeck.com ( #726 )
4 years ago
90491ab606
[artstation] improve embed extraction ( #720 )
4 years ago
999efec5cc
[deviantart] limit API wait times to 2**9=512 seconds ( #721 )
4 years ago
504de79d8b
[vsco] fix extraction
4 years ago
5e2974d699
[weibo] add 'videos' option
4 years ago
9f638c2e01
[twitter] add 'replies' option ( closes #705 )
4 years ago
fc3e54275b
[patreon] respect filters and sort order in query params ( #711 )
4 years ago
46b9a4d8ff
[patreon] improve hash extraction ( #693 , #713 )
...
Instead of accessing a specific part of a download URL, potentially
causing an exception if it doesn't exist, we're now searching through
all parts for a potential MD5 hash without ever raising an exception.
4 years ago
c56a751dae
[newgrounds] fix URLs produced by 'followng' extractors ( #684 )
4 years ago
a4fd620a25
[hiperdex] revert domain back to hiperdex.com
4 years ago
233b6f93a2
[patreon] recognize URLs with creator IDs ( #711 )
...
e.g. https://www.patreon.com/user/posts?u= …
4 years ago
38b6bd66b0
[500px] match 'web.500px.com' subdomains
4 years ago
d3b3b30107
update test results
4 years ago
5d7ca76885
retry Cloudflare challenges
4 years ago
3eab07739f
[twitter] ensure videos have a 'filename'
...
This usually gets set when invoking the 'ytdl' downloader, but when
that fails, the error message would use 'None' as filename.
4 years ago
c4371a6970
[twitter] add 'reply' metadata field ( #705 )
4 years ago
12ff23b6cc
[mastodon] improve account searches ( fixes #704 )
...
Searching for just the username ("@NAME") can produce multiple
unrelated results, so we now search for username + mastodon instance
("@NAME@INSTANCE")
4 years ago
400a0df661
[jaiminisbox] update decoding procedure ( fixes #702 )
4 years ago
8fe858eb0e
improve parameter extraction when solving Cloudflare challenge
4 years ago
fb98b567fa
[gelbooru] improve post ID extraction for pools
4 years ago
d6facdee7b
[mastodon] add tests ( #701 )
4 years ago
12eebb6f16
[xhamster] support xhamster.porncache.net domains ( closes #700 )
4 years ago
e749402191
[mastodon] fix pagination ( #701 )
4 years ago
921914141e
[imgbb] improve redirect handling
4 years ago
6cc800aad4
[instagram] add 'post_id' and 'num' metadata fields ( closes #698 )
4 years ago
a3de234e70
[hitomi] add extractor for tag searches ( closes #697 )
4 years ago
Mike Fährmann
