Mike Fährmann
be04e44e2c
[reddit] catch JSON decode errors ( #765 )
4 years ago
Mike Fährmann
cf863f60b3
[redgifs] add 'user' and 'search' extractors ( closes #724 )
4 years ago
Mike Fährmann
998d1d3a5c
[webtoons] generalize and improve comic extraction ( fixes #820 )
4 years ago
Mike Fährmann
036a40943a
[twitter] don't cache results of 'user_by_screen_name()'
...
A 'keyarg=1' argument to the memcache decorator would have worked as
well, but keeping the user object in memory isn't useful for the vast
majority of use cases and only wastes space.
(closes #817 )
4 years ago
Mike Fährmann
4442dfe7b8
[twitter] add 'reply_to' metadata to replies
4 years ago
Mike Fährmann
83b7bd0413
[nhentai] fix extraction ( closes #819 )
4 years ago
Mike Fährmann
d769bb4b80
[twitter] improve pagination
4 years ago
Mike Fährmann
5bc1097f9d
[twitter] metadata cleanup #2
...
- remove useless clutter by creating new tweet-data dicts instead of
reusing the original Tweet objects
- rename fields to how they were named before
('id_str' -> 'tweet_id', etc.)
- only include 'author' if it would differ from 'user'
- restore 'archive_fmt'
4 years ago
Mike Fährmann
c6c06c41f6
[deviantart] don't add journal text to description ( #712 )
4 years ago
Mike Fährmann
4aea5138dd
[sensescans] use https://
4 years ago
Mike Fährmann
3eed5f52d7
[twitter] small metadata cleanup
...
- add 'date' field
- remove 'entities' and 'extended_entities'
- don't include 'focus_fields' from 'original_info'
4 years ago
Mike Fährmann
655c98cbef
[twitter] skip unavailable tweets
4 years ago
Mike Fährmann
41d03160ff
[deviantart] also search journals for sta.sh links ( #712 )
...
when 'extra' is enabled
4 years ago
Mike Fährmann
2132e5461a
[twitter] restore TwitPic support
4 years ago
Mike Fährmann
bd0f21478a
[twitter] login using the mobile nojs login page
4 years ago
Mike Fährmann
a10f31dde5
[twitter] rewrite; use new interface ( #740 , #806 )
...
Everything except logging in with username & password and TwitPic
embeds should be working again.
Metadata per Tweet is massively different than before (mostly raw API
responses - might need some cleaning up) and the default 'archive_fmt'
changed.
4 years ago
Mike Fährmann
3bad1579ee
update extractor test results
4 years ago
Mike Fährmann
864f4220d9
update output of 'oauth:…' ( #616 )
4 years ago
Mike Fährmann
0f459f340b
[instagram] fix and re-enable login with username&password
...
This reverts commit 3e0848a482
.
(#756 , #771 , #797 , #803 )
https://github.com/althonos/InsaLooter/issues/287#issuecomment-630456522
4 years ago
Mike Fährmann
3e0848a482
[instagram] disable login with username&password ( #756 )
4 years ago
Mike Fährmann
a32aea41e1
[instagram] update 'query_hash' values
4 years ago
Mike Fährmann
2bff8dd465
[hentainexus] fix flake8 issues ( #787 )
4 years ago
Mike Fährmann
a63682a9c0
[instagram] simplify code & complete tests ( #743 )
4 years ago
墨焓
a4e3d40672
hentainexus.py minor fix ( #787 )
...
* rectify code of `join_title`, some minor fix.
* + hentainexus self.data
* fixed: call staticmethod join_title with data
4 years ago
Vrihub
62b65e59d0
Add instagram metadata: post_pageurl, post_tags ( #743 )
...
* Add instagram metadata: post_pageurl, post_tags
Add the following metadata for instagram:
- post_pageurl: json string with url of the post page
- post_tags: json array with instagram tags extracted from the post description
* Oops: rename post_tags to tags for --write-tags
This way, --write-tags will pick up the post tags.
* Rename to post_url, improve regex
* Add post_url and tags to tests
* Remove duplicate tags and sort them
* Bugfix: don't create empty tag lists
* Metadata: add location
* Metadata: add tagged_users for each media
* Move self._find_tags() to base class
* Make flake happy
4 years ago
Mike Fährmann
275cceeb6a
[redgifs] fix extraction ( #724 )
...
… and prepare for more potential extractors
4 years ago
Mike Fährmann
45baa13615
update extractor test results
...
- don't run Instagram tests on Travis anymore
- replace Twitter test because timeline was made private
- update Hiperdex domain to '.com' (again ...)
4 years ago
Mike Fährmann
dfcf2a2c91
write OAuth token to cache by default ( #616 )
4 years ago
Mike Fährmann
15c3d29062
move dump_response() into a separate function ( #737 )
4 years ago
Mike Fährmann
a363da4b43
include redirects and headers in --write-pages dumps ( #737 )
4 years ago
Mike Fährmann
6bcdb264e0
[imgur] treat 't/unmuted' URLs as galleries
4 years ago
Mike Fährmann
b6cee3e45b
[imgur] fix extraction of animated images without 'mp4' entry
4 years ago
Leonardo Taccari
bcac31b7c7
[webtoons] make archive_fmt unique ( #779 )
...
close #778
4 years ago
Mike Fährmann
e19f665a44
[danbooru] change default for 'ugoira' to 'false'
...
Downloading the pre-rendered versions should be a better default
than .zip files with individual frames.
4 years ago
Mike Fährmann
3201fe3521
add global SENTINEL object
4 years ago
Mike Fährmann
c8787647ed
add global WINDOWS bool
4 years ago
Mike Fährmann
6294e2c540
add 'text.ensure_http_scheme()'
4 years ago
Mike Fährmann
0378d079a5
[webtoons] fixes and simplifications ( #593 , #761 )
...
- fix episode listings for french comics
- allow input URLs without explicit scheme
- add 'lang'/'language' metadata
- use str.format() instead of '+' to assemble URLs
4 years ago
Mike Fährmann
ab11b1c896
[imagechest] simplify code ( #750 )
4 years ago
Mike Fährmann
846d3a2466
[sexcom] replace 404ed test
4 years ago
Mike Fährmann
9b4635917f
[gelbooru] simplify and fix pool extraction
...
use 'pool:<pool id>' as search tag to get pool posts
4 years ago
Leonardo Taccari
39cd389679
[webtoons] Add a new extractor for webtoons.com ( #761 )
...
The webtoons extractor can extract episode and entire comic (all
episodes) from webtoons.com.
All the logic of the extractors should be trivial except for a couple
of kludges needed:
- `ageGatePass' cookie is always set to avoid possible redirect and stop of
extraction, especially in the comic extractor
- The image URLs returned by the episode extractor could not be fetched
directly and the `Referer:' HTTP header needs to be passed to fetch them
Close #593 .
4 years ago
Bepis
7b5711ee04
[imagechest] Add new extractor for ImageChest ( #750 )
...
* [imagechest] Add new extractor for ImageChest
* [imagechest] Fix flake8 compliance issues
4 years ago
Mike Fährmann
a1e739b96c
reuse connection adapters from parent extractors
4 years ago
Mike Fährmann
f8f95e68a7
improve '--write-pages' ( #737 )
...
- move code into its own function
- add enumeration index to filenames
- dump responses regardless of status code
4 years ago
Mike Fährmann
09cc9dbec0
prevent flake8 errors from comments looking like type annotations
4 years ago
Mike Fährmann
2d6724180b
[hiperdex] update domain to hiperdex.info
4 years ago
Vrihub
4cc761c730
Implement --write-pages option ( #736 )
...
* Implement --write-pages option
* Fix long lines
* Fix file mode to binary
* Fix pattern for Windows compatibility
4 years ago
Mike Fährmann
f557cac074
[redgifs] add image extractor ( #724 )
4 years ago
Mike Fährmann
65b1cb7acd
[deviantart] use private access tokens for Journals ( fixes #738 )
4 years ago
Mike Fährmann
0bf0146bfe
[reddit] don't send OAuth headers for file downloads ( fixes #729 )
4 years ago
Mike Fährmann
d6a480682f
update test results
4 years ago
Leonardo Taccari
b47cfc5ac9
[speakerdeck] Add a new extractor for speakerdeck.com ( #726 )
4 years ago
Mike Fährmann
90491ab606
[artstation] improve embed extraction ( #720 )
4 years ago
Mike Fährmann
999efec5cc
[deviantart] limit API wait times to 2**9=512 seconds ( #721 )
4 years ago
Mike Fährmann
504de79d8b
[vsco] fix extraction
4 years ago
Mike Fährmann
5e2974d699
[weibo] add 'videos' option
4 years ago
Mike Fährmann
9f638c2e01
[twitter] add 'replies' option ( closes #705 )
4 years ago
Mike Fährmann
fc3e54275b
[patreon] respect filters and sort order in query params ( #711 )
4 years ago
Mike Fährmann
46b9a4d8ff
[patreon] improve hash extraction ( #693 , #713 )
...
Instead of accessing a specific part of a download URL, potentially
causing an exception if it doesn't exist, we're now searching through
all parts for a potential MD5 hash without ever raising an exception.
4 years ago
Mike Fährmann
c56a751dae
[newgrounds] fix URLs produced by 'followng' extractors ( #684 )
4 years ago
Mike Fährmann
a4fd620a25
[hiperdex] revert domain back to hiperdex.com
4 years ago
Mike Fährmann
233b6f93a2
[patreon] recognize URLs with creator IDs ( #711 )
...
e.g. https://www.patreon.com/user/posts?u= …
4 years ago
Mike Fährmann
38b6bd66b0
[500px] match 'web.500px.com' subdomains
4 years ago
Mike Fährmann
d3b3b30107
update test results
4 years ago
Mike Fährmann
5d7ca76885
retry Cloudflare challenges
4 years ago
Mike Fährmann
3eab07739f
[twitter] ensure videos have a 'filename'
...
This usually gets set when invoking the 'ytdl' downloader, but when
that fails, the error message would use 'None' as filename.
4 years ago
Mike Fährmann
c4371a6970
[twitter] add 'reply' metadata field ( #705 )
4 years ago
Mike Fährmann
12ff23b6cc
[mastodon] improve account searches ( fixes #704 )
...
Searching for just the username ("@NAME") can produce multiple
unrelated results, so we now search for username + mastodon instance
("@NAME@INSTANCE")
4 years ago
Mike Fährmann
400a0df661
[jaiminisbox] update decoding procedure ( fixes #702 )
4 years ago
Mike Fährmann
8fe858eb0e
improve parameter extraction when solving Cloudflare challenge
4 years ago
Mike Fährmann
fb98b567fa
[gelbooru] improve post ID extraction for pools
4 years ago
Mike Fährmann
d6facdee7b
[mastodon] add tests ( #701 )
4 years ago
Mike Fährmann
12eebb6f16
[xhamster] support xhamster.porncache.net domains ( closes #700 )
4 years ago
Mike Fährmann
e749402191
[mastodon] fix pagination ( #701 )
4 years ago
Mike Fährmann
921914141e
[imgbb] improve redirect handling
4 years ago
Mike Fährmann
6cc800aad4
[instagram] add 'post_id' and 'num' metadata fields ( closes #698 )
4 years ago
Mike Fährmann
a3de234e70
[hitomi] add extractor for tag searches ( closes #697 )
4 years ago
Mike Fährmann
456f6e8d05
[nozomi] move '_unpack()' method to global scope
4 years ago
Mike Fährmann
55ac408bdf
[hitomi] fix extraction of galleries without tags
4 years ago
Mike Fährmann
db6685eeae
[aryion] support downloading from folders ( fixes #694 )
4 years ago
Mike Fährmann
fa2952ac55
[furaffinity] add 'following' extractor ( #515 )
4 years ago
Mike Fährmann
9b194520db
[newgrounds] add 'following' extractor ( closes #684 )
4 years ago
Mike Fährmann
6386ee54e1
[deviantart] add extractor info to 'following' results
4 years ago
Mike Fährmann
d5273f9b0c
[hiperdex] update domain to hiperdex.net
4 years ago
Mike Fährmann
08674a91f3
[patreon] fix hash extraction from download URLs ( closes #693 )
...
The old method was assuming every URL path ends with '/1'. For URLs
where this is not the case, the segment containing the post ID was
used as file hash.
4 years ago
Mike Fährmann
a6286bb551
[hiperdex] add 'artist' extractor ( #606 )
5 years ago
Mike Fährmann
291033720a
[hiperdex] fix manga extraction
5 years ago
Mike Fährmann
dfc0557807
[vsco] fix collection extraction
5 years ago
Mike Fährmann
fd438f0d78
update extractor test results
5 years ago
Mike Fährmann
bae1e8ed12
[deviantart] fix JPEG quality replacement pattern
...
'q_\d+' would sometimes also replace something in the 'token' query
parameter, invalidating the URL.
5 years ago
Mike Fährmann
cf4cef3d63
[aryion] adjust 'date' to UTC time
5 years ago
Mike Fährmann
6c531be294
[aryion] fix malformed 'last-modified' headers ( #390 )
5 years ago
Mike Fährmann
dc65f7d8dc
[aryion] use generic download URLs ( #390 )
...
i.e. /g4/data.php?id=…
- get filename & extension from Content-Disposition header
- handle all downloadable file types (docx, swf, etc)
5 years ago
Mike Fährmann
96b78bcf04
[aryion] include path in default directory format ( #390 )
5 years ago
Mike Fährmann
6143050980
[aryion] add gallery and post extractors ( #390 , #673 )
5 years ago
Mike Fährmann
9e7dfc0cfc
[myportfolio] fix extraction of galleries without title
5 years ago
Mike Fährmann
88fca0a172
[mastodon] update OAuth credentials for pawoo.net ( #665 )
5 years ago
Mike Fährmann
4ae8a25567
[mastodon] use 'combine_dict()' to combine extractor info dicts
5 years ago
Mike Fährmann
220c06b86e
[mastodon] handle rate limits
5 years ago
Mike Fährmann
d02f7c1118
improve Extractor.wait()
...
- allow 'until' to be a datetime object
- do "time calculations" with UTC timestamps
- set a default 'reason'
5 years ago
Mike Fährmann
5d7404ab58
[oauth] use the new name for 'DeviantartAPI' ( fixes #670 )
5 years ago
Mike Fährmann
762c758af4
[hiperdex] fix extraction
5 years ago
Mike Fährmann
f9a590f92b
[deviantart] apply HTTP request limits in more places
...
"Request blocked" can also happen on sta.sh and for *any* HTTP
request directed at deviantart.com
5 years ago
Mike Fährmann
2587296deb
[mastodon] add access tokens for mastodon.social and baraag.net
...
(closes #665 )
5 years ago
Mike Fährmann
ff7c0b7eff
[deviantart] handle "Request blocked" errors ( #655 )
...
- add a 2 second wait time between requests to deviantart.com
- catch 403 "Request blocked" errors and wait for 3 minutes until
retrying
5 years ago
Mike Fährmann
c874684f05
[deviantart] retrieve *all* download URLs through OAuth API
...
'/extended_fetch' as well as Deviation webpages now again contain
Deviation UUIDs needed to grab Deviation info through the OAuth API,
meaning cookies are no longer necessary to grab original files.
The only instance were cookies are still needed are scraps marked as
"mature", since those entries are hidden for public users.
(#655 , #657 , #660 )
5 years ago
Mike Fährmann
5c27b25a8f
[deviantart] improve sta.sh extraction
...
Extract all sta.sh items in a single extractor run.
Don't spawn a new StashExtractor for each individual sta.sh item to
preserve the current requests.Session and its opened TCP connections.
5 years ago
Mike Fährmann
e2fc4eaa6f
[deviantart] detect stash folders ( fixes #659 )
5 years ago
Mike Fährmann
c034159701
[piczel] fix extraction for single images
5 years ago
Mike Fährmann
699036ea0c
[weibo] accept status URLs with non-numeric IDs ( #664 )
5 years ago
Mike Fährmann
fe96f99e4b
[hentainexus] reduce line length (flake8) & update test
5 years ago
墨焓
6f81cac8fa
Add metadata to hentainexus: circle, event, title_conventional. ( #661 )
5 years ago
Mike Fährmann
6f911aeb1c
[deviantart] add error message for cloudFront blocks ( #655 )
5 years ago
Mike Fährmann
7499d71d02
[simplyhentai] ignore certificate errors in video test
5 years ago
Mike Fährmann
4203dc0bdc
[mangapark] fix metadata extraction
5 years ago
Mike Fährmann
1b82d36ab2
[deviantart] handle decode errors for extended_fetch results ( #655 )
...
This isn't going to solve the underlying problem, but it should at
least provide the server response when those errors happen.
5 years ago
Mike Fährmann
09f2271528
[35photo] add 'tag' extractor
5 years ago
Mike Fährmann
77fda8190c
[35photo] simplify/remove tests for the 'genre' extractor
...
There is still a nice genre overview page (https://35photo.pro/genre/ )
but the individual sub-pages don't list photos anymore
5 years ago
Mike Fährmann
fb846c9ee5
[instagram] reduce line lengths and make flake8 happy
5 years ago
Mike Fährmann
ad2efa8509
[e621] derive from Danbooru extractors ( #651 )
...
- use extractor implementations from 'danbooru'
- use "page": "b[ID]" to paginate over results instead of
"tags": "id:<[ID]", avoiding infinite loops with certain
post orders
- bump User-Agent version
5 years ago
Mike Fährmann
9b39e1cd7e
[e621] fix bug in API rate limiting ( #651 )
5 years ago
Mike Fährmann
b607d0ad7f
[twitter] fix typo in 'x-twitter-auth-type' header ( #625 )
5 years ago
Mike Fährmann
2c3b9e1450
[nozomi] support multiple images per post ( #646 )
...
This changes the default filename format as well as archive IDs,
since those assumed that each post would only have one image.
5 years ago
Mike Fährmann
c606d0c854
[instagram] update pattern for user profile URLs
...
Allow for query parameters and fragments,
for example https://www.instagram.com/instagram/?hl=en
5 years ago
Mike Fährmann
2530db3f4d
[mangadex] transform 'date' timestamps to datetime objects
5 years ago
Mike Fährmann
ae2a33243b
[newgrounds] catch general Exceptions
5 years ago
Mike Fährmann
32e36d8f02
[sexcom] replace tests
5 years ago
Mike Fährmann
33b42dc847
[nozomi] sort search results ( fixes #646 )
5 years ago
Mike Fährmann
eaa60a438b
[piczel] fix extraction
...
- manually filter by folder_id
- extract data for single posts from embedded JSON, since the
'/api/gallery/image/<id>' endpoint is no longer available
5 years ago
Mike Fährmann
5bcc7184c9
[danbooru][e621] increase page limits
5 years ago
Mike Fährmann
90d15e3682
[instagram] use 'itertools.chain()'
5 years ago
Leonardo Taccari
160328d21c
[instagram] Add support for user's saved medias ( #644 )
...
* [instagram] Gracefully handle possible 'HttpErrorPage' in _extract_page()
`HttpErrorPage' is returned in shared_data at least when not authenticated or
when trying to fetch other users saved medias
(i.e. `instagram.com/<user>/saved/').
Gracefully handle it by returning nothing.
* [instagram] Add support for user's saved medias
(Please note that this need the user to be authenticated and they can
only see their saved media (not other users ones).)
Close #643 .
* [instagram] Bump copyright year
5 years ago
Mike Fährmann
d3482ace7f
[furaffinity] extract more metadata
...
- views
- favorites
- comments
- rating
- fa_category (since 'category' is already in use)
- theme
- species
- gender
- width
- height
5 years ago
Mike Fährmann
fdd2dd5136
[kabeuchi] add 'user' extractor ( closes #561 )
5 years ago
Mike Fährmann
59edcdc822
[hitomi] restore metadata fields from before f33b13a
...
... and add a 'metadata' option to disable
visiting the gallery page and extracting data from it
if this is not needed.
5 years ago
Mike Fährmann
2d5703c493
[twitter] use a simpler data structure to store cookies in cache
...
Use a dict with name-value pairs instead of an entire
RequestsCookieJar object.
5 years ago
Mike Fährmann
87d4f83597
[newgrounds] make post extraction nonfatal
5 years ago
Mike Fährmann
823fbeaae6
[newgrounds] add 'favorite' extractor ( #394 )
5 years ago
Mike Fährmann
a45fbc38ea
[pixiv] implement 'avatar' option ( #595 , #623 )
5 years ago
Mike Fährmann
a63a376ad2
[mangoxo] fix login
5 years ago
Mike Fährmann
ebc70e87ce
[e621] update to new interface / API endpoints ( closes #635 )
5 years ago
Mike Fährmann
d1cf7ccdb3
[instagram] add 'post_shortcode' metadata field ( #525 )
5 years ago
Mike Fährmann
32df8d06fe
[twitter] add 'bookmark' extractor ( closes #625 )
5 years ago
Mike Fährmann
3fb41c34c8
[bcy] reduce requests to '/item/detail/<id>' ( #613 )
...
The former implementation would try to use the embedded data from
'/item/detail/' pages for every post, even if that wasn't really
necessary.
This commit also fixes some issues with posts only visible to
logged in users.
5 years ago
Mike Fährmann
f33b13aacf
[hitomi] simplify metadata extraction
...
Use the data from https://ltn.hitomi.la/galleries/ <id>.js for both
image URLs and metadata and ignore any gallery or reader pages.
This removes 'artist', 'characters', 'group', and 'parody' metadata
fields since this information is, as for now, only available in
gallery pages.
5 years ago
Mike Fährmann
ce5e2a58fe
[imgbb] update test results
...
Image server domain changed from
https://image.ibb.co/ to https://i.ibb.co/
5 years ago
Mike Fährmann
f117e32910
[danbooru] restore 'popular' functionality
5 years ago
Mike Fährmann
39b48d665b
[hiperdex] use proper name for 'chapter_minor'
5 years ago
Mike Fährmann
8fbbaa54ff
[bcy] fix partial image URLs ( #613 )
...
Images from new posts can have incomplete/partial URLs (1)
without any filename extension when fetching their data from
'/apiv3/user/selfPosts', so now all data gets taken from
'/item/detail/ID' pages.
It is currently unknown how to get the non-watermarked original version
of these images, or if that is possible at all. (2)
Images with a watermark will have their 'filter' metadata field set to
"watermark". For original images this field is an empty string "".
Enabling the 'noop' option will, in addition to the watermarked version,
yield the the '~noop.image' filter version (3),
where 'filter' is set to "noop".
(1) "https://img-bcy-qn.pstatp.com/banciyuan/3ccdff22479c4060aadc86718209b281 "
(2) "https://p1-bcy.byteimg.com/img/banciyuan/3ccdff22479c4060aadc86718209b281~tplv-banciyuan-logo-v3:wqnpnLLlhZLlpKfprZTnjotfCuWNiuasoeWFgyAtIEFDR-eIseWlveiAheekvuWMug==.image "
(3) "https://p1-bcy.byteimg.com/img/banciyuan/3ccdff22479c4060aadc86718209b281~noop.image "
5 years ago
Mike Fährmann
86c00f9e66
[danbooru] move extractor logic from booru.py
5 years ago
Mike Fährmann
1d4a369ea2
update extractor test results
5 years ago
Mike Fährmann
7625912b31
[piczel] improve and update
...
- fix tag names
- fix a bug in _pagination()
- parse datetime in 'created_at' as 'date'
- rewrite main loop
- replace user profile test
5 years ago
Mike Fährmann
913b8333cc
write DeviantArt refresh-tokens to cache ( #616 )
...
Writing the token is currently disabled by default and must be
enabled with 'extractor.oauth.cache'.
'extractor.deviantart.refresh-token' must be set to '"cache"'
to use the cached token.
5 years ago
Mike Fährmann
2a4f227e08
warn about expired cookies
5 years ago
Mike Fährmann
4e361b3008
add tests for specific datetime values
5 years ago
Mike Fährmann
80ecb99089
[hitomi] fix extraction
5 years ago
Mike Fährmann
247c9e1416
[vsco] update gallery URL pattern
5 years ago
Mike Fährmann
19ae6f3fc4
update test results
...
- twitter:
Don't test the whole kwdict, only the actual content, since the
keyword hash changes whenever that user changes his display name.
- khinsider:
Download host changed
5 years ago
Mike Fährmann
cc5079c844
[hiperdex] add chapter and manga extractors ( closes #606 )
5 years ago
Mike Fährmann
64bdec8430
[deviantart] check availability of intermediary URLs ( fixes #609 )
5 years ago
Mike Fährmann
5607dd3646
[hitomi] follow multiple redirects
5 years ago
Mike Fährmann
765b2a0527
[hentaihand] add extractors ( closes #605 )
5 years ago
Mike Fährmann
d94215d119
[tumblr] replace '-' with ' ' in tag searches ( fixes #611 )
...
To search for tags with actual minus signs in them
(there shouldn't be too many,) manually replace those
with url-encoded minus characters ('-' -> '%2d')
before inputting them into gallery-dl:
https://s679874.tumblr.com/tagged/tag-with-minus
->
https://s679874.tumblr.com/tagged/tag%2dwith%2dminus
5 years ago
Mike Fährmann
e6cd49e78b
update extractor test results
5 years ago
Mike Fährmann
5d9437b398
[vsco] skip "invalid" entities
5 years ago
Mike Fährmann
650f2b6d58
[furaffinity] accept sfw.furaffinity.net URLs ( closes #608 )
...
Just as an alias for regular URLs with no extra content filtering.
5 years ago
Mike Fährmann
74e684e828
[twitter] change default value for 'videos' to 'true'
...
Every other 'videos' option defaulted to 'true', except Twitter.
5 years ago
Mike Fährmann
c7cf9dd111
[furaffinity] support classic layout ( #284 )
5 years ago
Mike Fährmann
138135c190
[furaffinity] add extractors ( #284 )
5 years ago
Mike Fährmann
b9c574bd1d
[patreon] log skipped files ( #590 )
5 years ago
Mike Fährmann
80ea9104b8
[8kun] adjust URL pattern
5 years ago
Mike Fährmann
ce26070231
[pixiv] reduce calls to '/user/detail'
5 years ago
Mike Fährmann
da0d5f6092
[oauth] add 'port' option ( #604 )
5 years ago
Mike Fährmann
719b63d0ca
[bcy] add user and post extractors ( #592 )
5 years ago
Mike Fährmann
6426e3efc7
[khinsider] fix and improve metadata extraction
5 years ago
Mike Fährmann
b7eb6cecbb
[pixiv] handle tags at the end of new bookmark URLs
5 years ago
Mike Fährmann
109f6c8685
[patreon] filter duplicate files per post ( #590 )
5 years ago
Mike Fährmann
b38cf59711
[sexcom] fix image URLs & parse 'date' fields
5 years ago
Mike Fährmann
1f4c9c5f9d
[8kun] add thread and board extractors ( closes #582 )
5 years ago
Mike Fährmann
facc5daa6d
[twitter] force old login page layout ( fixes #584 , fixes #598 )
5 years ago
Mike Fährmann
d1de7dc296
[hitomi] implement workaround for "broken" redirects
...
Some galleries redirect to a new "version" with different gallery id.
This new version might not be available any more, but the /reader/
page for the original gallery id can still work.
5 years ago
Mike Fährmann
40fe062851
[pixiv] fix user id for bookmarks API calls ( closes #596 )
5 years ago
Mike Fährmann
91aaaf1a9e
[pixiv] add 'rating' metadata field ( #595 )
...
A human-friendlier representation of 'x_restrict'
5 years ago
Mike Fährmann
dff33b260c
[reddit] add 'videos' option
5 years ago
Mike Fährmann
2ad43618cc
[piczel] fix extraction
5 years ago
Mike Fährmann
cf7a67d67f
[yaplog] remove module
...
Yaplog! ended its service on 2020-01-31
5 years ago
Mike Fährmann
e0dd073ce0
[twitter] replace embedded tweet test
...
the old one was deleted
5 years ago
Mike Fährmann
ec36df4851
[deviantart] fix video extraction from 'extended_fetch' results
...
DeviantArt is now serving videos from wixmp servers (1), instead of
the former film00.deviantart.com (2), even though those URLS are still
functional.
They seem to also have re-encoded those videos. The 10 MB 1080p video
from (2) is now only available in 720p at ~20 MB (with a higher
bitrate, but still …). Other videos are still available in 1080p, but
not this one for some reason.
(Changing the '720p' in (1) to '1080p' doesn't work.)
(1) https://wixmp-ed30a86b8c4ca887773594c2.wixmp.com/v/mp4/9feaa2c9-1baf-4fc2-84f7-f3384b34cefe/d5gxnb5-282a2e9a-b552-40ff-8542-b3c5eed823f5.720p.a837d7cec12c41be8ca2ee53152cea3a.mp4
(2) https://film00.deviantart.net/4c1d/v/mp4/2012/279/d/1/_video____brushes_i_use_in_paint_tool_sai_by_chi_u-d5gxnb5.mp4
5 years ago
Mike Fährmann
48be2266ed
[deviantart] better error message for 'extended_fetch' ( #585 )
5 years ago
Mike Fährmann
71851a6241
[pixiv] update URLs of followed users to the new format
5 years ago
Mike Fährmann
d086f30b42
[reddit] restore archive keys for i.redd.it images
5 years ago
Mike Fährmann
56f1c96168
implement 'parent-directory' option ( #551 )
5 years ago
Mike Fährmann
ae07f92f7e
[reddit] rewrite extractor logic ( closes #551 )
...
Handle images and videos hosted on Reddit "natively",
allowing them to use reddit-specific metadata to build directory
and file names.
5 years ago
Mike Fährmann
2852691d78
[paheal] replace test URL
...
searching for 'k-on' doesn't yield any results anymore
5 years ago
Mike Fährmann
2a9be48511
improve util.load/save_cookiestxt() and add tests
...
- take a file object as argument instead of an filename
- accept whitespace before comments (" # comment")
- map expiration "0" to None and not the number 0
5 years ago
Mike Fährmann
e35c2ea1a6
[weibo] use youtube-dl to download from m3u8 manifests
5 years ago
Mike Fährmann
6703b8a86b
[blogger] implement video extraction ( closes #587 )
5 years ago
Mike Fährmann
c1a6862863
implement functions to load/save cookies.txt files ( closes #586 )
...
The methods of the standard libraries' MozillaCookieJar have
several shortcomings (#HttpOnly_ cookies, 0 expiration timestamps, etc.)
and require construction of an ultimately pointless CookieJar object.
5 years ago
Mike Fährmann
25d5ec4ff3
[twitter] add option to extract TwitPic embeds ( #579 )
5 years ago
Mike Fährmann
32d7195d08
[pinterest] improve detection of invalid pin.it links
5 years ago
Mike Fährmann
174117f827
allow multiple hashes for content tests
5 years ago
Alice
f498a9057f
[twitter] Fix stop before real end ( #573 )
...
* [twitter] Fix stop before real end
Fix for https://github.com/mikf/gallery-dl/issues/544 . Makes sure that it really reached the end by checking that both "min_position" is null and "has_more_items" is false before stopping.
* [twitter] Fix stop before real end (update)
5 years ago
Mike Fährmann
8bb32ee188
[hitomi] fix image URLs
5 years ago
Mike Fährmann
bd5ce9855c
allow GalleryExtractors to set URL-independent extensions
5 years ago
Mike Fährmann
af42c75152
[mangadex] revert domain to 'mangadex.org'
5 years ago
Mike Fährmann
e89413da22
update test results
5 years ago
Mike Fährmann
33a6e0ac6e
[hentaifoundry] extract more metadata ( closes #565 )
5 years ago
Mike Fährmann
5cac79c3d9
[erolord] remove extractor
5 years ago
Mike Fährmann
b9cbf932b4
[pixiv] update URL patterns ( fixes #568 )
...
Pixiv now uses new URLs for
- user profiles and illustration listings:
- https://www.pixiv.net/en/users/ <ID>
- https://www.pixiv.net/en/users/ <ID>/artworks
- bookmarks:
- https://www.pixiv.net/en/users/ <ID>/bookmarks/artworks
5 years ago
Mike Fährmann
988cc2ec23
[mangadex] change domain to mangadex.cc ( closes #559 )
5 years ago
Mike Fährmann
f8e137d6b4
[deviantart] show warning about private deviations only once
...
… per call to '_pagination()'
5 years ago
Mike Fährmann
939fec8ecd
[deviantart] match new search/popular URLs ( closes #538 )
5 years ago
Mike Fährmann
09cc88b715
[deviantart] match '/favourites/all' URLs ( closes #555 )
5 years ago
Mike Fährmann
3811fd8a25
fix time formatting for Python 3.4 and 3.5
...
'datetime.time.isoformat()' only has an optional 'timespec' argument
since Python 3.6.
5 years ago
Mike Fährmann
43ab9572b4
[twitter] handle API rate limits ( #526 )
5 years ago
Mike Fährmann
569747a78d
implement extractor.wait()
5 years ago
Mike Fährmann
5532e9c158
[twitter] handle quoted tweets ( #526 )
...
… and categorize them as retweets
5 years ago
Mike Fährmann
0b4cb8e57a
[mangahere] send 'isAdult' cookie ( fixes #556 )
5 years ago
Mike Fährmann
1afb91363c
[imagefap] generalize URL patterns and add tests ( #552 )
5 years ago
Xope Totec
f701e9f33a
Handle beta.imagefap.com URLs ( #552 )
5 years ago
Mike Fährmann
ce54b8c04c
let extractors opt-out of cookie option usage
...
useful to avoid sending unnecessary cookies when all authentication
is done through OAuth tokens
5 years ago
Mike Fährmann
5ad92fc196
[newgrounds] fix tags metadata extraction
5 years ago
Mike Fährmann
82f7f4172a
update test results
5 years ago
Mike Fährmann
1f2a69f3c5
add '_extractor' information to redirect results
5 years ago
Mike Fährmann
a27f43dad1
[pixiv] wait and retry after rate limit error ( closes #535 )
5 years ago
Mike Fährmann
6b373cb7e2
[exhentai] restrict default directory name length ( #545 )
5 years ago
Mike Fährmann
b347bf68c7
[deviantart] add extractor for followed users ( #515 )
5 years ago
Mike Fährmann
c0f391a4e2
[pixiv] support listing followed users ( #515 )
5 years ago
Mike Fährmann
896896a490
[twitter] fix URLs forwarded to youtube-dl ( closes #540 )
...
Since commit 3bba763
data["user"] is an entire dict object
and no longer just the user nickname …
5 years ago
Mike Fährmann
1e2713b895
[artstation] fix search result pagination ( closes #537 )
5 years ago
Mike Fährmann
bf3df3d0b0
[directlink] send Referer headers ( closes #536 )
5 years ago
Mike Fährmann
9be7ff600e
[imagetwist] replace test image
...
the old one expired, it seems
5 years ago
Mike Fährmann
66905b1664
[foolslide] add fallback for chapter data extraction
5 years ago
Mike Fährmann
48e42e73fb
[reddit] change default value for 'comments' to '0'
5 years ago
Mike Fährmann
9c0928457a
[reddit] fix errors with 't1_…' submissions
5 years ago
Mike Fährmann
bf658fd84b
[vsco] implement 'videos' option
5 years ago
Mike Fährmann
95c90722ee
[instagram] implement 'videos' option ( closes #521 )
5 years ago
Mike Fährmann
d0920e84e9
update test results
5 years ago
Mike Fährmann
8c11e81c9f
Merge commit '63e6993716db8d8bedfb7b0d445c7161493046b6'
5 years ago
Mike Fährmann
63e6993716
merge 'bypost' functionality into metadata postprocessor
5 years ago
Mike Fährmann
31a29835ff
[realbooru] simplify extractors and update tests ( #514 )
5 years ago
The Oddball
9a4ce20b8e
[realbooru] Add Realbooru extractor ( #514 )
5 years ago
Mike Fährmann
72b8fbfbad
[instagram] make post-page extraction nonfatal
5 years ago
Mike Fährmann
922b8a9595
[weibo] raise NotFoundError for unavailable/deleted statuses
5 years ago
Mike Fährmann
0cd157300e
[patreon] fix regex pattern for posts
...
The previous one would match the first number in the URL slug as
post ID, which would fail for posts with numbers in their title.
5 years ago
Mike Fährmann
fe19e233f3
[xvideos] improve
...
- derive from GalleryExtractor
- match '…-channels' URLs
- "better" metadata structure
5 years ago
Mike Fährmann
d3e44e899d
raise NotFoundErrors for 404 responses in GalleryExtractors
5 years ago
Mike Fährmann
a4dd8b3dab
improve _check_cookies()
...
Only loop over all cookies once instead of calling
cookiejar._find() for each cookie name.
5 years ago
Mike Fährmann
76e60d10a6
[patreon] raise proper exception if creator/post doesn't exist
5 years ago
Mike Fährmann
9e63804347
[patreon] make retrieving user info nonfatal ( #508 )
...
… and fall back to the included data if an error occurs.
5 years ago
Mike Fährmann
964dc57286
[vsco] improve image resolutions
...
https://im.vsco.co/ URLs redirect to the appropriate CDN server
and occasionally insert a '/1200x1600/' into the image path,
limiting image dimensions.
This commit constructs redirect targets out of the given
im,vsco.co URLs without sending extra HTTP requests
and without any "builtin" resolution restrictions.
5 years ago
Mike Fährmann
0629fe8fa4
[vsco] fix user profile extraction … again
...
Given the pattern from last time, collections will also change
in due time and use cursor-based pagination.
5 years ago
Mike Fährmann
ab17ea9632
[deviantart] only print warning if 'original' is enabled
5 years ago
Mike Fährmann
2188db6284
[gelbooru] fix non-API tag extraction
5 years ago
Mike Fährmann
c4702ec9b6
simplify some logging calls
5 years ago
Gio
c0b9ad678d
Separate metadata from handle_url into handle_metadata, commenting
5 years ago
Mike Fährmann
c9ef1b21c3
[patreon] get partial user info without /api/user/<id> ( #507 )
...
It's a lot less data, but doesn't invoke any additional
HTTP requests with potential Cloudflare CAPTCHAs.
5 years ago
Mike Fährmann
0ab9bb1721
[4chan] add extractor for entire boards ( closes #510 )
5 years ago
Gio
cfc70a97ab
Added an additional channel for downloading the metadata of an entire post or gallery.
5 years ago
Mike Fährmann
15f9bb3d14
add option to disable pyOpenSSL usage ( #508 )
...
(pyOpenSSL is now disabled by default)
5 years ago
Mike Fährmann
c8e99e3b3b
[deviantart] fix crash on missing "token" field ( #505 )
5 years ago
Mike Fährmann
6ed2c7823c
[deviantart] disable original downloads if no cookies set
...
For 'deviation' and 'scraps' extractors only, since original file
downloads for those two will always fail with a 404 Not Found
when not logged in.
5 years ago
Mike Fährmann
50deab5265
[deviantart] fix URL generation from /extended_fetch results
...
(closes #505 )
5 years ago
Mike Fährmann
1f209da4c0
[pixiv] match new search URLs ( closes #507 )
5 years ago
Mike Fährmann
e17907ee2a
change default value of 'cookies-update' to 'true'
5 years ago
Mike Fährmann
07dafad26d
[twitter] attempt to fix infinite loops ( #499 )
...
(Hopefully this doesn't break anything else)
5 years ago
Mike Fährmann
71acbdabf4
[2chan] fix metadata extraction
5 years ago
Mike Fährmann
c0a1241648
[livedoor] force https:// for image URLs
5 years ago
Mike Fährmann
6e23c0da09
[imgur] add extractor for subreddit links ( closes #500 )
5 years ago
Mike Fährmann
372ffe95ee
[oauth] adjust Flickr redirect URI ( fixes #503 )
...
Flickr now automatically forces https:// for all redirect URIs.
5 years ago
Mike Fährmann
004812258d
[hentaifox] fix extraction
5 years ago
Mike Fährmann
e2710702d4
fix Cloudflare bypss
5 years ago
Mike Fährmann
8759403f37
[plurk] add delay between comment requests
5 years ago
Mike Fährmann
a28552fd19
update test results
...
- hbrowse: one tag got removed
- mangoxo: gallery changed owner
- photobucket: ?, but photo still downloads
5 years ago
Mike Fährmann
dcaa3d01bd
[imagefap] adapt to new image URL format
5 years ago
Mike Fährmann
e62c209ca0
[nijie] fix 'date' parsing
5 years ago
Mike Fährmann
3bba763ab9
[twitter] improve
...
- update metadata structure
- combine all user… entries into their own dict
- let 'user' always specify the Timeline owner
- add 'author' entry that specifies the original Tweet author
- create directories per post (closes #491 )
- fix username issues with /i/web/ URLs
5 years ago
Mike Fährmann
db35c3b581
[directlink] separate filenames from paths
...
With this, all default filename formats specify an '{extension}'
and PathFormat.set_extension() reliably works for all files.
5 years ago
Mike Fährmann
41a3169c67
[foolfuuka] use '{extension}' in default filename format
5 years ago
Mike Fährmann
e9aed62c91
[imgur] unescape image titles
5 years ago
Mike Fährmann
2c332edaad
[plurk] fix comment pagination
5 years ago
Mike Fährmann
a3fa45bbb1
[behance] get images from 'media_collection' modules
5 years ago
Mike Fährmann
359c3bc1c5
[deviantart] revert to getting download URLs from OAuth API
...
This commit (partially) reverts 27b5b24
, 94eb7c6
, and a437e78
.
Download URLs from the 'extended_fetch' endpoint are now only
usable for logged in users, while those from the respective
OAuth API endpoint are working again. Everything except
scraps and direct deviation links should be fixed, and those
two categories will work with exported cookies. (#488 )
TODO:
- "native" login with --username and --password
- better handling of internally stored cookies
5 years ago
Mike Fährmann
42b9633c7e
update test results
5 years ago
Mike Fährmann
b28bd1c73e
[bobx] set generated session cookie ( closes #482 )
...
This reverts commit 490831f
and also restores original image downloads
by setting a randomly generated session cookie. No login required.
5 years ago
Mike Fährmann
ae09f87602
improve SharedConfigMixin config lookups
5 years ago
Mike Fährmann
f5604492c3
update interface of config functions
5 years ago
Mike Fährmann
4ca883c66f
[smugmug] replace test for custom URLs
...
The old one (http://www.creativedogportraits.com/ ) is empty and/or
no longer handled by SmugMug.
5 years ago
Mike Fährmann
d45fabb79d
match user profile handling on deviantart and newgrounds
5 years ago
Mike Fährmann
ea80dadd09
[deviantart] restore archive keys
...
Commit 9fdc5e7
changed 'username' fields to have consistent
capitalization, but that invalidated the archive keys of several
extractors where 'username' was usually lowercase.
5 years ago
Mike Fährmann
ea094692c8
[vsco] fix collection extraction ( #480 )
5 years ago
Mike Fährmann
490831f84a
[bobx] "fix" image download URLs
...
Access to original images got restricted to (paid) members only.
All that's publicly accessible now are essentially preview pictures.
5 years ago
Mike Fährmann
978cb03f81
update misc test results
...
- Livedoor now uses https:// for its image URLs
- Instagram image URLs got simplified
5 years ago
Mike Fährmann
fca87974fe
[sexcom] fix video downloads by sending specific Referer headers
5 years ago
Mike Fährmann
edc080468d
[instagram] make 'video_url' fields optional ( fixes #479 )
...
[ci skip]
5 years ago
Mike Fährmann
9fdc5e74cb
[deviantart] ensure consistent username capitalization ( #455 )
...
The 'username' field was capitalized in a very inconsistent manner:
Either all lowercase, or as given by the input URL, or with the
"original" capitalization, depending on the extractor used among
other things.
Now usernames use their original capitalization for all extractors.
('UserName' instead of 'username' or 'uSeRnAmE')
5 years ago
Mike Fährmann
b1f0609de5
[newgrounds] rewrite ( #394 )
...
- restructure extractor hierarchy
- extract more metadata
- extract videos without youtube-dl
- be more resilient to errors
TODO:
- favorites
- games, but that might be near impossible for non-flash titles
5 years ago
Mike Fährmann
3ece3976ae
[newgrounds] implement login support ( #394 )
5 years ago
Mike Fährmann
3a07c06865
[newgrounds] update
...
- create directory per post
- rename variables and methods
5 years ago