Mike Fährmann
3918b69677
remove 'extractor.blacklist' context manager
4 years ago
Mike Fährmann
c78aa17506
add general 'blacklist' and 'whitelist' options ( #492 , #844 )
4 years ago
Mike Fährmann
abda352a5b
add '--no-skip' command-line option ( closes #986 )
4 years ago
Mike Fährmann
5912727b88
support format string replacement fields in archive paths
...
(closes #985 )
4 years ago
Mike Fährmann
2b8d57f0ab
[twitter] support '/intent/user?user_id=…' URLs ( #980 )
4 years ago
Mike Fährmann
a3b473bd2f
[twitter] support specifying users by ID ( #980 )
...
by using 'id:…' as their screen name, i.e.
https://www.twitter.com/id:2976459548/media
instead of
https://twitter.com/supernaturepics/media
The user ID can, for example, be obtained from the output of
$ gallery-dl -j --range 1 https://twitter.com/ <screen-name>
4 years ago
Mike Fährmann
a0d916ed41
[exhentai] update wait time before original image download ( #978 )
...
depend on 'wait-max', don't use a hard-coded value
4 years ago
Mike Fährmann
f6fd449b59
reduce wait time growth rate from exponential to linear
...
Waiting for 2**N seconds after each error grows too fast.
Simply waiting N seconds seems far more reasonable.
4 years ago
Mike Fährmann
bc48514d84
[aryion] get post ID via gallery-item ( fixes #981 , closes #982 )
...
this even works when fetching post IDs from '/latest.php?id='
4 years ago
Mike Fährmann
799ca07fc8
[imgur] update
...
- fix image/album detection for galleries
- use new API endpoints for image/album data
4 years ago
Mike Fährmann
b5243297ff
write skipped files to archive ( closes #550 )
4 years ago
Mike Fährmann
ac3036ef56
add 'filesize-min' and 'filesize-max' options ( closes #780 )
4 years ago
Mike Fährmann
7876a03ece
[tumblr] create directories for each post ( fixes #965 )
...
This changes the identifiers for directory format string fields.
Everything blog related is now inside a 'blog' object
and not at the "base level" anymore.
E.g. '{name}' for directories is now '{blog[name]}'
(or '{blog_name}', since that is also available)
4 years ago
Mike Fährmann
fd0685d9b5
[postprocessor:zip] defer zip file creation ( fixes #968 )
...
don't try to create zip files on postprocessor construction,
wait until directory creation during file download,
4 years ago
Mike Fährmann
33fe67b594
release version 1.14.5
4 years ago
Mike Fährmann
d50f3b333a
update extractor test results
4 years ago
Mike Fährmann
0f55b8e80a
[exhentai] fix type check from dbbbb21
( #940 )
...
'bool' is a subclass of 'int', and therefore
'isinstance(self.limits, int)' also returns True when
'self.limits' has a boolean value
4 years ago
Mike Fährmann
e33293fdd8
[hentaihand] update to new site layout
4 years ago
Mike Fährmann
fda9e296dd
[gelbooru] fix extraction without API
4 years ago
Mike Fährmann
69e4871005
update extractor test results
...
- sensescans: replace 404d chapters
- mangapark: replace 404d chapters
- subscribestar: update test for attached files
4 years ago
Mike Fährmann
ab1af66a97
[imgur] add 'search' extractor ( #934 )
4 years ago
Mike Fährmann
e4bbc1fb5c
[imgur] add 'tag' extractor ( #934 )
4 years ago
Mike Fährmann
deaacc70bb
[hitomi] update URL pattern for tag searches
4 years ago
ArtaxIsSleeping
0e941553ec
[aryion] Add username/password support ( #960 )
...
* Add username/password support to aryion extractor
* Update docs to match
* Fix code style
4 years ago
Mike Fährmann
84e04cc23b
[500px] fix extraction and update URL patterns ( fixes #956 )
...
- rewrite most API calls to GraphQL queries
- match '500px.com/p/<user>' URLs
4 years ago
Mike Fährmann
d4ff767291
[reddit] improve gallery extraction ( fixes #955 )
4 years ago
Mike Fährmann
7140fe7e6d
[hitomi] fix redirect processing
4 years ago
Mike Fährmann
a57b6b3c3a
[reddit] handle deleted galleries ( fixes #953 )
4 years ago
Mike Fährmann
063c71cd84
[furaffinity] add 'search' extractor ( closes #915 )
4 years ago
Mike Fährmann
dbbbb21180
[exhentai] add ability to specify custom image limit ( #940 )
4 years ago
Mike Fährmann
b2009ea39e
[aryion] update folder mime type list ( fixes #945 )
4 years ago
Mike Fährmann
688bd046fc
release version 1.14.4
4 years ago
Mike Fährmann
d06ad148c7
[shopify] use alternate regex for products on collection pages
...
when the first on doesn't yield any results
4 years ago
Mike Fährmann
7619152988
[reactor] sort 'tags'
...
to ensure a consistent order for test results
4 years ago
Mike Fährmann
cd9de613a2
[exhentai] adjust image limit costs ( #940 )
...
Each original file costs 10 points per 10^6 bytes,
not 10 per 2^20 == 1048576 bytes.
4 years ago
Mike Fährmann
2e6f6ee1c1
[mangoxo] fix login
4 years ago
Mike Fährmann
a6a080656c
[pixnet] detect password-protected albums ( #177 )
4 years ago
Mike Fährmann
67ac6667af
[mangareader] fix extraction
4 years ago
Mike Fährmann
2b88c90f6f
[blogger] add search extractor ( #925 )
4 years ago
Mike Fährmann
d5067c51c5
[instagram] support '/reel/' URLs
4 years ago
Mike Fährmann
2c9766b29f
fix UnboundLocalError in Extractor.request()
...
introduced in d6a271d
4 years ago
Mike Fährmann
aa64149583
[blogger] support searching posts by labels ( closes #925 )
4 years ago
Mike Fährmann
60ba3cb946
[reddit] support gallery posts ( closes #920 )
4 years ago
Mike Fährmann
0d84d3af55
[subscribestar] extract attached media files ( #852 )
4 years ago
Mike Fährmann
19bf76bcf8
update extractor test results
4 years ago
Mike Fährmann
0762d6b29c
[inkbunny] add 'num' field ( #283 )
4 years ago
Mike Fährmann
fbc4278fe4
[instagram] wait before GraphQL requests ( #901 )
4 years ago
Mike Fährmann
ec5870576d
[imgur] handle 403 overcapacity responses ( closes #910 )
4 years ago
Mike Fährmann
d6a271d2c7
add 'response' objects to 'HttpError's
4 years ago
Mike Fährmann
72c5578a27
[hentainexus] improve/simplify code
4 years ago
Mike Fährmann
627d2141d3
[xhamster] fix extraction ( closes #917 )
4 years ago
Mike Fährmann
3f73cc6855
allow 'parent-directory' to work recursively ( fixes #905 )
4 years ago
Mike Fährmann
27e31f4a16
[myportfolio] raise 'NotFoundError' for deleted posts
4 years ago
Mike Fährmann
f317a57c5e
[simplyhentai] fix 'gallery_id' extraction
4 years ago
Mike Fährmann
daeef8a5e3
[vsco] handle missing 'description' fields
4 years ago
Mike Fährmann
26a967cbd4
[pinterest] match 'pinterest.co.uk' URLs ( fixes #914 )
4 years ago
Mike Fährmann
c5aaa1de77
[inkbunny] simplify metadata structure ( #283 )
...
Just put everything at the top level,
instead of having a separate 'post' object.
4 years ago
Mike Fährmann
b921fee24d
[inkbunny] fix submission order ( #283 )
...
Getting detailed submission info via /api_submissions.php reordered the
input submissions and sorted them by ID. InkbunnyAPI.detail() now sorts
them back and ensures they are returned in their original order.
This commit also removes the 'metadata' option and always requests
submission descriptions.
4 years ago
Mike Fährmann
e50c75628c
[subscribestar] update 'date' parsing
4 years ago
Mike Fährmann
c4ed9f4faa
[inkbunny] add 'metadata' option ( #283 )
4 years ago
Mike Fährmann
493cadb1e7
[inkbunny] add 'orderby' option ( #283 )
4 years ago
Mike Fährmann
336e682a7a
[inkbunny] handle gallery/scraps URLs ( #283 )
4 years ago
Mike Fährmann
8dbf827649
[bobx] remove module
4 years ago
Mike Fährmann
8f64585ff2
[twitter] handle 429 responses without x-rate-limit-reset header
4 years ago
Mike Fährmann
d2e17e16bf
[inkbunny] update tests ( #283 )
4 years ago
Mike Fährmann
57f7d9b790
[inkbunny] improve error handling ( #283 )
4 years ago
Mike Fährmann
baf5d0e3c1
[gfycat] skip malformed gfycat responses ( closes #902 )
4 years ago
Mike Fährmann
453f3bc519
[blogger] improve error messages for missing posts/blogs ( #903 )
4 years ago
Mike Fährmann
87202b8d74
[inkbunny] add 'user' and 'post' extractors ( #283 )
4 years ago
Mike Fährmann
b62ea72533
release version 1.14.3
4 years ago
Mike Fährmann
2ecf1efb16
update extractor test results
...
- tumblr: remove deleted post
- jaiminisbox: replace removed manga/chapters
- smugmug: one inconsequential field got removed
4 years ago
Mike Fährmann
d5fcffcced
[subscribestar] add login capabilities ( #852 )
4 years ago
Mike Fährmann
ecaecc4064
[exhentai] add 'domain' option ( #897 )
4 years ago
Mike Fährmann
45c32213dc
[gfycat] retry 404'ed videos on redgifs ( closes #874 )
4 years ago
Mike Fährmann
cf44571fe0
[gfycat] add 'user' and 'search' extractors
4 years ago
Mike Fährmann
11b744d971
[mangakakalot] improve/fix chapter extraction
4 years ago
Mike Fährmann
2da71cb561
[twitter] raise proper exception if user doesn't exist ( #891 )
4 years ago
Leonardo Taccari
86e5a05e29
[twitter] add support for nitter.net URLs in pattern ( #890 )
...
Please note that URLs are only "translated", all requests are still
done always via the Twitter API.
4 years ago
Mike Fährmann
e17d4f44f6
[newgrounds] fix favorites extraction
4 years ago
Mike Fährmann
c51fbd72ba
update extractor test results
4 years ago
Mike Fährmann
9cd1bc6907
[mangakakalot] update URL patterns, fix flake8 errors ( #876 )
4 years ago
jakem72360
7dfdcc3fbf
[mangakakalot] Added extractors for MangaKakalot ( #876 )
4 years ago
Mike Fährmann
cb0132e441
[khinsider] add 'format' option ( closes #840 )
4 years ago
Mike Fährmann
d594977ca1
[artstation] add 'following' extractor ( closes #888 )
4 years ago
Mike Fährmann
3855d0dd3c
[twitter] add debug messages for all skipped Tweets ( #867 )
4 years ago
Mike Fährmann
27d163afb3
[imgur] support all '/t/...' URLs ( closes #880 )
...
… instead of just '/t/unmuted/'
4 years ago
Mike Fährmann
f5c9f1d066
[subscribestar] use current date instead of hard-coded '2020' ( #852 )
4 years ago
Mike Fährmann
5a6e750704
[reddit] fix AttributeError when using 'recursion' ( fixes #879 )
4 years ago
Mike Fährmann
94a08f0bcb
[reddit] limit title length in default filenames ( #873 )
4 years ago
Mike Fährmann
3424fb96c3
[redgifs] support gifsdeliverynetwork.com URLs ( #874 )
4 years ago
Mike Fährmann
f1344fe552
[patreon] yield images and attachments before postfiles ( #871 )
...
The reported filename of the 'postfile' entry of each post may differ
from the corresponding entry in the list of images or attachments,
and be outright "wrong".
4 years ago
Mike Fährmann
dbf841ebd1
prevent unhandled exception on Cloudflare challenges ( #868 )
...
The relatively new v2 challenges aren't supported (*), but retrying
often enough may yield a v1 challenge which can be solved.
(*) and probably never will. They are far too complicated to do without
a real browser.
4 years ago
Mike Fährmann
6e2af9a8d8
[twitter] improve error message formatting
4 years ago
Mike Fährmann
c28db7a6ea
[8muses] support 'comics.8muses.com' URLs
4 years ago
Mike Fährmann
4d8b3e4f70
defer directory creation ( fixes #722 )
...
Only call os.makedirs() before a file is getting downloaded,
and not immediately for every Directory message.
4 years ago
Mike Fährmann
d5bfb0b38c
set pseudo extension for Metadata messages ( #865 )
...
This prevents pathfmt.filename from potentially being empty.
4 years ago
Mike Fährmann
821524e4ee
[subscribestar] add 'user' and 'post' extractors ( #852 )
4 years ago
Mike Fährmann
4f16fd37fe
release version 1.14.2
4 years ago
Mike Fährmann
e62ebb4643
update CHANGELOG before building sdist and wheel packages
4 years ago
Mike Fährmann
f1ddbff0b5
[aryion] add 'recursive' option ( fixes #832 )
...
This is enabled by default and will recursively go through all
(sub)folders in an artist's gallery.
The old method of using "Latest Updates" lists can be restored by
disabling this option.
4 years ago
Mike Fährmann
699062b91f
Revert "[kissmanga] workaround for CAPTCHAs ( #818 )"
...
This reverts commit 4cf3d54718
.
4 years ago
Mike Fährmann
0cac14c3bd
update extractor test results
4 years ago
Mike Fährmann
5e5be67c26
[tumblr] prevent KeyErrors when using reblogs=same-blog
...
(fixes #851 )
4 years ago
Mike Fährmann
9da2bc67f8
[twitter] add option to filter media from quoted tweets ( #854 )
4 years ago
Mike Fährmann
56ab5fb8f4
[twitter] improve handling of quoted tweets ( #854 )
...
Split each "quote" into two parts:
- the original tweet
- the tweet that quoted the original
4 years ago
Mike Fährmann
bd0e1ca1a5
[imgur] build directory path for each file ( closes #842 )
4 years ago
Mike Fährmann
a8c2d997e8
[twitter] treat quoted tweets like retweets ( #833 )
...
- filter them when 'retweets' is disabled
- set 'author' to the creator of the quoted tweet
like it was before the rewrite
4 years ago
Mike Fährmann
aed1c63e51
[twitter] improve search results ( fixes #847 )
...
Adding 'tweet_search_mode=live' to the query parameters
is the most important part here.
4 years ago
Mike Fährmann
0e714b9a0e
[pinterest] add 'section' extractor ( #835 )
4 years ago
Mike Fährmann
53cc498d9c
improve config lookup when there are multiple possible locations
...
This specifically applies to all Mastodon extractors and all
extractors with a 'basecategory', i.e. 'booru', 'foolslide', etc.
Values inside those general config locations wouldn't be recognized
when a value with the same was set on the 'extractor' level.
For example 'extractor.mastodon.directory' should be used over
'extractor.directory' when both are set, but this was impossible
with the previous implementation.
(fixes #843 )
4 years ago
Mike Fährmann
1b3870a4be
flush after writing JSON in DataJob() ( #727 )
...
… and remove the dead handle_finalize() method,
which is never called since DataJob() overrides run().
4 years ago
Mike Fährmann
d81a8e6544
[twitter] update tests
4 years ago
Mike Fährmann
d39eedd9bb
[twitter] improve handling of deleted tweets ( fixes #838 )
4 years ago
Mike Fährmann
1ae1df0d27
update '--write-pages' ( #737 )
...
- fix infinite recursion for responses with multiple entries in
'history'
- hide values of Set-Cookie headers
- only write the response content by default
(use '-o write-pages=all' to also include HTTP headers)
4 years ago
Mike Fährmann
7e8a747c56
improve output of '-K' for parent extractors 2 ( #825 )
...
This is what bb882b8
was supposed to be, but I managed to
not include those changes in the first commit …
4 years ago
Mike Fährmann
dc16f73965
[twitter] move '_guest_token()' into TwitterAPI class
4 years ago
Mike Fährmann
3561d1020a
[twitter] always provide an 'author' field ( #831 , #833 )
...
The idea was to have less metadata clutter for most Tweets were
'author' and 'user' are the same (non-retweets), and only provide
a 'user' field.
The original Tweet author could be gotten with
{author[…]|user[…]}, but basically no one knows about that.
4 years ago
Mike Fährmann
7158bdd7c7
[weibo] improve extractor logic ( #829 )
4 years ago
Mike Fährmann
37d71f6e09
strip microseconds in text.parse_datetime()
4 years ago
Mike Fährmann
0371fd54a1
[artstation] add 'date' metadata field ( #839 )
4 years ago
Mike Fährmann
8c857052d7
[mastodon] ignore toots without media attachments
4 years ago
Mike Fährmann
de045d39b2
[mastodon] add 'date' metadata field ( #839 )
4 years ago
Mike Fährmann
d5d90a0450
[weibo] add 'date' field to 'status' objects ( #829 )
4 years ago
Mike Fährmann
5ba90f72ca
[pinterest] add support for sections ( closes #835 )
4 years ago
Mike Fährmann
c37a1c06c8
[twitter] add extractor for liked tweets ( closes #837 )
...
You need to be logged in to get access to anyone's liked tweets,
it seems.
4 years ago
Mike Fährmann
b94394104c
[twitter] don't download video previews ( #833 )
...
when 'videos' is set to False
4 years ago
Mike Fährmann
bb882b8cdb
improve output of '-K' for parent extractors ( #825 )
4 years ago
Mike Fährmann
6db7ed90cb
release version 1.14.1
4 years ago
Mike Fährmann
4cf3d54718
[kissmanga] workaround for CAPTCHAs ( fixes #818 )
...
Requesting the same page again when being redirected to a CAPTCHA
lets us access that page without solving it.
4 years ago
Mike Fährmann
7daef6ee70
update extractor test results
...
- certain posts on Instagram now return
https://static.cdninstagram.com/rsrc.php/null.jpg
for public users
- MangaDex is deploying its new MangaDex@Home network similar to
exhentai's Hentai@Home
- realbooru has a new site layout, but the underlying booru API still
works like before
4 years ago
Mike Fährmann
ffb6c5277a
[furaffinity] add 'artist_url' metadata field ( closes #821 )
4 years ago
Mike Fährmann
be04e44e2c
[reddit] catch JSON decode errors ( #765 )
4 years ago
Mike Fährmann
cf863f60b3
[redgifs] add 'user' and 'search' extractors ( closes #724 )
4 years ago
Mike Fährmann
998d1d3a5c
[webtoons] generalize and improve comic extraction ( fixes #820 )
4 years ago
Mike Fährmann
1489712325
resolve redirects after solving Cloudflare challanges
4 years ago
Mike Fährmann
b0b1feaa67
request 'transparent.gif' when solving Cloudflare challenges
...
This currently also works without, but they might be using these to
detect potential bots in the future.
4 years ago
Mike Fährmann
036a40943a
[twitter] don't cache results of 'user_by_screen_name()'
...
A 'keyarg=1' argument to the memcache decorator would have worked as
well, but keeping the user object in memory isn't useful for the vast
majority of use cases and only wastes space.
(closes #817 )
4 years ago
Mike Fährmann
4442dfe7b8
[twitter] add 'reply_to' metadata to replies
4 years ago
Mike Fährmann
83b7bd0413
[nhentai] fix extraction ( closes #819 )
4 years ago
Mike Fährmann
d769bb4b80
[twitter] improve pagination
4 years ago
Mike Fährmann
5bc1097f9d
[twitter] metadata cleanup #2
...
- remove useless clutter by creating new tweet-data dicts instead of
reusing the original Tweet objects
- rename fields to how they were named before
('id_str' -> 'tweet_id', etc.)
- only include 'author' if it would differ from 'user'
- restore 'archive_fmt'
4 years ago
Mike Fährmann
1fcf938f9c
implement a general 'delete_items()' function
4 years ago
Mike Fährmann
c6c06c41f6
[deviantart] don't add journal text to description ( #712 )
4 years ago
Mike Fährmann
4aea5138dd
[sensescans] use https://
4 years ago
Mike Fährmann
3eed5f52d7
[twitter] small metadata cleanup
...
- add 'date' field
- remove 'entities' and 'extended_entities'
- don't include 'focus_fields' from 'original_info'
4 years ago
Mike Fährmann
655c98cbef
[twitter] skip unavailable tweets
4 years ago
Mike Fährmann
41d03160ff
[deviantart] also search journals for sta.sh links ( #712 )
...
when 'extra' is enabled
4 years ago
Mike Fährmann
2132e5461a
[twitter] restore TwitPic support
4 years ago
Mike Fährmann
bd0f21478a
[twitter] login using the mobile nojs login page
4 years ago
Mike Fährmann
a10f31dde5
[twitter] rewrite; use new interface ( #740 , #806 )
...
Everything except logging in with username & password and TwitPic
embeds should be working again.
Metadata per Tweet is massively different than before (mostly raw API
responses - might need some cleaning up) and the default 'archive_fmt'
changed.
4 years ago
Mike Fährmann
f1ef9082bb
release version 1.14.0
4 years ago
Mike Fährmann
3bad1579ee
update extractor test results
4 years ago
Mike Fährmann
864f4220d9
update output of 'oauth:…' ( #616 )
4 years ago
Mike Fährmann
0f459f340b
[instagram] fix and re-enable login with username&password
...
This reverts commit 3e0848a482
.
(#756 , #771 , #797 , #803 )
https://github.com/althonos/InsaLooter/issues/287#issuecomment-630456522
4 years ago
Mike Fährmann
3e0848a482
[instagram] disable login with username&password ( #756 )
4 years ago
Mike Fährmann
a32aea41e1
[instagram] update 'query_hash' values
4 years ago
Mike Fährmann
2bff8dd465
[hentainexus] fix flake8 issues ( #787 )
4 years ago
Mike Fährmann
a63682a9c0
[instagram] simplify code & complete tests ( #743 )
4 years ago
墨焓
a4e3d40672
hentainexus.py minor fix ( #787 )
...
* rectify code of `join_title`, some minor fix.
* + hentainexus self.data
* fixed: call staticmethod join_title with data
4 years ago
Vrihub
62b65e59d0
Add instagram metadata: post_pageurl, post_tags ( #743 )
...
* Add instagram metadata: post_pageurl, post_tags
Add the following metadata for instagram:
- post_pageurl: json string with url of the post page
- post_tags: json array with instagram tags extracted from the post description
* Oops: rename post_tags to tags for --write-tags
This way, --write-tags will pick up the post tags.
* Rename to post_url, improve regex
* Add post_url and tags to tests
* Remove duplicate tags and sort them
* Bugfix: don't create empty tag lists
* Metadata: add location
* Metadata: add tagged_users for each media
* Move self._find_tags() to base class
* Make flake happy
4 years ago
Mike Fährmann
da22ea8ced
use %APPDATA%\gallery-dl for config/cache on Windows
4 years ago
Mike Fährmann
275cceeb6a
[redgifs] fix extraction ( #724 )
...
… and prepare for more potential extractors
4 years ago
Mike Fährmann
45baa13615
update extractor test results
...
- don't run Instagram tests on Travis anymore
- replace Twitter test because timeline was made private
- update Hiperdex domain to '.com' (again ...)
4 years ago
Mike Fährmann
dfcf2a2c91
write OAuth token to cache by default ( #616 )
4 years ago
Mike Fährmann
ddc253cf9a
implement a 'path-replace' option ( #662 , #755 )
4 years ago
Mike Fährmann
15c3d29062
move dump_response() into a separate function ( #737 )
4 years ago
Mike Fährmann
a363da4b43
include redirects and headers in --write-pages dumps ( #737 )
4 years ago
Mike Fährmann
6bcdb264e0
[imgur] treat 't/unmuted' URLs as galleries
4 years ago
Mike Fährmann
b6cee3e45b
[imgur] fix extraction of animated images without 'mp4' entry
4 years ago
Mike Fährmann
bc53302ad6
extend 'path-restrict' option
...
Allow its value to be a JSON object / Python dict that specifies
a mapping from invalid/unwanted input characters to specific
output characters.
For example {"/": "-", "*": "+"} will transform
"foo / ***bar***" into "foo - +++bar+++"
(closes #662 , #755 )
4 years ago
Leonardo Taccari
bcac31b7c7
[webtoons] make archive_fmt unique ( #779 )
...
close #778
4 years ago
Mike Fährmann
34929f673f
readd 'session' to base downloader class ( fixes #768 )
4 years ago
Mike Fährmann
e19f665a44
[danbooru] change default for 'ugoira' to 'false'
...
Downloading the pre-rendered versions should be a better default
than .zip files with individual frames.
4 years ago
Mike Fährmann
3201fe3521
add global SENTINEL object
4 years ago
Mike Fährmann
c8787647ed
add global WINDOWS bool
4 years ago
Mike Fährmann
6294e2c540
add 'text.ensure_http_scheme()'
4 years ago
Mike Fährmann
0378d079a5
[webtoons] fixes and simplifications ( #593 , #761 )
...
- fix episode listings for french comics
- allow input URLs without explicit scheme
- add 'lang'/'language' metadata
- use str.format() instead of '+' to assemble URLs
4 years ago
Mike Fährmann
ab11b1c896
[imagechest] simplify code ( #750 )
4 years ago
Mike Fährmann
ece73b5b2a
make 'path' and 'keywords' available in logging messages
...
Wrap all loggers used by job, extractor, downloader, and postprocessor
objects into a (custom) LoggerAdapter that provides access to the
underlying job, extractor, pathfmt, and kwdict objects and their
properties.
__init__() signatures for all downloader and postprocessor classes have
been changed to take the current Job object as their first argument,
instead of the current extractor or pathfmt.
(#574 , #575 )
4 years ago
Mike Fährmann
846d3a2466
[sexcom] replace 404ed test
4 years ago
Mike Fährmann
9b4635917f
[gelbooru] simplify and fix pool extraction
...
use 'pool:<pool id>' as search tag to get pool posts
4 years ago
Mike Fährmann
abbd8fbbd9
reset filenames on empty file extensions ( #733 )
4 years ago
Mike Fährmann
d17e9628b3
fix/improve Cloudflare bypass code ( #728 , #757 )
...
- support changing values for 'k'
- use XML parser to get request parameters
(some input fields are now embedded in an HTML comment)
4 years ago
Leonardo Taccari
39cd389679
[webtoons] Add a new extractor for webtoons.com ( #761 )
...
The webtoons extractor can extract episode and entire comic (all
episodes) from webtoons.com.
All the logic of the extractors should be trivial except for a couple
of kludges needed:
- `ageGatePass' cookie is always set to avoid possible redirect and stop of
extraction, especially in the comic extractor
- The image URLs returned by the episode extractor could not be fetched
directly and the `Referer:' HTTP header needs to be passed to fetch them
Close #593 .
4 years ago
Bepis
7b5711ee04
[imagechest] Add new extractor for ImageChest ( #750 )
...
* [imagechest] Add new extractor for ImageChest
* [imagechest] Fix flake8 compliance issues
4 years ago
Unit 193
b7ebf51229
Fix typo: defaut → default. ( #754 )
4 years ago
Mike Fährmann
f8661c6578
[downloader:ytdl] fix file extensions when merging into mkv
4 years ago
Mike Fährmann
a1e739b96c
reuse connection adapters from parent extractors
4 years ago
Mike Fährmann
f8f95e68a7
improve '--write-pages' ( #737 )
...
- move code into its own function
- add enumeration index to filenames
- dump responses regardless of status code
4 years ago
Mike Fährmann
dba87ca99e
[downloader:ytdl] change 'forward-cookies' default to 'false'
...
There are currently no situations where forwarding gallery-dl's cookies
to youtube-dl is necessary, and it only causes problems when forcing
youtube-dl for Twitter video downloads while logged in.
4 years ago
Mike Fährmann
09cc9dbec0
prevent flake8 errors from comments looking like type annotations
4 years ago
Mike Fährmann
2d6724180b
[hiperdex] update domain to hiperdex.info
4 years ago
Vrihub
4cc761c730
Implement --write-pages option ( #736 )
...
* Implement --write-pages option
* Fix long lines
* Fix file mode to binary
* Fix pattern for Windows compatibility
4 years ago
Mike Fährmann
f557cac074
[redgifs] add image extractor ( #724 )
4 years ago
Mike Fährmann
11d3247c85
use 'util.Formatter' when formatting logging output
4 years ago
Mike Fährmann
8b60bd6a91
mock 'time()' in cache tests
...
instead of calling 'sleep()' to let time advance.
This shortens the time needed to run those tests,
and ensures consistent results.
(Tests would randomly fail when using 'sleep()')
4 years ago
Mike Fährmann
42f29c3e11
improve and simplify attribute access in DownloadJob.initialize()
4 years ago
Mike Fährmann
65b1cb7acd
[deviantart] use private access tokens for Journals ( fixes #738 )
4 years ago
Mike Fährmann
0bf0146bfe
[reddit] don't send OAuth headers for file downloads ( fixes #729 )
4 years ago
Mike Fährmann
ba42ec321c
release version 1.13.6
4 years ago
Mike Fährmann
d6a480682f
update test results
4 years ago
Mike Fährmann
714566b6e3
fix/improve Cloudflare bypass code
4 years ago
Leonardo Taccari
b47cfc5ac9
[speakerdeck] Add a new extractor for speakerdeck.com ( #726 )
4 years ago
Mike Fährmann
90491ab606
[artstation] improve embed extraction ( #720 )
4 years ago
Mike Fährmann
999efec5cc
[deviantart] limit API wait times to 2**9=512 seconds ( #721 )
4 years ago
Mike Fährmann
504de79d8b
[vsco] fix extraction
4 years ago
Mike Fährmann
5e2974d699
[weibo] add 'videos' option
4 years ago
Mike Fährmann
9f638c2e01
[twitter] add 'replies' option ( closes #705 )
4 years ago
Mike Fährmann
19a7afdd9b
[downloader:http] add MIME types for .psd files ( closes #714 )
4 years ago
Mike Fährmann
fc3e54275b
[patreon] respect filters and sort order in query params ( #711 )
4 years ago
Mike Fährmann
46b9a4d8ff
[patreon] improve hash extraction ( #693 , #713 )
...
Instead of accessing a specific part of a download URL, potentially
causing an exception if it doesn't exist, we're now searching through
all parts for a potential MD5 hash without ever raising an exception.
4 years ago
Mike Fährmann
c56a751dae
[newgrounds] fix URLs produced by 'followng' extractors ( #684 )
4 years ago
Mike Fährmann
baa877d853
release version 1.13.5
4 years ago
Mike Fährmann
a4fd620a25
[hiperdex] revert domain back to hiperdex.com
4 years ago
Mike Fährmann
233b6f93a2
[patreon] recognize URLs with creator IDs ( #711 )
...
e.g. https://www.patreon.com/user/posts?u= …
4 years ago
Mike Fährmann
38b6bd66b0
[500px] match 'web.500px.com' subdomains
4 years ago
Mike Fährmann
d3b3b30107
update test results
4 years ago
Mike Fährmann
5d7ca76885
retry Cloudflare challenges
4 years ago
Mike Fährmann
3eab07739f
[twitter] ensure videos have a 'filename'
...
This usually gets set when invoking the 'ytdl' downloader, but when
that fails, the error message would use 'None' as filename.
4 years ago
Mike Fährmann
c4371a6970
[twitter] add 'reply' metadata field ( #705 )
4 years ago
Mike Fährmann
12ff23b6cc
[mastodon] improve account searches ( fixes #704 )
...
Searching for just the username ("@NAME") can produce multiple
unrelated results, so we now search for username + mastodon instance
("@NAME@INSTANCE")
4 years ago
Mike Fährmann
400a0df661
[jaiminisbox] update decoding procedure ( fixes #702 )
4 years ago
Mike Fährmann
8fe858eb0e
improve parameter extraction when solving Cloudflare challenge
4 years ago
Mike Fährmann
fb98b567fa
[gelbooru] improve post ID extraction for pools
4 years ago
Mike Fährmann
d6facdee7b
[mastodon] add tests ( #701 )
4 years ago
Mike Fährmann
12eebb6f16
[xhamster] support xhamster.porncache.net domains ( closes #700 )
4 years ago
Mike Fährmann
e749402191
[mastodon] fix pagination ( #701 )
4 years ago
Mike Fährmann
921914141e
[imgbb] improve redirect handling
4 years ago
Mike Fährmann
6cc800aad4
[instagram] add 'post_id' and 'num' metadata fields ( closes #698 )
4 years ago
Mike Fährmann
a3de234e70
[hitomi] add extractor for tag searches ( closes #697 )
4 years ago
Mike Fährmann
456f6e8d05
[nozomi] move '_unpack()' method to global scope
4 years ago
Mike Fährmann
55ac408bdf
[hitomi] fix extraction of galleries without tags
4 years ago
Mike Fährmann
db6685eeae
[aryion] support downloading from folders ( fixes #694 )
4 years ago
Mike Fährmann
fa2952ac55
[furaffinity] add 'following' extractor ( #515 )
4 years ago
Mike Fährmann
9b194520db
[newgrounds] add 'following' extractor ( closes #684 )
4 years ago
Mike Fährmann
6386ee54e1
[deviantart] add extractor info to 'following' results
4 years ago
Mike Fährmann
d5273f9b0c
[hiperdex] update domain to hiperdex.net
4 years ago
Mike Fährmann
08674a91f3
[patreon] fix hash extraction from download URLs ( closes #693 )
...
The old method was assuming every URL path ends with '/1'. For URLs
where this is not the case, the segment containing the post ID was
used as file hash.
4 years ago
Mike Fährmann
a31c1aae72
release version 1.13.4
5 years ago
Mike Fährmann
a6286bb551
[hiperdex] add 'artist' extractor ( #606 )
5 years ago
Mike Fährmann
291033720a
[hiperdex] fix manga extraction
5 years ago
Mike Fährmann
dfc0557807
[vsco] fix collection extraction
5 years ago
Mike Fährmann
fd438f0d78
update extractor test results
5 years ago
Mike Fährmann
bae1e8ed12
[deviantart] fix JPEG quality replacement pattern
...
'q_\d+' would sometimes also replace something in the 'token' query
parameter, invalidating the URL.
5 years ago
Mike Fährmann
cf4cef3d63
[aryion] adjust 'date' to UTC time
5 years ago
Mike Fährmann
a0f4c295c0
add optional 'utcoffset' argument to 'parse_datetime()'
5 years ago
Mike Fährmann
6c531be294
[aryion] fix malformed 'last-modified' headers ( #390 )
5 years ago
Mike Fährmann
38bc6430d3
[downloader:http] don't overwrite existing '_mtime' fields
5 years ago
Mike Fährmann
dc65f7d8dc
[aryion] use generic download URLs ( #390 )
...
i.e. /g4/data.php?id=…
- get filename & extension from Content-Disposition header
- handle all downloadable file types (docx, swf, etc)
5 years ago
Mike Fährmann
96b78bcf04
[aryion] include path in default directory format ( #390 )
5 years ago
Mike Fährmann
300264f676
read config files from PyInstaller exe directory ( closes #682 )
5 years ago
Mike Fährmann
6143050980
[aryion] add gallery and post extractors ( #390 , #673 )
5 years ago
Mike Fährmann
9e7dfc0cfc
[myportfolio] fix extraction of galleries without title
5 years ago
Mike Fährmann
88fca0a172
[mastodon] update OAuth credentials for pawoo.net ( #665 )
5 years ago
Mike Fährmann
4ae8a25567
[mastodon] use 'combine_dict()' to combine extractor info dicts
5 years ago
Mike Fährmann
220c06b86e
[mastodon] handle rate limits
5 years ago
Mike Fährmann
d02f7c1118
improve Extractor.wait()
...
- allow 'until' to be a datetime object
- do "time calculations" with UTC timestamps
- set a default 'reason'
5 years ago
Mike Fährmann
5d7404ab58
[oauth] use the new name for 'DeviantartAPI' ( fixes #670 )
5 years ago
Mike Fährmann
762c758af4
[hiperdex] fix extraction
5 years ago
Mike Fährmann
f9a590f92b
[deviantart] apply HTTP request limits in more places
...
"Request blocked" can also happen on sta.sh and for *any* HTTP
request directed at deviantart.com
5 years ago
Mike Fährmann
2587296deb
[mastodon] add access tokens for mastodon.social and baraag.net
...
(closes #665 )
5 years ago
Mike Fährmann
ff7c0b7eff
[deviantart] handle "Request blocked" errors ( #655 )
...
- add a 2 second wait time between requests to deviantart.com
- catch 403 "Request blocked" errors and wait for 3 minutes until
retrying
5 years ago
Mike Fährmann
c874684f05
[deviantart] retrieve *all* download URLs through OAuth API
...
'/extended_fetch' as well as Deviation webpages now again contain
Deviation UUIDs needed to grab Deviation info through the OAuth API,
meaning cookies are no longer necessary to grab original files.
The only instance were cookies are still needed are scraps marked as
"mature", since those entries are hidden for public users.
(#655 , #657 , #660 )
5 years ago
Mike Fährmann
5c27b25a8f
[deviantart] improve sta.sh extraction
...
Extract all sta.sh items in a single extractor run.
Don't spawn a new StashExtractor for each individual sta.sh item to
preserve the current requests.Session and its opened TCP connections.
5 years ago
Mike Fährmann
e2fc4eaa6f
[deviantart] detect stash folders ( fixes #659 )
5 years ago
Mike Fährmann
c034159701
[piczel] fix extraction for single images
5 years ago
Mike Fährmann
699036ea0c
[weibo] accept status URLs with non-numeric IDs ( #664 )
5 years ago
Mike Fährmann
fe96f99e4b
[hentainexus] reduce line length (flake8) & update test
5 years ago
墨焓
6f81cac8fa
Add metadata to hentainexus: circle, event, title_conventional. ( #661 )
5 years ago
Mike Fährmann
3ed72f82dc
release version 1.13.3
5 years ago
Mike Fährmann
6f911aeb1c
[deviantart] add error message for cloudFront blocks ( #655 )
5 years ago
Mike Fährmann
7499d71d02
[simplyhentai] ignore certificate errors in video test
5 years ago
Mike Fährmann
4203dc0bdc
[mangapark] fix metadata extraction
5 years ago
Mike Fährmann
6ecb0a19cf
handle sys.stdin being None when using '-' as input file ( #653 )
5 years ago
Mike Fährmann
1b82d36ab2
[deviantart] handle decode errors for extended_fetch results ( #655 )
...
This isn't going to solve the underlying problem, but it should at
least provide the server response when those errors happen.
5 years ago
Mike Fährmann
09f2271528
[35photo] add 'tag' extractor
5 years ago
Mike Fährmann
77fda8190c
[35photo] simplify/remove tests for the 'genre' extractor
...
There is still a nice genre overview page (https://35photo.pro/genre/ )
but the individual sub-pages don't list photos anymore
5 years ago
Mike Fährmann
4bc161ca0f
prevent crash when sys.stdout and co. are None ( #653 )
5 years ago
Mike Fährmann
fb846c9ee5
[instagram] reduce line lengths and make flake8 happy
5 years ago
Mike Fährmann
ad2efa8509
[e621] derive from Danbooru extractors ( #651 )
...
- use extractor implementations from 'danbooru'
- use "page": "b[ID]" to paginate over results instead of
"tags": "id:<[ID]", avoiding infinite loops with certain
post orders
- bump User-Agent version
5 years ago
Mike Fährmann
9b39e1cd7e
[e621] fix bug in API rate limiting ( #651 )
5 years ago
Mike Fährmann
b607d0ad7f
[twitter] fix typo in 'x-twitter-auth-type' header ( #625 )
5 years ago
Mike Fährmann
9159cb8fb3
remove trailing dots and spaces from directory names ( #647 )
5 years ago
Mike Fährmann
2c3b9e1450
[nozomi] support multiple images per post ( #646 )
...
This changes the default filename format as well as archive IDs,
since those assumed that each post would only have one image.
5 years ago
Mike Fährmann
c606d0c854
[instagram] update pattern for user profile URLs
...
Allow for query parameters and fragments,
for example https://www.instagram.com/instagram/?hl=en
5 years ago
Mike Fährmann
2530db3f4d
[mangadex] transform 'date' timestamps to datetime objects
5 years ago
Mike Fährmann
ae2a33243b
[newgrounds] catch general Exceptions
5 years ago
Mike Fährmann
32e36d8f02
[sexcom] replace tests
5 years ago
Mike Fährmann
33b42dc847
[nozomi] sort search results ( fixes #646 )
5 years ago
Mike Fährmann
eaa60a438b
[piczel] fix extraction
...
- manually filter by folder_id
- extract data for single posts from embedded JSON, since the
'/api/gallery/image/<id>' endpoint is no longer available
5 years ago
Mike Fährmann
5bcc7184c9
[danbooru][e621] increase page limits
5 years ago
Mike Fährmann
90d15e3682
[instagram] use 'itertools.chain()'
5 years ago
Leonardo Taccari
160328d21c
[instagram] Add support for user's saved medias ( #644 )
...
* [instagram] Gracefully handle possible 'HttpErrorPage' in _extract_page()
`HttpErrorPage' is returned in shared_data at least when not authenticated or
when trying to fetch other users saved medias
(i.e. `instagram.com/<user>/saved/').
Gracefully handle it by returning nothing.
* [instagram] Add support for user's saved medias
(Please note that this need the user to be authenticated and they can
only see their saved media (not other users ones).)
Close #643 .
* [instagram] Bump copyright year
5 years ago
Mike Fährmann
e0b0e8d62a
release version 1.13.2
5 years ago
Mike Fährmann
d3482ace7f
[furaffinity] extract more metadata
...
- views
- favorites
- comments
- rating
- fa_category (since 'category' is already in use)
- theme
- species
- gender
- width
- height
5 years ago
Mike Fährmann
f6c5edb76b
pre-compile regex pattern for remove_html() and split_html()
5 years ago
Mike Fährmann
fdd2dd5136
[kabeuchi] add 'user' extractor ( closes #561 )
5 years ago
Mike Fährmann
59edcdc822
[hitomi] restore metadata fields from before f33b13a
...
... and add a 'metadata' option to disable
visiting the gallery page and extracting data from it
if this is not needed.
5 years ago
Mike Fährmann
2d5703c493
[twitter] use a simpler data structure to store cookies in cache
...
Use a dict with name-value pairs instead of an entire
RequestsCookieJar object.
5 years ago
Mike Fährmann
87d4f83597
[newgrounds] make post extraction nonfatal
5 years ago
Mike Fährmann
823fbeaae6
[newgrounds] add 'favorite' extractor ( #394 )
5 years ago
Mike Fährmann
a45fbc38ea
[pixiv] implement 'avatar' option ( #595 , #623 )
5 years ago
Mike Fährmann
a63a376ad2
[mangoxo] fix login
5 years ago
Mike Fährmann
ebc70e87ce
[e621] update to new interface / API endpoints ( closes #635 )
5 years ago
Mike Fährmann
d1cf7ccdb3
[instagram] add 'post_shortcode' metadata field ( #525 )
5 years ago
Mike Fährmann
32df8d06fe
[twitter] add 'bookmark' extractor ( closes #625 )
5 years ago
Mike Fährmann
3fb41c34c8
[bcy] reduce requests to '/item/detail/<id>' ( #613 )
...
The former implementation would try to use the embedded data from
'/item/detail/' pages for every post, even if that wasn't really
necessary.
This commit also fixes some issues with posts only visible to
logged in users.
5 years ago
Mike Fährmann
f33b13aacf
[hitomi] simplify metadata extraction
...
Use the data from https://ltn.hitomi.la/galleries/ <id>.js for both
image URLs and metadata and ignore any gallery or reader pages.
This removes 'artist', 'characters', 'group', and 'parody' metadata
fields since this information is, as for now, only available in
gallery pages.
5 years ago
Mike Fährmann
115fd2c6f2
"fix" incomplete MIME types ( #632 )
...
e-/exhentai's original image downloads currently send
incomplete/invalid Content-Type headers, "jpg" instead
of "image/jpg" etc, since the last update.
(https://forums.e-hentai.org/index.php?showtopic=236113 )
This change prepends any Content-Type value missing a
media type specification with "image/", transforming it
into a valid MIME type.
(A global solution to a local problem, but it shouldn't
cause any issues anywhere else)
5 years ago
Mike Fährmann
72122eb9b3
release version 1.13.1
5 years ago
Mike Fährmann
adcd7cb24a
[downloader:http] add another MIME type for '.rar' files ( #628 )
5 years ago
Mike Fährmann
ce5e2a58fe
[imgbb] update test results
...
Image server domain changed from
https://image.ibb.co/ to https://i.ibb.co/
5 years ago
Mike Fährmann
f117e32910
[danbooru] restore 'popular' functionality
5 years ago
Mike Fährmann
39b48d665b
[hiperdex] use proper name for 'chapter_minor'
5 years ago
Mike Fährmann
8fbbaa54ff
[bcy] fix partial image URLs ( #613 )
...
Images from new posts can have incomplete/partial URLs (1)
without any filename extension when fetching their data from
'/apiv3/user/selfPosts', so now all data gets taken from
'/item/detail/ID' pages.
It is currently unknown how to get the non-watermarked original version
of these images, or if that is possible at all. (2)
Images with a watermark will have their 'filter' metadata field set to
"watermark". For original images this field is an empty string "".
Enabling the 'noop' option will, in addition to the watermarked version,
yield the the '~noop.image' filter version (3),
where 'filter' is set to "noop".
(1) "https://img-bcy-qn.pstatp.com/banciyuan/3ccdff22479c4060aadc86718209b281 "
(2) "https://p1-bcy.byteimg.com/img/banciyuan/3ccdff22479c4060aadc86718209b281~tplv-banciyuan-logo-v3:wqnpnLLlhZLlpKfprZTnjotfCuWNiuasoeWFgyAtIEFDR-eIseWlveiAheekvuWMug==.image "
(3) "https://p1-bcy.byteimg.com/img/banciyuan/3ccdff22479c4060aadc86718209b281~noop.image "
5 years ago
Mike Fährmann
86c00f9e66
[danbooru] move extractor logic from booru.py
5 years ago
Mike Fährmann
1d4a369ea2
update extractor test results
5 years ago
Mike Fährmann
7625912b31
[piczel] improve and update
...
- fix tag names
- fix a bug in _pagination()
- parse datetime in 'created_at' as 'date'
- rewrite main loop
- replace user profile test
5 years ago
Mike Fährmann
ec85bf90de
use context managers in cache.py & add tests
5 years ago
Mike Fährmann
913b8333cc
write DeviantArt refresh-tokens to cache ( #616 )
...
Writing the token is currently disabled by default and must be
enabled with 'extractor.oauth.cache'.
'extractor.deviantart.refresh-token' must be set to '"cache"'
to use the cached token.
5 years ago
Mike Fährmann
2a4f227e08
warn about expired cookies
5 years ago
Mike Fährmann
34887ae139
fix bugs in DatabaseCacheDecorator.update()/.invalidate()
...
- call db.commit() after changes have been made
- remove 'LIMIT 1' from the DELETE statement in invalidate()
(only available if SQLite3 was compiled with the right flags
enabled, syntax error otherwise)
5 years ago
Mike Fährmann
380b693fad
[downloader:http] add more MIME types for '.bmp' files ( #621 )
5 years ago
Mike Fährmann
4e361b3008
add tests for specific datetime values
5 years ago
Mike Fährmann
80ecb99089
[hitomi] fix extraction
5 years ago
Mike Fährmann
247c9e1416
[vsco] update gallery URL pattern
5 years ago
Mike Fährmann
19ae6f3fc4
update test results
...
- twitter:
Don't test the whole kwdict, only the actual content, since the
keyword hash changes whenever that user changes his display name.
- khinsider:
Download host changed
5 years ago
Mike Fährmann
cc5079c844
[hiperdex] add chapter and manga extractors ( closes #606 )
5 years ago
Mike Fährmann
64bdec8430
[deviantart] check availability of intermediary URLs ( fixes #609 )
5 years ago
Mike Fährmann
5607dd3646
[hitomi] follow multiple redirects
5 years ago
Mike Fährmann
765b2a0527
[hentaihand] add extractors ( closes #605 )
5 years ago
Mike Fährmann
d94215d119
[tumblr] replace '-' with ' ' in tag searches ( fixes #611 )
...
To search for tags with actual minus signs in them
(there shouldn't be too many,) manually replace those
with url-encoded minus characters ('-' -> '%2d')
before inputting them into gallery-dl:
https://s679874.tumblr.com/tagged/tag-with-minus
->
https://s679874.tumblr.com/tagged/tag%2dwith%2dminus
5 years ago
Mike Fährmann
5cdf1b1319
fix --verbose/--quiet
...
caused by 383795b
5 years ago
Mike Fährmann
78e8d33c97
release version 1.13.0
5 years ago
Mike Fährmann
e6cd49e78b
update extractor test results
5 years ago
Mike Fährmann
90e4c645ba
[formatter] allow multiple "special" format specifiers ( #595 )
...
It is now, for example, possible to specify multiple replacement
operations per format replacement field: {name:Ra/b/Rc/d/}
5 years ago
Mike Fährmann
5d9437b398
[vsco] skip "invalid" entities
5 years ago
Mike Fährmann
650f2b6d58
[furaffinity] accept sfw.furaffinity.net URLs ( closes #608 )
...
Just as an alias for regular URLs with no extra content filtering.
5 years ago
Mike Fährmann
219c4cc78c
[formatter] allow for numeric list and string indices
5 years ago
Mike Fährmann
7d1da614d9
[formatter] implement field name alternatives ( #525 )
...
The format string '{a|b|c}' will now try to use the value from 'a' and
fall back to 'b' and 'c' if accessing a field raises an exception or
if its value is None.
5 years ago
Mike Fährmann
74e684e828
[twitter] change default value for 'videos' to 'true'
...
Every other 'videos' option defaulted to 'true', except Twitter.
5 years ago
Mike Fährmann
c7cf9dd111
[furaffinity] support classic layout ( #284 )
5 years ago
Mike Fährmann
138135c190
[furaffinity] add extractors ( #284 )
5 years ago
Mike Fährmann
b9c574bd1d
[patreon] log skipped files ( #590 )
5 years ago
Mike Fährmann
80ea9104b8
[8kun] adjust URL pattern
5 years ago
Mike Fährmann
c76c8b765a
[cloudflare] unescape challenge URL
5 years ago
Mike Fährmann
ce26070231
[pixiv] reduce calls to '/user/detail'
5 years ago
Mike Fährmann
da0d5f6092
[oauth] add 'port' option ( #604 )
5 years ago
Mike Fährmann
719b63d0ca
[bcy] add user and post extractors ( #592 )
5 years ago
Mike Fährmann
6426e3efc7
[khinsider] fix and improve metadata extraction
5 years ago