aa1ca4ed35
[shopify] skip deleted products ( #175 )
...
Product pages which return a 4xx status code will now be skipped instead
of raising an exception.
5 years ago
096009367b
[xhamster] add gallery & user extractor ( #281 )
5 years ago
208202b962
[tumblr] improve error handling ( #297 )
...
In some cases Tumblr's API responds with an HTML document.
Trying to decode it as JSON would raise an uncaught exception.
5 years ago
c08c340178
[directlink] make pattern case insensitive ( fixes #296 )
5 years ago
95b4a53b9c
[keenspot] improve pagination ( #223 )
...
The old code would skip the last comic page for some series.
5 years ago
12c965d547
release version 1.8.5
5 years ago
731c7cbd5b
[keenspot] support all comics and "random" access ( #223 )
5 years ago
6a34f4b0c1
skip tests on read timeouts; print list of skipped tests
5 years ago
1c36e65e9b
[exhentai] choose site version depending on input URL ( #278 )
...
Use e-hentai.org as root and cookiedomain if the input URL is from
e-hentai (or g.e-hentai), use exhentai.org otherwise.
5 years ago
6da3e21237
[downloader:ytdl] provide 'filename' metadata ( closes #291 )
5 years ago
d33f5a7423
[wallhaven] rewrite
...
- use API
- remove login support, add 'api-key' option
- remove support for "alpha" subdomain - alpha.wallhaven.cc used numeric
IDs that can't be translated to the new ID system
- support direct links to wallpapers
5 years ago
5499934ae2
[ngomik] fix extraction
5 years ago
f1893b2b5b
[deviantart] add 'folders' option ( #276 )
5 years ago
c849574def
[keenspot] add comic extractor ( #223 )
...
Doesn't work for
- http://brawlinthefamily.keenspot.com/
- http://flipside.keenspot.com/
- http://lastblood.keenspot.com/
- http://mysticrevolution.keenspot.com/
- http://porcelain.keenspot.com/
- http://twokinds.keenspot.com/
yet, because of custom layouts.
5 years ago
2b1999476e
implement 'text.rextract()'
5 years ago
8bd5a19515
[hentainexus] add '_extractor' data
5 years ago
2a085a5e96
[sankakucomplex] fix 'date' values ( #258 )
5 years ago
bcd1801aa8
[sankakucomplex] add 'tag' extractor ( #258 )
5 years ago
74c2415138
[sankakucomplex] move article extractor to its own module ( #258 )
5 years ago
4465a3ea68
[kissmanga][readcomiconline] add 'captcha' option ( #279 )
...
to configure how to handle CAPTCHA page redirects:
- either interactively wait for the user to solve the CAPTCHA
- or raise StopExtraction like before
5 years ago
e30ada162d
fix cookie tests
...
update _get_extractor():
- always return an Extractor instance with a _login_impl() method
- use Extractor.from_url()
5 years ago
1e3e15c4f3
[sankaku] add article extractor ( #258 )
5 years ago
48233f00c0
[readcomiconline] detect 'AreYouHuman' redirects ( #279 )
5 years ago
1cde38110d
[livedoor] return 'date' as datetime object
5 years ago
e88824e1a7
[livedoor] fix adjustments for https:// URLs
5 years ago
2316e0ed3d
fix strptime workaround from b0e85a4
...
Don't return a modified version of 'date_time' if strptime fails.
5 years ago
b3e4664715
[hentainexus] fix extraction
5 years ago
399e8e965a
also update urllib3's cipher list for versions >= 1.25
5 years ago
f837ea98cb
[deviantart] don't call 'extend()' on folders ( fixes #271 )
5 years ago
bb32a2d490
[patreon] use file extensions from original filenames ( #268 )
5 years ago
efa805c5d7
[sankaku] update pagination end condition ( fixes #265 )
...
Pagination over popular listings (`date:...+order:popular") never
terminates, not even on the site itself, and at some point returns the
same results over and over again.
5 years ago
d514d49c72
release version 1.8.4
5 years ago
a4ba34c835
[booru] prevent crash when no tags are present ( #259 )
5 years ago
ca3bad1779
[patreon] small fixes and adjustments ( #226 )
...
- fix datetime parsing
- rename 'user' to 'creator'
- convert 'id' to integer
- improve tests
5 years ago
fb09dd962a
[instagram] Fix extraction after `rhx_gis' field removal
5 years ago
7a14aaed7d
[luscious] fix extraction
5 years ago
e82cadac61
[patreon] add extractors ( #226 )
5 years ago
4891f4a328
[hentainexus] add search extractor ( #256 )
5 years ago
c02f12ce2f
avoid Cloudflare CAPTCHAs for OpenSSL < 1.1.1
...
see https://github.com/Anorov/cloudflare-scrape/pull/242
5 years ago
0b4be57a10
[sankaku] fix error when no tags available ( closes #259 )
...
[ci skip]
5 years ago
6764847349
fix cookie tests
...
'cookies' is a CookieJar, not a dict,
and removing the call to '.keys()' doesn't have the same effect
5 years ago
9890bfdf23
[flickr] improve code and metadata
...
- simplify pagination
- add more metadata and slightly change its structure
- convert suitable values to int or list
- move keys from ["photo"] to the base level
- proper video support (#246 )
- rename method and variable names to better fit with other extractors
5 years ago
aa8e366b90
[luscious] fix tag extraction
5 years ago
a5b060765d
improve code in tests
...
- use 'assertRaises' as context manager
- remove calls to .keys()
5 years ago
ba8eb1ffec
[hentainexus] add gallery extractor ( #256 )
5 years ago
bd9cb3d191
improve job class selection code
...
+ consistent argument order for add_argument() calls
5 years ago
e64773ffdd
allow multiple post-processor command-line options ( #253 )
...
... without overwriting any previous ones
5 years ago
b1db194c14
[reactor] update and improve
...
- split 'tags' into a list
- parse 'date' into a datetime object
- fix webm/mp4 URLs
5 years ago
b0e85a42e3
apply workaround from 4736912 in parse_datetime() itself
5 years ago
523ebc9b0b
Fix serialization of 'datetime' objects in '--write-metadata'
...
Simplified universal serialization support in json.dump() can be achieved
by passing 'default=str', which was already the case in DataJob.run()
for -j/--dump-json, but not for the 'metadata' post-processor.
This commit introduces util.dump_json() that (more or less) unifies the
JSON output procedure of both --write-metadata and --dump-json.
(#251 , #252 )
5 years ago
Mike Fährmann