5499934ae2
[ngomik] fix extraction
5 years ago
a5b060765d
improve code in tests
...
- use 'assertRaises' as context manager
- remove calls to .keys()
5 years ago
5582b06ae4
fix tests with 'urllist' messages
5 years ago
5018781898
allow type tests by name
5 years ago
e25ebc4bff
don't disable certificate checks anymore
...
Executables generated with PyInstaller auto-include the root certificate
file and certificate checks now work out-of-the-box.
6 years ago
d6ddb74cde
update test results
...
- deviantart: 'index' is now an integer
- flickr: image file with lower quality
- paheal: image server name changed
- rule34: post got deleted
6 years ago
d9b94a585d
[mangoxo] add login support ( #184 )
...
A very recent change: It is now only possible to see more
than the first 5 images of an album if you are logged in.
6 years ago
e730fc9045
[twitter] add login support ( #214 )
6 years ago
790f15a56f
[photobucket] use HTTPS
6 years ago
c70b21248d
[wikiart] add extractors ( #179 )
...
for
- artists: https://www.wikiart.org/en/thomas-cole
- artist-listings: https://www.wikiart.org/en/artists-by-century/12
- artwork-listings: https://www.wikiart.org/en/paintings-by-media/grisaille
6 years ago
0c991a3155
add convenience targets to Makefile
6 years ago
973a720a7a
[weibo] fix unit test URL patterns
6 years ago
6f57d44ec2
[seaotterscans] remove extractor
...
http://seaotterscans.com/ now redirects to their MangaDex profile
6 years ago
0887fb61f4
[komikcast] update test results
6 years ago
976ccb267f
[myportfolio] combine gallery and user extractors
...
An URL alone isn't good enough to distinguish between a gallery or a
gallery-listing, so the new extractor decides what to do based on the
page's content.
6 years ago
9c0e2f294b
[shopify] add generic collection and product extractors ( #175 )
...
with fashionnova.com as a default domain
6 years ago
e687a6095e
[luscious] raise exception if album is not available
6 years ago
b09a8184ca
move TestJob into test module; test _extractor values
6 years ago
1f3422c28b
[mangahere] fix extraction
6 years ago
84ae72b8d8
[ngomik] fix extraction
6 years ago
bc0951d974
allow for simplified test data structures
...
Instead of a strict list of (URL, RESULTS)-tuples, extractor result
tests can now be a single (URL, RESULTS)-tuple, if it's just one test,
and "only matching" tests can now be a simple string.
6 years ago
347398f692
fix various tests
6 years ago
0c32dc5858
[hentaifox] add extractor for search results ( #160 )
6 years ago
217a0687ef
[behance] add 'collection' extractor ( closes #157 )
6 years ago
66460337f1
[mangapark] fix extraction
6 years ago
9bbbadd93a
[hbrowse] use HTTPS
6 years ago
98c6520384
[pinterest] update root URL of API calls
6 years ago
751e535948
[nhentai] fix extraction ( closes #156 )
...
Use JSON embedded in webpage since API endpoints have been disabled
6 years ago
1734a6c879
[reactor] detect "circular" redirects ( #148 )
6 years ago
e53cdfd6a8
update build_supportedsites.py
6 years ago
0afa913de4
[tumblr] add tests for hidden and private blogs ( #145 )
...
Hidden / dashboard-only blogs are pretty straightforward and "only"
require a valid 'access-token' and 'access-token-secret' for the given
'api-key' and 'api-secret', so that signed OAuth1.0 requests are possible.
Private / password protected blogs on the other hand are a bit
cumbersome. In addition to a valid 'access-token' and
'access-token-secret', they also require the account belonging to those
tokens to be a member of the blog itself. Knowing the password and
entering it in the website isn't enough to access a blog through the
API. Following a private blog is also impossible, so that option can't
work either.
6 years ago
fa7fa2f8ff
[deviantart1 update tests]
6 years ago
259123732f
[readcomiconline] improve comic-page parsing
6 years ago
6c71e9cf5d
[deviantart] add separate 'sta.sh' extractor ( #113 )
...
- supports multiple stashed deviations per page
- explicitly mentions sta.sh support on supportedsites.rst
6 years ago
4d73cc785d
update test results
6 years ago
010da8372a
[instagram] relax test pattern
6 years ago
15890930ea
[mangafox] fix extraction
...
use mobile version since desktop version is obfuscated
6 years ago
fb53b5dd55
fix control+c during -j and range tests
6 years ago
59bb434ba5
[flickr] add ability to download all albums of a user
...
for example with 'https://www.flickr.com/photos/shona_s/albums '
6 years ago
041bd501fc
[hentaifoundry] unescape YII_CSRF_TOKEN value
...
This fixes the POST requests to /site/filters
6 years ago
d4b2b73bef
release version 1.6.0
6 years ago
3c25fa2dad
update build_testresult_db.py script
6 years ago
7f6a0be982
adjust some tests
6 years ago
966a9ca3a0
update test results
6 years ago
c9861ca812
adjust message for status_code based exceptions
...
from: 5xx HTTP Error: Reason
to : 5xx: Reason
The "HTTP Error" part was in there to emulate Request's error messages
from response.raise_for_status(), but it reads a lot better without.
6 years ago
c00dce2adc
[behance] enable 'categorytransfer'
6 years ago
1532d1b690
fix 'range' tests and update a few test results
6 years ago
ca6ac4db6a
fix 'content' tests
6 years ago
d70db2d555
Revert "[komikcast] fix extraction"
...
This reverts commit 5507f5ce2e
6 years ago
5507f5ce2e
[komikcast] fix extraction
6 years ago
17611bfec0
update build_supportedsites.py script
6 years ago
e066f35118
update extractor tests
6 years ago
22ab509a70
[bobx] rename "model" to "idol" extractor
6 years ago
8a23b21d0e
[tests] let 'pattern' require at least 1 URL
6 years ago
0bc8ef51c8
[smugmug] Handle albums with no explicit owner ( #100 )
6 years ago
34b556922d
update/restore tests
6 years ago
e3055d356c
release version 1.5.1
6 years ago
f9ded38d89
[test:results] add support for "range" options in tests
6 years ago
7f4e41c989
increase timeout during extractor tests
...
cloudflare's 522 response takes longer than 30 seconds
6 years ago
b55e39d1ee
[mangadex] improve extraction
...
- cache manga API results
- add artist, author and date fields to chapter metadata
- remove Manga-/ChapterExtractor inheritance
- minor code simplifications and improvements
6 years ago
2a9f3341a2
[behance] fix title extraction
6 years ago
a86f2bfc80
[pinterest] update not-found redirects
6 years ago
7442d2940c
release version 1.5.0
6 years ago
b040ca0718
[rule34] small unit test fixes
6 years ago
f3793660ef
update tests
6 years ago
42a346413b
fix "re:" prefix for keyword tests
6 years ago
bb89a1e6d7
[mangahere] use http://
...
invalid SSL cert for quite some time now
6 years ago
ce34d82cb4
fix skipping tests on 5xx status codes
6 years ago
a6fe2bb594
[whatisthisimnotgoodwithcomputers] remove extractor
6 years ago
0ba93650e0
[8chan] replace unit test URL
...
the other thread is no longer accessible
6 years ago
269dc2bbd5
[sankaku] add 'tags' option ( #94 )
6 years ago
764331823b
release version 1.4.2
6 years ago
2eefaa99a3
[mangapark] support .net and .com mirrors
6 years ago
188e956c4e
[imagefap] use HTTPS + update test results
6 years ago
a699787d01
[deviantart] update URL patterns to new format
...
DeviantArt changed its URL format from
https://<name>.deviantart.com/...
to
https://www.deviantart.com/ <name>/...
With this change both formats will be supported.
6 years ago
b8c97d2295
use 'extractor.request()' for more HTTP requests
6 years ago
7a98cc9798
[smugmug] update tests
...
My test account expired and all uploaded images got deleted.
6 years ago
4eb94aca17
[postprocessor:ugoira] pass '-f' if not present
6 years ago
a9e276bc37
reset delete-flag
...
Since 'PathFormat' objects are being reused, setting `delete`
to True once caused all files downloaded after to be deleted as well.
6 years ago
6ac403c5d3
add postprocessor config example
6 years ago
a47c6136cd
[simplyhentai] avoid redirects for all-pages.json ( #89 )
6 years ago
0a1863fce3
[pixiv] respect more query parameters for user URLs
...
The API endpoint responsible for user illustrations does not
provide sufficient filter capabilities* to match the actual
website, so we are spinning our own filters.
Respected parameters are
'type': illust, manga, ugoira
'tag' : any image tag (this was already supported)
'p' : the page to start on
*
- API can filter for illustrations and manga, but not for ugoira.
- 'offset' is applied before filtering
- no 'tag' filter
6 years ago
4cea886177
[imgur] allow longer album hashes
6 years ago
e1e23165a0
[pinterest] catch JSON decode errors
6 years ago
e2157f594e
[mangadex] fix manga extraction ( closes #84 )
...
Chapter listings for manga now use
https://mangadex.org/manga/ <id>/_/chapters/2/
as URL instead of
https://mangadex.org/manga/ <id>/_//2/
6 years ago
3fe653d940
fix test_results for empty sets
...
{} is an empty dict and doesn't support set operations
6 years ago
d96b3474e5
[puremashiro] remove module
...
site has been unreachable for a couple of weeks
and now the DNS record is gone as well
6 years ago
b44a296404
[gomanga] remove module
...
site has been unreachable for a couple of weeks
and the cloudflare status page shows host errors
6 years ago
2395d870dd
[pinterest] unquote board and user names, better errors
6 years ago
55d4d23860
[pinterest] use Pinterest's "Web" API ( #83 )
...
no access tokens, no user credentials of any kind ...
7 years ago
10cc59f3b5
fix extractor names
7 years ago
df7e18399e
[luscious] fix image order
7 years ago
d10579edb5
[pinterest] improve PinterestAPI code; remove OAuth mentions
...
on another note: access_tokens have been set to only allow for
10 requests per hour (from 200 yesterday)
7 years ago
4bd182c107
[pinterest] implement `oauth:pinterest` ( #83 )
...
Pinterest access tokens are rate limited at 200 requests per
hour (or maybe per 2 or 3 hours?) so having just one access token
for all users isn't going to work in the long run.
7 years ago
dbe250f7e5
[pinterest] update access_token ( #83 )
7 years ago
48a83a89e9
[loveisover] remove module
...
archive.loveisover.me was shut down on 2018-03-29;
https://www.archiveteam.org/index.php?title=4chan#archive.loveisover.me
7 years ago
564e12ca8f
replace 'imgyt' with 'imxto'
...
https://img.yt/ wasn't available for a couple of days, but has now
re-emerged as https://imx.to/ with a new web-interface.
Links to older images still work (see tests).
7 years ago
d11fcf4804
smaller changes and fixes
...
- fix the cloudflare challenge result if the last decimal places
are zero (JS`s toFixed() removes trailing zeroes)
- fix downloading of kissmanga chapter-pages hosted on blogspot
(accessing blogspot with "kissmanga.com" as referrer yields a 401)
- disable certificate validation for 'mangahere' tests
- update flickr test result
7 years ago
759ba26fb0
[luscious] proper image order for picture albums
...
... and (try) to start with the first image instead of somewhere
in the middle of an album.
7 years ago
0381ae5318
replace error handlers for stdout and co.
...
Python3.5 and lower throw an UnicodeEncodeError when trying to print
not-encodable characters when not using 'utf-8' as encoding.
Setting their error handlers to 'replace' should help.
7 years ago
64d7c85b55
[exhentai] improve metadata
...
- add 'width', 'height' and 'size' (in bytes) for each image
- change the former 'size' and 'size_units' into 'gallery_size'
7 years ago
a112e3f2a0
[nijie] add doujin extractor
...
adds support for "https://nijie.info/members_dojin.php?id= <artist_id>"
7 years ago
f5c6a2d7f5
[nhentai] use API to get gallery info
7 years ago
8ef790de12
update .travis.yml
...
- restrict builds to master branch and release tags
- implement 'core' and 'results' test categories
7 years ago
Mike Fährmann