300264f676
read config files from PyInstaller exe directory ( closes #682 )
5 years ago
6143050980
[aryion] add gallery and post extractors ( #390 , #673 )
5 years ago
9e7dfc0cfc
[myportfolio] fix extraction of galleries without title
5 years ago
3b50c4f49d
add tests for "Extractors" in oauth.py ( #670 )
5 years ago
04bd0472de
add tests for Extractor.wait()
5 years ago
88fca0a172
[mastodon] update OAuth credentials for pawoo.net ( #665 )
5 years ago
4ae8a25567
[mastodon] use 'combine_dict()' to combine extractor info dicts
5 years ago
220c06b86e
[mastodon] handle rate limits
5 years ago
d02f7c1118
improve Extractor.wait()
...
- allow 'until' to be a datetime object
- do "time calculations" with UTC timestamps
- set a default 'reason'
5 years ago
5d7404ab58
[oauth] use the new name for 'DeviantartAPI' ( fixes #670 )
5 years ago
762c758af4
[hiperdex] fix extraction
5 years ago
f9a590f92b
[deviantart] apply HTTP request limits in more places
...
"Request blocked" can also happen on sta.sh and for *any* HTTP
request directed at deviantart.com
5 years ago
2587296deb
[mastodon] add access tokens for mastodon.social and baraag.net
...
(closes #665 )
5 years ago
ff7c0b7eff
[deviantart] handle "Request blocked" errors ( #655 )
...
- add a 2 second wait time between requests to deviantart.com
- catch 403 "Request blocked" errors and wait for 3 minutes until
retrying
5 years ago
c874684f05
[deviantart] retrieve *all* download URLs through OAuth API
...
'/extended_fetch' as well as Deviation webpages now again contain
Deviation UUIDs needed to grab Deviation info through the OAuth API,
meaning cookies are no longer necessary to grab original files.
The only instance were cookies are still needed are scraps marked as
"mature", since those entries are hidden for public users.
(#655 , #657 , #660 )
5 years ago
5c27b25a8f
[deviantart] improve sta.sh extraction
...
Extract all sta.sh items in a single extractor run.
Don't spawn a new StashExtractor for each individual sta.sh item to
preserve the current requests.Session and its opened TCP connections.
5 years ago
e2fc4eaa6f
[deviantart] detect stash folders ( fixes #659 )
5 years ago
c034159701
[piczel] fix extraction for single images
5 years ago
699036ea0c
[weibo] accept status URLs with non-numeric IDs ( #664 )
5 years ago
a0111edfd2
change Travis badge URL to .com
5 years ago
fe96f99e4b
[hentainexus] reduce line length (flake8) & update test
5 years ago
6f81cac8fa
Add metadata to hentainexus: circle, event, title_conventional. ( #661 )
5 years ago
3ed72f82dc
release version 1.13.3
5 years ago
6f911aeb1c
[deviantart] add error message for cloudFront blocks ( #655 )
5 years ago
7499d71d02
[simplyhentai] ignore certificate errors in video test
5 years ago
4203dc0bdc
[mangapark] fix metadata extraction
5 years ago
6ecb0a19cf
handle sys.stdin being None when using '-' as input file ( #653 )
5 years ago
c8db2a87e9
fix create_test_data.py script
5 years ago
1b82d36ab2
[deviantart] handle decode errors for extended_fetch results ( #655 )
...
This isn't going to solve the underlying problem, but it should at
least provide the server response when those errors happen.
5 years ago
88ebbadc58
remove dashes from subcategory names in supportedsites.rst
5 years ago
09f2271528
[35photo] add 'tag' extractor
5 years ago
77fda8190c
[35photo] simplify/remove tests for the 'genre' extractor
...
There is still a nice genre overview page (https://35photo.pro/genre/ )
but the individual sub-pages don't list photos anymore
5 years ago
4bc161ca0f
prevent crash when sys.stdout and co. are None ( #653 )
5 years ago
d47d0f757c
[travis] allow 'results' and 'snap' tests to fail
5 years ago
ce73796eaa
[travis] add flake8 job
5 years ago
fb846c9ee5
[instagram] reduce line lengths and make flake8 happy
5 years ago
ad2efa8509
[e621] derive from Danbooru extractors ( #651 )
...
- use extractor implementations from 'danbooru'
- use "page": "b[ID]" to paginate over results instead of
"tags": "id:<[ID]", avoiding infinite loops with certain
post orders
- bump User-Agent version
5 years ago
9b39e1cd7e
[e621] fix bug in API rate limiting ( #651 )
5 years ago
b607d0ad7f
[twitter] fix typo in 'x-twitter-auth-type' header ( #625 )
5 years ago
9159cb8fb3
remove trailing dots and spaces from directory names ( #647 )
5 years ago
2c3b9e1450
[nozomi] support multiple images per post ( #646 )
...
This changes the default filename format as well as archive IDs,
since those assumed that each post would only have one image.
5 years ago
c606d0c854
[instagram] update pattern for user profile URLs
...
Allow for query parameters and fragments,
for example https://www.instagram.com/instagram/?hl=en
5 years ago
2530db3f4d
[mangadex] transform 'date' timestamps to datetime objects
5 years ago
ae2a33243b
[newgrounds] catch general Exceptions
5 years ago
32e36d8f02
[sexcom] replace tests
5 years ago
33b42dc847
[nozomi] sort search results ( fixes #646 )
5 years ago
eaa60a438b
[piczel] fix extraction
...
- manually filter by folder_id
- extract data for single posts from embedded JSON, since the
'/api/gallery/image/<id>' endpoint is no longer available
5 years ago
5bcc7184c9
[danbooru][e621] increase page limits
5 years ago
90d15e3682
[instagram] use 'itertools.chain()'
5 years ago
160328d21c
[instagram] Add support for user's saved medias ( #644 )
...
* [instagram] Gracefully handle possible 'HttpErrorPage' in _extract_page()
`HttpErrorPage' is returned in shared_data at least when not authenticated or
when trying to fetch other users saved medias
(i.e. `instagram.com/<user>/saved/').
Gracefully handle it by returning nothing.
* [instagram] Add support for user's saved medias
(Please note that this need the user to be authenticated and they can
only see their saved media (not other users ones).)
Close #643 .
* [instagram] Bump copyright year
5 years ago
Mike Fährmann