d1cf7ccdb3
[instagram] add 'post_shortcode' metadata field ( #525 )
5 years ago
32df8d06fe
[twitter] add 'bookmark' extractor ( closes #625 )
5 years ago
3fb41c34c8
[bcy] reduce requests to '/item/detail/<id>' ( #613 )
...
The former implementation would try to use the embedded data from
'/item/detail/' pages for every post, even if that wasn't really
necessary.
This commit also fixes some issues with posts only visible to
logged in users.
5 years ago
f33b13aacf
[hitomi] simplify metadata extraction
...
Use the data from https://ltn.hitomi.la/galleries/ <id>.js for both
image URLs and metadata and ignore any gallery or reader pages.
This removes 'artist', 'characters', 'group', and 'parody' metadata
fields since this information is, as for now, only available in
gallery pages.
5 years ago
115fd2c6f2
"fix" incomplete MIME types ( #632 )
...
e-/exhentai's original image downloads currently send
incomplete/invalid Content-Type headers, "jpg" instead
of "image/jpg" etc, since the last update.
(https://forums.e-hentai.org/index.php?showtopic=236113 )
This change prepends any Content-Type value missing a
media type specification with "image/", transforming it
into a valid MIME type.
(A global solution to a local problem, but it shouldn't
cause any issues anywhere else)
5 years ago
72122eb9b3
release version 1.13.1
5 years ago
adcd7cb24a
[downloader:http] add another MIME type for '.rar' files ( #628 )
5 years ago
ce5e2a58fe
[imgbb] update test results
...
Image server domain changed from
https://image.ibb.co/ to https://i.ibb.co/
5 years ago
f117e32910
[danbooru] restore 'popular' functionality
5 years ago
39b48d665b
[hiperdex] use proper name for 'chapter_minor'
5 years ago
8fbbaa54ff
[bcy] fix partial image URLs ( #613 )
...
Images from new posts can have incomplete/partial URLs (1)
without any filename extension when fetching their data from
'/apiv3/user/selfPosts', so now all data gets taken from
'/item/detail/ID' pages.
It is currently unknown how to get the non-watermarked original version
of these images, or if that is possible at all. (2)
Images with a watermark will have their 'filter' metadata field set to
"watermark". For original images this field is an empty string "".
Enabling the 'noop' option will, in addition to the watermarked version,
yield the the '~noop.image' filter version (3),
where 'filter' is set to "noop".
(1) "https://img-bcy-qn.pstatp.com/banciyuan/3ccdff22479c4060aadc86718209b281 "
(2) "https://p1-bcy.byteimg.com/img/banciyuan/3ccdff22479c4060aadc86718209b281~tplv-banciyuan-logo-v3:wqnpnLLlhZLlpKfprZTnjotfCuWNiuasoeWFgyAtIEFDR-eIseWlveiAheekvuWMug==.image "
(3) "https://p1-bcy.byteimg.com/img/banciyuan/3ccdff22479c4060aadc86718209b281~noop.image "
5 years ago
86c00f9e66
[danbooru] move extractor logic from booru.py
5 years ago
1d4a369ea2
update extractor test results
5 years ago
7625912b31
[piczel] improve and update
...
- fix tag names
- fix a bug in _pagination()
- parse datetime in 'created_at' as 'date'
- rewrite main loop
- replace user profile test
5 years ago
ec85bf90de
use context managers in cache.py & add tests
5 years ago
913b8333cc
write DeviantArt refresh-tokens to cache ( #616 )
...
Writing the token is currently disabled by default and must be
enabled with 'extractor.oauth.cache'.
'extractor.deviantart.refresh-token' must be set to '"cache"'
to use the cached token.
5 years ago
2a4f227e08
warn about expired cookies
5 years ago
34887ae139
fix bugs in DatabaseCacheDecorator.update()/.invalidate()
...
- call db.commit() after changes have been made
- remove 'LIMIT 1' from the DELETE statement in invalidate()
(only available if SQLite3 was compiled with the right flags
enabled, syntax error otherwise)
5 years ago
380b693fad
[downloader:http] add more MIME types for '.bmp' files ( #621 )
5 years ago
4e361b3008
add tests for specific datetime values
5 years ago
80ecb99089
[hitomi] fix extraction
5 years ago
247c9e1416
[vsco] update gallery URL pattern
5 years ago
19ae6f3fc4
update test results
...
- twitter:
Don't test the whole kwdict, only the actual content, since the
keyword hash changes whenever that user changes his display name.
- khinsider:
Download host changed
5 years ago
cc5079c844
[hiperdex] add chapter and manga extractors ( closes #606 )
5 years ago
64bdec8430
[deviantart] check availability of intermediary URLs ( fixes #609 )
5 years ago
5607dd3646
[hitomi] follow multiple redirects
5 years ago
765b2a0527
[hentaihand] add extractors ( closes #605 )
5 years ago
d94215d119
[tumblr] replace '-' with ' ' in tag searches ( fixes #611 )
...
To search for tags with actual minus signs in them
(there shouldn't be too many,) manually replace those
with url-encoded minus characters ('-' -> '%2d')
before inputting them into gallery-dl:
https://s679874.tumblr.com/tagged/tag-with-minus
->
https://s679874.tumblr.com/tagged/tag%2dwith%2dminus
5 years ago
5cdf1b1319
fix --verbose/--quiet
...
caused by 383795b
5 years ago
78e8d33c97
release version 1.13.0
5 years ago
e6cd49e78b
update extractor test results
5 years ago
90e4c645ba
[formatter] allow multiple "special" format specifiers ( #595 )
...
It is now, for example, possible to specify multiple replacement
operations per format replacement field: {name:Ra/b/Rc/d/}
5 years ago
5d9437b398
[vsco] skip "invalid" entities
5 years ago
650f2b6d58
[furaffinity] accept sfw.furaffinity.net URLs ( closes #608 )
...
Just as an alias for regular URLs with no extra content filtering.
5 years ago
219c4cc78c
[formatter] allow for numeric list and string indices
5 years ago
7d1da614d9
[formatter] implement field name alternatives ( #525 )
...
The format string '{a|b|c}' will now try to use the value from 'a' and
fall back to 'b' and 'c' if accessing a field raises an exception or
if its value is None.
5 years ago
74e684e828
[twitter] change default value for 'videos' to 'true'
...
Every other 'videos' option defaulted to 'true', except Twitter.
5 years ago
c7cf9dd111
[furaffinity] support classic layout ( #284 )
5 years ago
138135c190
[furaffinity] add extractors ( #284 )
5 years ago
b9c574bd1d
[patreon] log skipped files ( #590 )
5 years ago
80ea9104b8
[8kun] adjust URL pattern
5 years ago
c76c8b765a
[cloudflare] unescape challenge URL
5 years ago
ce26070231
[pixiv] reduce calls to '/user/detail'
5 years ago
da0d5f6092
[oauth] add 'port' option ( #604 )
5 years ago
719b63d0ca
[bcy] add user and post extractors ( #592 )
5 years ago
6426e3efc7
[khinsider] fix and improve metadata extraction
5 years ago
4a3d2405de
[postprocessor:ugoira] small optimization
...
Use tuples instead of lists when extending the list of
command-line arguments.
5 years ago
b7eb6cecbb
[pixiv] handle tags at the end of new bookmark URLs
5 years ago
109f6c8685
[patreon] filter duplicate files per post ( #590 )
5 years ago
b38cf59711
[sexcom] fix image URLs & parse 'date' fields
5 years ago
1f4c9c5f9d
[8kun] add thread and board extractors ( closes #582 )
5 years ago
facc5daa6d
[twitter] force old login page layout ( fixes #584 , fixes #598 )
5 years ago
d1de7dc296
[hitomi] implement workaround for "broken" redirects
...
Some galleries redirect to a new "version" with different gallery id.
This new version might not be available any more, but the /reader/
page for the original gallery id can still work.
5 years ago
40fe062851
[pixiv] fix user id for bookmarks API calls ( closes #596 )
5 years ago
91aaaf1a9e
[pixiv] add 'rating' metadata field ( #595 )
...
A human-friendlier representation of 'x_restrict'
5 years ago
dff33b260c
[reddit] add 'videos' option
5 years ago
2ad43618cc
[piczel] fix extraction
5 years ago
cf7a67d67f
[yaplog] remove module
...
Yaplog! ended its service on 2020-01-31
5 years ago
e0dd073ce0
[twitter] replace embedded tweet test
...
the old one was deleted
5 years ago
ec36df4851
[deviantart] fix video extraction from 'extended_fetch' results
...
DeviantArt is now serving videos from wixmp servers (1), instead of
the former film00.deviantart.com (2), even though those URLS are still
functional.
They seem to also have re-encoded those videos. The 10 MB 1080p video
from (2) is now only available in 720p at ~20 MB (with a higher
bitrate, but still …). Other videos are still available in 1080p, but
not this one for some reason.
(Changing the '720p' in (1) to '1080p' doesn't work.)
(1) https://wixmp-ed30a86b8c4ca887773594c2.wixmp.com/v/mp4/9feaa2c9-1baf-4fc2-84f7-f3384b34cefe/d5gxnb5-282a2e9a-b552-40ff-8542-b3c5eed823f5.720p.a837d7cec12c41be8ca2ee53152cea3a.mp4
(2) https://film00.deviantart.net/4c1d/v/mp4/2012/279/d/1/_video____brushes_i_use_in_paint_tool_sai_by_chi_u-d5gxnb5.mp4
5 years ago
48be2266ed
[deviantart] better error message for 'extended_fetch' ( #585 )
5 years ago
383795b550
prevent superfluous calls to Logger.makeRecord()
...
… by setting an appropriate minimal logging level for the root Logger.
5 years ago
71851a6241
[pixiv] update URLs of followed users to the new format
5 years ago
d086f30b42
[reddit] restore archive keys for i.redd.it images
5 years ago
56f1c96168
implement 'parent-directory' option ( #551 )
5 years ago
ae07f92f7e
[reddit] rewrite extractor logic ( closes #551 )
...
Handle images and videos hosted on Reddit "natively",
allowing them to use reddit-specific metadata to build directory
and file names.
5 years ago
2852691d78
[paheal] replace test URL
...
searching for 'k-on' doesn't yield any results anymore
5 years ago
2a9be48511
improve util.load/save_cookiestxt() and add tests
...
- take a file object as argument instead of an filename
- accept whitespace before comments (" # comment")
- map expiration "0" to None and not the number 0
5 years ago
e35c2ea1a6
[weibo] use youtube-dl to download from m3u8 manifests
5 years ago
6703b8a86b
[blogger] implement video extraction ( closes #587 )
5 years ago
c1a6862863
implement functions to load/save cookies.txt files ( closes #586 )
...
The methods of the standard libraries' MozillaCookieJar have
several shortcomings (#HttpOnly_ cookies, 0 expiration timestamps, etc.)
and require construction of an ultimately pointless CookieJar object.
5 years ago
5d73b7f29c
release version 1.12.3
5 years ago
37247dbaff
miscellaneous fixes
5 years ago
0e9dc5c88e
fix AttributeError when accessing 'temppath'
...
[ci skip]
5 years ago
25d5ec4ff3
[twitter] add option to extract TwitPic embeds ( #579 )
5 years ago
254f7c3999
implement a post-processor module to compare file versions
...
(#530 )
5 years ago
32d7195d08
[pinterest] improve detection of invalid pin.it links
5 years ago
0b84068d84
remove temp files before downloading from fallback URLs
...
otherwise the next call to download() with a fallback URL could see
the partially downloaded "remains" from the previous, failed download
attempt and "continue" it, writing the second half of a potentially
different version of that file.
5 years ago
760b9b4db4
add remove_file() and remove_directory() helpers
...
these functions call os.unlink() or os.rmdir()
while catching and suppressing potential OSErrors
5 years ago
b2d542ad40
improve PathFormat._enum_file()
...
open only one try-except block for the whole loop,
instead of one for each iteration in os.path.exists()
5 years ago
174117f827
allow multiple hashes for content tests
5 years ago
f498a9057f
[twitter] Fix stop before real end ( #573 )
...
* [twitter] Fix stop before real end
Fix for https://github.com/mikf/gallery-dl/issues/544 . Makes sure that it really reached the end by checking that both "min_position" is null and "has_more_items" is false before stopping.
* [twitter] Fix stop before real end (update)
5 years ago
8bb32ee188
[hitomi] fix image URLs
5 years ago
bd5ce9855c
allow GalleryExtractors to set URL-independent extensions
5 years ago
af42c75152
[mangadex] revert domain to 'mangadex.org'
5 years ago
200aea308a
[downloader:common] enable 'job'/'extractor' for logging messages
...
(#574 )
5 years ago
e89413da22
update test results
5 years ago
33a6e0ac6e
[hentaifoundry] extract more metadata ( closes #565 )
5 years ago
5cac79c3d9
[erolord] remove extractor
5 years ago
b9cbf932b4
[pixiv] update URL patterns ( fixes #568 )
...
Pixiv now uses new URLs for
- user profiles and illustration listings:
- https://www.pixiv.net/en/users/ <ID>
- https://www.pixiv.net/en/users/ <ID>/artworks
- bookmarks:
- https://www.pixiv.net/en/users/ <ID>/bookmarks/artworks
5 years ago
9d369de592
release version 1.12.2
5 years ago
988cc2ec23
[mangadex] change domain to mangadex.cc ( closes #559 )
5 years ago
f8e137d6b4
[deviantart] show warning about private deviations only once
...
… per call to '_pagination()'
5 years ago
939fec8ecd
[deviantart] match new search/popular URLs ( closes #538 )
5 years ago
09cc88b715
[deviantart] match '/favourites/all' URLs ( closes #555 )
5 years ago
3811fd8a25
fix time formatting for Python 3.4 and 3.5
...
'datetime.time.isoformat()' only has an optional 'timespec' argument
since Python 3.6.
5 years ago
43ab9572b4
[twitter] handle API rate limits ( #526 )
5 years ago
569747a78d
implement extractor.wait()
5 years ago
5532e9c158
[twitter] handle quoted tweets ( #526 )
...
… and categorize them as retweets
5 years ago
0b4cb8e57a
[mangahere] send 'isAdult' cookie ( fixes #556 )
5 years ago
Mike Fährmann