e0b0e8d62a
release version 1.13.2
5 years ago
d3482ace7f
[furaffinity] extract more metadata
...
- views
- favorites
- comments
- rating
- fa_category (since 'category' is already in use)
- theme
- species
- gender
- width
- height
5 years ago
f6c5edb76b
pre-compile regex pattern for remove_html() and split_html()
5 years ago
fdd2dd5136
[kabeuchi] add 'user' extractor ( closes #561 )
5 years ago
59edcdc822
[hitomi] restore metadata fields from before f33b13a
...
... and add a 'metadata' option to disable
visiting the gallery page and extracting data from it
if this is not needed.
5 years ago
2d5703c493
[twitter] use a simpler data structure to store cookies in cache
...
Use a dict with name-value pairs instead of an entire
RequestsCookieJar object.
5 years ago
87d4f83597
[newgrounds] make post extraction nonfatal
5 years ago
823fbeaae6
[newgrounds] add 'favorite' extractor ( #394 )
5 years ago
a45fbc38ea
[pixiv] implement 'avatar' option ( #595 , #623 )
5 years ago
a63a376ad2
[mangoxo] fix login
5 years ago
ebc70e87ce
[e621] update to new interface / API endpoints ( closes #635 )
5 years ago
d1cf7ccdb3
[instagram] add 'post_shortcode' metadata field ( #525 )
5 years ago
32df8d06fe
[twitter] add 'bookmark' extractor ( closes #625 )
5 years ago
3fb41c34c8
[bcy] reduce requests to '/item/detail/<id>' ( #613 )
...
The former implementation would try to use the embedded data from
'/item/detail/' pages for every post, even if that wasn't really
necessary.
This commit also fixes some issues with posts only visible to
logged in users.
5 years ago
f33b13aacf
[hitomi] simplify metadata extraction
...
Use the data from https://ltn.hitomi.la/galleries/ <id>.js for both
image URLs and metadata and ignore any gallery or reader pages.
This removes 'artist', 'characters', 'group', and 'parody' metadata
fields since this information is, as for now, only available in
gallery pages.
5 years ago
115fd2c6f2
"fix" incomplete MIME types ( #632 )
...
e-/exhentai's original image downloads currently send
incomplete/invalid Content-Type headers, "jpg" instead
of "image/jpg" etc, since the last update.
(https://forums.e-hentai.org/index.php?showtopic=236113 )
This change prepends any Content-Type value missing a
media type specification with "image/", transforming it
into a valid MIME type.
(A global solution to a local problem, but it shouldn't
cause any issues anywhere else)
5 years ago
72122eb9b3
release version 1.13.1
5 years ago
adcd7cb24a
[downloader:http] add another MIME type for '.rar' files ( #628 )
5 years ago
ce5e2a58fe
[imgbb] update test results
...
Image server domain changed from
https://image.ibb.co/ to https://i.ibb.co/
5 years ago
f117e32910
[danbooru] restore 'popular' functionality
5 years ago
39b48d665b
[hiperdex] use proper name for 'chapter_minor'
5 years ago
8fbbaa54ff
[bcy] fix partial image URLs ( #613 )
...
Images from new posts can have incomplete/partial URLs (1)
without any filename extension when fetching their data from
'/apiv3/user/selfPosts', so now all data gets taken from
'/item/detail/ID' pages.
It is currently unknown how to get the non-watermarked original version
of these images, or if that is possible at all. (2)
Images with a watermark will have their 'filter' metadata field set to
"watermark". For original images this field is an empty string "".
Enabling the 'noop' option will, in addition to the watermarked version,
yield the the '~noop.image' filter version (3),
where 'filter' is set to "noop".
(1) "https://img-bcy-qn.pstatp.com/banciyuan/3ccdff22479c4060aadc86718209b281 "
(2) "https://p1-bcy.byteimg.com/img/banciyuan/3ccdff22479c4060aadc86718209b281~tplv-banciyuan-logo-v3:wqnpnLLlhZLlpKfprZTnjotfCuWNiuasoeWFgyAtIEFDR-eIseWlveiAheekvuWMug==.image "
(3) "https://p1-bcy.byteimg.com/img/banciyuan/3ccdff22479c4060aadc86718209b281~noop.image "
5 years ago
86c00f9e66
[danbooru] move extractor logic from booru.py
5 years ago
1d4a369ea2
update extractor test results
5 years ago
7625912b31
[piczel] improve and update
...
- fix tag names
- fix a bug in _pagination()
- parse datetime in 'created_at' as 'date'
- rewrite main loop
- replace user profile test
5 years ago
ec85bf90de
use context managers in cache.py & add tests
5 years ago
913b8333cc
write DeviantArt refresh-tokens to cache ( #616 )
...
Writing the token is currently disabled by default and must be
enabled with 'extractor.oauth.cache'.
'extractor.deviantart.refresh-token' must be set to '"cache"'
to use the cached token.
5 years ago
2a4f227e08
warn about expired cookies
5 years ago
34887ae139
fix bugs in DatabaseCacheDecorator.update()/.invalidate()
...
- call db.commit() after changes have been made
- remove 'LIMIT 1' from the DELETE statement in invalidate()
(only available if SQLite3 was compiled with the right flags
enabled, syntax error otherwise)
5 years ago
380b693fad
[downloader:http] add more MIME types for '.bmp' files ( #621 )
5 years ago
4e361b3008
add tests for specific datetime values
5 years ago
80ecb99089
[hitomi] fix extraction
5 years ago
247c9e1416
[vsco] update gallery URL pattern
5 years ago
19ae6f3fc4
update test results
...
- twitter:
Don't test the whole kwdict, only the actual content, since the
keyword hash changes whenever that user changes his display name.
- khinsider:
Download host changed
5 years ago
cc5079c844
[hiperdex] add chapter and manga extractors ( closes #606 )
5 years ago
64bdec8430
[deviantart] check availability of intermediary URLs ( fixes #609 )
5 years ago
5607dd3646
[hitomi] follow multiple redirects
5 years ago
765b2a0527
[hentaihand] add extractors ( closes #605 )
5 years ago
d94215d119
[tumblr] replace '-' with ' ' in tag searches ( fixes #611 )
...
To search for tags with actual minus signs in them
(there shouldn't be too many,) manually replace those
with url-encoded minus characters ('-' -> '%2d')
before inputting them into gallery-dl:
https://s679874.tumblr.com/tagged/tag-with-minus
->
https://s679874.tumblr.com/tagged/tag%2dwith%2dminus
5 years ago
5cdf1b1319
fix --verbose/--quiet
...
caused by 383795b
5 years ago
78e8d33c97
release version 1.13.0
5 years ago
e6cd49e78b
update extractor test results
5 years ago
90e4c645ba
[formatter] allow multiple "special" format specifiers ( #595 )
...
It is now, for example, possible to specify multiple replacement
operations per format replacement field: {name:Ra/b/Rc/d/}
5 years ago
5d9437b398
[vsco] skip "invalid" entities
5 years ago
650f2b6d58
[furaffinity] accept sfw.furaffinity.net URLs ( closes #608 )
...
Just as an alias for regular URLs with no extra content filtering.
5 years ago
219c4cc78c
[formatter] allow for numeric list and string indices
5 years ago
7d1da614d9
[formatter] implement field name alternatives ( #525 )
...
The format string '{a|b|c}' will now try to use the value from 'a' and
fall back to 'b' and 'c' if accessing a field raises an exception or
if its value is None.
5 years ago
74e684e828
[twitter] change default value for 'videos' to 'true'
...
Every other 'videos' option defaulted to 'true', except Twitter.
5 years ago
c7cf9dd111
[furaffinity] support classic layout ( #284 )
5 years ago
138135c190
[furaffinity] add extractors ( #284 )
5 years ago
b9c574bd1d
[patreon] log skipped files ( #590 )
5 years ago
80ea9104b8
[8kun] adjust URL pattern
5 years ago
c76c8b765a
[cloudflare] unescape challenge URL
5 years ago
ce26070231
[pixiv] reduce calls to '/user/detail'
5 years ago
da0d5f6092
[oauth] add 'port' option ( #604 )
5 years ago
719b63d0ca
[bcy] add user and post extractors ( #592 )
5 years ago
6426e3efc7
[khinsider] fix and improve metadata extraction
5 years ago
4a3d2405de
[postprocessor:ugoira] small optimization
...
Use tuples instead of lists when extending the list of
command-line arguments.
5 years ago
b7eb6cecbb
[pixiv] handle tags at the end of new bookmark URLs
5 years ago
109f6c8685
[patreon] filter duplicate files per post ( #590 )
5 years ago
b38cf59711
[sexcom] fix image URLs & parse 'date' fields
5 years ago
1f4c9c5f9d
[8kun] add thread and board extractors ( closes #582 )
5 years ago
facc5daa6d
[twitter] force old login page layout ( fixes #584 , fixes #598 )
5 years ago
d1de7dc296
[hitomi] implement workaround for "broken" redirects
...
Some galleries redirect to a new "version" with different gallery id.
This new version might not be available any more, but the /reader/
page for the original gallery id can still work.
5 years ago
40fe062851
[pixiv] fix user id for bookmarks API calls ( closes #596 )
5 years ago
91aaaf1a9e
[pixiv] add 'rating' metadata field ( #595 )
...
A human-friendlier representation of 'x_restrict'
5 years ago
dff33b260c
[reddit] add 'videos' option
5 years ago
2ad43618cc
[piczel] fix extraction
5 years ago
cf7a67d67f
[yaplog] remove module
...
Yaplog! ended its service on 2020-01-31
5 years ago
e0dd073ce0
[twitter] replace embedded tweet test
...
the old one was deleted
5 years ago
ec36df4851
[deviantart] fix video extraction from 'extended_fetch' results
...
DeviantArt is now serving videos from wixmp servers (1), instead of
the former film00.deviantart.com (2), even though those URLS are still
functional.
They seem to also have re-encoded those videos. The 10 MB 1080p video
from (2) is now only available in 720p at ~20 MB (with a higher
bitrate, but still …). Other videos are still available in 1080p, but
not this one for some reason.
(Changing the '720p' in (1) to '1080p' doesn't work.)
(1) https://wixmp-ed30a86b8c4ca887773594c2.wixmp.com/v/mp4/9feaa2c9-1baf-4fc2-84f7-f3384b34cefe/d5gxnb5-282a2e9a-b552-40ff-8542-b3c5eed823f5.720p.a837d7cec12c41be8ca2ee53152cea3a.mp4
(2) https://film00.deviantart.net/4c1d/v/mp4/2012/279/d/1/_video____brushes_i_use_in_paint_tool_sai_by_chi_u-d5gxnb5.mp4
5 years ago
48be2266ed
[deviantart] better error message for 'extended_fetch' ( #585 )
5 years ago
383795b550
prevent superfluous calls to Logger.makeRecord()
...
… by setting an appropriate minimal logging level for the root Logger.
5 years ago
71851a6241
[pixiv] update URLs of followed users to the new format
5 years ago
d086f30b42
[reddit] restore archive keys for i.redd.it images
5 years ago
56f1c96168
implement 'parent-directory' option ( #551 )
5 years ago
ae07f92f7e
[reddit] rewrite extractor logic ( closes #551 )
...
Handle images and videos hosted on Reddit "natively",
allowing them to use reddit-specific metadata to build directory
and file names.
5 years ago
2852691d78
[paheal] replace test URL
...
searching for 'k-on' doesn't yield any results anymore
5 years ago
2a9be48511
improve util.load/save_cookiestxt() and add tests
...
- take a file object as argument instead of an filename
- accept whitespace before comments (" # comment")
- map expiration "0" to None and not the number 0
5 years ago
e35c2ea1a6
[weibo] use youtube-dl to download from m3u8 manifests
5 years ago
6703b8a86b
[blogger] implement video extraction ( closes #587 )
5 years ago
c1a6862863
implement functions to load/save cookies.txt files ( closes #586 )
...
The methods of the standard libraries' MozillaCookieJar have
several shortcomings (#HttpOnly_ cookies, 0 expiration timestamps, etc.)
and require construction of an ultimately pointless CookieJar object.
5 years ago
5d73b7f29c
release version 1.12.3
5 years ago
37247dbaff
miscellaneous fixes
5 years ago
0e9dc5c88e
fix AttributeError when accessing 'temppath'
...
[ci skip]
5 years ago
25d5ec4ff3
[twitter] add option to extract TwitPic embeds ( #579 )
5 years ago
254f7c3999
implement a post-processor module to compare file versions
...
(#530 )
5 years ago
32d7195d08
[pinterest] improve detection of invalid pin.it links
5 years ago
0b84068d84
remove temp files before downloading from fallback URLs
...
otherwise the next call to download() with a fallback URL could see
the partially downloaded "remains" from the previous, failed download
attempt and "continue" it, writing the second half of a potentially
different version of that file.
5 years ago
760b9b4db4
add remove_file() and remove_directory() helpers
...
these functions call os.unlink() or os.rmdir()
while catching and suppressing potential OSErrors
5 years ago
b2d542ad40
improve PathFormat._enum_file()
...
open only one try-except block for the whole loop,
instead of one for each iteration in os.path.exists()
5 years ago
174117f827
allow multiple hashes for content tests
5 years ago
f498a9057f
[twitter] Fix stop before real end ( #573 )
...
* [twitter] Fix stop before real end
Fix for https://github.com/mikf/gallery-dl/issues/544 . Makes sure that it really reached the end by checking that both "min_position" is null and "has_more_items" is false before stopping.
* [twitter] Fix stop before real end (update)
5 years ago
8bb32ee188
[hitomi] fix image URLs
5 years ago
bd5ce9855c
allow GalleryExtractors to set URL-independent extensions
5 years ago
af42c75152
[mangadex] revert domain to 'mangadex.org'
5 years ago
200aea308a
[downloader:common] enable 'job'/'extractor' for logging messages
...
(#574 )
5 years ago
e89413da22
update test results
5 years ago
33a6e0ac6e
[hentaifoundry] extract more metadata ( closes #565 )
5 years ago
5cac79c3d9
[erolord] remove extractor
5 years ago
b9cbf932b4
[pixiv] update URL patterns ( fixes #568 )
...
Pixiv now uses new URLs for
- user profiles and illustration listings:
- https://www.pixiv.net/en/users/ <ID>
- https://www.pixiv.net/en/users/ <ID>/artworks
- bookmarks:
- https://www.pixiv.net/en/users/ <ID>/bookmarks/artworks
5 years ago
9d369de592
release version 1.12.2
5 years ago
988cc2ec23
[mangadex] change domain to mangadex.cc ( closes #559 )
5 years ago
f8e137d6b4
[deviantart] show warning about private deviations only once
...
… per call to '_pagination()'
5 years ago
939fec8ecd
[deviantart] match new search/popular URLs ( closes #538 )
5 years ago
09cc88b715
[deviantart] match '/favourites/all' URLs ( closes #555 )
5 years ago
3811fd8a25
fix time formatting for Python 3.4 and 3.5
...
'datetime.time.isoformat()' only has an optional 'timespec' argument
since Python 3.6.
5 years ago
43ab9572b4
[twitter] handle API rate limits ( #526 )
5 years ago
569747a78d
implement extractor.wait()
5 years ago
5532e9c158
[twitter] handle quoted tweets ( #526 )
...
… and categorize them as retweets
5 years ago
0b4cb8e57a
[mangahere] send 'isAdult' cookie ( fixes #556 )
5 years ago
025f6e3398
add fallback for missing WITHOUT ROWID support ( #553 )
5 years ago
87c8b89ddd
[postprocessor:metadata] add 'directory' option ( #520 )
5 years ago
1afb91363c
[imagefap] generalize URL patterns and add tests ( #552 )
5 years ago
f701e9f33a
Handle beta.imagefap.com URLs ( #552 )
5 years ago
ce54b8c04c
let extractors opt-out of cookie option usage
...
useful to avoid sending unnecessary cookies when all authentication
is done through OAuth tokens
5 years ago
5ad92fc196
[newgrounds] fix tags metadata extraction
5 years ago
82f7f4172a
update test results
5 years ago
1f2a69f3c5
add '_extractor' information to redirect results
5 years ago
2d4887b75b
improve KeywordJob output for "parent" extractors ( closes #548 )
5 years ago
a27f43dad1
[pixiv] wait and retry after rate limit error ( closes #535 )
5 years ago
6b373cb7e2
[exhentai] restrict default directory name length ( #545 )
5 years ago
b347bf68c7
[deviantart] add extractor for followed users ( #515 )
5 years ago
c0f391a4e2
[pixiv] support listing followed users ( #515 )
5 years ago
2e2fc7f0ad
prevent infinite recursion when spawning extractors ( closes #489 )
5 years ago
896896a490
[twitter] fix URLs forwarded to youtube-dl ( closes #540 )
...
Since commit 3bba763
5 years ago
1e2713b895
[artstation] fix search result pagination ( closes #537 )
5 years ago
bf3df3d0b0
[directlink] send Referer headers ( closes #536 )
5 years ago
83909ab5d4
release version 1.12.1
5 years ago
9be7ff600e
[imagetwist] replace test image
...
the old one expired, it seems
5 years ago
66905b1664
[foolslide] add fallback for chapter data extraction
5 years ago
48e42e73fb
[reddit] change default value for 'comments' to '0'
5 years ago
9c0928457a
[reddit] fix errors with 't1_…' submissions
5 years ago
58391d492d
cache archive keys generated in __contains__() ( #524 )
...
To avoid writing a different key to the archive than what was checked
against before the file download.
5 years ago
bf658fd84b
[vsco] implement 'videos' option
5 years ago
95c90722ee
[instagram] implement 'videos' option ( closes #521 )
5 years ago
1921c127a5
make OSErrors during file downloads nonfatal ( closes #512 )
...
… except ENOSPC (No space left on device), since there is no reason to
continue downloading in that case.
All other errors that would prevent downloading data and writing it to
disk get already raised during directory creation and are therefore not
checked here.
5 years ago
d0920e84e9
update test results
5 years ago
8c11e81c9f
Merge commit '63e6993716db8d8bedfb7b0d445c7161493046b6'
5 years ago
63e6993716
merge 'bypost' functionality into metadata postprocessor
5 years ago
31a29835ff
[realbooru] simplify extractors and update tests ( #514 )
5 years ago
9a4ce20b8e
[realbooru] Add Realbooru extractor ( #514 )
5 years ago
f9e74320de
retain trailing zeroes in Cloudflare challenge answers
5 years ago
72b8fbfbad
[instagram] make post-page extraction nonfatal
5 years ago
922b8a9595
[weibo] raise NotFoundError for unavailable/deleted statuses
5 years ago
0cd157300e
[patreon] fix regex pattern for posts
...
The previous one would match the first number in the URL slug as
post ID, which would fail for posts with numbers in their title.
5 years ago
fe19e233f3
[xvideos] improve
...
- derive from GalleryExtractor
- match '…-channels' URLs
- "better" metadata structure
5 years ago
d3e44e899d
raise NotFoundErrors for 404 responses in GalleryExtractors
5 years ago
a4dd8b3dab
improve _check_cookies()
...
Only loop over all cookies once instead of calling
cookiejar._find() for each cookie name.
5 years ago
76e60d10a6
[patreon] raise proper exception if creator/post doesn't exist
5 years ago
Mike Fährmann