cf7a67d67f
[yaplog] remove module
...
Yaplog! ended its service on 2020-01-31
5 years ago
e0dd073ce0
[twitter] replace embedded tweet test
...
the old one was deleted
5 years ago
ec36df4851
[deviantart] fix video extraction from 'extended_fetch' results
...
DeviantArt is now serving videos from wixmp servers (1), instead of
the former film00.deviantart.com (2), even though those URLS are still
functional.
They seem to also have re-encoded those videos. The 10 MB 1080p video
from (2) is now only available in 720p at ~20 MB (with a higher
bitrate, but still …). Other videos are still available in 1080p, but
not this one for some reason.
(Changing the '720p' in (1) to '1080p' doesn't work.)
(1) https://wixmp-ed30a86b8c4ca887773594c2.wixmp.com/v/mp4/9feaa2c9-1baf-4fc2-84f7-f3384b34cefe/d5gxnb5-282a2e9a-b552-40ff-8542-b3c5eed823f5.720p.a837d7cec12c41be8ca2ee53152cea3a.mp4
(2) https://film00.deviantart.net/4c1d/v/mp4/2012/279/d/1/_video____brushes_i_use_in_paint_tool_sai_by_chi_u-d5gxnb5.mp4
5 years ago
48be2266ed
[deviantart] better error message for 'extended_fetch' ( #585 )
5 years ago
71851a6241
[pixiv] update URLs of followed users to the new format
5 years ago
d086f30b42
[reddit] restore archive keys for i.redd.it images
5 years ago
56f1c96168
implement 'parent-directory' option ( #551 )
5 years ago
ae07f92f7e
[reddit] rewrite extractor logic ( closes #551 )
...
Handle images and videos hosted on Reddit "natively",
allowing them to use reddit-specific metadata to build directory
and file names.
5 years ago
2852691d78
[paheal] replace test URL
...
searching for 'k-on' doesn't yield any results anymore
5 years ago
2a9be48511
improve util.load/save_cookiestxt() and add tests
...
- take a file object as argument instead of an filename
- accept whitespace before comments (" # comment")
- map expiration "0" to None and not the number 0
5 years ago
e35c2ea1a6
[weibo] use youtube-dl to download from m3u8 manifests
5 years ago
6703b8a86b
[blogger] implement video extraction ( closes #587 )
5 years ago
c1a6862863
implement functions to load/save cookies.txt files ( closes #586 )
...
The methods of the standard libraries' MozillaCookieJar have
several shortcomings (#HttpOnly_ cookies, 0 expiration timestamps, etc.)
and require construction of an ultimately pointless CookieJar object.
5 years ago
25d5ec4ff3
[twitter] add option to extract TwitPic embeds ( #579 )
5 years ago
32d7195d08
[pinterest] improve detection of invalid pin.it links
5 years ago
174117f827
allow multiple hashes for content tests
5 years ago
f498a9057f
[twitter] Fix stop before real end ( #573 )
...
* [twitter] Fix stop before real end
Fix for https://github.com/mikf/gallery-dl/issues/544 . Makes sure that it really reached the end by checking that both "min_position" is null and "has_more_items" is false before stopping.
* [twitter] Fix stop before real end (update)
5 years ago
8bb32ee188
[hitomi] fix image URLs
5 years ago
bd5ce9855c
allow GalleryExtractors to set URL-independent extensions
5 years ago
af42c75152
[mangadex] revert domain to 'mangadex.org'
5 years ago
e89413da22
update test results
5 years ago
33a6e0ac6e
[hentaifoundry] extract more metadata ( closes #565 )
5 years ago
5cac79c3d9
[erolord] remove extractor
5 years ago
b9cbf932b4
[pixiv] update URL patterns ( fixes #568 )
...
Pixiv now uses new URLs for
- user profiles and illustration listings:
- https://www.pixiv.net/en/users/ <ID>
- https://www.pixiv.net/en/users/ <ID>/artworks
- bookmarks:
- https://www.pixiv.net/en/users/ <ID>/bookmarks/artworks
5 years ago
988cc2ec23
[mangadex] change domain to mangadex.cc ( closes #559 )
5 years ago
f8e137d6b4
[deviantart] show warning about private deviations only once
...
… per call to '_pagination()'
5 years ago
939fec8ecd
[deviantart] match new search/popular URLs ( closes #538 )
5 years ago
09cc88b715
[deviantart] match '/favourites/all' URLs ( closes #555 )
5 years ago
3811fd8a25
fix time formatting for Python 3.4 and 3.5
...
'datetime.time.isoformat()' only has an optional 'timespec' argument
since Python 3.6.
5 years ago
43ab9572b4
[twitter] handle API rate limits ( #526 )
5 years ago
569747a78d
implement extractor.wait()
5 years ago
5532e9c158
[twitter] handle quoted tweets ( #526 )
...
… and categorize them as retweets
5 years ago
0b4cb8e57a
[mangahere] send 'isAdult' cookie ( fixes #556 )
5 years ago
1afb91363c
[imagefap] generalize URL patterns and add tests ( #552 )
5 years ago
f701e9f33a
Handle beta.imagefap.com URLs ( #552 )
5 years ago
ce54b8c04c
let extractors opt-out of cookie option usage
...
useful to avoid sending unnecessary cookies when all authentication
is done through OAuth tokens
5 years ago
5ad92fc196
[newgrounds] fix tags metadata extraction
5 years ago
82f7f4172a
update test results
5 years ago
1f2a69f3c5
add '_extractor' information to redirect results
5 years ago
a27f43dad1
[pixiv] wait and retry after rate limit error ( closes #535 )
5 years ago
6b373cb7e2
[exhentai] restrict default directory name length ( #545 )
5 years ago
b347bf68c7
[deviantart] add extractor for followed users ( #515 )
5 years ago
c0f391a4e2
[pixiv] support listing followed users ( #515 )
5 years ago
896896a490
[twitter] fix URLs forwarded to youtube-dl ( closes #540 )
...
Since commit 3bba763
5 years ago
1e2713b895
[artstation] fix search result pagination ( closes #537 )
5 years ago
bf3df3d0b0
[directlink] send Referer headers ( closes #536 )
5 years ago
9be7ff600e
[imagetwist] replace test image
...
the old one expired, it seems
5 years ago
66905b1664
[foolslide] add fallback for chapter data extraction
5 years ago
48e42e73fb
[reddit] change default value for 'comments' to '0'
5 years ago
9c0928457a
[reddit] fix errors with 't1_…' submissions
5 years ago
bf658fd84b
[vsco] implement 'videos' option
5 years ago
95c90722ee
[instagram] implement 'videos' option ( closes #521 )
5 years ago
d0920e84e9
update test results
5 years ago
8c11e81c9f
Merge commit '63e6993716db8d8bedfb7b0d445c7161493046b6'
5 years ago
63e6993716
merge 'bypost' functionality into metadata postprocessor
5 years ago
31a29835ff
[realbooru] simplify extractors and update tests ( #514 )
5 years ago
9a4ce20b8e
[realbooru] Add Realbooru extractor ( #514 )
5 years ago
72b8fbfbad
[instagram] make post-page extraction nonfatal
5 years ago
922b8a9595
[weibo] raise NotFoundError for unavailable/deleted statuses
5 years ago
0cd157300e
[patreon] fix regex pattern for posts
...
The previous one would match the first number in the URL slug as
post ID, which would fail for posts with numbers in their title.
5 years ago
fe19e233f3
[xvideos] improve
...
- derive from GalleryExtractor
- match '…-channels' URLs
- "better" metadata structure
5 years ago
d3e44e899d
raise NotFoundErrors for 404 responses in GalleryExtractors
5 years ago
a4dd8b3dab
improve _check_cookies()
...
Only loop over all cookies once instead of calling
cookiejar._find() for each cookie name.
5 years ago
76e60d10a6
[patreon] raise proper exception if creator/post doesn't exist
5 years ago
9e63804347
[patreon] make retrieving user info nonfatal ( #508 )
...
… and fall back to the included data if an error occurs.
5 years ago
964dc57286
[vsco] improve image resolutions
...
https://im.vsco.co/ URLs redirect to the appropriate CDN server
and occasionally insert a '/1200x1600/' into the image path,
limiting image dimensions.
This commit constructs redirect targets out of the given
im,vsco.co URLs without sending extra HTTP requests
and without any "builtin" resolution restrictions.
5 years ago
0629fe8fa4
[vsco] fix user profile extraction … again
...
Given the pattern from last time, collections will also change
in due time and use cursor-based pagination.
5 years ago
ab17ea9632
[deviantart] only print warning if 'original' is enabled
5 years ago
2188db6284
[gelbooru] fix non-API tag extraction
5 years ago
c4702ec9b6
simplify some logging calls
5 years ago
c0b9ad678d
Separate metadata from handle_url into handle_metadata, commenting
5 years ago
c9ef1b21c3
[patreon] get partial user info without /api/user/<id> ( #507 )
...
It's a lot less data, but doesn't invoke any additional
HTTP requests with potential Cloudflare CAPTCHAs.
5 years ago
0ab9bb1721
[4chan] add extractor for entire boards ( closes #510 )
5 years ago
cfc70a97ab
Added an additional channel for downloading the metadata of an entire post or gallery.
5 years ago
15f9bb3d14
add option to disable pyOpenSSL usage ( #508 )
...
(pyOpenSSL is now disabled by default)
5 years ago
c8e99e3b3b
[deviantart] fix crash on missing "token" field ( #505 )
5 years ago
6ed2c7823c
[deviantart] disable original downloads if no cookies set
...
For 'deviation' and 'scraps' extractors only, since original file
downloads for those two will always fail with a 404 Not Found
when not logged in.
5 years ago
50deab5265
[deviantart] fix URL generation from /extended_fetch results
...
(closes #505 )
5 years ago
1f209da4c0
[pixiv] match new search URLs ( closes #507 )
5 years ago
e17907ee2a
change default value of 'cookies-update' to 'true'
5 years ago
07dafad26d
[twitter] attempt to fix infinite loops ( #499 )
...
(Hopefully this doesn't break anything else)
5 years ago
71acbdabf4
[2chan] fix metadata extraction
5 years ago
c0a1241648
[livedoor] force https:// for image URLs
5 years ago
6e23c0da09
[imgur] add extractor for subreddit links ( closes #500 )
5 years ago
372ffe95ee
[oauth] adjust Flickr redirect URI ( fixes #503 )
...
Flickr now automatically forces https:// for all redirect URIs.
5 years ago
004812258d
[hentaifox] fix extraction
5 years ago
e2710702d4
fix Cloudflare bypss
5 years ago
8759403f37
[plurk] add delay between comment requests
5 years ago
a28552fd19
update test results
...
- hbrowse: one tag got removed
- mangoxo: gallery changed owner
- photobucket: ?, but photo still downloads
5 years ago
dcaa3d01bd
[imagefap] adapt to new image URL format
5 years ago
e62c209ca0
[nijie] fix 'date' parsing
5 years ago
3bba763ab9
[twitter] improve
...
- update metadata structure
- combine all user… entries into their own dict
- let 'user' always specify the Timeline owner
- add 'author' entry that specifies the original Tweet author
- create directories per post (closes #491 )
- fix username issues with /i/web/ URLs
5 years ago
db35c3b581
[directlink] separate filenames from paths
...
With this, all default filename formats specify an '{extension}'
and PathFormat.set_extension() reliably works for all files.
5 years ago
41a3169c67
[foolfuuka] use '{extension}' in default filename format
5 years ago
e9aed62c91
[imgur] unescape image titles
5 years ago
2c332edaad
[plurk] fix comment pagination
5 years ago
a3fa45bbb1
[behance] get images from 'media_collection' modules
5 years ago
359c3bc1c5
[deviantart] revert to getting download URLs from OAuth API
...
This commit (partially) reverts 27b5b2494eb7c6a437e78#488 )
TODO:
- "native" login with --username and --password
- better handling of internally stored cookies
5 years ago
42b9633c7e
update test results
5 years ago
b28bd1c73e
[bobx] set generated session cookie ( closes #482 )
...
This reverts commit 490831f
5 years ago
Mike Fährmann