Mike Fährmann
56f1c96168
implement 'parent-directory' option ( #551 )
5 years ago
Mike Fährmann
ae07f92f7e
[reddit] rewrite extractor logic ( closes #551 )
...
Handle images and videos hosted on Reddit "natively",
allowing them to use reddit-specific metadata to build directory
and file names.
5 years ago
Mike Fährmann
2852691d78
[paheal] replace test URL
...
searching for 'k-on' doesn't yield any results anymore
5 years ago
Mike Fährmann
2a9be48511
improve util.load/save_cookiestxt() and add tests
...
- take a file object as argument instead of an filename
- accept whitespace before comments (" # comment")
- map expiration "0" to None and not the number 0
5 years ago
Mike Fährmann
e35c2ea1a6
[weibo] use youtube-dl to download from m3u8 manifests
5 years ago
Mike Fährmann
6703b8a86b
[blogger] implement video extraction ( closes #587 )
5 years ago
Mike Fährmann
c1a6862863
implement functions to load/save cookies.txt files ( closes #586 )
...
The methods of the standard libraries' MozillaCookieJar have
several shortcomings (#HttpOnly_ cookies, 0 expiration timestamps, etc.)
and require construction of an ultimately pointless CookieJar object.
5 years ago
Mike Fährmann
5d73b7f29c
release version 1.12.3
5 years ago
Mike Fährmann
37247dbaff
miscellaneous fixes
5 years ago
Mike Fährmann
0e9dc5c88e
fix AttributeError when accessing 'temppath'
...
[ci skip]
5 years ago
Mike Fährmann
25d5ec4ff3
[twitter] add option to extract TwitPic embeds ( #579 )
5 years ago
Mike Fährmann
254f7c3999
implement a post-processor module to compare file versions
...
(#530 )
5 years ago
Mike Fährmann
32d7195d08
[pinterest] improve detection of invalid pin.it links
5 years ago
Mike Fährmann
0b84068d84
remove temp files before downloading from fallback URLs
...
otherwise the next call to download() with a fallback URL could see
the partially downloaded "remains" from the previous, failed download
attempt and "continue" it, writing the second half of a potentially
different version of that file.
5 years ago
Mike Fährmann
760b9b4db4
add remove_file() and remove_directory() helpers
...
these functions call os.unlink() or os.rmdir()
while catching and suppressing potential OSErrors
5 years ago
Mike Fährmann
b2d542ad40
improve PathFormat._enum_file()
...
open only one try-except block for the whole loop,
instead of one for each iteration in os.path.exists()
5 years ago
Mike Fährmann
174117f827
allow multiple hashes for content tests
5 years ago
Alice
f498a9057f
[twitter] Fix stop before real end ( #573 )
...
* [twitter] Fix stop before real end
Fix for https://github.com/mikf/gallery-dl/issues/544 . Makes sure that it really reached the end by checking that both "min_position" is null and "has_more_items" is false before stopping.
* [twitter] Fix stop before real end (update)
5 years ago
Mike Fährmann
8bb32ee188
[hitomi] fix image URLs
5 years ago
Mike Fährmann
bd5ce9855c
allow GalleryExtractors to set URL-independent extensions
5 years ago
Mike Fährmann
af42c75152
[mangadex] revert domain to 'mangadex.org'
5 years ago
Mike Fährmann
200aea308a
[downloader:common] enable 'job'/'extractor' for logging messages
...
(#574 )
5 years ago
Mike Fährmann
e89413da22
update test results
5 years ago
Mike Fährmann
33a6e0ac6e
[hentaifoundry] extract more metadata ( closes #565 )
5 years ago
Mike Fährmann
5cac79c3d9
[erolord] remove extractor
5 years ago
Mike Fährmann
b9cbf932b4
[pixiv] update URL patterns ( fixes #568 )
...
Pixiv now uses new URLs for
- user profiles and illustration listings:
- https://www.pixiv.net/en/users/ <ID>
- https://www.pixiv.net/en/users/ <ID>/artworks
- bookmarks:
- https://www.pixiv.net/en/users/ <ID>/bookmarks/artworks
5 years ago
Mike Fährmann
9d369de592
release version 1.12.2
5 years ago
Mike Fährmann
988cc2ec23
[mangadex] change domain to mangadex.cc ( closes #559 )
5 years ago
Mike Fährmann
f8e137d6b4
[deviantart] show warning about private deviations only once
...
… per call to '_pagination()'
5 years ago
Mike Fährmann
939fec8ecd
[deviantart] match new search/popular URLs ( closes #538 )
5 years ago
Mike Fährmann
09cc88b715
[deviantart] match '/favourites/all' URLs ( closes #555 )
5 years ago
Mike Fährmann
3811fd8a25
fix time formatting for Python 3.4 and 3.5
...
'datetime.time.isoformat()' only has an optional 'timespec' argument
since Python 3.6.
5 years ago
Mike Fährmann
43ab9572b4
[twitter] handle API rate limits ( #526 )
5 years ago
Mike Fährmann
569747a78d
implement extractor.wait()
5 years ago
Mike Fährmann
5532e9c158
[twitter] handle quoted tweets ( #526 )
...
… and categorize them as retweets
5 years ago
Mike Fährmann
0b4cb8e57a
[mangahere] send 'isAdult' cookie ( fixes #556 )
5 years ago
Mike Fährmann
025f6e3398
add fallback for missing WITHOUT ROWID support ( #553 )
5 years ago
Mike Fährmann
87c8b89ddd
[postprocessor:metadata] add 'directory' option ( #520 )
5 years ago
Mike Fährmann
1afb91363c
[imagefap] generalize URL patterns and add tests ( #552 )
5 years ago
Xope Totec
f701e9f33a
Handle beta.imagefap.com URLs ( #552 )
5 years ago
Mike Fährmann
ce54b8c04c
let extractors opt-out of cookie option usage
...
useful to avoid sending unnecessary cookies when all authentication
is done through OAuth tokens
5 years ago
Mike Fährmann
5ad92fc196
[newgrounds] fix tags metadata extraction
5 years ago
Mike Fährmann
82f7f4172a
update test results
5 years ago
Mike Fährmann
1f2a69f3c5
add '_extractor' information to redirect results
5 years ago
Mike Fährmann
2d4887b75b
improve KeywordJob output for "parent" extractors ( closes #548 )
5 years ago
Mike Fährmann
a27f43dad1
[pixiv] wait and retry after rate limit error ( closes #535 )
5 years ago
Mike Fährmann
6b373cb7e2
[exhentai] restrict default directory name length ( #545 )
5 years ago
Mike Fährmann
b347bf68c7
[deviantart] add extractor for followed users ( #515 )
5 years ago
Mike Fährmann
c0f391a4e2
[pixiv] support listing followed users ( #515 )
5 years ago
Mike Fährmann
2e2fc7f0ad
prevent infinite recursion when spawning extractors ( closes #489 )
5 years ago
Mike Fährmann
896896a490
[twitter] fix URLs forwarded to youtube-dl ( closes #540 )
...
Since commit 3bba763
data["user"] is an entire dict object
and no longer just the user nickname …
5 years ago
Mike Fährmann
1e2713b895
[artstation] fix search result pagination ( closes #537 )
5 years ago
Mike Fährmann
bf3df3d0b0
[directlink] send Referer headers ( closes #536 )
5 years ago
Mike Fährmann
83909ab5d4
release version 1.12.1
5 years ago
Mike Fährmann
9be7ff600e
[imagetwist] replace test image
...
the old one expired, it seems
5 years ago
Mike Fährmann
66905b1664
[foolslide] add fallback for chapter data extraction
5 years ago
Mike Fährmann
48e42e73fb
[reddit] change default value for 'comments' to '0'
5 years ago
Mike Fährmann
9c0928457a
[reddit] fix errors with 't1_…' submissions
5 years ago
Mike Fährmann
58391d492d
cache archive keys generated in __contains__() ( #524 )
...
To avoid writing a different key to the archive than what was checked
against before the file download.
5 years ago
Mike Fährmann
bf658fd84b
[vsco] implement 'videos' option
5 years ago
Mike Fährmann
95c90722ee
[instagram] implement 'videos' option ( closes #521 )
5 years ago
Mike Fährmann
1921c127a5
make OSErrors during file downloads nonfatal ( closes #512 )
...
… except ENOSPC (No space left on device), since there is no reason to
continue downloading in that case.
All other errors that would prevent downloading data and writing it to
disk get already raised during directory creation and are therefore not
checked here.
5 years ago
Mike Fährmann
d0920e84e9
update test results
5 years ago
Mike Fährmann
8c11e81c9f
Merge commit '63e6993716db8d8bedfb7b0d445c7161493046b6'
5 years ago
Mike Fährmann
63e6993716
merge 'bypost' functionality into metadata postprocessor
5 years ago
Mike Fährmann
31a29835ff
[realbooru] simplify extractors and update tests ( #514 )
5 years ago
The Oddball
9a4ce20b8e
[realbooru] Add Realbooru extractor ( #514 )
5 years ago
Mike Fährmann
f9e74320de
retain trailing zeroes in Cloudflare challenge answers
5 years ago
Mike Fährmann
72b8fbfbad
[instagram] make post-page extraction nonfatal
5 years ago
Mike Fährmann
922b8a9595
[weibo] raise NotFoundError for unavailable/deleted statuses
5 years ago
Mike Fährmann
0cd157300e
[patreon] fix regex pattern for posts
...
The previous one would match the first number in the URL slug as
post ID, which would fail for posts with numbers in their title.
5 years ago
Mike Fährmann
fe19e233f3
[xvideos] improve
...
- derive from GalleryExtractor
- match '…-channels' URLs
- "better" metadata structure
5 years ago
Mike Fährmann
d3e44e899d
raise NotFoundErrors for 404 responses in GalleryExtractors
5 years ago
Mike Fährmann
a4dd8b3dab
improve _check_cookies()
...
Only loop over all cookies once instead of calling
cookiejar._find() for each cookie name.
5 years ago
Mike Fährmann
76e60d10a6
[patreon] raise proper exception if creator/post doesn't exist
5 years ago
Mike Fährmann
9e63804347
[patreon] make retrieving user info nonfatal ( #508 )
...
… and fall back to the included data if an error occurs.
5 years ago
Mike Fährmann
964dc57286
[vsco] improve image resolutions
...
https://im.vsco.co/ URLs redirect to the appropriate CDN server
and occasionally insert a '/1200x1600/' into the image path,
limiting image dimensions.
This commit constructs redirect targets out of the given
im,vsco.co URLs without sending extra HTTP requests
and without any "builtin" resolution restrictions.
5 years ago
Mike Fährmann
0629fe8fa4
[vsco] fix user profile extraction … again
...
Given the pattern from last time, collections will also change
in due time and use cursor-based pagination.
5 years ago
Mike Fährmann
ab17ea9632
[deviantart] only print warning if 'original' is enabled
5 years ago
Mike Fährmann
2188db6284
[gelbooru] fix non-API tag extraction
5 years ago
Mike Fährmann
c4702ec9b6
simplify some logging calls
5 years ago
Gio
c0b9ad678d
Separate metadata from handle_url into handle_metadata, commenting
5 years ago
Mike Fährmann
c9ef1b21c3
[patreon] get partial user info without /api/user/<id> ( #507 )
...
It's a lot less data, but doesn't invoke any additional
HTTP requests with potential Cloudflare CAPTCHAs.
5 years ago
Mike Fährmann
0ab9bb1721
[4chan] add extractor for entire boards ( closes #510 )
5 years ago
Mike Fährmann
c59b98c81b
[downloader:http] improve rate limit handling
...
- Move the download "logic" with rate limit checks into its own
method that only gets used if a rate limit should be enforced
- Fix an issue where suspending gallery-dl during a download would
basically ignore the rate limit for the remaining download when
resuming its execution.
5 years ago
Mike Fährmann
bbbafc1c24
[downloader:http] catch both possible SSLException instances
...
With pyOpenSSL installed, but disabled, the SSLError exception
would be set to the one from pyOpenSSL, which could never get raised.
This commit solves this problem by catching both, the native SSLError
exception as well as the one from pyOpenSSL (if available.1)
5 years ago
Gio
c20bb5c338
Naming convention, as per travis.
5 years ago
Gio
6ed4fc07ff
Don't print intentional metadata skips to the console.
5 years ago
Gio
cfc70a97ab
Added an additional channel for downloading the metadata of an entire post or gallery.
5 years ago
Mike Fährmann
f451be48c3
release version 1.12.0
5 years ago
Mike Fährmann
15f9bb3d14
add option to disable pyOpenSSL usage ( #508 )
...
(pyOpenSSL is now disabled by default)
5 years ago
Mike Fährmann
c8e99e3b3b
[deviantart] fix crash on missing "token" field ( #505 )
5 years ago
Mike Fährmann
6ed2c7823c
[deviantart] disable original downloads if no cookies set
...
For 'deviation' and 'scraps' extractors only, since original file
downloads for those two will always fail with a 404 Not Found
when not logged in.
5 years ago
Mike Fährmann
50deab5265
[deviantart] fix URL generation from /extended_fetch results
...
(closes #505 )
5 years ago
Mike Fährmann
1f209da4c0
[pixiv] match new search URLs ( closes #507 )
5 years ago
Mike Fährmann
e17907ee2a
change default value of 'cookies-update' to 'true'
5 years ago
Mike Fährmann
07dafad26d
[twitter] attempt to fix infinite loops ( #499 )
...
(Hopefully this doesn't break anything else)
5 years ago
Mike Fährmann
71acbdabf4
[2chan] fix metadata extraction
5 years ago
Mike Fährmann
c0a1241648
[livedoor] force https:// for image URLs
5 years ago
Mike Fährmann
6e23c0da09
[imgur] add extractor for subreddit links ( closes #500 )
5 years ago
Mike Fährmann
38c05df290
[oauth] add custom/default indicator to log messages ( #501 )
5 years ago
Mike Fährmann
372ffe95ee
[oauth] adjust Flickr redirect URI ( fixes #503 )
...
Flickr now automatically forces https:// for all redirect URIs.
5 years ago
Mike Fährmann
004812258d
[hentaifox] fix extraction
5 years ago
Mike Fährmann
e2710702d4
fix Cloudflare bypss
5 years ago
Mike Fährmann
8759403f37
[plurk] add delay between comment requests
5 years ago
Mike Fährmann
a28552fd19
update test results
...
- hbrowse: one tag got removed
- mangoxo: gallery changed owner
- photobucket: ?, but photo still downloads
5 years ago
Mike Fährmann
dcaa3d01bd
[imagefap] adapt to new image URL format
5 years ago
Mike Fährmann
e62c209ca0
[nijie] fix 'date' parsing
5 years ago
Mike Fährmann
3bba763ab9
[twitter] improve
...
- update metadata structure
- combine all user… entries into their own dict
- let 'user' always specify the Timeline owner
- add 'author' entry that specifies the original Tweet author
- create directories per post (closes #491 )
- fix username issues with /i/web/ URLs
5 years ago
Mike Fährmann
26d2334550
[postprocessor:metadata] rename 'format' to 'content-format'
...
Just to be consistent with the other 'extension-format' option name,
and only 'format' is also still accepted.
5 years ago
Mike Fährmann
a412531451
[postprocessor:metadata] implement 'extension-format' option
...
closes #477
5 years ago
Mike Fährmann
0f1538af78
split filename formatting into its own function
5 years ago
Mike Fährmann
db35c3b581
[directlink] separate filenames from paths
...
With this, all default filename formats specify an '{extension}'
and PathFormat.set_extension() reliably works for all files.
5 years ago
Mike Fährmann
41a3169c67
[foolfuuka] use '{extension}' in default filename format
5 years ago
Mike Fährmann
e9aed62c91
[imgur] unescape image titles
5 years ago
Mike Fährmann
bca2222559
add '--exec-after'
5 years ago
Mike Fährmann
ed6592ea1a
remove '--abort-on-skip'
5 years ago
Mike Fährmann
2c332edaad
[plurk] fix comment pagination
5 years ago
Mike Fährmann
a3fa45bbb1
[behance] get images from 'media_collection' modules
5 years ago
Mike Fährmann
359c3bc1c5
[deviantart] revert to getting download URLs from OAuth API
...
This commit (partially) reverts 27b5b24
, 94eb7c6
, and a437e78
.
Download URLs from the 'extended_fetch' endpoint are now only
usable for logged in users, while those from the respective
OAuth API endpoint are working again. Everything except
scraps and direct deviation links should be fixed, and those
two categories will work with exported cookies. (#488 )
TODO:
- "native" login with --username and --password
- better handling of internally stored cookies
5 years ago
Mike Fährmann
42b9633c7e
update test results
5 years ago
Mike Fährmann
b28bd1c73e
[bobx] set generated session cookie ( closes #482 )
...
This reverts commit 490831f
and also restores original image downloads
by setting a randomly generated session cookie. No login required.
5 years ago
Mike Fährmann
ae09f87602
improve SharedConfigMixin config lookups
5 years ago
Mike Fährmann
b5c964332b
improve config.py test coverage
5 years ago
Mike Fährmann
f5604492c3
update interface of config functions
5 years ago
Mike Fährmann
4ca883c66f
[smugmug] replace test for custom URLs
...
The old one (http://www.creativedogportraits.com/ ) is empty and/or
no longer handled by SmugMug.
5 years ago
Mike Fährmann
d45fabb79d
match user profile handling on deviantart and newgrounds
5 years ago
Mike Fährmann
ea80dadd09
[deviantart] restore archive keys
...
Commit 9fdc5e7
changed 'username' fields to have consistent
capitalization, but that invalidated the archive keys of several
extractors where 'username' was usually lowercase.
5 years ago
Mike Fährmann
3fc1e12949
[postprocessor:metadata] filter private entries
...
i.e. keys starting with an underscore
5 years ago
Mike Fährmann
ea094692c8
[vsco] fix collection extraction ( #480 )
5 years ago
Mike Fährmann
490831f84a
[bobx] "fix" image download URLs
...
Access to original images got restricted to (paid) members only.
All that's publicly accessible now are essentially preview pictures.
5 years ago
Mike Fährmann
978cb03f81
update misc test results
...
- Livedoor now uses https:// for its image URLs
- Instagram image URLs got simplified
5 years ago
Mike Fährmann
fca87974fe
[sexcom] fix video downloads by sending specific Referer headers
5 years ago
Mike Fährmann
bbbeff4c41
[downloader.http] implement file-specific HTTP headers
5 years ago
Mike Fährmann
edc080468d
[instagram] make 'video_url' fields optional ( fixes #479 )
...
[ci skip]
5 years ago
Mike Fährmann
9fdc5e74cb
[deviantart] ensure consistent username capitalization ( #455 )
...
The 'username' field was capitalized in a very inconsistent manner:
Either all lowercase, or as given by the input URL, or with the
"original" capitalization, depending on the extractor used among
other things.
Now usernames use their original capitalization for all extractors.
('UserName' instead of 'username' or 'uSeRnAmE')
5 years ago
Mike Fährmann
b1f0609de5
[newgrounds] rewrite ( #394 )
...
- restructure extractor hierarchy
- extract more metadata
- extract videos without youtube-dl
- be more resilient to errors
TODO:
- favorites
- games, but that might be near impossible for non-flash titles
5 years ago
Mike Fährmann
3ece3976ae
[newgrounds] implement login support ( #394 )
5 years ago
Mike Fährmann
3a07c06865
[newgrounds] update
...
- create directory per post
- rename variables and methods
5 years ago
Mike Fährmann
5513b66eb0
[vsco] fix user profile extraction
5 years ago
Mike Fährmann
abfcb356fc
[flickr] support 3k, 4k, 5k, and 6k photo sizes ( closes #472 )
5 years ago
Mike Fährmann
521fcd2eb9
[imgbb] fix error in galleries without user info ( closes #471 )
5 years ago
Mike Fährmann
8061263d4c
[imgbb] improve pagination logic
...
- avoid unnecessary API calls for small or empty galleries
- combine duplicate code
5 years ago
Mike Fährmann
da6789b2b0
disable unique archive id checks for some tests
...
- same image twice in a livedoor blog post
- unreliable results for related pinterest items
5 years ago
Mike Fährmann
67e54ed8ea
release version 1.11.1
5 years ago
Mike Fährmann
ce98a86c0e
fix data file inclusion in source distributions
5 years ago
Mike Fährmann
6c86fbfe2a
release version 1.11.0
5 years ago
Mike Fährmann
94a94f3b86
miscellaneous stuff
5 years ago
Mike Fährmann
b0197098e6
[imgur] get title from webpage if missing in API response
...
(closes #467 )
5 years ago
Mike Fährmann
dd5d2b2eac
[deviantart] add user profile extractor ( #377 , #419 )
5 years ago
Mike Fährmann
a437e78620
[deviantart] minimize cookie usage during scraps extraction
...
(#445 )
5 years ago
Mike Fährmann
1a197d2195
store the original cookiejar as Extractor._cookiejar
5 years ago
Mike Fährmann
de83ae4576
make 'method' argument of Extractor.request keyword-only
5 years ago
Mike Fährmann
a5be08a830
[downloader:ytdl] forward proxy settings
5 years ago
Mike Fährmann
4325695d74
[luscious] expand GraphQL queries
5 years ago
Mike Fährmann
94dbdbf506
[nijie] change default filename format
...
… to be consistent with Pixiv filenames
5 years ago
Mike Fährmann
9e88e7a344
[postprocessor:exec] improve ( #421 , #413 )
...
- add 'final' option
- include job status in pp finalization
- improve and extend documentation
5 years ago
Mike Fährmann
c18fadc221
[instagram] extract videos without youtube-dl ( #391 )
5 years ago
Mike Fährmann
f15eedb634
[sexcom] set Referer header for file downloads ( closes #464 )
5 years ago
Mike Fährmann
2a3bd4e3c7
rename extractor classes starting with a digit
5 years ago
Mike Fährmann
b3b9da6d74
[photobucket] replace test URL
...
The other user deleted all of is images.
5 years ago
Mike Fährmann
64786363be
[4chan] simplify
...
- remove 'chan.py'
- slight adjustments to directory and filenames
5 years ago
Mike Fährmann
557e2c018b
[8chan] remove module
5 years ago
Mike Fährmann
e14782a948
[instagram] simplify graphql extraction for post pages
5 years ago
Mike Fährmann
c01ff78467
[twitter] extend 'videos' option to force extraction with ytdl
...
(closes #459 )
5 years ago
Mike Fährmann
f8ac67ce50
[hitomi] extend URL pattern + follow redirects
5 years ago
Mike Fährmann
e877ca97c3
[naver] adjust directory names and metadata structure
5 years ago
Mike Fährmann
702f2fbd1f
[issuu] add publication and user extractors ( #413 )
5 years ago
Mike Fährmann
8361d874d7
[hitomi] fix extraction
5 years ago
Mike Fährmann
5fa6ff04dd
[instagram] extract '__additionalDataLoaded' ( #391 )
...
The '_sharedData' of Post pages is missing its 'graphql' part for
logged in users. This data is now included in the parameters of a
function call to '__additionalDataLoaded(...)'
And, of course, video extraction with youtube-dl broke because of
this change as well.
5 years ago
Mike Fährmann
5af291ba5c
include failed downloads and child extractors in exit status
5 years ago
Mike Fährmann
322c2e7ed4
renaming variables
...
mostly 'keyword(s)' to 'kwdict'
5 years ago
Mike Fährmann
87a87bff7e
[simplyhentai] fix image URLs
5 years ago
Mike Fährmann
4409d00141
embed error messages in StopExtraction exceptions
5 years ago
Mike Fährmann
d5e3910270
adjust 'util.raises()'
5 years ago
Mike Fährmann
d44f790e81
adjust output for HTTP status related errors
5 years ago
Mike Fährmann
03e0cec715
return with non-zero exit status on error
5 years ago
Mike Fährmann
c887493a80
overhaul exception stuff
5 years ago
Mike Fährmann
109718a5e3
[blogger] add blog and post extractors ( closes #364 )
5 years ago
Mike Fährmann
244d396b0b
add '--ugoira-conv-lossless' command-line option ( #432 )
...
and cleanup the arguments for the regular '--ugoira-conv':
- remove '-an'
- enable two-pass encoding
5 years ago
Mike Fährmann
49a6b1b6c0
[twitter] extract video stream info without youtube-dl ( #452 )
...
This should allow video downloads when logged in without
'forward-cookies' disabled and from protected tweets.
youtube-dl still gets used to download HLS playlists, but the data
extraction part, which doesn't work with youtube-dl at the moment,
now gets handled by gallery-dl itself.
5 years ago
Mike Fährmann
9f0dbf2a72
[twitter] raise proper exception for protected Tweets
5 years ago
Mike Fährmann
083e14ad9a
[downloader:ytdl] add data from '_ytdl_extra' to info_dicts
5 years ago
Mike Fährmann
6e08ada4fe
[luscious] simplify some metadata entries
5 years ago
Mike Fährmann
9e3a8607ee
[deviantart] update usernames ( #455 )
...
In the case that a user changed his username, requesting deviations
with an old name might cause problems (missing deviations, etc.)
The internal 'username' value therefore now gets updated to the
current username taken from the user profile.
5 years ago
Mike Fährmann
2eb38810c5
[twitter] fix image extraction when logged in ( #452 )
...
... for individual tweets.
To get a Tweet page with the old Twitter layout, an Internet
Explorer User-Agent (e.g. Mozilla/5.0 (Windows NT 6.1; WOW64;
Trident/7.0; rv:11.0) like Gecko) as well as a Referer header
pointing to the page itself is required. The "app_shell_visited"
cookie appears to be optional at the moment, but that is what
a regular web browser would send.
5 years ago
Mike Fährmann
8f38a35b91
[imgur] use API with "public" client_id ( #446 )
...
Using the API endpoints makes it possible to access NSFW content
without logging in.
5 years ago
Mike Fährmann
b23c822b23
[luscious] use GraphQL
5 years ago
Mike Fährmann
ef17d94469
update test results
5 years ago
Mike Fährmann
2057c6ba29
[naver] add blog and post extractors ( closes #447 )
5 years ago
Mike Fährmann
389d2d7e38
implement 'cookies-update' option ( #445 )
5 years ago
Mike Fährmann
fbc0a6a059
[nozomi] skip unavailable posts ( #388 )
5 years ago
Mike Fährmann
ae98dbcbb3
[nozomi] implement searching for negated terms ( #388 )
...
It's incredibly slow and resource intensive (> 1GB of memory),
but that is also how it is implemented on nozomi.la itself.
5 years ago
Mike Fährmann
1c03a389df
[twitter] small improvements to search extractor
...
- put search results in separate directories
- set 'max_position' to '-1' for first request
-> prevent duplicate results
- add a test
- flake8
5 years ago
Mike Fährmann
c3042978b8
[deviantart] match "/gallery/all" ( closes #449 )
5 years ago
Alice
bcddcca6db
Add search downloading to twitter.py ( #448 )
...
Adds the functionality to download search results on twitter.com/search. Since twitter only allows downloading of up to 3,200 of a users most recent tweets, you will be unable to download old images from users with a lot of tweets. To bypass this, you can use the twitter search to get the tweets from the sections in time you were stopped at. An example search would be "from:user since:2015-01-01 until:2016-01-01 filter:images". The URL you would use will look something like this https://twitter.com/search?f=tweets&q=from%3Asupernaturepics%20since%3A2015-01-01%20until%3A2016-01-01%20filter%3Aimages&src=typd&lang=en
The _tweets_from_api function had to be changed because it would not get the next page of results using the last "data-tweet-id". It would return the same JSON but with a "min_position" string added. Using this string for the "max_position" param from the second page onwards correctly returned the next pages. This change does not interfere with how the other extractors work as far as I know. The 2 regex patterns in the extractors had to be changed to not match the search URL.
5 years ago
Mike Fährmann
1693d97bd3
update extractor class hierarchies
...
- let the GalleryExtractor class inherit directly from Extractor
- make ChapterExtractor a subclass of GalleryExtractor
- change enumeration field names of GalleryExtractors to 'num'
5 years ago
Mike Fährmann
7ebd984e8d
[imgur] print error message if no JSON data is found ( #446 )
5 years ago
Mike Fährmann
5882b00f2f
[imgur] implement login support ( #446 )
5 years ago
Mike Fährmann
91643ca54b
[nozomi] add search extractor ( #388 )
5 years ago
Mike Fährmann
df2b3c6888
restore OAuth2 authentication error messages
5 years ago
Mike Fährmann
6779512fc7
[nozomi] add post and tag extractors ( #388 )
5 years ago
Mike Fährmann
6abe5f5bbb
[patreon] fix pagination ( #444 )
...
The Patreon-provided URLs for the next set of posts aren't
always complete, i.e. they can be missing their scheme and
the subsequent double slash: "www.patreon.com/…"
5 years ago
Mike Fährmann
ff1e4a86aa
release version 1.10.6
5 years ago
Mike Fährmann
d4ffd6c952
[yaplog] improve metadata extraction ( #443 )
...
- provide a fallback if there is no numerical image ID
- add a 'filename' field
- convert 'date' to an actual datetime object
5 years ago
Mike Fährmann
15af2f8464
[hitomi] fallback to /reader/ page if main page returns 404
...
Some galleries return a 404: Not Found error when trying to access
them through the main gallery URL, but their content is still
available on the respective /reader/ page.
5 years ago
Mike Fährmann
8af59a4bba
fix & update docs
...
- update Requests links
- add example for --exec
- set '-dev' version
5 years ago
Mike Fährmann
dc6ad81e2e
[yaplog] prevent crash on empty posts ( #443 )
5 years ago
Mike Fährmann
94eb7c6cad
[deviantart] fix sta.sh extraction (436)
5 years ago
Mike Fährmann
1032cfa34b
[downloader:http] extend mimetype map with archive formats
5 years ago
Mike Fährmann
27b5b2497e
[deviantart] fix download URLs ( #436 )
...
... except for sta.sh content.
Instead of using the old '/api/v1/oauth2/deviation/download' endpoint,
which started delivering URLs to 404 pages a while ago,
it is also possible to get a download URL from the relatively new
'/_napi/da-browse/shared_api/deviation/extended_fetch' endpoint
used by DeviantArt's Eclipse interface.
The current strategy is therefore:
- Iterate over deviations using the OAuth2 API
- Fetch original download URLs with the new NAPI/Shared API
5 years ago
Mike Fährmann
93aac8dfea
[yaplog] fix incomplete image URLs ( #443 )
5 years ago
Mike Fährmann
a782b009b8
[yaplog] match blog names with '-' ( #443 )
5 years ago
Mike Fährmann
cf5e716b9d
[hitomi] fix image URLs
5 years ago
Mike Fährmann
ad81c07204
[postprocessor] match logger names of downloader modules
...
The logger name for a postprocessor object got changed to
"postprocessor.<module-name>" instead of just
"postprocessor"
5 years ago
Mike Fährmann
03bc8adfc7
[postprocessor:exec] run after file moved to target location
...
(#421 )
5 years ago
Mike Fährmann
35958bebd4
[postprocessor:exec] fix filename quoting on Windows ( #421 )
5 years ago
Mike Fährmann
b06c372e4d
[postprocessor:exec] improve; add command-line option ( #421 )
5 years ago
Mike Fährmann
5a54efa025
[xhamster] unescape 'title' and 'description'
5 years ago
Mike Fährmann
1b9bf4fc6e
[behance] fix 'tags' extraction
5 years ago
Mike Fährmann
bb97e87989
[komikcast] ignore banner image
5 years ago
Mike Fährmann
0ff90a3f7d
[gfycat] include title in default filenames ( closes #434 )
5 years ago
Mike Fährmann
fabdc3b0c6
release version 1.10.5
5 years ago
Mike Fährmann
de4e2029d1
[nsfwalbum] update test album
...
the old one is no longer available
5 years ago
Mike Fährmann
1faec285d1
[nijie] further improvements ( closes #423 )
...
- provide a 'user_name' metadata field
- usually the same as 'artist_id', except for favorite downloads
- extract the whole description text and properly escape HTML entities
- fixed an issue with titles or tags containing double quotes
5 years ago
Mike Fährmann
6d0a533d68
[reddit] respect 'comments:0' for single submissions ( #429 )
5 years ago
Mike Fährmann
803d8f814e
[oauth] update scope for reddit tokens ( #428 )
...
'/user/<username>/...' requires the 'history' scope to be accessible
(https://www.reddit.com/dev/api/#GET_user_{username}_{where} )
5 years ago
Mike Fährmann
46ba173ded
[reddit] fix documentation inconsistencies ( closes #429 )
...
- Require 'reddit.comments' to be a number and convert it to an
integer to be extra sure
- Link to the README's OAuth section were appropriate
5 years ago
Mike Fährmann
20eb6c401f
[nijie] improvements and fixes ( #423 )
...
- ignore unavailable image pages
- more metadata fields: artist_name, date, tags
- rename 'index' to 'num'
- improved code structure
5 years ago
Mike Fährmann
d1ea08c67d
[weibo] fixes and improvements
...
- ignore unavailable videos (fixes #427 )
- handle empty 'geo' fields
- consistent metadata fields for images and videos
5 years ago
Mike Fährmann
38d97f3da6
[deviantart] add debug message about API credentials ( #424 )
5 years ago
Mike Fährmann
80c2104fb5
[deviantart] fix 429 handling if 'fatal' is False ( closes #424 )
5 years ago
Mike Fährmann
913460240d
[reddit] fix 'extractor.blacklist()' arguments
...
The second argument must support 'append()'.
5 years ago
Mike Fährmann
22bac14452
[pixiv] match '/artworks/' URLs
5 years ago
Mike Fährmann
66cac207ac
[twitter] match and use 'i/web' status URLs
5 years ago
Mike Fährmann
946f2751e2
[reddit] add 'user' extractor ( closes #350 )
5 years ago
Mike Fährmann
c14abb9fb8
[reddit] improve URL parameter handling for subreddit links
5 years ago
Mike Fährmann
ee8b654464
[instagram] implement 'highlights' option ( closes #329 )
5 years ago
Mike Fährmann
f63c3097a9
[instagram] rework some code paths
...
- combine fetching an HTML page and extracting its 'shared_data'
- move 'shared_data' and field access info out of '_extract_page()'
- introduce a '_request_graphql()' method
5 years ago
Mike Fährmann
4330133114
[imgur] add 'favorite' extractor ( closes #420 )
...
… and use a newer site-internal API endpoint for user posts
5 years ago
Mike Fährmann
ee5e20221f
[imgth] fix image URLs
5 years ago
Mike Fährmann
b63b126808
[hentaicafe] extend URL pattern
5 years ago
Mike Fährmann
d780f0357e
[imgur] add user extractor
5 years ago
Mike Fährmann
11ea689013
[simplyhentai] fix image and video URLs
5 years ago
Mike Fährmann
15632a1570
[tsumino] fix extraction
5 years ago
Mike Fährmann
d92802fd37
[luscious] fix detection of unavailable galleries
5 years ago
Mike Fährmann
f99da2b866
[imgbb] detect invalid album and user profile links
...
and update test results, since the old album got deleted
5 years ago
Mike Fährmann
01bc7adadc
[deviantart] improve journal detection ( #419 )
...
Some journal-like posts are not reported to be journals (isJournal
is set to False), even though they have a textContent field.
https://www.deviantart.com/gliitchlord/art/brashstrokes-812942668
5 years ago
Mike Fährmann
776e9e073f
close archive on job completion ( #417 )
5 years ago
Mike Fährmann
5ac9732adc
call 'sys.exit()' on Ctrl+c
5 years ago