Mike Fährmann
2076d40681
[ytdl] improve error handling ( #1680 )
3 years ago
Mike Fährmann
8eaedb0bd3
[ytdl] fix some compatibility issues ( #1680 )
3 years ago
Mike Fährmann
2aaac3c997
[instagram] include user metadata for 'tagged' downloads ( #2024 )
...
Adds
- tagged_owner_id
- tagged_full_name
- tagged_username
containing the values for the user profile the URL originated from,
e.g. 'instagram' for https://www.instagram.com/instagram/tagged/ .
3 years ago
Mike Fährmann
cfa4876848
[philomena] support furbooru.org ( closes #1995 )
3 years ago
Mike Fährmann
4377f1c284
[twitter] distinguish between fatal & nonfatal errors ( #2020 )
...
only show a warning for nonfatal errors
and do not raise a StopExtraction exception
3 years ago
Mike Fährmann
19403a7fff
[downloader:ytdl] prevent crash in '_progress_hook()' ( #1680 )
...
'speed' is not guaranteed to be defined or convertible to 'int'
3 years ago
Mike Fährmann
01b28f3674
[ytdl] fix syntax for Python 3.4
3 years ago
Kyle Anthony Williams
a14b72be21
[webtoons] Use swebtoon-phinf.pstatic.net instead of webtoon-phinf.pstatic.net ( #2005 )
...
* [webtoons] Use swebtoon-phinf.pstatic.net instead of webtoon-phinf.pstatic.net
This trick to avoid having to set a Referer header comes from
Webtoon's RSS feeds. The two URLs below are equivalent in content:
https://webtoon-phinf.pstatic.net/20210929_153/1632867980912DmcGK_JPEG/16328679808882705182.jpg?type=q90
https://swebtoon-phinf.pstatic.net/20210929_153/1632867980912DmcGK_JPEG/16328679808882705182.jpg?type=q90
The URL with the domain "webtoon-phinf.pstatic.net" needs a Referer
header, and the domain "swebtoon-phinf.pstatic.net" does not. This
is because of the environment "swebtoon" images live in, one without
explicit network control: RSS feeds on sites such as Feedly. This change should
make it easier for gallery-dl developers to embed Webtoon comics without
worrying about headers.
3 years ago
Mike Fährmann
6e3658ef52
[kemonoparty] provide 'date' metadata for gumroad ( #2007 )
...
Not the 'published' or 'edited' values since they are 'null',
but still better then nothing at all.
3 years ago
Mike Fährmann
f4d201f626
[ytdl] fix syntax for Python 3.4
3 years ago
Mike Fährmann
37c9dedee1
[seisoparty] remove module
3 years ago
Mike Fährmann
efa178cc91
[ytdl] implement parsing ytdl command-line options ( #1680 )
...
- adds 'config-file' and 'cmdline-args' options
for both ytdl downloader and extractor
- create 'ytdl' helper module, which combines YoutubeDL creation
and option parsing.
- most likely a buggy mess due to incompatibilities between the
original youtube-dl and yt-dlp.
3 years ago
Mike Fährmann
a881305357
release version 1.19.2
3 years ago
Mike Fährmann
7cb303d745
[redgifs] improve URL extraction
...
Fields inside 'urls' can be None, which would have caused an exception
with the old method.
3 years ago
Mike Fährmann
2befed1a96
[redgifs] update search URL pattern ( #1984 )
3 years ago
Mike Fährmann
b315a0ecef
[redgifs] update to API v2 ( #1984 )
3 years ago
Mike Fährmann
f0fc3b0ba1
[kemonoparty] add 'comments' option ( #1980 )
3 years ago
Mike Fährmann
1fac74b14d
[reddit] prevent crash for galleries with no 'media_metadata'
...
(fixes #2001 )
3 years ago
Mike Fährmann
211de95dd0
update extractor test results
3 years ago
Mike Fährmann
8bea02c38c
[deviantart] fix 'index' values for stashed deviations
3 years ago
Mike Fährmann
dd88a7d980
{cyberdrop] restore video extraction ( fixes #1993 )
...
fixes a regression introduced in f33c2ef7
3 years ago
Mike Fährmann
fa5646eadc
[mangoxo] fix login and extraction
3 years ago
Mike Fährmann
4c49174579
[mangakakalot] update domain and fix extraction
3 years ago
YongChan Cho
14852f7050
[hitomi] fix image path ( #1988 )
3 years ago
Mike Fährmann
46e17c5e61
support accessing the current local datetime in format strings
...
{_now}, {_now:%Y-%m-%d}, etc
(#1968 )
3 years ago
Mike Fährmann
dad2875a3e
fix calculating retry sleep times ( fixes #1990 )
3 years ago
Mike Fährmann
9156e90f1f
[twitter] add 'pinned' option
3 years ago
Mike Fährmann
06b414c9a3
[redgifs] 'gfyId' -> 'id' ( #1984 )
3 years ago
Ryu juheon
d4614e5ba4
[hitomi] fix image URLs ( #1982 )
3 years ago
Mike Fährmann
6434ccf9e8
[redgifs] split from 'gfycat' ( #1984 )
...
Update API endpoints and metadata names - mostly 'gfycat' -> 'gif' -
and remove some obsolete checks.
3 years ago
Mike Fährmann
38193dba46
support accessing environment variables in format strings ( #1968 )
...
{_env[HOME]} to get the value of $HOME
every other format string feature is supported as well
3 years ago
Mike Fährmann
e4696b40ba
[instagram] update query hashes
3 years ago
Alice
bfd7401b1e
[skeb] add 'user' and 'post' extractors ( #1031 ) ( #1971 )
...
* Create skeb.py
* Update __init__.py
* Update supportedsites.py
* Update supportedsites.md
* Update supportedsites.py
* Update skeb.py
3 years ago
Ryu juheon
6b6d92d51c
[hitomi]: fix image URLs ( #1975 )
3 years ago
Mike Fährmann
dcb201ff19
[gfycat] show warning when there are no available formats
3 years ago
Mike Fährmann
e436a2607b
[gfycat] consistent 'userName' values for 'user' downloads ( #1962 )
...
by using the name from the input URL and not relying on possibly faulty
or incomplete API results.
'userData[username]', if available, will still have the original name.
3 years ago
Mike Fährmann
ba9579c504
release version 1.19.1
3 years ago
Mike Fährmann
f1487a3cfa
[kemonoparty:discord] improve 'inline' extraction ( #1940 )
...
- extract media.discordapp.*NET* URLs
- rewrite media.discordapp.net to cdn.discordapp.com
- use a more restricted set of characters for the URL path
3 years ago
Mike Fährmann
02a247f4e5
[deviantart] full resolution for non-downloadable images ( #293 )
...
Many thanks to @Ironchest337 for discovering this method
and providing a well-documented implementation.
3 years ago
Mike Fährmann
a7ddb5f5fa
[deviantart] update 'search' argument handling ( fixes #1911 )
...
- use 'alltime' by default
- support newer 'order' values (most-recent, this-week, etc)
3 years ago
Mike Fährmann
c19e762fdf
[vk] add 'album' extractor ( #474 , fixes #1952 )
...
todo: better metadata for albums
3 years ago
Mike Fährmann
8bb442f20d
[redgifs][gfycat] provide fallback URLs ( fixes #1962 )
...
and extend the 'format' option
3 years ago
Mike Fährmann
b6443c576d
[kemonoparty:discord] extract 'inline' files
3 years ago
Mike Fährmann
232ab626a7
[downloader:ytdl] prevent crash in '_progress_hook()'
...
https://github.com/mikf/gallery-dl/discussions/1964#discussioncomment-1516702
3 years ago
Mike Fährmann
bcbf9bcf36
[kemonoparty] split 'discord' extractor ( #1940 )
...
in 'server' and 'channel'
3 years ago
Mike Fährmann
db857b40d8
[kemonoparty] improve inline extraction ( #1899 )
3 years ago
Mike Fährmann
975e0a4fe0
[furaffinity] unquote search queries ( #1958 )
...
instead of unescape
(unquote -> url params, unescape -> html entities)
3 years ago
Mike Fährmann
8d676151b7
[patreon] implement 'files' option ( #1935 )
3 years ago
Mike Fährmann
6695ef2e10
[patreon] better filenames for 'content' images ( #1954 )
3 years ago
Mike Fährmann
70005e3275
[kemonoparty:discord] support downloading from a specific channel
...
https://kemono.party/discord/server/ <server-id>#<channel-name>>
3 years ago
Mike Fährmann
003f25931d
[kemonoparty:discord] provide a 'channel_name'
3 years ago
Mike Fährmann
28bdd58e6d
[nhentai] simplify
3 years ago
Mike Fährmann
50098762e3
[nhentai] add 'tag' extractor ( closes #1950 )
3 years ago
Mike Fährmann
fe6ce5495a
[kemonoparty] add 'discord' extractor ( #1827 , #1940 )
3 years ago
Mike Fährmann
f2d6b3e6b4
run tests without using 'nose'
...
run_tests.sh -> run_tests.py
3 years ago
Mike Fährmann
918fc9974d
[picarto] add 'gallery' extractor ( closes #1931 )
3 years ago
Mike Fährmann
e33125ad39
[pixiv] add 'sketch' extractor ( #1497 )
3 years ago
Mike Fährmann
e9dc6ff262
[inkbunny] add 'following' extractor ( #515 )
3 years ago
Mike Fährmann
9c8fc6e7b4
[inkbunny] match "long" URLs for pools and favorites ( #1937 )
3 years ago
Mike Fährmann
f33c2ef73b
[cyberdrop] extract direct download URLs ( #1943 )
...
do not rely on redirects from f.cyberdrop.cc
3 years ago
Mike Fährmann
b93915c113
[inkbunny] add 'pool' extractor ( #1937 )
3 years ago
Mike Fährmann
373d3e1c57
[seisoparty] implement login with username & password ( #1906 )
3 years ago
Mike Fährmann
7c5f62b453
[seisoparty] add 'favorite' extractor ( #1906 )
3 years ago
Mike Fährmann
d93b5474c3
[mangadex] update parameter handling for API requests
...
- move common parameters into '_pagination()'
- add 'ratings' (#1908 ) and 'api-parameters' options
3 years ago
Mike Fährmann
cd66c3c415
[twitter] add 'size' option ( #1881 )
3 years ago
Mike Fährmann
df8050b81d
[postprocessor:compare] add 'equal' option ( #1592 )
...
Move functionality from cdd72e14
to its own option,
where it can be used with any 'action'
3 years ago
Mike Fährmann
f8410203ef
release version 1.19.0
3 years ago
Mike Fährmann
cdd72e1413
[postprocessor:compare] extend 'action' option ( #1592 )
...
allow setting it to "abort", "terminate", or "exit" as with 'skip'
3 years ago
Mike Fährmann
fb98b3fdaf
[redgifs][gfycat] remove webtoken code ( fixes #1907 )
3 years ago
Mike Fährmann
96215c926e
[mangadex] fix retrieving chapters from 'pornographic' titles
...
(fixes #1908 )
3 years ago
Mike Fährmann
da9685609c
[kemonoparty] update file download URLs
...
(closes #1902 , fixes #1903 )
3 years ago
Mike Fährmann
783eae6fc5
[hiperdex] fix extraction
3 years ago
Mike Fährmann
28f1c36da2
simplify and adjust download progress indicator ( #1519 )
3 years ago
Mike Fährmann
e0bdacd932
[fappic] add 'image' extractor ( closes #1898 )
3 years ago
Mike Fährmann
c22ff97743
remove 'unit' argument from 'util.format_value()'
3 years ago
Mike Fährmann
d0761454b1
implement a download progress indicator ( #1519 )
3 years ago
Mike Fährmann
cad85640de
move 'util.PathFormat' into its own 'path' module
...
to prevent circular imports between 'formatter' and 'util'
3 years ago
Mike Fährmann
0038a8c1a4
implement a way to specify 'extended' format strings
...
Starting a format string with '\f<Type> ' allows to set a different
format string type than the default. Available ones are
- T: A template file containing the actual format string
"\fT ~/.templates/booru.txt
- E: An arbitrary Python expression
"\fE title.upper().replace(' ', '-')"
- M: Name of a Python module followed by a function name inside it.
This function gets called with the current metadata dict as
argument and should return a string.
"\fM my_module:generate_text"
'\f' was chosen since it is highly unlikely that a regular format string
would start with it, but that character could be changed to for example
'\\' or '/' etc.
3 years ago
Mike Fährmann
74145467dd
move 'util.Formatter' into its own 'formatter' module
3 years ago
Mike Fährmann
9377543162
[mastodon] add 'following' extractor ( #1891 )
3 years ago
Mike Fährmann
2c2932973c
[mastodon] support specifying accounts by ID
...
Same as a3b473bd
for Twitter
Instead of just
https://instance.tld/@user
it is now also possible to refer to that account with
https://instance.tld/users/user
https://instance.tld/@id:12345
https://instance.tld/users/id:12345
3 years ago
Mike Fährmann
94143eb86c
[twitter] add 'quote_by' metadata field ( #1481 )
...
Only present for tweets quoted by another tweet.
Represents the tweet_id of said tweet quoting this one.
3 years ago
Mike Fährmann
1cc73e11aa
add '--write-infojson' command-line option
3 years ago
Mike Fährmann
a23f5d45d7
[deviantart] fix bug with fetching premium content ( #1879 )
...
When a user has both 'watchers' and 'paid' folders and one of them is
inaccessible, the other one could get handled as inaccessible as well.
3 years ago
Mike Fährmann
ada36c2044
[deviantart] update default archive_fmt for single deviations
...
(#1874 )
use the same as gallery downloads
3 years ago
Mike Fährmann
da16eabb82
[twitter] ensure card entries have a 'url' ( #1868 )
3 years ago
Mike Fährmann
e69ee41f25
implement 'page-reverse' option ( #1854 )
3 years ago
cyberdrop-me
c83668c2ff
[CyberDrop] Change directory name format ( #1871 )
...
Album IDs are random, organization would be much better having the album name then the identifier at the end
3 years ago
Mike Fährmann
e4684c5cb9
[desktopography] simplify ( #1740 )
3 years ago
Giacomo Rossetto
4a7d7899ff
Implement desktopography extractor ( #1740 )
3 years ago
Alice
9992ff38da
[fantia] add 'date' metadata field ( #1853 )
3 years ago
Mike Fährmann
fba95c3a9e
[nozomi] preserve case of search tags ( fixes #1860 )
3 years ago
Mike Fährmann
4b3e309b90
[aryion] update/improve pagination ( #1849 )
...
Manually increment the 'p' query parameter,
instead of relying on a "Next" link which only works up to page 200.
3 years ago
Mike Fährmann
266ed9b62e
[aryion] add 'tag' extractor ( closes #1849 )
3 years ago
Mike Fährmann
6bbeaac029
[mangadex] fix extraction ( fixes #1852 )
3 years ago
Mike Fährmann
e9bf8d2591
[instagram] update default delay to 6-12 seconds ( #1835 )
3 years ago
Mike Fährmann
c9e6693530
allow specifying a minimum/maximum for 'sleep-*' options ( #1835 )
...
for example '"sleep-request": [5.0, 10.0]' to wait between 5 and 10
seconds between each HTTP request
3 years ago
Mike Fährmann
bd845303ad
implement a way to shorten filenames with east-asian characters
...
(#1377 )
Setting 'output.shorten' to "eaw" (East-Asian Width) uses a slower
algorithm that also considers characters with a width > 1.
3 years ago
Mike Fährmann
2ff2974353
[common] update default argument handling in Extractor.request()
...
more lines of code, but slightly less execution time
3 years ago
Mike Fährmann
0fd959a2a7
[twitter] support '/with_replies' URLs ( closes #1833 )
3 years ago