1e3dd7330e
merge SharedConfigMixin functionality into Extractor
4 years ago
ddfb4fd07a
[twitter] use ' https://twitter.com/i/api/ ' for logged in users
...
Doesn't seem to make a difference from what I can tell,
i.e. downloaded files are the same, but the website does it.
4 years ago
42ccae53c4
[mangadex] switch to API v2
...
https://mangadex.org/api/v2/
https://mangadex.org/thread/351011
4 years ago
ca44111726
[flickr] update
...
- ensure every photo has an 'owner' (#828 )
- change default directories to a more consistent schema
- create directory for each photo
4 years ago
9b1bd09454
change 'extension-map' default
...
Replace all JPEG filename extensions with 'jpg'.
4 years ago
e5438b8a29
release version 1.15.3
4 years ago
de0c57886d
[twitter] add 'list-members' extractor ( closes #1096 )
4 years ago
904ba08568
[gfycat] fix default filename format
4 years ago
a46561bc16
[500px] update query hashes
4 years ago
2e3a0dff21
[8kun] fix file URLs of older posts ( fixes #1101 )
4 years ago
00825cddf5
[hentaifoundry] use scheme from input URL ( fixes #1095 )
...
Let the user choose between http and https,
instead of always forcing https.
4 years ago
8a98d3549a
[weasyl] create directory for each favorite submission
...
(#1032 )
4 years ago
91db8df1c7
[deviantart] add 'index_base36' metadata field ( closes #1099 )
...
This is the same ID as found in 'filename' without the 'd' in front,
which is just 'index' encoded in base36.
4 years ago
b9bfa4c675
update extractor test results
4 years ago
1b5b789401
[mangoxo] fix metadata extraction
4 years ago
41d4968866
[twitter] add 'list' extractor ( #1096 )
4 years ago
5d10520f4c
[twitter] update GraphQL endpoint & fix width/height entries
4 years ago
bc7b1d91bc
fix rST markup in configuration.rst
...
[ci skip]
4 years ago
9b2e5f72d6
[exhentai] update image URL parsing ( #1094 )
4 years ago
e3480bc8de
implement 'extension-map' option ( #318 )
4 years ago
98a4d86a01
[sankakucomplex] extract videos and embeds ( closes #308 )
4 years ago
c3f01dc4e6
implement 'util.unique()'
4 years ago
558cde139c
[paheal] fix extraction ( fixes #1088 )
4 years ago
0211af7ca8
[hentaifoundry] update 'YII_CSRF_TOKEN' cookie handling
...
(fixes #1083 )
4 years ago
d83b95fd28
[postprocessor:metadata] accept a string-list for 'content-format'
...
(closes #1080 )
4 years ago
198c33ec36
also collect post processors from 'basecategory' entries
...
(fixes #1084 )
4 years ago
350b1afe1c
speed up _list_classes() after iterating over all modules once
4 years ago
5bcf28de93
add a 'extractor.modules' option
4 years ago
18213dc5ba
release version 1.15.2
4 years ago
de4a1e45c9
improve 'generate_csrf_token()'
...
no need to use hashlib.md5()
4 years ago
b788712844
[fallenangels] fix extraction of '.5' chapters
4 years ago
28d8541cb3
[mangafox] ensure download URLs have a scheme
4 years ago
8e3a324c91
[mangakakalot] ignore "Go Home" buttons in chapter pages
4 years ago
c14c5d82d6
[newgrounds] use generator for fallback URLs
4 years ago
a09f42f6b3
improve filename_from_url() performance
...
Manually extracting the part between the last '/' and '?' instead of
relying on the standard libraries' 'urllib.parse.urlsplit()' increases
performance by ~400%.
urlsplit() : 3.64 secs per 1.000.000 iterations
partition(): 0.87 secs per 1.000.000 iterations
4 years ago
968d3e8465
remove '&' from URL patterns
...
'/?&#' -> '/?#' and '?&#' -> '?#'
According to https://www.ietf.org/rfc/rfc3986.txt , URLs are
"organized hierarchically" by using "the slash ("/"), question
mark ("?"), and number sign ("#") characters to delimit components"
4 years ago
1686dc1757
[twitter] support media from Cards ( #1005 , #937 )
...
Can be enabled with 'extractor.twitter.cards', but for now disabled by
default because cards can redirect to rather large videos from YouTube
or Twitch.
4 years ago
ffd38215a4
[hitomi] fix image URLs and URL pattern
...
- non-webp files are now hosted on [a-c]b.hitomi.la
- removed ampersand from invalid slug characters
4 years ago
bac8af69e8
update configuration.rst
...
- add some lines to better explain post processor usage
- syntax highlighting for JSON blocks and other smaller stuff
4 years ago
05d7009cc6
rename 'Authentication' entries in supportedsites.rst
...
- change 'Optional' to 'Supported'
- use 'OAuth' and 'Cookies' in their own
- add link to weasyl API key option
4 years ago
286718950c
[mangahere] ensure download URLs have a scheme ( fixes #1070 )
4 years ago
76dfa11a65
[reddit] add 'date' metadata field ( closes #1068 )
4 years ago
3f2ba629ea
[newgrounds] provide fallback URLs for video downloads ( #1042 )
4 years ago
a3ca2f6080
update fallback URL handling
...
remove Message.Urllist and use a '_fallback' field inside a kwdict
4 years ago
43dab3a228
[mangadex] unescape more metadata fields ( fixes #1066 )
...
like 'manga', 'author', 'artist', etc.
4 years ago
ec61696316
add 't' format string conversion ( closes #1065 )
...
to Trim whitespace from the beginning and end of strings.
Example: '{field!t}' becomes 'foo' for 'field' == " \nfoo\t\r"
4 years ago
5565025221
[xhamster] fix user profile extraction
4 years ago
07432d6262
[seiga] fix flake8 and cookie test ( #1063 )
4 years ago
d1c6d78477
fix rST markup in configuration.rst
4 years ago
b8daabc3ca
[pinterest] implement login support ( closes #1055 )
...
being logged allows access to secret/protected boards
4 years ago
Mike Fährmann