1f9b16a70b
replace static 'sleep-request' defaults with dynamic ones
9 months ago
3ecb512722
send Referer headers by default
1 year ago
a453335a9f
remove test results in extractor modules
...
and add generic example URLs
1 year ago
1d2b5d0c60
update test comment positions
...
always put them above the test they're referring to
1 year ago
a383eca7f6
decouple extractor initialization
...
Introduce an 'initialize()' function that does the actual init
(session, cookies, config options) and can called separately from
the constructor __init__().
This allows, for example, to adjust config access inside a Job
before most of it already happened when calling 'extractor.find()'.
1 year ago
dd884b02ee
replace json.loads with direct calls to JSONDecoder.decode
2 years ago
b0cb4a1b9c
replace 'text.extract()' with 'text.extr()' where possible
2 years ago
ea8113ff36
[reactor] match 'best', 'new', 'all' URLs ( #3073 )
2 years ago
d26da3b9e5
add pre-generated 'pattern' for supported BaseExtractor sites
2 years ago
addb72e1bb
[reactor] support thatpervert.com ( closes #2029 )
3 years ago
d8d9502e1e
[reactor] inherit from BaseExtractor
3 years ago
bd08ee2859
remove most 'yield Message.Version' statements
...
only leave them in oauth.py as noop results
3 years ago
fa6af46756
Added ability to download GIFs instead of mp4 from Luscious and Reactor ( #1701 )
3 years ago
21c2da454f
update extractor test results
3 years ago
2c60c7d798
[reactor] skip deleted/empty posts
3 years ago
bae874f370
replace 'wait-min/-max' with 'sleep-request'
...
on exhentai, idolcomplex, reactor
4 years ago
3df527ee2c
update extractor test results
4 years ago
65ca923b4e
fix 'whitelist' option for BaseExtractor instances
4 years ago
912eea29bc
update extractor test results
4 years ago
1e3dd7330e
merge SharedConfigMixin functionality into Extractor
4 years ago
968d3e8465
remove '&' from URL patterns
...
'/?&#' -> '/?#' and '?&#' -> '?#'
According to https://www.ietf.org/rfc/rfc3986.txt , URLs are
"organized hierarchically" by using "the slash ("/"), question
mark ("?"), and number sign ("#") characters to delimit components"
4 years ago
43b156fb40
[reactor] match URLs without subdomain ( #1053 )
4 years ago
7619152988
[reactor] sort 'tags'
...
to ensure a consistent order for test results
4 years ago
c50d60a53d
[reactor] fix image URLs
5 years ago
b1db194c14
[reactor] update and improve
...
- split 'tags' into a list
- parse 'date' into a datetime object
- fix webm/mp4 URLs
5 years ago
0f02e85961
[reactor] use "/full/" URLs ( closes #210 )
...
Putting a "/full/" in image URLs potentially gives higher resolution
and better quality.
6 years ago
5530871b5a
change results of text.nameext_from_url()
...
Instead of getting a complete 'filename' from an URL and splitting that
into 'name' and 'extension', the new approach gets rid of the complete
version and renames 'name' to 'filename'. (Using anything other than
{extension} for a filename extension doesn't really work anyway)
Example: "https://example.org/path/filename.ext "
before:
- filename : filename.ext
- name : filename
- extension: ext
now:
- filename : filename
- extension: ext
6 years ago
2e516a1e3e
store the full original URL in Extractor.url
6 years ago
4b1880fa5e
propagate 'match' to base extractor constructor
6 years ago
6284731107
simplify extractor constants
...
- single strings for URL patterns
- tuples instead of lists for 'directory_fmt' and 'test'
- single-tuple tests where applicable
6 years ago
050bc1aa4a
[reactor] simplify tests
...
Some posts have, for whatever reason, a slightly different text
formatting the first time they are accessed that day
compared to any further time.
6 years ago
4d656a81ca
replace SharedConfigExtractor class with a Mixin
6 years ago
1734a6c879
[reactor] detect "circular" redirects ( #148 )
6 years ago
e53cdfd6a8
update build_supportedsites.py
6 years ago
e95b24f056
[reactor] add wait-min & -max options ( #148 )
6 years ago
8e01cf0ef8
[reactor] generalize extractors ( #148 )
...
- support *.reactor.cc domains
- combine joyreactor and pornreactor modules
6 years ago
Mike Fährmann