Mike Fährmann
bddcec49f1
implement 'text.root_from_url()'
...
use domain from input URL for kemono
3 years ago
Mike Fährmann
f5b2b9333f
fix another bug in _check:cookies ( #2160 )
...
regression introduced in ed317bfc
Added a couple of tests to hopefully catch such bugs
before they land in a release.
3 years ago
Mike Fährmann
563bd0ecf4
[danbooru] inherit from BaseExtractor
...
- merge danbooru and e621 code
- support booru.allthefallen.moe (closes #2283 )
- remove support for old e621 tag search URLs
3 years ago
Mike Fährmann
b5b4f5a168
use 'build_extractor_filter' in test_results.py
3 years ago
Mike Fährmann
64cf26eaf4
allow specifying sleep-* options as string
...
either as single value or as range: "3.5", "2.1 - 5.0"
3 years ago
Mike Fährmann
010d65dcec
extend blacklist/whitelist syntax ( #2025 )
...
Each entry in such a list can now also include a subcategory
'<category>:<subcategory>'
and it is possible to use '*' or an empty string as placeholder
'*:<subcategory>', ':<subcategory>', '<category>:*'
For example
"blacklist": "imgur,*:tag,gfycat:user" or
"blacklist": ["imgur", "*:tag", "gfycat:user"]
will filter all 'imgur' extractors, all extractors with a 'tag'
subcategory (e.g. https://danbooru.donmai.us/posts?tags=bonocho ),
and all 'gfycat' user extractors.
3 years ago
Mike Fährmann
af6424f398
allow testing metadata in list elements
3 years ago
Mike Fährmann
3842cdcd8f
[formatter] implement 'D' format specifier
...
To be able to parse any string into a 'datetime' object
and format it as necessary.
Example:
{created_at:D%Y-%m-%dT%H:%M:%S%z}
->
"2010-01-01 00:00:00"
{created_at:D%Y-%m-%dT%H:%M:%S%z/%b %d %Y %I:%M %p}
->
"Jan 01 2010 12:00 AM"
with 'created_at' == "2010-01-01T01:00:00+0100"
3 years ago
Mike Fährmann
2ab190ce08
add tests for special format strings
3 years ago
Mike Fährmann
46e17c5e61
support accessing the current local datetime in format strings
...
{_now}, {_now:%Y-%m-%d}, etc
(#1968 )
3 years ago
Mike Fährmann
38193dba46
support accessing environment variables in format strings ( #1968 )
...
{_env[HOME]} to get the value of $HOME
every other format string feature is supported as well
3 years ago
Mike Fährmann
f2d6b3e6b4
run tests without using 'nose'
...
run_tests.sh -> run_tests.py
3 years ago
Mike Fährmann
12fc646c53
fix filename formatting tests
3 years ago
Mike Fährmann
e0bdacd932
[fappic] add 'image' extractor ( closes #1898 )
3 years ago
Mike Fährmann
c22ff97743
remove 'unit' argument from 'util.format_value()'
3 years ago
Mike Fährmann
cad85640de
move 'util.PathFormat' into its own 'path' module
...
to prevent circular imports between 'formatter' and 'util'
3 years ago
Mike Fährmann
74145467dd
move 'util.Formatter' into its own 'formatter' module
3 years ago
Mike Fährmann
9377543162
[mastodon] add 'following' extractor ( #1891 )
3 years ago
Mike Fährmann
bd845303ad
implement a way to shorten filenames with east-asian characters
...
(#1377 )
Setting 'output.shorten' to "eaw" (East-Asian Width) uses a slower
algorithm that also considers characters with a width > 1.
3 years ago
Mike Fährmann
292fffc83c
add 'j' format string conversion
...
to convert to a JSON formatted string
3 years ago
Mike Fährmann
bb6a130942
automatically set required DDoS-GUARD cookies ( #1779 )
...
for kemono.party and seiso.party
3 years ago
Mike Fährmann
2792ed6e4b
implement 'util.format_value()'
3 years ago
Mike Fährmann
9e42cd58ea
replace ChainPredicate class with 'functools.partial'
3 years ago
Mike Fährmann
36ac2197db
[ytdl] add extractor for sites supported by youtube-dl
...
(#1680 , #878 )
Can be used by prefixing any URL with 'ytdl:',
or by setting 'extractor,ytdl.enabled' to 'true'.
3 years ago
Mike Fährmann
64240c8d42
[imagevenue] fix extraction
...
(closes #1677 )
3 years ago
Mike Fährmann
0179581340
add 'T' format string conversion ( #1646 )
...
to convert 'date'/datetime to timestamp
3 years ago
Mike Fährmann
f74cf52e2b
[seisoparty] add 'user' and 'post' extractors ( #1635 )
3 years ago
Mike Fährmann
759735fb02
[kemonoparty] fix 'username' extraction ( fixes #1652 )
...
The site's <title> content changed from
<title>NAME | Kemono</title>
to
<title>
NAME | Kemono
</title>
3 years ago
Mike Fährmann
07c8adbd8b
[mangadex] implement login with username & password ( #1535 )
3 years ago
Mike Fährmann
4a747a31a3
[postprocessor:metadata] handle dicts in mode;tags ( fixes #1598 )
3 years ago
Mike Fährmann
3cbbefd4ed
support 'filter' option for post processors ( #1460 )
3 years ago
Mike Fährmann
0abad8bc12
implement 'compile_expression()'
3 years ago
Mike Fährmann
da6806a161
fix job tests for Python 3.4 and 3.5
...
assert_called() and assert_not_called() got added in Python 3.6
3 years ago
Mike Fährmann
8fd8126117
fix ISO 639-1 code for Japanese
...
"jp" -> "ja"
3 years ago
Mike Fährmann
af9dba4684
add DataJob tests
3 years ago
Mike Fährmann
adf4d661b3
use '_extractor' info in UrlJobs
3 years ago
Mike Fährmann
1eabfa5c7a
[pillowfort] implement login with username & password ( #846 )
3 years ago
Mike Fährmann
559462789d
add some tests for job.py
3 years ago
Mike Fährmann
c5ca7905ce
add 'noop()' and 'identity()' functions
3 years ago
Mike Fährmann
bc868e7bb8
consider apparently long extensions as part of the filename
...
(#1516 )
3 years ago
Mike Fährmann
bdfcc9c4b1
update extractor test results
3 years ago
Mike Fährmann
387fe415d5
unescape items in text.split_html()
4 years ago
Mike Fährmann
78fd63b8f0
remove 'text.clean_xml()'
...
was not used anywhere
4 years ago
Mike Fährmann
8553b218d9
replace calls to 'os.path.splitext()' with 'str.rpartition()'
...
Makes functions who used it more than twice as fast
and we can get rid of an import as well.
4 years ago
Mike Fährmann
bff71cde80
implement 'util.unique_squence()'
4 years ago
Mike Fährmann
5f1a6ff6fa
remove unneeded 'TRAVIS_SKIP' from test_results.py
4 years ago
Mike Fährmann
8821dceb79
use __import__() to dynamically load modules
4 years ago
Mike Fährmann
36bf76fa44
update 'oauth:mastodon:<instance>' code
4 years ago
Mike Fährmann
91308140ec
make 'generate_token()' compatible with Python 3.4
4 years ago
Mike Fährmann
780b6adb91
rename 'generate_csrf_token()' to just 'generate_token()'
...
and add a 'size' argument
4 years ago