b6706b373a
[downloader:http] add signature checks for some formats
...
also add the MIME type for .obj files
2 years ago
71d3143c35
fix bug in test_extractors.py
...
pattern matching tests would succeed
if there is exactly one match
but for the wrong extractor
2 years ago
fa144f38ed
[ytdl} fix dfe4f00c for legacy yt-dlp
2 years ago
dfe4f00ca2
[ytdl] update for yt-dlp changes
2 years ago
d651d45239
implement specifying ranges in slice notation ( #918 , #2865 )
...
e.g.
- '1:101' or ':101' or ':101:' for files 1 to 100
- '1::2' or '::2' for every second file
- '1:101:5' or ':101:5' for files 1, 6, 11, ..., 91, 96
(the second argument specifies the first index NOT included)
2 years ago
3616adfc75
implement '--range' with Python ranges
2 years ago
1800bd7d14
allow '*-filter' options to be a list of expressions
2 years ago
43c211f1a7
extend and rename util.CustomNone
2 years ago
42481aed59
[formatter] implement 'S' format specifier ( #3266 )
...
to Sort lists
2 years ago
6e08ad26f7
update downloader tests
2 years ago
05255f5be0
add 'default' argument to 'text.extr()'
2 years ago
8124c16a50
split 'build_path' from 'set_filename' and 'set_extension'
...
Do not automatically build a new path
when setting file metadata or updating its extension.
2 years ago
eb33e6cf2d
add 'text.extr()'
...
a stripped-down version of text.extract() that
- always returns a string (like 'extract_from')
- only returns a string
- does not deal with 'pos' arguments
- is ~20% faster
2 years ago
460095adca
update downloader tests
2 years ago
f037429fa4
attempt to improve '-K' output for lists
...
- use [N] instead if [] to indicate a Number needs to be placed there
- enumerate list items
2 years ago
062ef238a6
add support for aibooru (using danbooru extractor) ( #3075 )
2 years ago
b57015cf0a
[postprocessor:metadata] assume 'mode: custom' when format is set
...
{"name": "metadata", "format": "foobar"}
will now implicitly use mode:custom and no longer mode:json like before
2 years ago
f7ba19a1c0
[nana] add 'nana' extractors ( #2967 )
2 years ago
b36125333f
[postprocessor:zip] implement 'files' option ( #2872 )
2 years ago
67bad04dda
[formatter] add 'g' conversion to sluGify a string ( #2410 )
2 years ago
6990ad0ba8
[formatter] do NOT apply :J to strings ( #2833 )
2 years ago
c0051d7d4c
fix test
2 years ago
dd3a6a9fd1
make 'enumerate_reversed()' work with generators ( #2795 )
2 years ago
0c73914848
[postprocessor:metadata] implement 'mode: modify' ( #2640 )
2 years ago
f3de6b7a87
[postprocessor:metadata] implement 'mode: delete' ( #2640 )
2 years ago
9704c04172
[postprocessor:zip] ensure target directory exists ( #2758 )
2 years ago
74865adae5
implement 'format-separator' option ( #2737 )
...
a global option, that servers as a workaround for shortcomings due to
lack of a proper format string parser
2 years ago
117eeefda0
[postprocessor:mtime] add 'value' option ( #2739 )
2 years ago
90ae48c40c
[formatter] implement 'O' format specifier ( #2736 )
...
to apply a UTC offset to 'date' values and other datetime objects
2 years ago
04bed1eba3
[formatter] allow for custom "format" functions ( #2721 )
2 years ago
54525d2e21
[formatter] implement slice operator as format specifier
...
this allows using a slice operator alongside other (special) format
specifiers like J, to first join list elements to a string and then
trimming that with a slice.
{tags:J, /[:50]}
2 years ago
241e82e18d
[horne] add support for horne.red ( #2700 )
2 years ago
42525cfe8d
fix '{…!j}' for otherwise non-serializable types (##2624)
...
like 'datetime'
2 years ago
5b43faffed
[postprocessor:metadata] write to stdout by setting filename to "-"
...
(#2624 )
2 years ago
6ad39f2b68
add ytdl tests
...
they only run when youtube-dl or yt-dlp are installed,
i.e. if __import__("<ytdl-package>") succeeds
2 years ago
688d6553b4
replace calls to print() with stdout_write() ( #2529 )
2 years ago
f3408a9d92
implement string literals in replacement fields
...
- either {_lit[foo]} or {'foo'}
- useful as alternative for empty metadata fields: {title|'no title'}
- due to using '_string.formatter_field_name_split()' to parse format
strings, using certain characters will result in an error: [].:!
2 years ago
c4b9f7bab8
update functions working with cookies.txt files
...
- rename
- load_cookiestxt -> cookiestxt_load
- save_cookiestxt -< cookiestxt_store
- in cookiestxt_load, add cookies directly to a cookie jar
instead of storing them in a list first
- other unnoticeable performance increases
2 years ago
ca3a364db7
fix build_duration_func() ( #2533 )
...
for extractors with request_interval_min > 0
2 years ago
7fe54bab2a
attempt to fix some issues with 'contains()' ( #2446 )
...
add a third argument that gets used
when the values o search are given as a string
2 years ago
d78a2c7163
re.escape() arguments for 'contains()' ( #2446 )
2 years ago
413b77757b
implement 'contains()' ( #2446 )
...
and add it to globals() in compiled expressions for --filter etc
3 years ago
e7b30866d0
[postprocessor:mtime] fix timestamps from datetime objects ( #2307 )
...
'datetime.timestamp()', which got used to convert datetime objects to
POSIX timestamps, assumes naive datetimes represent LOCAL time, while
datetimes in 'date' metadata fields represent UTC time.
Ref: https://docs.python.org/3/library/datetime.html#datetime.datetime.timestamp
> Naive datetime instances are assumed to represent local time
> you can obtain the POSIX timestamp by … calculating the timestamp directly
3 years ago
29db716a63
implement 'datetime_to_timestamp()'
...
and rename 'to_timestamp()'
to the more descriptive 'datetime_to_timestamp_string()'
3 years ago
8295bc6d97
fix loading/storing cookies without domain
3 years ago
500a479026
fix a third(!) bug in _check_cookies() ( #2372 )
...
turns out tests are worthless if you get em wrong ...
3 years ago
cf44aba333
[formatter] allow evaluating f-string literals
...
by starting a format string with '\fF'.
This was technically already possible with '\fE',
but this makes it a bit more convenient.
3 years ago
94452761ed
fix cookies tests
3 years ago
bddcec49f1
implement 'text.root_from_url()'
...
use domain from input URL for kemono
3 years ago
f5b2b9333f
fix another bug in _check:cookies ( #2160 )
...
regression introduced in ed317bfc
3 years ago
563bd0ecf4
[danbooru] inherit from BaseExtractor
...
- merge danbooru and e621 code
- support booru.allthefallen.moe (closes #2283 )
- remove support for old e621 tag search URLs
3 years ago
b5b4f5a168
use 'build_extractor_filter' in test_results.py
3 years ago
64cf26eaf4
allow specifying sleep-* options as string
...
either as single value or as range: "3.5", "2.1 - 5.0"
3 years ago
010d65dcec
extend blacklist/whitelist syntax ( #2025 )
...
Each entry in such a list can now also include a subcategory
'<category>:<subcategory>'
and it is possible to use '*' or an empty string as placeholder
'*:<subcategory>', ':<subcategory>', '<category>:*'
For example
"blacklist": "imgur,*:tag,gfycat:user" or
"blacklist": ["imgur", "*:tag", "gfycat:user"]
will filter all 'imgur' extractors, all extractors with a 'tag'
subcategory (e.g. https://danbooru.donmai.us/posts?tags=bonocho ),
and all 'gfycat' user extractors.
3 years ago
af6424f398
allow testing metadata in list elements
3 years ago
3842cdcd8f
[formatter] implement 'D' format specifier
...
To be able to parse any string into a 'datetime' object
and format it as necessary.
Example:
{created_at:D%Y-%m-%dT%H:%M:%S%z}
->
"2010-01-01 00:00:00"
{created_at:D%Y-%m-%dT%H:%M:%S%z/%b %d %Y %I:%M %p}
->
"Jan 01 2010 12:00 AM"
with 'created_at' == "2010-01-01T01:00:00+0100"
3 years ago
2ab190ce08
add tests for special format strings
3 years ago
46e17c5e61
support accessing the current local datetime in format strings
...
{_now}, {_now:%Y-%m-%d}, etc
(#1968 )
3 years ago
38193dba46
support accessing environment variables in format strings ( #1968 )
...
{_env[HOME]} to get the value of $HOME
every other format string feature is supported as well
3 years ago
f2d6b3e6b4
run tests without using 'nose'
...
run_tests.sh -> run_tests.py
3 years ago
12fc646c53
fix filename formatting tests
3 years ago
e0bdacd932
[fappic] add 'image' extractor ( closes #1898 )
3 years ago
c22ff97743
remove 'unit' argument from 'util.format_value()'
3 years ago
cad85640de
move 'util.PathFormat' into its own 'path' module
...
to prevent circular imports between 'formatter' and 'util'
3 years ago
74145467dd
move 'util.Formatter' into its own 'formatter' module
3 years ago
9377543162
[mastodon] add 'following' extractor ( #1891 )
3 years ago
bd845303ad
implement a way to shorten filenames with east-asian characters
...
(#1377 )
Setting 'output.shorten' to "eaw" (East-Asian Width) uses a slower
algorithm that also considers characters with a width > 1.
3 years ago
292fffc83c
add 'j' format string conversion
...
to convert to a JSON formatted string
3 years ago
bb6a130942
automatically set required DDoS-GUARD cookies ( #1779 )
...
for kemono.party and seiso.party
3 years ago
2792ed6e4b
implement 'util.format_value()'
3 years ago
9e42cd58ea
replace ChainPredicate class with 'functools.partial'
3 years ago
36ac2197db
[ytdl] add extractor for sites supported by youtube-dl
...
(#1680 , #878 )
Can be used by prefixing any URL with 'ytdl:',
or by setting 'extractor,ytdl.enabled' to 'true'.
3 years ago
64240c8d42
[imagevenue] fix extraction
...
(closes #1677 )
3 years ago
0179581340
add 'T' format string conversion ( #1646 )
...
to convert 'date'/datetime to timestamp
3 years ago
f74cf52e2b
[seisoparty] add 'user' and 'post' extractors ( #1635 )
3 years ago
759735fb02
[kemonoparty] fix 'username' extraction ( fixes #1652 )
...
The site's <title> content changed from
<title>NAME | Kemono</title>
to
<title>
NAME | Kemono
</title>
3 years ago
07c8adbd8b
[mangadex] implement login with username & password ( #1535 )
3 years ago
4a747a31a3
[postprocessor:metadata] handle dicts in mode;tags ( fixes #1598 )
3 years ago
3cbbefd4ed
support 'filter' option for post processors ( #1460 )
3 years ago
0abad8bc12
implement 'compile_expression()'
3 years ago
da6806a161
fix job tests for Python 3.4 and 3.5
...
assert_called() and assert_not_called() got added in Python 3.6
3 years ago
8fd8126117
fix ISO 639-1 code for Japanese
...
"jp" -> "ja"
3 years ago
af9dba4684
add DataJob tests
3 years ago
adf4d661b3
use '_extractor' info in UrlJobs
3 years ago
1eabfa5c7a
[pillowfort] implement login with username & password ( #846 )
3 years ago
559462789d
add some tests for job.py
3 years ago
c5ca7905ce
add 'noop()' and 'identity()' functions
3 years ago
bc868e7bb8
consider apparently long extensions as part of the filename
...
(#1516 )
3 years ago
bdfcc9c4b1
update extractor test results
3 years ago
387fe415d5
unescape items in text.split_html()
4 years ago
78fd63b8f0
remove 'text.clean_xml()'
...
was not used anywhere
4 years ago
8553b218d9
replace calls to 'os.path.splitext()' with 'str.rpartition()'
...
Makes functions who used it more than twice as fast
and we can get rid of an import as well.
4 years ago
bff71cde80
implement 'util.unique_squence()'
4 years ago
5f1a6ff6fa
remove unneeded 'TRAVIS_SKIP' from test_results.py
4 years ago
8821dceb79
use __import__() to dynamically load modules
4 years ago
36bf76fa44
update 'oauth:mastodon:<instance>' code
4 years ago
91308140ec
make 'generate_token()' compatible with Python 3.4
4 years ago
780b6adb91
rename 'generate_csrf_token()' to just 'generate_token()'
...
and add a 'size' argument
4 years ago
0fdaea00a3
[postprocessor:metadata] sanitize filenames
4 years ago
aac00a2024
add 'd' conversion for format strings
...
to convert a timestamp to a formattable 'datetime' object.
For example '{created_at!d:%Y-%m-%d}'
transforms the timestamp in 'created_at' into a 'datetime' object
and then formats its content using '%Y-%m-%d' as template.
1262304000 -> datetime(2010, 1, 1) -> "2010-01-01"
4 years ago
ClosedPort22