continuation of d37e7f48
but more versatile and extendable
Example:
"actions": [
# change debug messages to info
["debug", "level ~info"],
# change exit status to a non-zero value
["info:^No results for", "status |= 1"],
# exit with status 2 on 429
["warning:429", "exit 2"],
# restart extractor when no cookies found
["warning:^[Nn]o .*cookies", "restart"]
]
Very much a work in progress.
At the moment, it allows to
- wait and restart an extractor (#3338)
- change the exit code (#3630)
- change the log level of a logging message
based on the contents of a logging message
work in progress
the same output as produced by "mode": "color" can be achieved with
"output": {
"mode": {
"start" : "{}",
"success": "\r\u001b[1;32m{}\u001b[0m\n",
"skip" : "\u001b[2m{}\u001b[0m\n",
"progress" : "\r{0:>7}B {1:>7}B/s ",
"progress-total": "\r{3:>3}% {0:>7}B {1:>7}B/s "
}
}
to make 'output.shorten' work correctly, it is necessary to manually
specify the number of extra characters:
"start" : [12, "Downloading {}"]
Wrap all loggers used by job, extractor, downloader, and postprocessor
objects into a (custom) LoggerAdapter that provides access to the
underlying job, extractor, pathfmt, and kwdict objects and their
properties.
__init__() signatures for all downloader and postprocessor classes have
been changed to take the current Job object as their first argument,
instead of the current extractor or pathfmt.
(#574, #575)
This allows for stuff like "{extractor.url}" and "{extractor.category}"
in logging format strings.
Accessing 'extractor' and 'job' in any way will return "None" if those
fields aren't defined, i.e. in general logging messages.
- safeprint() was used to print values which might have caused a
UnicodeEncodeError, but that is no longer necessary (0381ae5)
- errors are now handled via logging output (f94e370)
Mike Fährmann