c8787647ed
add global WINDOWS bool
4 years ago
ece73b5b2a
make 'path' and 'keywords' available in logging messages
...
Wrap all loggers used by job, extractor, downloader, and postprocessor
objects into a (custom) LoggerAdapter that provides access to the
underlying job, extractor, pathfmt, and kwdict objects and their
properties.
__init__() signatures for all downloader and postprocessor classes have
been changed to take the current Job object as their first argument,
instead of the current extractor or pathfmt.
(#574 , #575 )
4 years ago
4a3d2405de
[postprocessor:ugoira] small optimization
...
Use tuples instead of lists when extending the list of
command-line arguments.
5 years ago
254f7c3999
implement a post-processor module to compare file versions
...
(#530 )
5 years ago
760b9b4db4
add remove_file() and remove_directory() helpers
...
these functions call os.unlink() or os.rmdir()
while catching and suppressing potential OSErrors
5 years ago
87c8b89ddd
[postprocessor:metadata] add 'directory' option ( #520 )
5 years ago
63e6993716
merge 'bypost' functionality into metadata postprocessor
5 years ago
c0b9ad678d
Separate metadata from handle_url into handle_metadata, commenting
5 years ago
c20bb5c338
Naming convention, as per travis.
5 years ago
cfc70a97ab
Added an additional channel for downloading the metadata of an entire post or gallery.
5 years ago
26d2334550
[postprocessor:metadata] rename 'format' to 'content-format'
...
Just to be consistent with the other 'extension-format' option name,
and only 'format' is also still accepted.
5 years ago
a412531451
[postprocessor:metadata] implement 'extension-format' option
...
closes #477
5 years ago
3fc1e12949
[postprocessor:metadata] filter private entries
...
i.e. keys starting with an underscore
5 years ago
9e88e7a344
[postprocessor:exec] improve ( #421 , #413 )
...
- add 'final' option
- include job status in pp finalization
- improve and extend documentation
5 years ago
ad81c07204
[postprocessor] match logger names of downloader modules
...
The logger name for a postprocessor object got changed to
"postprocessor.<module-name>" instead of just
"postprocessor"
5 years ago
03bc8adfc7
[postprocessor:exec] run after file moved to target location
...
(#421 )
5 years ago
35958bebd4
[postprocessor:exec] fix filename quoting on Windows ( #421 )
5 years ago
b06c372e4d
[postprocessor:exec] improve; add command-line option ( #421 )
5 years ago
5fcebb69c2
[postprocessor:ugoira] improve error messages ( #406 )
5 years ago
60c8e090da
[postprocessor:zip] fix archive names ( closes #397 )
...
Remove the trailing path separator introduced in 3284c62
5 years ago
3284c62f22
ensure PathFormat.directory ends with a path separator
...
... plus some other small optimizations
5 years ago
2495b99347
[postprocessor:classify] improve path generation ( fixes #138 )
...
It still doesn't work for converted ugoira animations thanks to how
those files are handled, but everything else, including files with
unknown or changing file extension, now works as it should.
5 years ago
5f8621b29d
improve output of active post processor modules
5 years ago
0bb873757a
update PathFormat class
...
- change 'has_extension' from a simple flag/bool to a field that
contains the original filename extension
- rename 'keywords' to 'kwdict' and some other stuff as well
- inline 'adjust_path()'
- put enumeration index before filename extension (#306 )
5 years ago
a90280f4e7
[postprocessor:zip] add 'mode' option ( #355 )
5 years ago
c41ff9441e
improve find() for downloaders and postprocessors
5 years ago
16c582aaf9
implement 'mtime' post-processor ( #332 )
...
This can set a file's modification time according to a UNIX timestamp
or a datetime object from its metadata.
5 years ago
a01f99728c
[postprocessor:zip] delete empty archives when done ( #316 )
5 years ago
b1db194c14
[reactor] update and improve
...
- split 'tags' into a list
- parse 'date' into a datetime object
- fix webm/mp4 URLs
5 years ago
523ebc9b0b
Fix serialization of 'datetime' objects in '--write-metadata'
...
Simplified universal serialization support in json.dump() can be achieved
by passing 'default=str', which was already the case in DataJob.run()
for -j/--dump-json, but not for the 'metadata' post-processor.
This commit introduces util.dump_json() that (more or less) unifies the
JSON output procedure of both --write-metadata and --dump-json.
(#251 , #252 )
5 years ago
7973419b54
restrict downloader and postprocessor module imports
6 years ago
8aba2bdebf
[postprocessor:metadata] add 'tags' and 'custom' modes ( #135 )
6 years ago
5f38ac9609
[postprocessor:exec] add a better error message ( #155 )
6 years ago
38500ad697
[postprocessor:metadata] first implementation ( #135 )
6 years ago
79f6755c60
[postprocessor:classify] handle missing "extension" ( #138 )
6 years ago
d3d7f01543
add 'prepare()' step for post-processors
...
This allows post-processors to modify the destination path before
checking if a file already exists.
6 years ago
09d2f3e5e7
[postprocessor:ugoira] improve libx264 detection
6 years ago
55f5c87160
[postprocessor:ugoira] add 'libx264-prevent-odd' option
...
A rather crude workaround for "width/height not divisible by 2"
errors when using libx264.
6 years ago
e9ae6fd080
improve downloader/postprocessor module loading
...
- handle arguments of any type without propagating an exception
- prevent potential security risk through relative imports
6 years ago
6ecb36d88c
[postprocessor:ugoira] add 'ffmpeg-output' option
6 years ago
02a4a67f6d
[postprocessor:ugoira] support danbooru sources
6 years ago
54a0d72dc8
[postprocessor:ugoira] improve frame rate handling
...
By default FFmpeg assumes a 25 FPS input frame rate, leading to dropped
frames if the source requires a higher frame rate than that.
This commit adds a `framerate` option (default "auto"), which allows to
automatically assign a (more or less) fitting frame rate based on
delays between ugoira frames and avoids dropped frames.
6 years ago
4eb94aca17
[postprocessor:ugoira] pass '-f' if not present
6 years ago
0c1c4557dd
[postprocessor:ugoira] add option for two-pass encoding
6 years ago
eb7a1f3b98
[pixiv] rework ugoira handling
...
Frame information now gets attached to the ZIP file's keyword dict
instead of being written to a separate text file.
6 years ago
613b692275
[postprocessor:ugoira] add a few options
...
- ffmpeg-location: path to the ffmpeg (or avconv) executable
- ffmpeg-args: additional command line args for ffmpeg
- extension: filename extension of the resulting video file
6 years ago
a444755979
[postprocessor] add 'ugoira' to convert pixiv animations to webm
6 years ago
eec081dd3e
[postprocessor:zip] delete directory ( #85 )
6 years ago
baccf8a958
improve postprocessor handling
...
- add pathfmt argument for __init__()
- add finalization step
- add option to keep or delete zipped files
6 years ago
2628911ba0
[pp:exec] add 'async' option
6 years ago
Mike Fährmann