5503ac4d5e
replace json.dumps with direct calls to JSONEncoder.encode
2 years ago
762a68996b
implement 'archive-pragma' option
2 years ago
f58215705a
add '-O/--postprocessor-option' command-line option ( #3565 )
2 years ago
b14b33f19e
Implement `version-metadata` option ( #3201 )
2 years ago
226d778294
do not try to fetch 'http-metadata' for ytdl URLs ( #3257 )
2 years ago
133412bd62
remove previous 'http-metadata' entries from kwdict
2 years ago
8124c16a50
split 'build_path' from 'set_filename' and 'set_extension'
...
Do not automatically build a new path
when setting file metadata or updating its extension.
2 years ago
39d9c362e4
include 'http-metadata' in '-K' output
2 years ago
c12a97bcde
[postprocessor] add 'post-after' event ( #3117 )
2 years ago
f037429fa4
attempt to improve '-K' output for lists
...
- use [N] instead if [] to indicate a Number needs to be placed there
- enumerate list items
2 years ago
88f8975ab9
Fix duplicated metadata bug ( #3033 )
2 years ago
8b1fe0bcf1
emit debug logging messages before calling time.sleep() ( #2982 )
2 years ago
7d1a95ada6
implement 'path-metadata' option ( #2734 )
2 years ago
5806a1851e
add --no-postprocessors command-line option ( #2725 )
2 years ago
44ffc017ea
remove useless 'tries' argument from out.success
2 years ago
64d3ad2e7a
detect circular references with -K ( fixes #2609 )
2 years ago
688d6553b4
replace calls to print() with stdout_write() ( #2529 )
2 years ago
71bba774da
respect 'output.private' in '-K/--list-keywords' output
3 years ago
9bd27b1b8d
[postprocessor:metadata] implement archive options ( #2421 )
...
'archive', 'archive-format', and 'archive-prefix'
3 years ago
bb3e182562
overhaul session initialization
...
- share adapter & connection pool across sessions with the same
ssl options, ssl ciphers, and source address
- simplify browser emulation to just a list of headers and ciphers
3 years ago
6e0a6c484f
apply SPECIAL_EXTRACTORS only for blacklist settings
...
as was the case before 010d65dc
3 years ago
010d65dcec
extend blacklist/whitelist syntax ( #2025 )
...
Each entry in such a list can now also include a subcategory
'<category>:<subcategory>'
and it is possible to use '*' or an empty string as placeholder
'*:<subcategory>', ':<subcategory>', '<category>:*'
For example
"blacklist": "imgur,*:tag,gfycat:user" or
"blacklist": ["imgur", "*:tag", "gfycat:user"]
will filter all 'imgur' extractors, all extractors with a 'tag'
subcategory (e.g. https://danbooru.donmai.us/posts?tags=bonocho ),
and all 'gfycat' user extractors.
3 years ago
cad85640de
move 'util.PathFormat' into its own 'path' module
...
to prevent circular imports between 'formatter' and 'util'
3 years ago
74145467dd
move 'util.Formatter' into its own 'formatter' module
3 years ago
c9e6693530
allow specifying a minimum/maximum for 'sleep-*' options ( #1835 )
...
for example '"sleep-request": [5.0, 10.0]' to wait between 5 and 10
seconds between each HTTP request
3 years ago
d79bcb6236
allow extractors to register a 'finalize()' method
3 years ago
72c0cd30c7
do not return with a nonzero exit status when no results found
...
also change loglevel from 'warning' to 'info'
(#1789 )
3 years ago
bd08ee2859
remove most 'yield Message.Version' statements
...
only leave them in oauth.py as noop results
3 years ago
bdfdabf498
show warning if extractor doesn't yield any results ( #1759 )
3 years ago
d320ee6251
implement a 'fallback' option ( closes #1770 )
3 years ago
1b2f9050fb
rename all instances of 'kwds' to 'kwdict'
3 years ago
b9783403d9
add 'url-metadata' option ( #1659 , #1073 )
3 years ago
e95f99882f
extend 'parent-metadata' functionality ( #1687 , #1651 , #1364 )
3 years ago
64986f9435
fix depth counter in UrlJob
...
regression from adf4d661
3 years ago
83fc4c1098
update post processor config capabilities
...
This change makes it possible to specify just the name of a post processor
in the "postprocessors" list instead of a dict with all of its options.
The options for it will then be taken from inside the "postprocessor"
block similar to "extractor", "downloader", or "output" blocks.
This makes it possible to for example override the default settings for
--write-metadata by specifying a custom "metadata" block, or to set a
custom post processor block ("cbz") and then use it by referencing just
its name in "postprocessors" lists.
{
"postprocessor":
{
"metadata": {
"name": "metadata",
"event": "post",
"filename": "{tweet_id|post_id|id}.json"
},
"cbz": {
"name" : "zip",
"compression": "store",
"extension" : "cbz"
}
}
}
3 years ago
3cbbefd4ed
support 'filter' option for post processors ( #1460 )
3 years ago
adf4d661b3
use '_extractor' info in UrlJobs
3 years ago
b50b8e6cf4
refactor applying 'parent-…' options
3 years ago
7ab8374385
add 'parent-skip' option ( #1399 )
3 years ago
c693db5b1a
add '"skip": "terminate"' option
...
Stops not only the current extractor/job,
but all parent extractors/jobs as well.
3 years ago
c5ca7905ce
add 'noop()' and 'identity()' functions
3 years ago
5b4da4b4bf
reorder config access in Job constructor
...
(#1111 )
3 years ago
b4ed7cb961
fix 'category-transfer' ( #1111 )
...
broken since commit 055c32e0
3 years ago
a86ffb04bb
add 'output.fallback' option
...
to enable/disable fallback URLs for -g/--get-urls
3 years ago
a75e485461
add archive format to InfoJob output ( #875 )
4 years ago
bf241811dd
allow '_extractor' fields to be None or empty
4 years ago
23641742a3
improve 'parent-directory' ( #1364 )
...
Allow forwarding metadata from the top-level extractor to all children
if 'parent-directory' is enabled for all extractors along the way.
For example 'reddit' -> 'gfycat' -> 'redgifs'
4 years ago
df94182e11
implement 'parent-metadata' option ( #1364 )
...
experimental, might not work as expected, etc.
4 years ago
b6719becf1
ensure '-s/--simulate' always prints filenames ( #1360 )
...
by assuming a potentially wrong filename extension in cases where the
correct one would only get known after a download started
4 years ago
c963741860
add '-E/--extractor-info' command-line option ( #875 )
4 years ago
Mike Fährmann