4700051562
rework and extend input file processing ( #4732 )
...
- add 2 command-line options to modify input file contents
- -I/--input-file-comment
- -x/--input-file-delete
- implement InputManager class
- move code from util.py to __init__.py
(mainly to avoid import cycles)
10 months ago
72b18d701f
represent util.NONE as 'null' in JSON output
...
was '"None"' before
11 months ago
899df8f237
remove another '*' for keyword-only arguments
...
076380e0
1 year ago
26d06e0bb2
move executable check into util.py
2 years ago
d788e6c60c
implement 'globals' option
2 years ago
3d8777fbc1
move user agent string to util.py
2 years ago
56039d2456
add 'hash_md5' and 'hash_sha1' functions ( #3679 )
...
... to global eval namespace
2 years ago
d4232f3a8b
implement restarting an extractor ( #3338 )
2 years ago
1aae72773f
put argument init on separate lines
2 years ago
5503ac4d5e
replace json.dumps with direct calls to JSONEncoder.encode
2 years ago
dd884b02ee
replace json.loads with direct calls to JSONDecoder.decode
2 years ago
762a68996b
implement 'archive-pragma' option
2 years ago
d6793b2c7d
include request body in 'write-pages=all' output
2 years ago
d651d45239
implement specifying ranges in slice notation ( #918 , #2865 )
...
e.g.
- '1:101' or ':101' or ':101:' for files 1 to 100
- '1::2' or '::2' for every second file
- '1:101:5' or ':101:5' for files 1, 6, 11, ..., 91, 96
(the second argument specifies the first index NOT included)
2 years ago
3616adfc75
implement '--range' with Python ranges
2 years ago
1800bd7d14
allow '*-filter' options to be a list of expressions
2 years ago
43c211f1a7
extend and rename util.CustomNone
2 years ago
39d9c362e4
include 'http-metadata' in '-K' output
2 years ago
597b63d922
move git head functionality to function in util.py
2 years ago
455e34113e
Improve compatibility of DownloadArchive ( #3078 )
...
Other programs can add additional columns to the table without affecting
gallery-dl
2 years ago
a6e2d96dde
fix bug when processing input file comments ( #2808 )
...
and move 'parse_inputfile()' to util.py
2 years ago
dd3a6a9fd1
make 'enumerate_reversed()' work with generators ( #2795 )
2 years ago
a96f9d29b5
create missing directories for archive files ( #2597 )
2 years ago
c4b9f7bab8
update functions working with cookies.txt files
...
- rename
- load_cookiestxt -> cookiestxt_load
- save_cookiestxt -< cookiestxt_store
- in cookiestxt_load, add cookies directly to a cookie jar
instead of storing them in a list first
- other unnoticeable performance increases
2 years ago
4c8e28db82
avoid circular import in util.py
2 years ago
61afa973b7
use extended format strings for archive keys
2 years ago
ca3a364db7
fix build_duration_func() ( #2533 )
...
for extractors with request_interval_min > 0
2 years ago
7fe54bab2a
attempt to fix some issues with 'contains()' ( #2446 )
...
add a third argument that gets used
when the values o search are given as a string
2 years ago
d78a2c7163
re.escape() arguments for 'contains()' ( #2446 )
2 years ago
413b77757b
implement 'contains()' ( #2446 )
...
and add it to globals() in compiled expressions for --filter etc
3 years ago
29db716a63
implement 'datetime_to_timestamp()'
...
and rename 'to_timestamp()'
to the more descriptive 'datetime_to_timestamp_string()'
3 years ago
9bd27b1b8d
[postprocessor:metadata] implement archive options ( #2421 )
...
'archive', 'archive-format', and 'archive-prefix'
3 years ago
8295bc6d97
fix loading/storing cookies without domain
3 years ago
47cf05c4ab
refactor proxy handling code ( #2357 )
...
- allow gallery-dl proxy settings to overwrite environment proxies
- allow specifying different proxies for data extraction and download
- add 'downloader.proxy' option
- '-o extractor.proxy=–PROXY_URL -o downloader.proxy=null'
now has the same effect as youtube-dl's '--geo-verification-proxy'
3 years ago
00fbc942e5
add 'timedelta' class to the global namespace in filter expressions
...
(https://github.com/mikf/gallery-dl/issues/2308#issuecomment-1046135924 )
3 years ago
7958995398
[postprocessor:metadata] implement 'mtime' option ( #2307 )
3 years ago
64cf26eaf4
allow specifying sleep-* options as string
...
either as single value or as range: "3.5", "2.1 - 5.0"
3 years ago
010d65dcec
extend blacklist/whitelist syntax ( #2025 )
...
Each entry in such a list can now also include a subcategory
'<category>:<subcategory>'
and it is possible to use '*' or an empty string as placeholder
'*:<subcategory>', ':<subcategory>', '<category>:*'
For example
"blacklist": "imgur,*:tag,gfycat:user" or
"blacklist": ["imgur", "*:tag", "gfycat:user"]
will filter all 'imgur' extractors, all extractors with a 'tag'
subcategory (e.g. https://danbooru.donmai.us/posts?tags=bonocho ),
and all 'gfycat' user extractors.
3 years ago
c22ff97743
remove 'unit' argument from 'util.format_value()'
3 years ago
cad85640de
move 'util.PathFormat' into its own 'path' module
...
to prevent circular imports between 'formatter' and 'util'
3 years ago
74145467dd
move 'util.Formatter' into its own 'formatter' module
3 years ago
e69ee41f25
implement 'page-reverse' option ( #1854 )
3 years ago
c9e6693530
allow specifying a minimum/maximum for 'sleep-*' options ( #1835 )
...
for example '"sleep-request": [5.0, 10.0]' to wait between 5 and 10
seconds between each HTTP request
3 years ago
292fffc83c
add 'j' format string conversion
...
to convert to a JSON formatted string
3 years ago
d3eab417ed
implement a 'path-strip' option
3 years ago
2792ed6e4b
implement 'util.format_value()'
3 years ago
13d4045a8a
add 'archive-prefix' option ( #1711 )
3 years ago
9e42cd58ea
replace ChainPredicate class with 'functools.partial'
3 years ago
1b2f9050fb
rename all instances of 'kwds' to 'kwdict'
3 years ago
0179581340
add 'T' format string conversion ( #1646 )
...
to convert 'date'/datetime to timestamp
3 years ago
Mike Fährmann