e348da7a06
[poringa] add support
9 months ago
74c225f94e
[bato] add support
9 months ago
f9544194c0
[paheal] restore 'extension' metadata ( #4976 )
9 months ago
77d46e6f0c
[lynxchan] update 'bbw-chan' domain ( #4970 )
9 months ago
108c978073
merge #4919 : [postmill] add support ( #4917 )
9 months ago
2a60645095
[deviantart] set 'is_original' for intermediary URLs to 'false'
9 months ago
01bb75f6cb
merge #4945 : {shimmie2[ support 'rule34hentai.net' ( #861 , #4789 )
9 months ago
79e4606893
[rule34hentai] cleanup
...
- fix using 'self._posts_rule34hentai'
- fix 'file_url' for posts
- update docs/supportedsites
- add tests
9 months ago
627ed794a2
[danbooru] provide 'tags' as list ( #4942 )
...
keep the old 'tag_string' values around, similar to sankaku
a lot of repeat code ...
would be a lot less bad if "".split(" ") returned an empty list
9 months ago
99aa923322
[inkbunny] improve '/submissionsviewall.php' patterns ( #4934 )
...
allow 'mode=…' to be in any position
don't require it to be somewhere in the middle
9 months ago
3f9c113d78
[mastodon] Support non-numeric status IDs ( #4936 )
9 months ago
2852404e49
[inkbunny] add 'unread' extractor ( #4934 )
9 months ago
a37b7759bc
[myhentaigallery] recognize '/g/' URLs ( #4920 )
9 months ago
fbe14a2745
[postmill] add support
9 months ago
bf74eb5c46
merge #4886 : [urlgalleries] add 'gallery' extractor ( #919 , #1184 , #2905 )
10 months ago
ade93c5397
[urlgalleries] add tests
10 months ago
4eb3590103
[nijie] fix image URLs of multi-image posts ( #4876 )
10 months ago
c83fbe6c2d
merge #4855 : [nitter] fix video extraction ( #4853 )
10 months ago
1137d72d48
[tests] skip test_init for BaseExtractor classes without instances
10 months ago
625e94fa7d
update extractor test results
...
still not everything, but good enough for now
10 months ago
1e9bacd169
[nitter] fix video extraction
10 months ago
95c1dfb089
[tests] swap assertEqual argument order
...
before this, it would show test failures as
+ test value
- extracted value
when it should be the other way round
10 months ago
bdb3ce7217
[foolslide] remove 'powermanga.org'
10 months ago
f9dac43be9
[warosu] fix file URLs
10 months ago
645b4627ef
[sankaku] update URL patterns
10 months ago
119755a5a3
[tests] implement skipping/failing tests when pressing ctrl+c
10 months ago
1ae43d8123
merge #4841 : [fapello] support '.su' TLD ( #4840 )
10 months ago
e1404827a6
[pixeldrain] add 'file' and 'album' extractors ( #4839 )
10 months ago
2402162e8a
[fapello] support '.su' TLD
10 months ago
725c8dd55a
[tmohentai] 'categories' -> 'genres'
...
quite likely that the site meant 'genres' by "Genders"
10 months ago
ce7c4cb544
merge #4832 : [tmohentai] add 'gallery' extractor ( #4808 )
10 months ago
c4a201ed42
[tmohentai] simplify + tests
10 months ago
e17a48fe56
[blogger] inherit from BaseExtractor
...
- support www.micmicidol.club (#4759 )
10 months ago
0fa85360a0
merge #4812 : [erome] add 'count' metadata field
10 months ago
a43cf78bb7
[erome] tests
10 months ago
07cb584231
[behance] add 'modules' option ( #4799 )
10 months ago
ea78f67860
[downloader:http] skip files not passing filesize-min/-max ( #4821 )
...
instead of failing the download
10 months ago
3f591d5a4e
[mastodon] update test results
10 months ago
6402f2950f
[pp:metadata] ignore non-string tag values ( #4764 )
11 months ago
007c433677
[patreon] support 'id:<campaign_id>' in place of a user name
...
https://patreon.com/id:12345
… and remove 'campaign-id' config option
11 months ago
43a3d93467
merge #4755 : [twitter] recognize fixupx.com URLs
11 months ago
cdf77e326f
[twitter] add test for fixupx.com
11 months ago
fc8f86bf24
[hitomi] recognize 'imageset' gallery URLs ( #4756 )
11 months ago
72b18d701f
represent util.NONE as 'null' in JSON output
...
was '"None"' before
11 months ago
68e72a836c
[exhentai] fix extraction ( #4730 )
...
- update to new API response layout
- use proper API server URL
- fix 'filesize' metadata
11 months ago
fd8f58ad76
[behance] unescape embed URLs ( #4742 )
11 months ago
c9a2be36d4
[sankaku] support '/posts/' tag search URLs ( #4740 )
11 months ago
218295a4c6
[twitter] fix avatars without 'date' information ( #4696 )
11 months ago
d0effcae20
[kemonoparty] add 'revision_index' metadata field ( #4727 )
11 months ago
3bbaa875f1
[kemonoparty] fix parsing of non-standard 'dates' ( #4676 )
11 months ago
a09df34bcf
merge #4714 : [4archive] add 'thread' and 'board' extractors
...
(#1262 , #2418 , #4400 , #4710 )
11 months ago
acb713b95a
[4archive] update
11 months ago
6766877524
merge #4693 : [reddit] support Reddit Mobile share links
11 months ago
1042278bec
[misskey] support 'misskey.design' ( #4713 )
11 months ago
c0714d5585
[4archive] add 'thread' and 'board' extractors
11 months ago
b68aad3dab
[reddit] implement Reddit Mobile share links
11 months ago
7958ab1946
[newgrounds] support 'imageData' files ( #4642 )
11 months ago
b52fd91ac6
[sankaku] support '/posts/' URLs ( #4688 )
11 months ago
b2c3db3e24
[bunkr] add extractor for media URLs ( #4684 )
11 months ago
6e830ffc9e
[kemonoparty] support post searches ( #3385 , #4057 )
11 months ago
aaf539009b
[kemonoparty] initial support for post revisions ( #4498 , #4597 )
...
- single revision
https://kemono.party/SERVICE/user/12345/post/12345/revision/12345
- all revisions
https://kemono.party/SERVICE/user/12345/post/12345/revisions
11 months ago
174191cb79
[kemonoparty] restore discord pagination ( #4676 )
11 months ago
c9a976d8a6
[kemonoparty] various updates and fixes ( #4676 , #4681 )
...
- fix pagination
- fix 'date' metadata
- fix discord channel API endpoint
11 months ago
bfdc07632a
[deviantart] expand nested comment replies ( #4653 )
11 months ago
9bc5ad4784
[tests] implement 'len:'
11 months ago
a1977a698e
[tests] fix spurious failures in '_assert_isotime()'
11 months ago
390d14dbcc
[chevereto] support 'img.kiwi' and 'deltaporno.com' ( #4664 , #1381 )
11 months ago
727c8eec6c
merge #4667 : [redgifs] fix 'niches' extraction ( #4666 )
11 months ago
2911ed1240
[chevereto] add generic extractors ( #4664 )
...
- support jpgfish
- support pixl.li / pixl.is (#3179 , #4357 )
11 months ago
db3363ac0b
[redgifs] fix 'niches' extraction
11 months ago
ade8347ead
[kemonoparty] fix DM dates
11 months ago
6dfe200ae4
[kemonoparty] support discord URLs with channel IDs ( #4662 )
11 months ago
c6a3892210
[imgbb] update username extraction ( #4626 )
11 months ago
13ce3a9acb
[warosu] fix extraction ( #4634 )
11 months ago
c4c4e4d2f4
[newgrounds] improve 'art-image' extraction ( #4642 )
...
- download files in original resolution
- replace .webp with extension of first file
11 months ago
833dce141f
[fantia] add 'content_count' and 'content_num' metadata fields ( #4627 )
11 months ago
2d41702762
[deviantart] implement '"group": "skip"' ( #4630 )
11 months ago
a9c3442d4e
[deviantart] add a couple 'deactivated account' test URLs
11 months ago
2974b8e3c8
[moebooru] add 'metadata' option ( #4646 )
...
for extended 'pool' metadata
11 months ago
67ba4ee842
[pp:exec] support more replacement fields for '--exec' ( #4633 )
...
- {_directory}
- {_filename}
- {_path} (alias for {})
12 months ago
9a008523ac
[hentaifoundry] fix '.swf' file downloads ( #4641 )
12 months ago
15f940819b
[newgrounds] support 'art-image' files ( #4642 )
12 months ago
efaab4fbfa
[twitter] fix crash due to missing 'source' ( #4620 )
...
regression caused by 06aaedde
12 months ago
84fbbd96aa
[shimmie2] remove 'meme.museum'
12 months ago
0b150d45db
[tests] add 'msg' arguments to assert statements
12 months ago
27da3f2958
[tests] re-implement filtering by basecategory
12 months ago
c7bd9925d9
[tests] use fallback URLs for content tests ( #3163 )
12 months ago
b92645cd37
[bunkr] fix extraction ( #4514 , #4532 , #4529 , #4540 )
12 months ago
bd3f7a5bbc
[tests] support one regex per URL for #pattern
12 months ago
0c5d8b1505
[deviantart] re-add 'quality' option and 'intermediary' transform
1 year ago
dbd820d7c5
[tests] allow checking for exact URL results
1 year ago
642998504d
[tests] support 'range()' for #count and metadata checks
1 year ago
1e31fce37b
[pillowfort] support '/tagged/' URLs ( #4570 )
1 year ago
1d2fd0b831
[pillowfort] extract 'b2_lg_url' media ( #4570 )
1 year ago
50e2ebaff0
[danbooru] support 'donmai.moe' URLs
1 year ago
918ba4f847
[redgifs] match gfycat image URLs ( #4558 )
1 year ago
2cd801232b
fix --range causing crashes ( #4557 )
...
regression caused by a383eca7
1 year ago
27ec653991
fix bug in test_init and update example URLs
1 year ago
24a1d46391
[mastodon] support '/@USER/following' URLs
...
Previously, only '/users/USER/following' got matched.
1 year ago
ac00d47a16
update test/test_results.py
1 year ago
65b6011cc5
update test/test_extractor.py
1 year ago
a833c244c8
add exported extractor results
1 year ago
93a7a89cf6
[formatter] use value of last alternative ( #4492 )
...
fixes {fieldname|''} evaluating to the value of 'keywords-default'
instead of an empty string
1 year ago
f2de70f254
[gfycat] remove module
1 year ago
d319777a24
[tests] skip 'test_init_ytdl' on Python<3.6
...
It passes without error in a Python 3.4/3.5 venv on my own machine,
but fails for some inexplicable reason on Github Actions.
1 year ago
0ef1fcab20
[postprocessor] update 'finalize' events
...
Add 'finalize-error' and 'finalize-success' events that trigger
depending on whether error(s) did or did not happen.
'finalize' itself now always triggers regardless of error status.
(was supposed to have the same behavior as the new 'finalize-success')
1 year ago
d50c312ff0
prevent test failure when there's no 'ytdl' module ( #4364 )
...
split of ytdl into its own test function and
skip it when there's an ImportError similar to test_ytdl.py
1 year ago
48ef062867
fix issues with 'Extractor.finalize()'
...
- prevent crash in InstagramUserExtractor (#4359 )
- call it at the end of every DownloadJob
- add it to tests
1 year ago
255d08b79e
add test for 'Extractor.initialize()' ( #4359 )
1 year ago
a383eca7f6
decouple extractor initialization
...
Introduce an 'initialize()' function that does the actual init
(session, cookies, config options) and can called separately from
the constructor __init__().
This allows, for example, to adjust config access inside a Job
before most of it already happened when calling 'extractor.find()'.
1 year ago
f0203b7559
[postprocessor:python] add tests
1 year ago
d97b8c2fba
consistent cookie-related names
...
- rename every cookie variable or method to 'cookies_*'
- simplify '.session.cookies' to just '.cookies'
- more consistent 'login()' structure
1 year ago
c5565f79f7
merge #4096 : [danbooru] add support for booru.borvar.art instance
1 year ago
63326e3168
[danbooru] add tests for booruvar
1 year ago
5171d8975c
[E621] support 'e6ai.net' ( #4320 )
1 year ago
7444fc125b
[gfycat] implement login support ( #3770 , #4271 )
...
For the record: '/webtoken' and '/weblogin' are not the same ...
1 year ago
25c5a6ffcb
no f-strings
1 year ago
ec64cbefeb
[postprocessor:exec] add tests
1 year ago
ce93c460a6
[formatter] implement 'H' conversion ( #4164 )
...
to remove HTML tags and unescape HTML entities
1 year ago
deff3b434d
[vipergirls] implement login support ( #4166 )
1 year ago
69865dcc05
[formatter] implement slicing strings as bytes ( #4087 )
...
prefixing a slice '[10:30]' with a lowercase b '[b10:30]' encodes
the string to bytes in filesystem encoding before applying the slice
1 year ago
df11214281
[ytdl] improve --xff/--geo-bypass detection ( #3989 )
...
check if --xff is supported in a try-except block
and select expected results accordingly
1 year ago
aa731c4298
[ytdl] run yt-dlp tests with latest code from master ( #3989 )
...
Only use PyPI version for Python 3.6, since that's no longer supported
by the current codebase.
1 year ago
43f4bd9faa
[ytdl] fix tests
...
tests pass with latest Git HEAD, but not with the current PyPI version
1 year ago
61a65d5bb9
[ytdl] fix crash due to --geo-bypass deprecation ( #3975 )
1 year ago
a96745368e
"fix" tests on Python 3.4 and 3.5
...
can't rely on dict insertion order
1 year ago
3905f05f00
[postprocessor:metadata] support putting keys in quotes
...
for mode 'modify' and 'delete'
based on fe41a2b1
1 year ago
7459e4abce
[postprocessor:metadata] fix traversing more than 1 level deep
...
for mode 'modify' and 'delete'
1 year ago
2edcdee32f
[downloader:http] add MIME type and signature for .heic files
...
(#3915 )
https://github.com/strukturag/libheif/issues/83
1 year ago
082d55de16
fix circular reference detection for -K
2 years ago
2ab66ad899
update -K output to include quotes around keys
2 years ago
fe41a2b159
[formatter] support putting keys in quotes
...
i.e. obj["key"] or obj['key']
as in f-strings
2 years ago
46fdf46f21
[formatter] support loading an f-string from a template file
...
"\fTF ~/path/to/file.txt"
2 years ago
1a4d4a799b
[formatter] support filesystem paths for \fM
2 years ago
00f0233b28
[postprocessor:metadata] add 'skip' option ( #3786 )
2 years ago
8f8b4de0e8
[ytdl] fix '--parse-metadata' ( #3663 )
2 years ago
7610d9cf82
merge #3675 : [pixiv] fix --write-tags for '"tags": "original"'
2 years ago
83e7a25b6b
extend OAuth tests
2 years ago
d788e6c60c
implement 'globals' option
2 years ago
56039d2456
add 'hash_md5' and 'hash_sha1' functions ( #3679 )
...
... to global eval namespace
2 years ago
e1df7f73b1
[deviantart] add 'search' extractor
...
(#538 , #1264 , #2954 , #2970 , #3577 )
Requires login to fetch any results, since the API endpoint raises an
error for not logged in requests.
TODO: parse HTML search results
2 years ago
38a6389e2c
Fix lint.
2 years ago
56cbae92ec
Use more pythony naming.
2 years ago
8e2ba4f32e
Add test.
2 years ago
dd884b02ee
replace json.loads with direct calls to JSONDecoder.decode
2 years ago
b7337d810e
[postprocessor:metadata] add 'sort' and 'separators' options
2 years ago
3436c6b117
[postprocessor:metadata] speed up JSON encoding
2 years ago
925b467496
split e621 from danbooru module ( #3425 )
2 years ago
c2bc70593e
implement ability to load external extractor classes
...
- -X/--extractors
- extractor.module-sources
2 years ago
b6706b373a
[downloader:http] add signature checks for some formats
...
also add the MIME type for .obj files
2 years ago
71d3143c35
fix bug in test_extractors.py
...
pattern matching tests would succeed
if there is exactly one match
but for the wrong extractor
2 years ago
fa144f38ed
[ytdl} fix dfe4f00c for legacy yt-dlp
2 years ago
dfe4f00ca2
[ytdl] update for yt-dlp changes
2 years ago
d651d45239
implement specifying ranges in slice notation ( #918 , #2865 )
...
e.g.
- '1:101' or ':101' or ':101:' for files 1 to 100
- '1::2' or '::2' for every second file
- '1:101:5' or ':101:5' for files 1, 6, 11, ..., 91, 96
(the second argument specifies the first index NOT included)
2 years ago
3616adfc75
implement '--range' with Python ranges
2 years ago
1800bd7d14
allow '*-filter' options to be a list of expressions
2 years ago
43c211f1a7
extend and rename util.CustomNone
2 years ago
42481aed59
[formatter] implement 'S' format specifier ( #3266 )
...
to Sort lists
2 years ago
6e08ad26f7
update downloader tests
2 years ago
05255f5be0
add 'default' argument to 'text.extr()'
2 years ago
8124c16a50
split 'build_path' from 'set_filename' and 'set_extension'
...
Do not automatically build a new path
when setting file metadata or updating its extension.
2 years ago
eb33e6cf2d
add 'text.extr()'
...
a stripped-down version of text.extract() that
- always returns a string (like 'extract_from')
- only returns a string
- does not deal with 'pos' arguments
- is ~20% faster
2 years ago
460095adca
update downloader tests
2 years ago
f037429fa4
attempt to improve '-K' output for lists
...
- use [N] instead if [] to indicate a Number needs to be placed there
- enumerate list items
2 years ago
062ef238a6
add support for aibooru (using danbooru extractor) ( #3075 )
2 years ago
b57015cf0a
[postprocessor:metadata] assume 'mode: custom' when format is set
...
{"name": "metadata", "format": "foobar"}
will now implicitly use mode:custom and no longer mode:json like before
2 years ago
f7ba19a1c0
[nana] add 'nana' extractors ( #2967 )
2 years ago
b36125333f
[postprocessor:zip] implement 'files' option ( #2872 )
2 years ago
67bad04dda
[formatter] add 'g' conversion to sluGify a string ( #2410 )
2 years ago
6990ad0ba8
[formatter] do NOT apply :J to strings ( #2833 )
2 years ago
c0051d7d4c
fix test
2 years ago
dd3a6a9fd1
make 'enumerate_reversed()' work with generators ( #2795 )
2 years ago
0c73914848
[postprocessor:metadata] implement 'mode: modify' ( #2640 )
2 years ago
f3de6b7a87
[postprocessor:metadata] implement 'mode: delete' ( #2640 )
2 years ago
9704c04172
[postprocessor:zip] ensure target directory exists ( #2758 )
2 years ago
74865adae5
implement 'format-separator' option ( #2737 )
...
a global option, that servers as a workaround for shortcomings due to
lack of a proper format string parser
2 years ago
117eeefda0
[postprocessor:mtime] add 'value' option ( #2739 )
2 years ago
90ae48c40c
[formatter] implement 'O' format specifier ( #2736 )
...
to apply a UTC offset to 'date' values and other datetime objects
2 years ago
04bed1eba3
[formatter] allow for custom "format" functions ( #2721 )
2 years ago
54525d2e21
[formatter] implement slice operator as format specifier
...
this allows using a slice operator alongside other (special) format
specifiers like J, to first join list elements to a string and then
trimming that with a slice.
{tags:J, /[:50]}
2 years ago
241e82e18d
[horne] add support for horne.red ( #2700 )
2 years ago
42525cfe8d
fix '{…!j}' for otherwise non-serializable types (##2624)
...
like 'datetime'
2 years ago
5b43faffed
[postprocessor:metadata] write to stdout by setting filename to "-"
...
(#2624 )
2 years ago
6ad39f2b68
add ytdl tests
...
they only run when youtube-dl or yt-dlp are installed,
i.e. if __import__("<ytdl-package>") succeeds
2 years ago
688d6553b4
replace calls to print() with stdout_write() ( #2529 )
2 years ago
f3408a9d92
implement string literals in replacement fields
...
- either {_lit[foo]} or {'foo'}
- useful as alternative for empty metadata fields: {title|'no title'}
- due to using '_string.formatter_field_name_split()' to parse format
strings, using certain characters will result in an error: [].:!
2 years ago
c4b9f7bab8
update functions working with cookies.txt files
...
- rename
- load_cookiestxt -> cookiestxt_load
- save_cookiestxt -< cookiestxt_store
- in cookiestxt_load, add cookies directly to a cookie jar
instead of storing them in a list first
- other unnoticeable performance increases
2 years ago
ca3a364db7
fix build_duration_func() ( #2533 )
...
for extractors with request_interval_min > 0
2 years ago
7fe54bab2a
attempt to fix some issues with 'contains()' ( #2446 )
...
add a third argument that gets used
when the values o search are given as a string
2 years ago
d78a2c7163
re.escape() arguments for 'contains()' ( #2446 )
2 years ago
413b77757b
implement 'contains()' ( #2446 )
...
and add it to globals() in compiled expressions for --filter etc
3 years ago
e7b30866d0
[postprocessor:mtime] fix timestamps from datetime objects ( #2307 )
...
'datetime.timestamp()', which got used to convert datetime objects to
POSIX timestamps, assumes naive datetimes represent LOCAL time, while
datetimes in 'date' metadata fields represent UTC time.
Ref: https://docs.python.org/3/library/datetime.html#datetime.datetime.timestamp
> Naive datetime instances are assumed to represent local time
> you can obtain the POSIX timestamp by … calculating the timestamp directly
3 years ago
29db716a63
implement 'datetime_to_timestamp()'
...
and rename 'to_timestamp()'
to the more descriptive 'datetime_to_timestamp_string()'
3 years ago
8295bc6d97
fix loading/storing cookies without domain
3 years ago
500a479026
fix a third(!) bug in _check_cookies() ( #2372 )
...
turns out tests are worthless if you get em wrong ...
3 years ago
cf44aba333
[formatter] allow evaluating f-string literals
...
by starting a format string with '\fF'.
This was technically already possible with '\fE',
but this makes it a bit more convenient.
3 years ago
94452761ed
fix cookies tests
3 years ago
bddcec49f1
implement 'text.root_from_url()'
...
use domain from input URL for kemono
3 years ago
f5b2b9333f
fix another bug in _check:cookies ( #2160 )
...
regression introduced in ed317bfc
3 years ago
563bd0ecf4
[danbooru] inherit from BaseExtractor
...
- merge danbooru and e621 code
- support booru.allthefallen.moe (closes #2283 )
- remove support for old e621 tag search URLs
3 years ago
b5b4f5a168
use 'build_extractor_filter' in test_results.py
3 years ago
64cf26eaf4
allow specifying sleep-* options as string
...
either as single value or as range: "3.5", "2.1 - 5.0"
3 years ago
010d65dcec
extend blacklist/whitelist syntax ( #2025 )
...
Each entry in such a list can now also include a subcategory
'<category>:<subcategory>'
and it is possible to use '*' or an empty string as placeholder
'*:<subcategory>', ':<subcategory>', '<category>:*'
For example
"blacklist": "imgur,*:tag,gfycat:user" or
"blacklist": ["imgur", "*:tag", "gfycat:user"]
will filter all 'imgur' extractors, all extractors with a 'tag'
subcategory (e.g. https://danbooru.donmai.us/posts?tags=bonocho ),
and all 'gfycat' user extractors.
3 years ago
af6424f398
allow testing metadata in list elements
3 years ago
3842cdcd8f
[formatter] implement 'D' format specifier
...
To be able to parse any string into a 'datetime' object
and format it as necessary.
Example:
{created_at:D%Y-%m-%dT%H:%M:%S%z}
->
"2010-01-01 00:00:00"
{created_at:D%Y-%m-%dT%H:%M:%S%z/%b %d %Y %I:%M %p}
->
"Jan 01 2010 12:00 AM"
with 'created_at' == "2010-01-01T01:00:00+0100"
3 years ago
2ab190ce08
add tests for special format strings
3 years ago
46e17c5e61
support accessing the current local datetime in format strings
...
{_now}, {_now:%Y-%m-%d}, etc
(#1968 )
3 years ago
38193dba46
support accessing environment variables in format strings ( #1968 )
...
{_env[HOME]} to get the value of $HOME
every other format string feature is supported as well
3 years ago
f2d6b3e6b4
run tests without using 'nose'
...
run_tests.sh -> run_tests.py
3 years ago
12fc646c53
fix filename formatting tests
3 years ago
e0bdacd932
[fappic] add 'image' extractor ( closes #1898 )
3 years ago
c22ff97743
remove 'unit' argument from 'util.format_value()'
3 years ago
cad85640de
move 'util.PathFormat' into its own 'path' module
...
to prevent circular imports between 'formatter' and 'util'
3 years ago
74145467dd
move 'util.Formatter' into its own 'formatter' module
3 years ago
9377543162
[mastodon] add 'following' extractor ( #1891 )
3 years ago
bd845303ad
implement a way to shorten filenames with east-asian characters
...
(#1377 )
Setting 'output.shorten' to "eaw" (East-Asian Width) uses a slower
algorithm that also considers characters with a width > 1.
3 years ago
292fffc83c
add 'j' format string conversion
...
to convert to a JSON formatted string
3 years ago
bb6a130942
automatically set required DDoS-GUARD cookies ( #1779 )
...
for kemono.party and seiso.party
3 years ago
2792ed6e4b
implement 'util.format_value()'
3 years ago
9e42cd58ea
replace ChainPredicate class with 'functools.partial'
3 years ago
36ac2197db
[ytdl] add extractor for sites supported by youtube-dl
...
(#1680 , #878 )
Can be used by prefixing any URL with 'ytdl:',
or by setting 'extractor,ytdl.enabled' to 'true'.
3 years ago
64240c8d42
[imagevenue] fix extraction
...
(closes #1677 )
3 years ago
0179581340
add 'T' format string conversion ( #1646 )
...
to convert 'date'/datetime to timestamp
3 years ago
f74cf52e2b
[seisoparty] add 'user' and 'post' extractors ( #1635 )
3 years ago
759735fb02
[kemonoparty] fix 'username' extraction ( fixes #1652 )
...
The site's <title> content changed from
<title>NAME | Kemono</title>
to
<title>
NAME | Kemono
</title>
3 years ago
07c8adbd8b
[mangadex] implement login with username & password ( #1535 )
3 years ago
4a747a31a3
[postprocessor:metadata] handle dicts in mode;tags ( fixes #1598 )
3 years ago
3cbbefd4ed
support 'filter' option for post processors ( #1460 )
3 years ago
0abad8bc12
implement 'compile_expression()'
3 years ago
da6806a161
fix job tests for Python 3.4 and 3.5
...
assert_called() and assert_not_called() got added in Python 3.6
3 years ago
8fd8126117
fix ISO 639-1 code for Japanese
...
"jp" -> "ja"
3 years ago
af9dba4684
add DataJob tests
3 years ago
adf4d661b3
use '_extractor' info in UrlJobs
3 years ago
1eabfa5c7a
[pillowfort] implement login with username & password ( #846 )
3 years ago
559462789d
add some tests for job.py
3 years ago
c5ca7905ce
add 'noop()' and 'identity()' functions
3 years ago
bc868e7bb8
consider apparently long extensions as part of the filename
...
(#1516 )
3 years ago
bdfcc9c4b1
update extractor test results
3 years ago
387fe415d5
unescape items in text.split_html()
4 years ago
78fd63b8f0
remove 'text.clean_xml()'
...
was not used anywhere
4 years ago
8553b218d9
replace calls to 'os.path.splitext()' with 'str.rpartition()'
...
Makes functions who used it more than twice as fast
and we can get rid of an import as well.
4 years ago
bff71cde80
implement 'util.unique_squence()'
4 years ago
5f1a6ff6fa
remove unneeded 'TRAVIS_SKIP' from test_results.py
4 years ago
8821dceb79
use __import__() to dynamically load modules
4 years ago
36bf76fa44
update 'oauth:mastodon:<instance>' code
4 years ago
91308140ec
make 'generate_token()' compatible with Python 3.4
4 years ago
780b6adb91
rename 'generate_csrf_token()' to just 'generate_token()'
...
and add a 'size' argument
4 years ago
0fdaea00a3
[postprocessor:metadata] sanitize filenames
4 years ago
aac00a2024
add 'd' conversion for format strings
...
to convert a timestamp to a formattable 'datetime' object.
For example '{created_at!d:%Y-%m-%d}'
transforms the timestamp in 'created_at' into a 'datetime' object
and then formats its content using '%Y-%m-%d' as template.
1262304000 -> datetime(2010, 1, 1) -> "2010-01-01"
4 years ago
912eea29bc
update extractor test results
4 years ago
Antonio