d651d45239
implement specifying ranges in slice notation ( #918 , #2865 )
...
e.g.
- '1:101' or ':101' or ':101:' for files 1 to 100
- '1::2' or '::2' for every second file
- '1:101:5' or ':101:5' for files 1, 6, 11, ..., 91, 96
(the second argument specifies the first index NOT included)
2 years ago
3616adfc75
implement '--range' with Python ranges
2 years ago
1800bd7d14
allow '*-filter' options to be a list of expressions
2 years ago
43c211f1a7
extend and rename util.CustomNone
2 years ago
42481aed59
[formatter] implement 'S' format specifier ( #3266 )
...
to Sort lists
2 years ago
6e08ad26f7
update downloader tests
2 years ago
05255f5be0
add 'default' argument to 'text.extr()'
2 years ago
8124c16a50
split 'build_path' from 'set_filename' and 'set_extension'
...
Do not automatically build a new path
when setting file metadata or updating its extension.
2 years ago
eb33e6cf2d
add 'text.extr()'
...
a stripped-down version of text.extract() that
- always returns a string (like 'extract_from')
- only returns a string
- does not deal with 'pos' arguments
- is ~20% faster
2 years ago
460095adca
update downloader tests
2 years ago
f037429fa4
attempt to improve '-K' output for lists
...
- use [N] instead if [] to indicate a Number needs to be placed there
- enumerate list items
2 years ago
062ef238a6
add support for aibooru (using danbooru extractor) ( #3075 )
2 years ago
b57015cf0a
[postprocessor:metadata] assume 'mode: custom' when format is set
...
{"name": "metadata", "format": "foobar"}
will now implicitly use mode:custom and no longer mode:json like before
2 years ago
f7ba19a1c0
[nana] add 'nana' extractors ( #2967 )
2 years ago
b36125333f
[postprocessor:zip] implement 'files' option ( #2872 )
2 years ago
67bad04dda
[formatter] add 'g' conversion to sluGify a string ( #2410 )
2 years ago
6990ad0ba8
[formatter] do NOT apply :J to strings ( #2833 )
2 years ago
c0051d7d4c
fix test
2 years ago
dd3a6a9fd1
make 'enumerate_reversed()' work with generators ( #2795 )
2 years ago
0c73914848
[postprocessor:metadata] implement 'mode: modify' ( #2640 )
2 years ago
f3de6b7a87
[postprocessor:metadata] implement 'mode: delete' ( #2640 )
2 years ago
9704c04172
[postprocessor:zip] ensure target directory exists ( #2758 )
2 years ago
74865adae5
implement 'format-separator' option ( #2737 )
...
a global option, that servers as a workaround for shortcomings due to
lack of a proper format string parser
2 years ago
117eeefda0
[postprocessor:mtime] add 'value' option ( #2739 )
2 years ago
90ae48c40c
[formatter] implement 'O' format specifier ( #2736 )
...
to apply a UTC offset to 'date' values and other datetime objects
2 years ago
04bed1eba3
[formatter] allow for custom "format" functions ( #2721 )
2 years ago
54525d2e21
[formatter] implement slice operator as format specifier
...
this allows using a slice operator alongside other (special) format
specifiers like J, to first join list elements to a string and then
trimming that with a slice.
{tags:J, /[:50]}
2 years ago
241e82e18d
[horne] add support for horne.red ( #2700 )
2 years ago
42525cfe8d
fix '{…!j}' for otherwise non-serializable types (##2624)
...
like 'datetime'
2 years ago
5b43faffed
[postprocessor:metadata] write to stdout by setting filename to "-"
...
(#2624 )
2 years ago
6ad39f2b68
add ytdl tests
...
they only run when youtube-dl or yt-dlp are installed,
i.e. if __import__("<ytdl-package>") succeeds
2 years ago
688d6553b4
replace calls to print() with stdout_write() ( #2529 )
2 years ago
f3408a9d92
implement string literals in replacement fields
...
- either {_lit[foo]} or {'foo'}
- useful as alternative for empty metadata fields: {title|'no title'}
- due to using '_string.formatter_field_name_split()' to parse format
strings, using certain characters will result in an error: [].:!
2 years ago
c4b9f7bab8
update functions working with cookies.txt files
...
- rename
- load_cookiestxt -> cookiestxt_load
- save_cookiestxt -< cookiestxt_store
- in cookiestxt_load, add cookies directly to a cookie jar
instead of storing them in a list first
- other unnoticeable performance increases
2 years ago
ca3a364db7
fix build_duration_func() ( #2533 )
...
for extractors with request_interval_min > 0
2 years ago
7fe54bab2a
attempt to fix some issues with 'contains()' ( #2446 )
...
add a third argument that gets used
when the values o search are given as a string
2 years ago
d78a2c7163
re.escape() arguments for 'contains()' ( #2446 )
2 years ago
413b77757b
implement 'contains()' ( #2446 )
...
and add it to globals() in compiled expressions for --filter etc
3 years ago
e7b30866d0
[postprocessor:mtime] fix timestamps from datetime objects ( #2307 )
...
'datetime.timestamp()', which got used to convert datetime objects to
POSIX timestamps, assumes naive datetimes represent LOCAL time, while
datetimes in 'date' metadata fields represent UTC time.
Ref: https://docs.python.org/3/library/datetime.html#datetime.datetime.timestamp
> Naive datetime instances are assumed to represent local time
> you can obtain the POSIX timestamp by … calculating the timestamp directly
3 years ago
29db716a63
implement 'datetime_to_timestamp()'
...
and rename 'to_timestamp()'
to the more descriptive 'datetime_to_timestamp_string()'
3 years ago
8295bc6d97
fix loading/storing cookies without domain
3 years ago
500a479026
fix a third(!) bug in _check_cookies() ( #2372 )
...
turns out tests are worthless if you get em wrong ...
3 years ago
cf44aba333
[formatter] allow evaluating f-string literals
...
by starting a format string with '\fF'.
This was technically already possible with '\fE',
but this makes it a bit more convenient.
3 years ago
94452761ed
fix cookies tests
3 years ago
bddcec49f1
implement 'text.root_from_url()'
...
use domain from input URL for kemono
3 years ago
f5b2b9333f
fix another bug in _check:cookies ( #2160 )
...
regression introduced in ed317bfc
3 years ago
563bd0ecf4
[danbooru] inherit from BaseExtractor
...
- merge danbooru and e621 code
- support booru.allthefallen.moe (closes #2283 )
- remove support for old e621 tag search URLs
3 years ago
b5b4f5a168
use 'build_extractor_filter' in test_results.py
3 years ago
64cf26eaf4
allow specifying sleep-* options as string
...
either as single value or as range: "3.5", "2.1 - 5.0"
3 years ago
010d65dcec
extend blacklist/whitelist syntax ( #2025 )
...
Each entry in such a list can now also include a subcategory
'<category>:<subcategory>'
and it is possible to use '*' or an empty string as placeholder
'*:<subcategory>', ':<subcategory>', '<category>:*'
For example
"blacklist": "imgur,*:tag,gfycat:user" or
"blacklist": ["imgur", "*:tag", "gfycat:user"]
will filter all 'imgur' extractors, all extractors with a 'tag'
subcategory (e.g. https://danbooru.donmai.us/posts?tags=bonocho ),
and all 'gfycat' user extractors.
3 years ago
af6424f398
allow testing metadata in list elements
3 years ago
3842cdcd8f
[formatter] implement 'D' format specifier
...
To be able to parse any string into a 'datetime' object
and format it as necessary.
Example:
{created_at:D%Y-%m-%dT%H:%M:%S%z}
->
"2010-01-01 00:00:00"
{created_at:D%Y-%m-%dT%H:%M:%S%z/%b %d %Y %I:%M %p}
->
"Jan 01 2010 12:00 AM"
with 'created_at' == "2010-01-01T01:00:00+0100"
3 years ago
2ab190ce08
add tests for special format strings
3 years ago
46e17c5e61
support accessing the current local datetime in format strings
...
{_now}, {_now:%Y-%m-%d}, etc
(#1968 )
3 years ago
38193dba46
support accessing environment variables in format strings ( #1968 )
...
{_env[HOME]} to get the value of $HOME
every other format string feature is supported as well
3 years ago
f2d6b3e6b4
run tests without using 'nose'
...
run_tests.sh -> run_tests.py
3 years ago
12fc646c53
fix filename formatting tests
3 years ago
e0bdacd932
[fappic] add 'image' extractor ( closes #1898 )
3 years ago
c22ff97743
remove 'unit' argument from 'util.format_value()'
3 years ago
cad85640de
move 'util.PathFormat' into its own 'path' module
...
to prevent circular imports between 'formatter' and 'util'
3 years ago
74145467dd
move 'util.Formatter' into its own 'formatter' module
3 years ago
9377543162
[mastodon] add 'following' extractor ( #1891 )
3 years ago
bd845303ad
implement a way to shorten filenames with east-asian characters
...
(#1377 )
Setting 'output.shorten' to "eaw" (East-Asian Width) uses a slower
algorithm that also considers characters with a width > 1.
3 years ago
292fffc83c
add 'j' format string conversion
...
to convert to a JSON formatted string
3 years ago
bb6a130942
automatically set required DDoS-GUARD cookies ( #1779 )
...
for kemono.party and seiso.party
3 years ago
2792ed6e4b
implement 'util.format_value()'
3 years ago
9e42cd58ea
replace ChainPredicate class with 'functools.partial'
3 years ago
36ac2197db
[ytdl] add extractor for sites supported by youtube-dl
...
(#1680 , #878 )
Can be used by prefixing any URL with 'ytdl:',
or by setting 'extractor,ytdl.enabled' to 'true'.
3 years ago
64240c8d42
[imagevenue] fix extraction
...
(closes #1677 )
3 years ago
0179581340
add 'T' format string conversion ( #1646 )
...
to convert 'date'/datetime to timestamp
3 years ago
f74cf52e2b
[seisoparty] add 'user' and 'post' extractors ( #1635 )
3 years ago
759735fb02
[kemonoparty] fix 'username' extraction ( fixes #1652 )
...
The site's <title> content changed from
<title>NAME | Kemono</title>
to
<title>
NAME | Kemono
</title>
3 years ago
07c8adbd8b
[mangadex] implement login with username & password ( #1535 )
3 years ago
4a747a31a3
[postprocessor:metadata] handle dicts in mode;tags ( fixes #1598 )
3 years ago
3cbbefd4ed
support 'filter' option for post processors ( #1460 )
3 years ago
0abad8bc12
implement 'compile_expression()'
3 years ago
da6806a161
fix job tests for Python 3.4 and 3.5
...
assert_called() and assert_not_called() got added in Python 3.6
3 years ago
8fd8126117
fix ISO 639-1 code for Japanese
...
"jp" -> "ja"
3 years ago
af9dba4684
add DataJob tests
3 years ago
adf4d661b3
use '_extractor' info in UrlJobs
3 years ago
1eabfa5c7a
[pillowfort] implement login with username & password ( #846 )
3 years ago
559462789d
add some tests for job.py
3 years ago
c5ca7905ce
add 'noop()' and 'identity()' functions
3 years ago
bc868e7bb8
consider apparently long extensions as part of the filename
...
(#1516 )
3 years ago
bdfcc9c4b1
update extractor test results
3 years ago
387fe415d5
unescape items in text.split_html()
4 years ago
78fd63b8f0
remove 'text.clean_xml()'
...
was not used anywhere
4 years ago
8553b218d9
replace calls to 'os.path.splitext()' with 'str.rpartition()'
...
Makes functions who used it more than twice as fast
and we can get rid of an import as well.
4 years ago
bff71cde80
implement 'util.unique_squence()'
4 years ago
5f1a6ff6fa
remove unneeded 'TRAVIS_SKIP' from test_results.py
4 years ago
8821dceb79
use __import__() to dynamically load modules
4 years ago
36bf76fa44
update 'oauth:mastodon:<instance>' code
4 years ago
91308140ec
make 'generate_token()' compatible with Python 3.4
4 years ago
780b6adb91
rename 'generate_csrf_token()' to just 'generate_token()'
...
and add a 'size' argument
4 years ago
0fdaea00a3
[postprocessor:metadata] sanitize filenames
4 years ago
aac00a2024
add 'd' conversion for format strings
...
to convert a timestamp to a formattable 'datetime' object.
For example '{created_at!d:%Y-%m-%d}'
transforms the timestamp in 'created_at' into a 'datetime' object
and then formats its content using '%Y-%m-%d' as template.
1262304000 -> datetime(2010, 1, 1) -> "2010-01-01"
4 years ago
912eea29bc
update extractor test results
4 years ago
1f9121fecb
release version 1.16.0
4 years ago
b2c55f0a72
[sankaku] remove login support
...
The old login method for 'https://chan.sankakucomplex.com/user/login '
and the cookies it produces have no effect on the results from
'beta.sankakucomplex.com'.
4 years ago
547107307e
fix 'Metadata' messages in result tests
4 years ago
578dcf805c
[mangapanda] don't force https://
4 years ago
ca59bd691c
[postprocessor:metadata] add 'event' and 'filename' options
4 years ago
9fffa9c343
rework post processor callbacks
4 years ago
1e3dd7330e
merge SharedConfigMixin functionality into Extractor
4 years ago
e5438b8a29
release version 1.15.3
4 years ago
b9bfa4c675
update extractor test results
4 years ago
c3f01dc4e6
implement 'util.unique()'
4 years ago
d83b95fd28
[postprocessor:metadata] accept a string-list for 'content-format'
...
(closes #1080 )
4 years ago
350b1afe1c
speed up _list_classes() after iterating over all modules once
4 years ago
18213dc5ba
release version 1.15.2
4 years ago
ec61696316
add 't' format string conversion ( closes #1065 )
...
to Trim whitespace from the beginning and end of strings.
Example: '{field!t}' becomes 'foo' for 'field' == " \nfoo\t\r"
4 years ago
07432d6262
[seiga] fix flake8 and cookie test ( #1063 )
4 years ago
b8daabc3ca
[pinterest] implement login support ( closes #1055 )
...
being logged allows access to secret/protected boards
4 years ago
7e0e872f4f
[seiga] Add metadata for single image downloads ( #1063 )
...
* [seiga] Support image metadata.
* [seiga] Update test data.
* [seiga] Fix cookie check.
* [test_cookies] [seiga] Fit test_cookies.py to the last commit.
4 years ago
844793847c
update extractor test results
4 years ago
c874071f5a
[kissmanga] remove module
4 years ago
844502cad5
update extractor test results
4 years ago
7cd383c0f9
update extractor test results
4 years ago
65744a7a31
use alternative for all falsey values in format strings
...
… and not just None (#525 )
It would be better to consistently use None for all non-existent
fields and/or fields without a valid value, but this is a good
enough workaround for now.
4 years ago
f5b7ae01c1
update extractor test results
4 years ago
392d022b04
implement 'config.accumulate()' ( #994 )
4 years ago
3108e85b89
[worldthree] remove extractors
...
http://www.slide.world-three.org/ hasn't been accessible for a long time.
4 years ago
3918b69677
remove 'extractor.blacklist' context manager
4 years ago
ac3036ef56
add 'filesize-min' and 'filesize-max' options ( closes #780 )
4 years ago
fd0685d9b5
[postprocessor:zip] defer zip file creation ( fixes #968 )
...
don't try to create zip files on postprocessor construction,
wait until directory creation during file download,
4 years ago
d50f3b333a
update extractor test results
4 years ago
e33293fdd8
[hentaihand] update to new site layout
4 years ago
69e4871005
update extractor test results
...
- sensescans: replace 404d chapters
- mangapark: replace 404d chapters
- subscribestar: update test for attached files
4 years ago
688bd046fc
release version 1.14.4
4 years ago
422e69f187
skip external OAuth tests ( closes #908 )
4 years ago
8dbf827649
[bobx] remove module
4 years ago
87202b8d74
[inkbunny] add 'user' and 'post' extractors ( #283 )
4 years ago
2ecf1efb16
update extractor test results
...
- tumblr: remove deleted post
- jaiminisbox: replace removed manga/chapters
- smugmug: one inconsequential field got removed
4 years ago
e62ebb4643
update CHANGELOG before building sdist and wheel packages
4 years ago
0cac14c3bd
update extractor test results
4 years ago
53cc498d9c
improve config lookup when there are multiple possible locations
...
This specifically applies to all Mastodon extractors and all
extractors with a 'basecategory', i.e. 'booru', 'foolslide', etc.
Values inside those general config locations wouldn't be recognized
when a value with the same was set on the 'extractor' level.
For example 'extractor.mastodon.directory' should be used over
'extractor.directory' when both are set, but this was impossible
with the previous implementation.
(fixes #843 )
4 years ago
d81a8e6544
[twitter] update tests
4 years ago
37d71f6e09
strip microseconds in text.parse_datetime()
4 years ago
6db7ed90cb
release version 1.14.1
4 years ago
087e3184dc
use a non-twitter URL when testing snap creation
4 years ago
7daef6ee70
update extractor test results
...
- certain posts on Instagram now return
https://static.cdninstagram.com/rsrc.php/null.jpg
for public users
- MangaDex is deploying its new MangaDex@Home network similar to
exhentai's Hentai@Home
- realbooru has a new site layout, but the underlying booru API still
works like before
4 years ago
3bad1579ee
update extractor test results
4 years ago
45baa13615
update extractor test results
...
- don't run Instagram tests on Travis anymore
- replace Twitter test because timeline was made private
- update Hiperdex domain to '.com' (again ...)
4 years ago
dfcf2a2c91
write OAuth token to cache by default ( #616 )
4 years ago
6294e2c540
add 'text.ensure_http_scheme()'
4 years ago
ece73b5b2a
make 'path' and 'keywords' available in logging messages
...
Wrap all loggers used by job, extractor, downloader, and postprocessor
objects into a (custom) LoggerAdapter that provides access to the
underlying job, extractor, pathfmt, and kwdict objects and their
properties.
__init__() signatures for all downloader and postprocessor classes have
been changed to take the current Job object as their first argument,
instead of the current extractor or pathfmt.
(#574 , #575 )
4 years ago
4b606b68e4
skip OAuth tests when server is unreachable
4 years ago
8b60bd6a91
mock 'time()' in cache tests
...
instead of calling 'sleep()' to let time advance.
This shortens the time needed to run those tests,
and ensures consistent results.
(Tests would randomly fail when using 'sleep()')
4 years ago
8f2c1da041
skip example config tests if files are not available ( #730 )
4 years ago
5df8f2959b
insert local directory into PYTHONPATH when running tests
4 years ago
ff47641b13
test whether default/example config files contain valid JSON
4 years ago
d6facdee7b
[mastodon] add tests ( #701 )
4 years ago
fd438f0d78
update extractor test results
5 years ago
a0f4c295c0
add optional 'utcoffset' argument to 'parse_datetime()'
5 years ago
406449b0d6
ensure keys for mastodon instances are available during tests
...
Calls to config.clear() from other tests are removing the API
credentials set when importing mastodon.py for the first time.
5 years ago
9e7dfc0cfc
[myportfolio] fix extraction of galleries without title
5 years ago
3b50c4f49d
add tests for "Extractors" in oauth.py ( #670 )
5 years ago
04bd0472de
add tests for Extractor.wait()
5 years ago
7499d71d02
[simplyhentai] ignore certificate errors in video test
5 years ago
4203dc0bdc
[mangapark] fix metadata extraction
5 years ago
77fda8190c
[35photo] simplify/remove tests for the 'genre' extractor
...
There is still a nice genre overview page (https://35photo.pro/genre/ )
but the individual sub-pages don't list photos anymore
5 years ago
32e36d8f02
[sexcom] replace tests
5 years ago
e0b0e8d62a
release version 1.13.2
5 years ago
a63a376ad2
[mangoxo] fix login
5 years ago
ebc70e87ce
[e621] update to new interface / API endpoints ( closes #635 )
5 years ago
72122eb9b3
release version 1.13.1
5 years ago
ce5e2a58fe
[imgbb] update test results
...
Image server domain changed from
https://image.ibb.co/ to https://i.ibb.co/
5 years ago
1d4a369ea2
update extractor test results
5 years ago
ec85bf90de
use context managers in cache.py & add tests
5 years ago
4e361b3008
add tests for specific datetime values
5 years ago
90e4c645ba
[formatter] allow multiple "special" format specifiers ( #595 )
...
It is now, for example, possible to specify multiple replacement
operations per format replacement field: {name:Ra/b/Rc/d/}
5 years ago
219c4cc78c
[formatter] allow for numeric list and string indices
5 years ago
7d1da614d9
[formatter] implement field name alternatives ( #525 )
...
The format string '{a|b|c}' will now try to use the value from 'a' and
fall back to 'b' and 'c' if accessing a field raises an exception or
if its value is None.
5 years ago
c7cf9dd111
[furaffinity] support classic layout ( #284 )
5 years ago
40fe062851
[pixiv] fix user id for bookmarks API calls ( closes #596 )
5 years ago
2852691d78
[paheal] replace test URL
...
searching for 'k-on' doesn't yield any results anymore
5 years ago
2a9be48511
improve util.load/save_cookiestxt() and add tests
...
- take a file object as argument instead of an filename
- accept whitespace before comments (" # comment")
- map expiration "0" to None and not the number 0
5 years ago
b3b5754f2d
update test_cookies.py
5 years ago
174117f827
allow multiple hashes for content tests
5 years ago
60a43f0264
fix downloader tests
5 years ago
e89413da22
update test results
5 years ago
5cac79c3d9
[erolord] remove extractor
5 years ago
988cc2ec23
[mangadex] change domain to mangadex.cc ( closes #559 )
5 years ago
87c8b89ddd
[postprocessor:metadata] add 'directory' option ( #520 )
5 years ago
82f7f4172a
update test results
5 years ago
d0920e84e9
update test results
5 years ago
9e63804347
[patreon] make retrieving user info nonfatal ( #508 )
...
… and fall back to the included data if an error occurs.
5 years ago
15f9bb3d14
add option to disable pyOpenSSL usage ( #508 )
...
(pyOpenSSL is now disabled by default)
5 years ago
50deab5265
[deviantart] fix URL generation from /extended_fetch results
...
(closes #505 )
5 years ago
004812258d
[hentaifox] fix extraction
5 years ago
a412531451
[postprocessor:metadata] implement 'extension-format' option
...
closes #477
5 years ago
b5c964332b
improve config.py test coverage
5 years ago
f5604492c3
update interface of config functions
5 years ago
3fc1e12949
[postprocessor:metadata] filter private entries
...
i.e. keys starting with an underscore
5 years ago
978cb03f81
update misc test results
...
- Livedoor now uses https:// for its image URLs
- Instagram image URLs got simplified
5 years ago
bbbeff4c41
[downloader.http] implement file-specific HTTP headers
5 years ago
3ece3976ae
[newgrounds] implement login support ( #394 )
5 years ago
abfcb356fc
[flickr] support 3k, 4k, 5k, and 6k photo sizes ( closes #472 )
5 years ago
da6789b2b0
disable unique archive id checks for some tests
...
- same image twice in a livedoor blog post
- unreliable results for related pinterest items
5 years ago
ba083b30b2
fix snap build
...
… hopefully
5 years ago
94a94f3b86
miscellaneous stuff
5 years ago
9e88e7a344
[postprocessor:exec] improve ( #421 , #413 )
...
- add 'final' option
- include job status in pp finalization
- improve and extend documentation
5 years ago
2a3bd4e3c7
rename extractor classes starting with a digit
5 years ago
64786363be
[4chan] simplify
...
- remove 'chan.py'
- slight adjustments to directory and filenames
5 years ago
557e2c018b
[8chan] remove module
5 years ago
322c2e7ed4
renaming variables
...
mostly 'keyword(s)' to 'kwdict'
5 years ago
87a87bff7e
[simplyhentai] fix image URLs
5 years ago
d5e3910270
adjust 'util.raises()'
5 years ago
b23c822b23
[luscious] use GraphQL
5 years ago
1693d97bd3
update extractor class hierarchies
...
- let the GalleryExtractor class inherit directly from Extractor
- make ChapterExtractor a subclass of GalleryExtractor
- change enumeration field names of GalleryExtractors to 'num'
5 years ago
7ebd984e8d
[imgur] print error message if no JSON data is found ( #446 )
5 years ago
de4e2029d1
[nsfwalbum] update test album
...
the old one is no longer available
5 years ago
913460240d
[reddit] fix 'extractor.blacklist()' arguments
...
The second argument must support 'append()'.
5 years ago
1848788970
update test results etc
5 years ago
d5fbb2d9de
[tumblr] ignore audio links from Spotify etc.
5 years ago
c6c5cb1898
improve 'deviantart.quality' description
5 years ago
c9b97dbca3
extend post processor tests
5 years ago
49f6d7176d
[deviantart] restore filenames ( #392 )
...
<title>_by_<user>_<id> --> <title>_by_<user>-<id>
5 years ago
e528f3cb77
adjust postprocessor test results
...
see 2495b99
5 years ago
cb7f149974
fix mtime datetime test
...
datetime.timestamp() uses local time for a naive datetime object
5 years ago
23251356cb
require 'extension' data for each URL ( #382 )
5 years ago
dd72ae7164
add postprocessor tests
5 years ago
0bb873757a
update PathFormat class
...
- change 'has_extension' from a simple flag/bool to a field that
contains the original filename extension
- rename 'keywords' to 'kwdict' and some other stuff as well
- inline 'adjust_path()'
- put enumeration index before filename extension (#306 )
5 years ago
748e37554c
update .travis.yml
...
- install pyOpenSSL before running tests
- simplify snap tests
5 years ago
b7fb93e2b2
[downloader:http] add 'adjust-extensions' option
5 years ago
eb7da159e2
[imagebam] update URL test results
...
Image URLs are now using https://, but the website itself is still
served as http://.
5 years ago
fa60109e97
[exhentai] don't use e-hentai.org for exhentai URLs
5 years ago
4a0c98bfc9
miscellaneous fixes and adjustments
5 years ago
40637556fa
[ngomik] fix extraction
5 years ago
d9d44ad953
[tsumino] update test results
5 years ago
b1bea8aaeb
add 'restrict-filenames' option ( #348 )
5 years ago
b3851e01d9
release version 1.9.0
5 years ago
12da6bd0c9
[simplyhentai] fix/improve extraction
5 years ago
b89f0d8d3c
update extractor result tests
5 years ago
40da44b17f
Merge branch 'v1.9.0'
5 years ago
7a99e85943
[kissmanga] fix download URLs and file extensions
...
The current Blogspot image URLs hosted on Kissmanga end with an
"invalid" query parameter (/000.png&upx=...), which doesn't get
recognized by 'spliturl()' and 'parseurl()' as such and gets therefore
included in the 'extension' field from 'text.nameext_from_url()'.
5 years ago
a9c89085fb
[instagram] implement login support ( #195 )
5 years ago
b1985d6579
test default format strings during extractor result tests
...
A missing value or an invalid "syntax" for a format replacement field
will raise an exception.
5 years ago
95b1e4c3c0
implement R<old>/<new>/ format option ( #318 )
5 years ago
70713f0f28
fix extractor result tests
5 years ago
ee4d7c3d89
update downloader.find() and related code
...
Instead of replacing 'https' with 'http' for every URL in
'get_downloader()', this now only happens once during downloader
initialization. Also unit tests.
5 years ago
179d112083
[downloader] overhaul http and text modules
...
Get rid of the modular structure and simplify/specialize those modules.
5 years ago
a77340c647
[keenspot] fix extraction for "TwoKinds"
5 years ago
b171befa87
implement 'parse_unicode_escapes()'
5 years ago
e05a96db5e
[deviantart] rename 'stash' to 'extra' ( #302 )
...
'stash' is already used as a name for the StashExtractor and therefore
expected to be a dictionary.
5 years ago
7c6cb908f9
[xhamster] update test results
5 years ago
62335b9015
[paheal] adjust test results
5 years ago
6a34f4b0c1
skip tests on read timeouts; print list of skipped tests
5 years ago
d33f5a7423
[wallhaven] rewrite
...
- use API
- remove login support, add 'api-key' option
- remove support for "alpha" subdomain - alpha.wallhaven.cc used numeric
IDs that can't be translated to the new ID system
- support direct links to wallpapers
5 years ago
5499934ae2
[ngomik] fix extraction
5 years ago
2b1999476e
implement 'text.rextract()'
5 years ago
e30ada162d
fix cookie tests
...
update _get_extractor():
- always return an Extractor instance with a _login_impl() method
- use Extractor.from_url()
5 years ago
2316e0ed3d
fix strptime workaround from b0e85a4
...
Don't return a modified version of 'date_time' if strptime fails.
5 years ago
6764847349
fix cookie tests
...
'cookies' is a CookieJar, not a dict,
and removing the call to '.keys()' doesn't have the same effect
5 years ago
a5b060765d
improve code in tests
...
- use 'assertRaises' as context manager
- remove calls to .keys()
5 years ago
b0e85a42e3
apply workaround from 4736912 in parse_datetime() itself
5 years ago
4736912d4e
[pixiv] work around strptime limitations in Python < 3.7
...
"%z" doesn't allow a colon separator in older Python versions:
- "+0900" is OK
- "+09:00" raises an exception
5 years ago
d09864b581
implement text.parse_datetime()
5 years ago
5582b06ae4
fix tests with 'urllist' messages
5 years ago
5018781898
allow type tests by name
5 years ago
6264a46212
use 'utcfromtimestamp()'
...
'fromtimestamp()' converts its results to the local timezone and causes
problems when running tests on a different machine.
5 years ago
d670de0344
implement 'text.parse_timestamp()'
5 years ago
21a7e395a7
implement convenience wrapper for text.extract functionality
6 years ago
e25ebc4bff
don't disable certificate checks anymore
...
Executables generated with PyInstaller auto-include the root certificate
file and certificate checks now work out-of-the-box.
6 years ago
d6ddb74cde
update test results
...
- deviantart: 'index' is now an integer
- flickr: image file with lower quality
- paheal: image server name changed
- rule34: post got deleted
6 years ago
d9b94a585d
[mangoxo] add login support ( #184 )
...
A very recent change: It is now only possible to see more
than the first 5 images of an album if you are logged in.
6 years ago
e730fc9045
[twitter] add login support ( #214 )
6 years ago
790f15a56f
[photobucket] use HTTPS
6 years ago
c70b21248d
[wikiart] add extractors ( #179 )
...
for
- artists: https://www.wikiart.org/en/thomas-cole
- artist-listings: https://www.wikiart.org/en/artists-by-century/12
- artwork-listings: https://www.wikiart.org/en/paintings-by-media/grisaille
6 years ago
0c991a3155
add convenience targets to Makefile
6 years ago
6277a739e4
[35photo] add user-, genre-, and image-extractors ( #162 )
6 years ago
973a720a7a
[weibo] fix unit test URL patterns
6 years ago
6f57d44ec2
[seaotterscans] remove extractor
...
http://seaotterscans.com/ now redirects to their MangaDex profile
6 years ago
0887fb61f4
[komikcast] update test results
6 years ago
a881537b91
more util.py tests
6 years ago
976ccb267f
[myportfolio] combine gallery and user extractors
...
An URL alone isn't good enough to distinguish between a gallery or a
gallery-listing, so the new extractor decides what to do based on the
page's content.
6 years ago
9c0e2f294b
[shopify] add generic collection and product extractors ( #175 )
...
with fashionnova.com as a default domain
6 years ago
176b7253a1
update function signature for config.load()
6 years ago
e687a6095e
[luscious] raise exception if album is not available
6 years ago
b09a8184ca
move TestJob into test module; test _extractor values
6 years ago
5530871b5a
change results of text.nameext_from_url()
...
Instead of getting a complete 'filename' from an URL and splitting that
into 'name' and 'extension', the new approach gets rid of the complete
version and renames 'name' to 'filename'. (Using anything other than
{extension} for a filename extension doesn't really work anyway)
Example: "https://example.org/path/filename.ext "
before:
- filename : filename.ext
- name : filename
- extension: ext
now:
- filename : filename
- extension: ext
6 years ago
148b8f15d0
update tests for util.py
6 years ago
4b1880fa5e
propagate 'match' to base extractor constructor
6 years ago
1f3422c28b
[mangahere] fix extraction
6 years ago
84ae72b8d8
[ngomik] fix extraction
6 years ago
9a9cd32461
implement alternative constructor for extractors
6 years ago
abbd45d0f4
update handling of extractor URL patterns
...
When loading extractor classes during 'extractor.find(…)', their
'pattern' attribute will be replaced with a compiled version of itself.
6 years ago
6284731107
simplify extractor constants
...
- single strings for URL patterns
- tuples instead of lists for 'directory_fmt' and 'test'
- single-tuple tests where applicable
6 years ago
bc0951d974
allow for simplified test data structures
...
Instead of a strict list of (URL, RESULTS)-tuples, extractor result
tests can now be a single (URL, RESULTS)-tuple, if it's just one test,
and "only matching" tests can now be a simple string.
6 years ago
347398f692
fix various tests
6 years ago
e1d3e9a926
add 'ext_from_url' to text.py
6 years ago
2d2953a5bf
add 'text.parse_float()' + cleanup in text.py
6 years ago
0c32dc5858
[hentaifox] add extractor for search results ( #160 )
6 years ago
217a0687ef
[behance] add 'collection' extractor ( closes #157 )
6 years ago
b8fed34548
add generalized extractors for Mastodon instances ( #144 )
...
Extractors for Mastodon instances can now be dynamically generated,
based on the instance names in the 'extractor.mastodon.*' config path.
Example:
{
"extractor": {
"mastodon": {
"pawoo.net": { ... },
"mastodon.xyz": { ... },
"tabletop.social": { ... },
...
}
}
}
Each entry requires an 'access-token' value, which can be generated with
'gallery-dl oauth:mastodon:<instance URL>'.
An 'access-token' (as well as a 'client-id' and 'client-secret') for
pawoo.net is always available, but can be overwritten as necessary.
6 years ago
66460337f1
[mangapark] fix extraction
6 years ago
79c01ec7ae
implement J<separator>/ format option
...
J joins list elements by calling <separator>.join(list):
Example:
{f:J - /} -> "a - b - c" (if "f" is ["a", "b", "c"])
6 years ago
9bbbadd93a
[hbrowse] use HTTPS
6 years ago
98c6520384
[pinterest] update root URL of API calls
6 years ago
751e535948
[nhentai] fix extraction ( closes #156 )
...
Use JSON embedded in webpage since API endpoints have been disabled
6 years ago
1734a6c879
[reactor] detect "circular" redirects ( #148 )
6 years ago
e53cdfd6a8
update build_supportedsites.py
6 years ago
0afa913de4
[tumblr] add tests for hidden and private blogs ( #145 )
...
Hidden / dashboard-only blogs are pretty straightforward and "only"
require a valid 'access-token' and 'access-token-secret' for the given
'api-key' and 'api-secret', so that signed OAuth1.0 requests are possible.
Private / password protected blogs on the other hand are a bit
cumbersome. In addition to a valid 'access-token' and
'access-token-secret', they also require the account belonging to those
tokens to be a member of the blog itself. Knowing the password and
entering it in the website isn't enough to access a blog through the
API. Following a private blog is also impossible, so that option can't
work either.
6 years ago
fa7fa2f8ff
[deviantart1 update tests]
6 years ago
259123732f
[readcomiconline] improve comic-page parsing
6 years ago
6c71e9cf5d
[deviantart] add separate 'sta.sh' extractor ( #113 )
...
- supports multiple stashed deviations per page
- explicitly mentions sta.sh support on supportedsites.rst
6 years ago
c5d4f558c9
allow missing field access keys in format strings ( #136 )
6 years ago
4d73cc785d
update test results
6 years ago
010da8372a
[instagram] relax test pattern
6 years ago
15890930ea
[mangafox] fix extraction
...
use mobile version since desktop version is obfuscated
6 years ago
fb53b5dd55
fix control+c during -j and range tests
6 years ago
59bb434ba5
[flickr] add ability to download all albums of a user
...
for example with 'https://www.flickr.com/photos/shona_s/albums '
6 years ago
041bd501fc
[hentaifoundry] unescape YII_CSRF_TOKEN value
...
This fixes the POST requests to /site/filters
6 years ago
d4b2b73bef
release version 1.6.0
6 years ago
3c25fa2dad
update build_testresult_db.py script
6 years ago
7f6a0be982
adjust some tests
6 years ago
966a9ca3a0
update test results
6 years ago
c9861ca812
adjust message for status_code based exceptions
...
from: 5xx HTTP Error: Reason
to : 5xx: Reason
The "HTTP Error" part was in there to emulate Request's error messages
from response.raise_for_status(), but it reads a lot better without.
6 years ago
c00dce2adc
[behance] enable 'categorytransfer'
6 years ago
1532d1b690
fix 'range' tests and update a few test results
6 years ago
0514d6a0ae
make --filter and --range config-file options
...
The functionality of --(chapter-)filter and --(chapter-)range are now
also exposed as the following config-file options:
- extractor.*.image-filter
- extractor.*.image-range
- extractor.*.chapter-filter
- extractor.*.chapter-range
TODO: update configuration.rst
6 years ago
4a348990f4
adjust value resolution for retries/timeout/verify options
...
This change introduces 'extractor.*.retries/timeout/verify' options
as a general way to set these values for all HTTP requests.
'downloader.http.retries/timeout/verify' is a way to override these
options for file downloads only and will fall back to 'extractor.*.…*
values if they haven't been explicitly set.
Also: downloader classes now take an extractor object as first argument
instead of a requests.session.
6 years ago
ca6ac4db6a
fix 'content' tests
6 years ago
d70db2d555
Revert "[komikcast] fix extraction"
...
This reverts commit 5507f5ce2e
6 years ago
5507f5ce2e
[komikcast] fix extraction
6 years ago
17611bfec0
update build_supportedsites.py script
6 years ago
e066f35118
update extractor tests
6 years ago
22ab509a70
[bobx] rename "model" to "idol" extractor
6 years ago
8a23b21d0e
[tests] let 'pattern' require at least 1 URL
6 years ago
0bc8ef51c8
[smugmug] Handle albums with no explicit owner ( #100 )
6 years ago
590c0b3ad5
re-implement and improve filename formatter
...
A format string now gets parsed only once instead of re-parsing it each
time it is applied to a set of data.
The initial parsing causes directory path creation to be at about 2x
slower than before, since each format string there is used only once,
but building a filename, the more common operation, is at least 2x
faster. The "directory slowness" cancels at about 5 filenames and
everything above that is significantly faster.
6 years ago
34b556922d
update/restore tests
6 years ago
e3055d356c
release version 1.5.1
6 years ago
f9ded38d89
[test:results] add support for "range" options in tests
6 years ago
c9e6ccbd7c
[test:extractor] small fixes and improvements
6 years ago
7f4e41c989
increase timeout during extractor tests
...
cloudflare's 522 response takes longer than 30 seconds
6 years ago
b55e39d1ee
[mangadex] improve extraction
...
- cache manga API results
- add artist, author and date fields to chapter metadata
- remove Manga-/ChapterExtractor inheritance
- minor code simplifications and improvements
6 years ago
2a9f3341a2
[behance] fix title extraction
6 years ago
a86f2bfc80
[pinterest] update not-found redirects
6 years ago
7442d2940c
release version 1.5.0
6 years ago
b040ca0718
[rule34] small unit test fixes
6 years ago
f3793660ef
update tests
6 years ago
42a346413b
fix "re:" prefix for keyword tests
6 years ago
e0dd8dff5f
implement L<maxlen>/<replacement>/ format option
...
The L option allows for the contents of a format field to be replaced
with <replacement> if its length is greater than <maxlen>.
Example:
{f:L5/too long/} -> "foo" (if "f" is "foo")
-> "too long" (if "f" is "foobar")
(#92 ) (#94 )
6 years ago
bb89a1e6d7
[mangahere] use http://
...
invalid SSL cert for quite some time now
6 years ago
ce34d82cb4
fix skipping tests on 5xx status codes
6 years ago
a6fe2bb594
[whatisthisimnotgoodwithcomputers] remove extractor
6 years ago
0ba93650e0
[8chan] replace unit test URL
...
the other thread is no longer accessible
6 years ago
8fe9056b16
implement string slicing for format strings
...
It is now possible to slice string (or list) values of format string
replacement fields with the same syntax as in regular Python code.
"{digits}" -> "0123456789"
"{digits[2:-2]}" -> "234567"
"{digits[:5]}" -> "01234"
The optional third parameter (step) has been left out to simplify things.
6 years ago
269dc2bbd5
[sankaku] add 'tags' option ( #94 )
6 years ago
764331823b
release version 1.4.2
6 years ago
2eefaa99a3
[mangapark] support .net and .com mirrors
6 years ago
188e956c4e
[imagefap] use HTTPS + update test results
6 years ago
a699787d01
[deviantart] update URL patterns to new format
...
DeviantArt changed its URL format from
https://<name>.deviantart.com/...
to
https://www.deviantart.com/ <name>/...
With this change both formats will be supported.
6 years ago
0055fdd714
change OAuth test server
...
DNS record for oauthbin.com expired
6 years ago
b8c97d2295
use 'extractor.request()' for more HTTP requests
6 years ago
7a98cc9798
[smugmug] update tests
...
My test account expired and all uploaded images got deleted.
6 years ago
4eb94aca17
[postprocessor:ugoira] pass '-f' if not present
6 years ago
a9e276bc37
reset delete-flag
...
Since 'PathFormat' objects are being reused, setting `delete`
to True once caused all files downloaded after to be deleted as well.
6 years ago
6ac403c5d3
add postprocessor config example
6 years ago
2403c405e3
Merge branch 'postprocessor'
6 years ago
b344f2290f
fix downloader tests
6 years ago
a47c6136cd
[simplyhentai] avoid redirects for all-pages.json ( #89 )
6 years ago
ae9a37a528
implement text.split_html()
6 years ago
0a1863fce3
[pixiv] respect more query parameters for user URLs
...
The API endpoint responsible for user illustrations does not
provide sufficient filter capabilities* to match the actual
website, so we are spinning our own filters.
Respected parameters are
'type': illust, manga, ugoira
'tag' : any image tag (this was already supported)
'p' : the page to start on
*
- API can filter for illustrations and manga, but not for ugoira.
- 'offset' is applied before filtering
- no 'tag' filter
6 years ago
7f899bd5d8
Merge branch 'master' into 1.4-dev
6 years ago
4cea886177
[imgur] allow longer album hashes
6 years ago
e1e23165a0
[pinterest] catch JSON decode errors
6 years ago
6a31ada9e3
re-implement OAuth1.0 code
...
OAuth support for SmugMug needs some additional features
(auth-rebuild on redirect, query parameters in URL, ...)
and fixing this in the old code wouldn't work all that well.
6 years ago
e2157f594e
[mangadex] fix manga extraction ( closes #84 )
...
Chapter listings for manga now use
https://mangadex.org/manga/ <id>/_/chapters/2/
as URL instead of
https://mangadex.org/manga/ <id>/_//2/
6 years ago
69a5e6ddb3
Merge branch 'master' into 1.4-dev
6 years ago
3fe653d940
fix test_results for empty sets
...
{} is an empty dict and doesn't support set operations
6 years ago
d96b3474e5
[puremashiro] remove module
...
site has been unreachable for a couple of weeks
and now the DNS record is gone as well
6 years ago
b44a296404
[gomanga] remove module
...
site has been unreachable for a couple of weeks
and the cloudflare status page shows host errors
6 years ago
2395d870dd
[pinterest] unquote board and user names, better errors
6 years ago
55d4d23860
[pinterest] use Pinterest's "Web" API ( #83 )
...
no access tokens, no user credentials of any kind ...
7 years ago
f471161920
Merge branch 'master' into 1.4-dev
7 years ago
cc36f88586
rename safe_int to parse_int; move parse_* to text module
7 years ago
10cc59f3b5
fix extractor names
7 years ago
b1325d4d2c
fix extractor docstrings
7 years ago
df7e18399e
[luscious] fix image order
7 years ago
d10579edb5
[pinterest] improve PinterestAPI code; remove OAuth mentions
...
on another note: access_tokens have been set to only allow for
10 requests per hour (from 200 yesterday)
7 years ago
4bd182c107
[pinterest] implement `oauth:pinterest` ( #83 )
...
Pinterest access tokens are rate limited at 200 requests per
hour (or maybe per 2 or 3 hours?) so having just one access token
for all users isn't going to work in the long run.
7 years ago
dbe250f7e5
[pinterest] update access_token ( #83 )
7 years ago
5c487300ee
improve 'parse_query()' and add tests
...
- another irrelevant micro-optimization !
- use urllib.parse.parse_qsl directly instead of parse_qs, which
just packs the results of parse_qsl in a different data structure
- reduced memory requirements since no additional dict and lists are
created
7 years ago
4ffa94f634
remove 'shorten_path()' and 'shorten_filename()'
7 years ago
27eab4e467
rewrite text tests and improve functions
...
- test more edge cases
- consistently return an empty string for invalid arguments
- remove the ungreedy-flag in 'remove_html()'
7 years ago
e3f2bd4087
add tests for 'text.clean_xml()' and improve it
7 years ago
6d8b191ea7
improve 'parse_query()' and add tests
...
- another irrelevant micro-optimization !
- use urllib.parse.parse_qsl directly instead of parse_qs, which
just packs the results of parse_qsl in a different data structure
- reduced memory requirements since no additional dict and lists are
created
7 years ago
51ea699083
add 'abort()' as function to filter expressions
...
calling 'abort()' in a filter aborts the current extractor run
in a cleaner way than using something like 1/0, which
causes an error message to be printed
7 years ago
48a83a89e9
[loveisover] remove module
...
archive.loveisover.me was shut down on 2018-03-29;
https://www.archiveteam.org/index.php?title=4chan#archive.loveisover.me
7 years ago
564e12ca8f
replace 'imgyt' with 'imxto'
...
https://img.yt/ wasn't available for a couple of days, but has now
re-emerged as https://imx.to/ with a new web-interface.
Links to older images still work (see tests).
7 years ago
d11fcf4804
smaller changes and fixes
...
- fix the cloudflare challenge result if the last decimal places
are zero (JS`s toFixed() removes trailing zeroes)
- fix downloading of kissmanga chapter-pages hosted on blogspot
(accessing blogspot with "kissmanga.com" as referrer yields a 401)
- disable certificate validation for 'mangahere' tests
- update flickr test result
7 years ago
759ba26fb0
[luscious] proper image order for picture albums
...
... and (try) to start with the first image instead of somewhere
in the middle of an album.
7 years ago
0381ae5318
replace error handlers for stdout and co.
...
Python3.5 and lower throw an UnicodeEncodeError when trying to print
not-encodable characters when not using 'utf-8' as encoding.
Setting their error handlers to 'replace' should help.
7 years ago
64d7c85b55
[exhentai] improve metadata
...
- add 'width', 'height' and 'size' (in bytes) for each image
- change the former 'size' and 'size_units' into 'gallery_size'
7 years ago
a112e3f2a0
[nijie] add doujin extractor
...
adds support for "https://nijie.info/members_dojin.php?id= <artist_id>"
7 years ago
299ae24996
[test] add a few downloader tests
7 years ago
dd314279fb
[test] add unit tests for extractor module functions
7 years ago
f5c6a2d7f5
[nhentai] use API to get gallery info
7 years ago
8ef790de12
update .travis.yml
...
- restrict builds to master branch and release tags
- implement 'core' and 'results' test categories
7 years ago
557cb94f81
[deviantart] use proper exponential backoff on API errors
...
... and use separate API credentials for unit tests.
7 years ago
b69cc94f0e
[util] implement bencode()
7 years ago
4d74749496
[tests] rework filters for extractor tests
...
CI incompatible tests will now only be skipped if tests are run in
a CI environment.
7 years ago
32bbd12f08
update extractor tests
7 years ago
749fbbfa6c
[mangadex] add chapter- and manga-extractor
7 years ago
5008e105ee
update archive IDs
...
... to behave in a more straightforward way when dealing with
bookmarks/favourites/etc.
specific IDs are now grouped by their owner, album-id, ... to
allow for duplicates when it would be expected.
7 years ago
2fad0b1f1b
add 'U' conversion for format strings to unquote their content
...
(#74 )
7 years ago
8f338347b6
[imagehosts] cleanup
...
removed
- chronos.to - unable to resolve hostname
- coreimg.net - same
- imgmaid.net - same
- hosturimage.com - everything returns 404
- imageontime.org - redirects to some shady site
- imgupload.yt - cloudflare error 522, host down
- img4ever.net - read timeout
7 years ago
e1e0668ca8
add option to set default replacement field value
...
Missing or undefined keywords will now be replaced with the value
set for 'keywords-default'. The default is Python's 'None', which
is equivalent to setting this option to JSON's 'null'.
7 years ago
ac3da8115e
[util] don't add text: URLs to list of downloaded URLs
7 years ago
89440382ad
[tumblr] use separate API key for unit tests
7 years ago
b50bdbf3d7
change config specifiers in input file format
...
Instead of a dictionary/object, input file options are now specified
by a 'key=value' pair starting with '-' for options only applying to
the next URL or '-G' for Global options applying to all following URLs.
See the docstring of parse_inputfile() for details.
Example option specifiers:
- filename = "{id}.{extension}"
- extractor.pixiv.user.directory = ["Pixiv Users", "{user[id]}"]
-spaces="are_optional"
-G keywords = {"global": "option"}
7 years ago
be3ea4425d
test archive-id creation and uniqueness
7 years ago
b73b8b4f50
add OAuth unittests
7 years ago
f5f2d29f56
[nijie] fix dojin extraction
...
- correctly extract artist_id
- set extension to "jpg" if it was empty and let filetype checks do
the rest
7 years ago
7a412f5c32
implement generic manga-chapter extractor
7 years ago
aa38eab2be
allow not-defined fields in format strings
...
... and replace them with "None", for now
7 years ago
619387cbb1
update extractor unittest results
7 years ago
f94e3706a8
use logging module for error messages during downloads
7 years ago
0dd48d644f
update test results
...
nothing broke, but things got updated or changed
7 years ago
1e93955170
[batoto] remove module
...
Site officially shut down on 2018.01.18
7 years ago
f10ffc0839
update extractor blacklist to also allow classes
7 years ago
35e09869d1
[mangapark] fix image URLs and use HTTPS
7 years ago
4edb25346e
[slideshare] support mobile URLs ( closes #67 )
7 years ago
b33efc99a4
[idolcomplex] add support for idol.sankakucomplex.com
7 years ago
1a70857a12
update extractor-unittest capabilities
...
- "count" can now be a string defining a comparison in the form of
'<operator> <value>', for example: '> 12' or '!= 1'. If its value
is not a string, it is assumed to be a concrete integer as before.
- "keyword" can now be a dictionary defining tests for individual keys.
These tests can either be a type, a concrete value or a regex
starting with "re:". Dictionaries can be stacked inside each other.
Optional keys can be indicated with a "?" before its name.
For example:
"keyword:" {
"image_id": int,
"gallery_id", 123,
"name": "re:pattern",
"user": {
"id": 321,
},
"?optional": None,
}
7 years ago
28cd78aae0
[kissmanga] extend chapter-string regex ( closes #58 )
7 years ago
fc7d165c97
[deviantart] add support for OAuth2 authentication
...
Some user galleries [*] require you to be either logged in or
authenticated via OAuth2 to access their deviations.
[*] e.g. https://polinaegorussia.deviantart.com/gallery/
--------------
known issue:
A deviantart 'refresh_token' can only be used once and gets updated
whenever it is used to request a new 'access_token', so storing its
initial value in a config file and reusing it again and again is not
possible.
7 years ago
0a9a07a6e1
[slideshare] improve metadata; flake8
...
- added 'views' and 'published' keywords
- fixed longer titles and descriptions
7 years ago
291369eab2
various smaller changes/additions
7 years ago
300346ecdf
[mangazuki] remove extractors
...
This site has been in "rebuild"-mode for a fairly long time and the
current extractor code isn't going to work for the new version either.
7 years ago
93482a1f88
implement 'util.advance()'
7 years ago
a718c6c6cd
implement 'util.parse_bytes()'
7 years ago
214972bc9a
[gelbooru] use manual extraction
...
... to compensate for their disabled API.
(https://gelbooru.com/index.php?page=forum&s=view&id=3875 )
This also adds an extractor for image-pools.
7 years ago
b14de6ffc2
[tumblr] small improvements
...
- don't transform inline GIF URLs
- set 'type' parameter for API calls if there is only
one post type selected
7 years ago
b8cdd42cab
[senmanga] fix extraction (again)
...
this is basically a re-revert of 2ace5c7
7 years ago
6913eeaa40
[powermanga] replace manga extractor unit test
...
My Hero Academia is gone
7 years ago
f72318e593
[seiga] support more than 200 images
...
Due to API restrictions and/or missing knowledge about and
documentation of API usage, it was only possible to retrieve the
latest 200 images of a niconico seiga user with said API.
The new approach manually visits each HTML page and gets its
information from there.
7 years ago
2457b71633
skip tests on 5xx status codes
7 years ago
305da540c3
[mangahere] fix metadata extraction
7 years ago
035ef655f1
[imagefap] update unit tests
...
old gallery/image has been deleted
7 years ago
caf26412dd
add option to set alternate location of .part files ( #29 )
...
Note: The path set for 'downloader.*.part-directory' needs to point to an
already existing directory.
7 years ago
27c026543f
re-enable download unit tests
7 years ago
b0353aa02d
rewrite download modules ( #29 )
...
- use '.part' files during file-download
- implement continuation of incomplete downloads
- check if file size matches the one reported by server
7 years ago
6af921a952
[sankaku] rewrite/improve ( fixes #44 )
...
- add wait-time between HTTP requests similar to exhentai
- add 'wait-min' and 'wait-max' options
- increase retry-count for HTTP requests to 10
- implement user authentication (non-authenticated users can only view
images up to page 25)
- implement 'skip()' functionality (only works up to page 50)
- implement image-retrieval for pages >= 51
- fix issue with multiple tags
7 years ago
75d3a1f72f
[deviantart] always download original images
...
Deviation-objects returned by the DeviantArt API don't always contain
the URL and metadata of the original image ([1]). Getting this
information requires an additional API call [2], which is indicated by
the 'is_downloadable' and 'download_filesize' metadata within a
deviation-object.
[1] https://myria-moon.deviantart.com/art/Aime-Moi-part-en-vadrouille-261986576
[2] https://www.deviantart.com/developers/http/v1/20160316/deviation_download/bed6982b88949bdb08b52cd6763fcafd
7 years ago
8e6a767109
[util] restructure formatter for better exception propagation
7 years ago
0386503c80
fix (sub)category-transfer for DownloadJob instances ( #41 )
...
... and extend "parent" parameters to TestJob- and DataJob-classes
as well.
7 years ago
41adb99e9c
[pawoo] fix extraction
...
- changed access_token
- use account-search instead of general search
7 years ago
Mike Fährmann
