3d8777fbc1
move user agent string to util.py
2 years ago
cbe4769246
[danbooru] use gallery-dl UA ( #3665 )
...
this removes the ability to set a custom UA via 'user-agent' option
for extractor requests
2 years ago
bbf0911a46
[e621] implement 'notes' and 'pools' metadata extraction
...
(#3425 )
2 years ago
925b467496
split e621 from danbooru module ( #3425 )
2 years ago
c87bd1a752
[danbooru] extend 'metadata' option
...
make it possible to specify a custom list of metadata includes
2 years ago
f82ee93676
[danbooru] extract uploader metadata (if metadata is set)
2 years ago
dd4a4a3fa6
[e621] softcode the pagination threshold
2 years ago
9faa4ed738
[e621] refactor pagination control
...
as suggested by @mikf
2 years ago
d0ad6d0e67
[e621] implement manual pagination mode
2 years ago
e99ce99284
[danbooru] remove stray 'print()'
2 years ago
4e26bf98f5
[aibooru] support 'safe' subdomain ( #3110 )
2 years ago
062ef238a6
add support for aibooru (using danbooru extractor) ( #3075 )
2 years ago
d0d4ce1a13
[danbooru] fix ugoira metadata extraction ( #3056 )
2 years ago
f362d4a3c7
[e621] fix 'popular' extraction
2 years ago
c6a9bab019
update extractor test results
2 years ago
3f02e483c6
[e621] fix applying request_interval_min ( #2533 )
...
Setting this property after calling Extractor.__init__() has no effect.
2 years ago
563bd0ecf4
[danbooru] inherit from BaseExtractor
...
- merge danbooru and e621 code
- support booru.allthefallen.moe (closes #2283 )
- remove support for old e621 tag search URLs
3 years ago
cadfad4eea
[danbooru] add 'external' option ( closes #1747 )
3 years ago
221015e586
[downloader:http] disable filename extension changes for ugoira
...
(#1507 )
3 years ago
224b883ff4
[danbooru] add option for extended metadata extraction ( #1458 )
...
* [danbooru] add option for extended metadata extraction
* appease linter
* [danbooru] update docs/configuration.rst
* [danbooru] rename extended-metadata -> metadata
3 years ago
d781e6ac44
[e621] return pool posts in order ( closes #1195 )
...
… and add a 'num' enumeration index.
A bit more code than the PR version, but it prints some helpful messages
and doesn't call 'metadata()' twice.
4 years ago
e7d446a8f7
[danbooru] slight code refactoring
4 years ago
1e3dd7330e
merge SharedConfigMixin functionality into Extractor
4 years ago
2626629117
[danbooru] handle posts without 'id' ( fixes #1004 )
4 years ago
e19f665a44
[danbooru] change default for 'ugoira' to 'false'
...
Downloading the pre-rendered versions should be a better default
than .zip files with individual frames.
4 years ago
ad2efa8509
[e621] derive from Danbooru extractors ( #651 )
...
- use extractor implementations from 'danbooru'
- use "page": "b[ID]" to paginate over results instead of
"tags": "id:<[ID]", avoiding infinite loops with certain
post orders
- bump User-Agent version
5 years ago
5bcc7184c9
[danbooru][e621] increase page limits
5 years ago
f117e32910
[danbooru] restore 'popular' functionality
5 years ago
86c00f9e66
[danbooru] move extractor logic from booru.py
5 years ago
f02a768b5c
[danbooru] add 'ugoira' option ( #406 )
...
to choose between ZIP archives or converted video files
for Ugoira posts
5 years ago
6284731107
simplify extractor constants
...
- single strings for URL patterns
- tuples instead of lists for 'directory_fmt' and 'test'
- single-tuple tests where applicable
6 years ago
1e4d351ad3
[danbooru] add authentication support ( closes #151 )
...
... via HTTP Basic Auth with username and "password".
The password value in this case is not the account password itself,
but the"api_key" found in your user profile.
6 years ago
e1d306cc48
update unit test results
6 years ago
6ac403c5d3
add postprocessor config example
6 years ago
3905474805
[booru] call update_page() with correct dict ( closes #82 )
7 years ago
974e73bdbb
[booru] smaller code adjustments
7 years ago
1219ebb7f5
[danbooru] use alternate subdomains; support safebooru
7 years ago
9e8a84ab6c
[booru] rewrite using Mixin classes ( #59 )
...
- improved code structure
- improved URL patterns
- better pagination to work around page limits on
- Danbooru
- e621
- 3dbooru
7 years ago
289d6b65d2
[danbooru] extend and improve URL regex
...
- add support for danbooru mirrors:
- hijiribe.donmai.us
- sonohara.donmai.us
- todo: actually use these domains instead of redirecting everything
to danbooru itself
- improve handling of query string parameters
7 years ago
65997d835b
replace popular/ranking tests with older ones
...
Metadata of several year old lists shouldn't change as much as it
would for newer ones, which makes metadata-comparisons of the output
of build_testresult_db.oy easier.
7 years ago
07214f4007
[booru] place subcategories into base classes
7 years ago
47bcf53ec1
implement support for additional unit test result types
...
- "pattern" matches all resulting URLs against the given regex
- "count" allows to specify the amount of returned URLs
7 years ago
18e6ed1c7e
[booru] add extractors for "Popular" images
7 years ago
c921b4f32a
code cleanup and fixing tests
7 years ago
cf79a47b59
update unit tests
8 years ago
ed94d9b92d
fix/improve various things
8 years ago
94e10f249a
code adjustments according to pep8 nr2
8 years ago
d7e168799d
consistent extractor naming scheme + docstrings
8 years ago
6f7d42b974
update tests
8 years ago
616e0aedd6
update booru testdata
9 years ago
Mike Fährmann