Mike Fährmann
fd0e1ffd6e
[danbooru] improve 75666cf9
( #4002 )
...
Search for direct post IDs instead of trying to
replicate the same results as the initial request.
1 year ago
Mike Fährmann
708f478d15
[danbooru][e621] add 'date' metadata field ( #4047 )
1 year ago
Mike Fährmann
75666cf9c3
[danbooru] reduce API requests for fetching extended 'metadata'
...
Instead of using one additional API request per post object (N+1),
this requires only one request per 200-post batch.
1 year ago
Mike Fährmann
3d8777fbc1
move user agent string to util.py
2 years ago
Mike Fährmann
cbe4769246
[danbooru] use gallery-dl UA ( #3665 )
...
this removes the ability to set a custom UA via 'user-agent' option
for extractor requests
2 years ago
Mike Fährmann
bbf0911a46
[e621] implement 'notes' and 'pools' metadata extraction
...
(#3425 )
2 years ago
Mike Fährmann
925b467496
split e621 from danbooru module ( #3425 )
2 years ago
Mike Fährmann
c87bd1a752
[danbooru] extend 'metadata' option
...
make it possible to specify a custom list of metadata includes
2 years ago
blankie
f82ee93676
[danbooru] extract uploader metadata (if metadata is set)
2 years ago
ClosedPort22
dd4a4a3fa6
[e621] softcode the pagination threshold
2 years ago
ClosedPort22
9faa4ed738
[e621] refactor pagination control
...
as suggested by @mikf
2 years ago
ClosedPort22
d0ad6d0e67
[e621] implement manual pagination mode
2 years ago
Mike Fährmann
e99ce99284
[danbooru] remove stray 'print()'
2 years ago
Mike Fährmann
4e26bf98f5
[aibooru] support 'safe' subdomain ( #3110 )
2 years ago
thatfuckingbird
062ef238a6
add support for aibooru (using danbooru extractor) ( #3075 )
2 years ago
Mike Fährmann
d0d4ce1a13
[danbooru] fix ugoira metadata extraction ( #3056 )
2 years ago
Mike Fährmann
f362d4a3c7
[e621] fix 'popular' extraction
2 years ago
Mike Fährmann
c6a9bab019
update extractor test results
2 years ago
Mike Fährmann
3f02e483c6
[e621] fix applying request_interval_min ( #2533 )
...
Setting this property after calling Extractor.__init__() has no effect.
2 years ago
Mike Fährmann
563bd0ecf4
[danbooru] inherit from BaseExtractor
...
- merge danbooru and e621 code
- support booru.allthefallen.moe (closes #2283 )
- remove support for old e621 tag search URLs
3 years ago
Mike Fährmann
cadfad4eea
[danbooru] add 'external' option ( closes #1747 )
3 years ago
Mike Fährmann
221015e586
[downloader:http] disable filename extension changes for ugoira
...
(#1507 )
3 years ago
thatfuckingbird
224b883ff4
[danbooru] add option for extended metadata extraction ( #1458 )
...
* [danbooru] add option for extended metadata extraction
* appease linter
* [danbooru] update docs/configuration.rst
* [danbooru] rename extended-metadata -> metadata
3 years ago
Mike Fährmann
d781e6ac44
[e621] return pool posts in order ( closes #1195 )
...
… and add a 'num' enumeration index.
A bit more code than the PR version, but it prints some helpful messages
and doesn't call 'metadata()' twice.
4 years ago
Mike Fährmann
e7d446a8f7
[danbooru] slight code refactoring
4 years ago
Mike Fährmann
1e3dd7330e
merge SharedConfigMixin functionality into Extractor
4 years ago
Mike Fährmann
2626629117
[danbooru] handle posts without 'id' ( fixes #1004 )
4 years ago
Mike Fährmann
e19f665a44
[danbooru] change default for 'ugoira' to 'false'
...
Downloading the pre-rendered versions should be a better default
than .zip files with individual frames.
4 years ago
Mike Fährmann
ad2efa8509
[e621] derive from Danbooru extractors ( #651 )
...
- use extractor implementations from 'danbooru'
- use "page": "b[ID]" to paginate over results instead of
"tags": "id:<[ID]", avoiding infinite loops with certain
post orders
- bump User-Agent version
5 years ago
Mike Fährmann
5bcc7184c9
[danbooru][e621] increase page limits
5 years ago
Mike Fährmann
f117e32910
[danbooru] restore 'popular' functionality
5 years ago
Mike Fährmann
86c00f9e66
[danbooru] move extractor logic from booru.py
5 years ago
Mike Fährmann
f02a768b5c
[danbooru] add 'ugoira' option ( #406 )
...
to choose between ZIP archives or converted video files
for Ugoira posts
5 years ago
Mike Fährmann
6284731107
simplify extractor constants
...
- single strings for URL patterns
- tuples instead of lists for 'directory_fmt' and 'test'
- single-tuple tests where applicable
6 years ago
Mike Fährmann
1e4d351ad3
[danbooru] add authentication support ( closes #151 )
...
... via HTTP Basic Auth with username and "password".
The password value in this case is not the account password itself,
but the"api_key" found in your user profile.
6 years ago
Mike Fährmann
e1d306cc48
update unit test results
6 years ago
Mike Fährmann
6ac403c5d3
add postprocessor config example
6 years ago
Mike Fährmann
3905474805
[booru] call update_page() with correct dict ( closes #82 )
7 years ago
Mike Fährmann
974e73bdbb
[booru] smaller code adjustments
7 years ago
Mike Fährmann
1219ebb7f5
[danbooru] use alternate subdomains; support safebooru
7 years ago
Mike Fährmann
9e8a84ab6c
[booru] rewrite using Mixin classes ( #59 )
...
- improved code structure
- improved URL patterns
- better pagination to work around page limits on
- Danbooru
- e621
- 3dbooru
7 years ago
Mike Fährmann
289d6b65d2
[danbooru] extend and improve URL regex
...
- add support for danbooru mirrors:
- hijiribe.donmai.us
- sonohara.donmai.us
- todo: actually use these domains instead of redirecting everything
to danbooru itself
- improve handling of query string parameters
7 years ago
Mike Fährmann
65997d835b
replace popular/ranking tests with older ones
...
Metadata of several year old lists shouldn't change as much as it
would for newer ones, which makes metadata-comparisons of the output
of build_testresult_db.oy easier.
7 years ago
Mike Fährmann
07214f4007
[booru] place subcategories into base classes
7 years ago
Mike Fährmann
47bcf53ec1
implement support for additional unit test result types
...
- "pattern" matches all resulting URLs against the given regex
- "count" allows to specify the amount of returned URLs
7 years ago
Mike Fährmann
18e6ed1c7e
[booru] add extractors for "Popular" images
7 years ago
Mike Fährmann
c921b4f32a
code cleanup and fixing tests
7 years ago
Mike Fährmann
cf79a47b59
update unit tests
8 years ago
Mike Fährmann
ed94d9b92d
fix/improve various things
8 years ago
Mike Fährmann
94e10f249a
code adjustments according to pep8 nr2
8 years ago