mirror_maker
/
gallery-dl
mirror of https://github.com/mikf/gallery-dl.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

[foolfuuka] add 'board' extractors (closes #1044)

Browse Source
pull/1229/head
Mike Fährmann 4 years ago
parent 0594821fcd
commit fb64183d53
No known key found for this signature in database
GPG Key ID: 5680CA389D365A88
2 changed files with 77 additions and 27 deletions
Show Stats Download Patch File Download Diff File

18
docs/supportedsites.rst
Unescape View File

@ -8,21 +8,21 @@ Site URL Capabilities
35PHOTO https://35photo.pro/ |35photo-C| 35PHOTO https://35photo.pro/ |35photo-C|
3dbooru http://behoimi.org/ Pools, Popular Images, Posts, Tag Searches 3dbooru http://behoimi.org/ Pools, Popular Images, Posts, Tag Searches
4chan https://www.4chan.org/ Boards, Threads 4chan https://www.4chan.org/ Boards, Threads
4plebs https://archive.4plebs.org/ Threads 4plebs https://archive.4plebs.org/ Boards, Threads
500px https://500px.com/ Galleries, individual Images, User Profiles 500px https://500px.com/ Galleries, individual Images, User Profiles
8kun https://8kun.top/ Boards, Threads 8kun https://8kun.top/ Boards, Threads
8muses https://comics.8muses.com/ Albums 8muses https://comics.8muses.com/ Albums
Adobe Portfolio https://www.myportfolio.com/ Galleries Adobe Portfolio https://www.myportfolio.com/ Galleries
Adult Empire https://www.adultempire.com/ Galleries Adult Empire https://www.adultempire.com/ Galleries
arch.b4k.co https://arch.b4k.co/ Threads arch.b4k.co https://arch.b4k.co/ Boards, Threads
Archive of Sins https://archiveofsins.com/ Threads Archive of Sins https://archiveofsins.com/ Boards, Threads
Archived.Moe https://archived.moe/ Threads Archived.Moe https://archived.moe/ Boards, Threads
ArtStation https://www.artstation.com/ |artstation-C| ArtStation https://www.artstation.com/ |artstation-C|
baraag https://baraag.net/ Images from Statuses, User Profiles `OAuth <https://github.com/mikf/gallery-dl#oauth>`__ baraag https://baraag.net/ Images from Statuses, User Profiles `OAuth <https://github.com/mikf/gallery-dl#oauth>`__
Behance https://www.behance.net/ Collections, Galleries, User Profiles Behance https://www.behance.net/ Collections, Galleries, User Profiles
Blogger https://www.blogger.com/ Blogs, Posts, Search Results Blogger https://www.blogger.com/ Blogs, Posts, Search Results
Danbooru https://danbooru.donmai.us/ Pools, Popular Images, Posts, Tag Searches Supported Danbooru https://danbooru.donmai.us/ Pools, Popular Images, Posts, Tag Searches Supported
Desuarchive https://desuarchive.org/ Threads Desuarchive https://desuarchive.org/ Boards, Threads
DeviantArt https://www.deviantart.com/ |deviantart-C| `OAuth <https://github.com/mikf/gallery-dl#oauth>`__ DeviantArt https://www.deviantart.com/ |deviantart-C| `OAuth <https://github.com/mikf/gallery-dl#oauth>`__
Doki Reader https://kobato.hologfx.com/reader/ Chapters, Manga Doki Reader https://kobato.hologfx.com/reader/ Chapters, Manga
Dynasty Reader https://dynasty-scans.com/ Chapters, individual Images, Search Results Dynasty Reader https://dynasty-scans.com/ Chapters, individual Images, Search Results
@ -32,7 +32,7 @@ Eka's Portal https://aryion.com/ Galleries, Posts
ExHentai https://exhentai.org/ Favorites, Galleries, Search Results Supported ExHentai https://exhentai.org/ Favorites, Galleries, Search Results Supported
Fallen Angels Scans https://www.fascans.com/ Chapters, Manga Fallen Angels Scans https://www.fascans.com/ Chapters, Manga
Fashion Nova https://www.fashionnova.com/ Collections, Products Fashion Nova https://www.fashionnova.com/ Collections, Products
Fireden https://boards.fireden.net/ Threads Fireden https://boards.fireden.net/ Boards, Threads
Flickr https://www.flickr.com/ |flickr-C| `OAuth <https://github.com/mikf/gallery-dl#oauth>`__ Flickr https://www.flickr.com/ |flickr-C| `OAuth <https://github.com/mikf/gallery-dl#oauth>`__
Fur Affinity https://www.furaffinity.net/ |furaffinity-C| `Cookies <https://github.com/mikf/gallery-dl#cookies>`__ Fur Affinity https://www.furaffinity.net/ |furaffinity-C| `Cookies <https://github.com/mikf/gallery-dl#cookies>`__
Fuskator https://fuskator.com/ Galleries, Search Results Fuskator https://fuskator.com/ Galleries, Search Results
@ -89,7 +89,7 @@ Niconico Seiga https://seiga.nicovideo.jp/ individual Images, User
nijie https://nijie.info/ |nijie-C| Required nijie https://nijie.info/ |nijie-C| Required
Nozomi.la https://nozomi.la/ Posts, Search Results, Tag Searches Nozomi.la https://nozomi.la/ Posts, Search Results, Tag Searches
NSFWalbum.com https://nsfwalbum.com/ Albums NSFWalbum.com https://nsfwalbum.com/ Albums
Nyafuu Archive https://archive.nyafuu.org/ Threads Nyafuu Archive https://archive.nyafuu.org/ Boards, Threads
Patreon https://www.patreon.com/ Creators, Posts, User Profiles `Cookies <https://github.com/mikf/gallery-dl#cookies>`__ Patreon https://www.patreon.com/ Creators, Posts, User Profiles `Cookies <https://github.com/mikf/gallery-dl#cookies>`__
Pawoo https://pawoo.net/ Images from Statuses, User Profiles `OAuth <https://github.com/mikf/gallery-dl#oauth>`__ Pawoo https://pawoo.net/ Images from Statuses, User Profiles `OAuth <https://github.com/mikf/gallery-dl#oauth>`__
Photobucket https://photobucket.com/ Albums, individual Images Photobucket https://photobucket.com/ Albums, individual Images
@ -104,7 +104,7 @@ PowerManga https://read.powermanga.org/ Chapters, Manga
Pururin https://pururin.io/ Galleries Pururin https://pururin.io/ Galleries
Read Comic Online https://readcomiconline.to/ Comic Issues, Comics Read Comic Online https://readcomiconline.to/ Comic Issues, Comics
Realbooru https://realbooru.com/ Pools, Posts, Tag Searches Realbooru https://realbooru.com/ Pools, Posts, Tag Searches
RebeccaBlackTech https://rbt.asia/ Threads RebeccaBlackTech https://rbt.asia/ Boards, Threads
Reddit https://www.reddit.com/ |reddit-C| `OAuth <https://github.com/mikf/gallery-dl#oauth>`__ Reddit https://www.reddit.com/ |reddit-C| `OAuth <https://github.com/mikf/gallery-dl#oauth>`__
RedGIFs https://redgifs.com/ individual Images, Search Results, User Profiles RedGIFs https://redgifs.com/ individual Images, Search Results, User Profiles
rule #34 https://rule34.paheal.net/ Posts, Tag Searches rule #34 https://rule34.paheal.net/ Posts, Tag Searches
@ -122,7 +122,7 @@ SlideShare https://www.slideshare.net/ Presentations
SmugMug https://www.smugmug.com/ |smugmug-C| `OAuth <https://github.com/mikf/gallery-dl#oauth>`__ SmugMug https://www.smugmug.com/ |smugmug-C| `OAuth <https://github.com/mikf/gallery-dl#oauth>`__
Speaker Deck https://speakerdeck.com/ Presentations Speaker Deck https://speakerdeck.com/ Presentations
SubscribeStar https://www.subscribestar.com/ Posts, User Profiles Supported SubscribeStar https://www.subscribestar.com/ Posts, User Profiles Supported
The /b/ Archive https://thebarchive.com/ Threads The /b/ Archive https://thebarchive.com/ Boards, Threads
Tsumino https://www.tsumino.com/ Galleries, Search Results Supported Tsumino https://www.tsumino.com/ Galleries, Search Results Supported
Tumblr https://www.tumblr.com/ Likes, Posts, Tag Searches, User Profiles `OAuth <https://github.com/mikf/gallery-dl#oauth>`__ Tumblr https://www.tumblr.com/ Likes, Posts, Tag Searches, User Profiles `OAuth <https://github.com/mikf/gallery-dl#oauth>`__
Twitter https://twitter.com/ |twitter-C| Supported Twitter https://twitter.com/ |twitter-C| Supported

86
gallery_dl/extractor/foolfuuka.py
Unescape View File

@ -1,6 +1,6 @@
# -*- coding: utf-8 -*- # -*- coding: utf-8 -*-
# Copyright 2019 Mike Fährmann # Copyright 2019-2021 Mike Fährmann
# #
# This program is free software; you can redistribute it and/or modify # This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License version 2 as # it under the terms of the GNU General Public License version 2 as
@ -14,19 +14,13 @@ import itertools
import operator import operator
class FoolfuukaThreadExtractor(Extractor): class FoolfuukaExtractor(Extractor):
"""Base extractor for FoolFuuka based boards/archives""" """Base extractor for FoolFuuka based boards/archives"""
basecategory = "foolfuuka" basecategory = "foolfuuka"
subcategory = "thread"
directory_fmt = ("{category}", "{board[shortname]}",
"{thread_num}{title:? - //}")
archive_fmt = "{board[shortname]}_{num}_{timestamp}"
pattern_fmt = r"/([^/]+)/thread/(\d+)"
external = "default" external = "default"
def __init__(self, match): def __init__(self, match):
Extractor.__init__(self, match) Extractor.__init__(self, match)
self.board, self.thread = match.groups()
self.session.headers["Referer"] = self.root self.session.headers["Referer"] = self.root
if self.external == "direct": if self.external == "direct":
self.remote = self._remote_direct self.remote = self._remote_direct
@ -54,16 +48,7 @@ class FoolfuukaThreadExtractor(Extractor):
yield Message.Url, url, post yield Message.Url, url, post
def posts(self): def posts(self):
"""Return an iterable with all posts in this thread""" """Return an iterable with all relevant posts"""
url = self.root + "/_/api/chan/thread/"
params = {"board": self.board, "num": self.thread}
data = self.request(url, params=params).json()[self.thread]
# sort post-objects by key
posts = sorted(data.get("posts", {}).items())
posts = map(operator.itemgetter(1), posts)
return itertools.chain((data["op"],), posts)
def remote(self, media): def remote(self, media):
"""Resolve a remote media link""" """Resolve a remote media link"""
@ -76,6 +61,60 @@ class FoolfuukaThreadExtractor(Extractor):
return media["remote_media_link"] return media["remote_media_link"]
class FoolfuukaThreadExtractor(FoolfuukaExtractor):
"""Base extractor for threads on FoolFuuka based boards/archives"""
subcategory = "thread"
directory_fmt = ("{category}", "{board[shortname]}",
"{thread_num}{title:? - //}")
archive_fmt = "{board[shortname]}_{num}_{timestamp}"
pattern_fmt = r"/([^/?#]+)/thread/(\d+)"
def __init__(self, match):
FoolfuukaExtractor.__init__(self, match)
self.board, self.thread = match.groups()
def posts(self):
url = self.root + "/_/api/chan/thread/"
params = {"board": self.board, "num": self.thread}
data = self.request(url, params=params).json()[self.thread]
# sort post objects by key
posts = sorted(data.get("posts", {}).items())
posts = map(operator.itemgetter(1), posts)
return itertools.chain((data["op"],), posts)
class FoolfuukaBoardExtractor(FoolfuukaExtractor):
"""Base extractor for FoolFuuka based boards/archives"""
subcategory = "board"
pattern_fmt = r"/([^/?#]+)/\d*$"
def __init__(self, match):
FoolfuukaExtractor.__init__(self, match)
self.board = match.group(1)
def items(self):
index_base = "{}/_/api/chan/index/?board={}&page=".format(
self.root, self.board)
thread_base = "{}/{}/thread/".format(self.root, self.board)
for page in itertools.count(1):
with self.request(index_base + format(page)) as response:
try:
threads = response.json()
except ValueError:
threads = None
if not threads:
return
for num, thread in threads.items():
thread["url"] = thread_base + format(num)
thread["_extractor"] = self.childclass
yield Message.Queue, thread["url"], thread
EXTRACTORS = { EXTRACTORS = {
"4plebs": { "4plebs": {
"name": "_4plebs", "name": "_4plebs",
@ -84,6 +123,7 @@ EXTRACTORS = {
"test-thread": ("https://archive.4plebs.org/tg/thread/54059290", { "test-thread": ("https://archive.4plebs.org/tg/thread/54059290", {
"url": "07452944164b602502b02b24521f8cee5c484d2a", "url": "07452944164b602502b02b24521f8cee5c484d2a",
}), }),
"test-board": ("https://archive.4plebs.org/tg/",),
}, },
"archivedmoe": { "archivedmoe": {
"root": "https://archived.moe", "root": "https://archived.moe",
@ -96,6 +136,7 @@ EXTRACTORS = {
"url": "ffec05a1a1b906b5ca85992513671c9155ee9e87", "url": "ffec05a1a1b906b5ca85992513671c9155ee9e87",
}), }),
), ),
"test-board": ("https://archived.moe/gd/",),
}, },
"archiveofsins": { "archiveofsins": {
"root": "https://archiveofsins.com", "root": "https://archiveofsins.com",
@ -104,6 +145,7 @@ EXTRACTORS = {
"url": "f612d287087e10a228ef69517cf811539db9a102", "url": "f612d287087e10a228ef69517cf811539db9a102",
"content": "0dd92d0d8a7bf6e2f7d1f5ac8954c1bcf18c22a4", "content": "0dd92d0d8a7bf6e2f7d1f5ac8954c1bcf18c22a4",
}), }),
"test-board": ("https://archiveofsins.com/h/",),
}, },
"b4k": { "b4k": {
"root": "https://arch.b4k.co", "root": "https://arch.b4k.co",
@ -111,18 +153,21 @@ EXTRACTORS = {
"test-thread": ("https://arch.b4k.co/meta/thread/196/", { "test-thread": ("https://arch.b4k.co/meta/thread/196/", {
"url": "d309713d2f838797096b3e9cb44fe514a9c9d07a", "url": "d309713d2f838797096b3e9cb44fe514a9c9d07a",
}), }),
"test-board": ("https://arch.b4k.co/meta/",),
}, },
"desuarchive": { "desuarchive": {
"root": "https://desuarchive.org", "root": "https://desuarchive.org",
"test-thread": ("https://desuarchive.org/a/thread/159542679/", { "test-thread": ("https://desuarchive.org/a/thread/159542679/", {
"url": "3ae1473f6916ac831efe5cc4d4e7d3298ce79406", "url": "3ae1473f6916ac831efe5cc4d4e7d3298ce79406",
}), }),
"test-board": ("https://desuarchive.org/a/",),
}, },
"fireden": { "fireden": {
"root": "https://boards.fireden.net", "root": "https://boards.fireden.net",
"test-thread": ("https://boards.fireden.net/sci/thread/11264294/", { "test-thread": ("https://boards.fireden.net/sci/thread/11264294/", {
"url": "3adfe181ee86a8c23021c705f623b3657a9b0a43", "url": "3adfe181ee86a8c23021c705f623b3657a9b0a43",
}), }),
"test-board": ("https://boards.fireden.net/sci/",),
}, },
"nyafuu": { "nyafuu": {
"root": "https://archive.nyafuu.org", "root": "https://archive.nyafuu.org",
@ -130,6 +175,7 @@ EXTRACTORS = {
"test-thread": ("https://archive.nyafuu.org/c/thread/2849220/", { "test-thread": ("https://archive.nyafuu.org/c/thread/2849220/", {
"url": "bbe6f82944a45e359f5c8daf53f565913dc13e4f", "url": "bbe6f82944a45e359f5c8daf53f565913dc13e4f",
}), }),
"test-board": ("https://archive.nyafuu.org/c/",),
}, },
"rbt": { "rbt": {
"root": "https://rbt.asia", "root": "https://rbt.asia",
@ -142,6 +188,7 @@ EXTRACTORS = {
"url": "61896d9d9a2edb556b619000a308a984307b6d30", "url": "61896d9d9a2edb556b619000a308a984307b6d30",
}), }),
), ),
"test-board": ("https://rbt.asia/g/",),
}, },
"thebarchive": { "thebarchive": {
"root": "https://thebarchive.com", "root": "https://thebarchive.com",
@ -149,9 +196,12 @@ EXTRACTORS = {
"test-thread": ("https://thebarchive.com/b/thread/739772332/", { "test-thread": ("https://thebarchive.com/b/thread/739772332/", {
"url": "e8b18001307d130d67db31740ce57c8561b5d80c", "url": "e8b18001307d130d67db31740ce57c8561b5d80c",
}), }),
"test-board": ("https://thebarchive.com/b/",),
}, },
"_ckey": "childclass",
} }
generate_extractors(EXTRACTORS, globals(), ( generate_extractors(EXTRACTORS, globals(), (
FoolfuukaThreadExtractor, FoolfuukaThreadExtractor,
FoolfuukaBoardExtractor,
)) ))

Write Preview
Loading…
Cancel
Save