Merge pull request #27 from fotoente/dev

Update to MiPA

Dockerfile

@@ -12,7 +12,7 @@ WORKDIR /usr/src/app
 COPY requirements.txt ./
 RUN pip install --upgrade pip
 RUN pip install --no-cache-dir -r requirements.txt
-RUN pip install git+https://github.com/yupix/Mi.py.git@v3.9.91
+#RUN pip install git+https://github.com/yupix/Mi.py.git@v3.9.91
 
 COPY . .
 

README.md

@@ -4,7 +4,7 @@ Misskey eBooks Bot with Markov Chain
 [Example @roboduck@ente.fun](https://ente.fun/@roboduck)
 
 ## Introduction
-This small python script is a Markov Chain eBooks bot based on the framework of [mi.py](https://github.com/yupix/Mi.py.git)
+This small python script is a Markov Chain eBooks bot based on the framework of [MiPA](https://github.com/yupix/MiPA.git)
 
 It can only read and write from and to Misskey. Reading from Mastodon or Pleroma is not (yet) implemented.
 
@@ -17,27 +17,31 @@ After this he only updates the database with new posts. The upgrading is threade
 ## Installation
 
 ### Host Installation
-To run `mi.py` you must install `python3.9` and `python3.9-dev` onto your system. (Please be aware of the requirements for mi.py!)
-`mi.py` is still under development and a lot of things change there quickly so please be aware that there could be chances that something changed, that I haven't implemented in the bot at the moment.
-to install `mi.py`please use the following command.
-`pip install git+https://github.com/yupix/Mi.py.git`
+To run `MiPA` you must install `python3.10`onto your system. (Please be aware that not all programs on your system might work under Python3.10!)
+`MiPA` is still under development and a lot of things change there quickly so please be aware that there could be changes, that haven't been implemented in the bot yet! (I try my best to keep it up to date!)
+to install `MiPA`please use the following commands:
+`python3.10 -m pip install git+https://github.com/yupix/MiPA.git`
+`python3.10 -m pip install git+https://github.com/yupix/MiPAC.git`
 
-For the bot to run you also need two additional packages
+For the bot to run you also need a few additional packages
 ```
 markovify
 configparser
+ujson
+requests
+msgpack
 regex
 ```
 
-or just use the command `pip install -r requirements.txt` in the local folder where you cloned the repo.
+or just use the command `python3.10 -m pip install -r requirements.txt` in the local folder where you cloned the repo.
 
 Before starting the bot, please copy `example-bot.cfg` to `bot.cfg` and
 configure it according to the configuration section below.
 
 The best way to run it would be a `systemd` unit file and run it as a daemon.
-Just to test it you can use `nohup python3.9 rdbot.py &` in the directory the bot is located in.
+Just to test it you can use `nohup python3.10 rdbot.py &` in the directory the bot is located in.
 
-### Docker
+### Docker (Will be updated later, so might be broken at the moment!)
 
 To host this image with docker, copy the `docker-compose.yml` file to the directory that you want to host it from.
 
@@ -60,6 +64,7 @@ Following things can be edited:
 |instance_write|domain.tld|Put here the domain of the Misskey instance your bot is running. Only domain name and TLD, no `/`,`:` or `https`
 |token|`String`|The token from your bot. Needs right to write notes and read notification|
 |cw|`String` or `None`|If the markov bot post should be posted with a content warning. Any String given here will show up as CW text. "none" to deactivate.|
+|exclude_links|`boolean`|Should every link starting with `http://` and `https://` be removed? `false` as default|
 |min_notes|`interger`|How many posts should be read at the initial start. Please state a number in 100 increments. Higher number means more variety but also more load when loading those and a bigger database and json file. 5000 notes resulted in ~3 MB disk space used. Default `5000`|
 |max_notes|`interger`|How many posts should be stored in the database. Everything over this number will be deleted during an update cycle Default `5000`|
 |includeReplies|`boolean`|Should reply included into the markov chain? Default `True`|
@@ -74,14 +79,14 @@ Following things can be edited:
 
 Changes to the Markov chain section of the .cfg-file will have immediate effect.
 Changes to the misskey part of the *.cfg-file, requires a restart of the bot.
+If an option is missing from the `misskey` part of the config file, the default values will be used.
 
 ## Known Quirks
 - The startup needs quite some time. On my system about 10 seconds. You know that everything runs well when the first Note is posted.
-- When the bot is started, he runs in a timeout in the first 60 seconds. To prevent that, just mention the bot, and he will stay in a loop.
 
 ## Contributors
 [Shibao](https://github.com/shibaobun) - Docker support and bug hunting<br />
-[Yupix](https://github.com/yupix) - Mi.py framework and clean code<br />
+[Yupix](https://github.com/yupix) - MiPA framework and clean code<br />
 [Nullobsi](https://github.com/nullobsi) - Added multi-user support<br />
 [ThatOneCalculator](https://github.com/ThatOneCalculator) - Option to CW the posts<br />
 

example-bot.cfg

@@ -5,6 +5,7 @@ token=[token here]
 cw=[content warning here; make "none" for no cw]
 
 [markov]
+exclude_links=false
 min_notes=5000
 max_notes=10000
 includeReplies=true

rdbot.py

@@ -1,54 +1,57 @@
 import asyncio
 import threading
 
-from mi.ext import commands, tasks
-from mi.framework import Note
-from mi.framework.router import Router
+from mipa.ext import commands, tasks
+from mipa.router import Router
+from mipac.models import Note
+from mipac.util import check_multi_arg
 
-from roboduck import *
+import roboduck
 
 # Load Misskey configuration
-config = configparser.ConfigParser()
-config.read(Path(__file__).parent.joinpath('bot.cfg'))
-uri = "https://" + config.get("misskey", "instance_write")
+config = roboduck.configparser.ConfigParser()
+config.read(roboduck.Path(__file__).parent.joinpath('bot.cfg'))
+url = "https://" + config.get("misskey", "instance_write")
 token = config.get("misskey", "token")
 
 try:
     contentwarning = config.get("misskey", "cw")
     if contentwarning.lower() == "none":
         contentwarning = None
-except (TypeError, ValueError):
+except (TypeError, ValueError, roboduck.configparser.NoOptionError):
     contentwarning = None
 
+if not check_multi_arg(url, token):
+    raise Exception("Misskey instance and token are required.")
+
 class MyBot(commands.Bot):
-    text_model = None  # Holds the markov object, so it won't be recreated everytime
 
     def __init__(self):
         super().__init__()
 
     @tasks.loop(3600)
     async def loop_1h(self):
-        text = create_sentence()
-        await bot.client.note.send(content=text, visibility="home", cw=contentwarning)
+        text = roboduck.create_sentence()
+        await bot.client.note.action.send(content=text, visibility="public", cw=contentwarning)
 
     @tasks.loop(43200)
     async def loop_12h(self):
-        thread_update = threading.Thread(target=update)
+        thread_update = threading.Thread(target=roboduck.update())
         thread_update.daemon = True
         thread_update.start()
 
     async def on_ready(self, ws):
         await Router(ws).connect_channel(["global", "main"])  # Connect to global and main channels
-        await bot.client.note.send(content=datetime.now().strftime('%Y-%m-%d %H:%M:%S') + " :roboduck: Bot started!",
+        await self.client.note.action.send(content=roboduck.datetime.now().strftime('%Y-%m-%d %H:%M:%S') + " Roboduck Bot started!",
                                    visibility="specified")
         self.loop_12h.start()  # Launching renew posts every 12 hours
         self.loop_1h.start()  #
-        print(datetime.now().strftime('%Y-%m-%d %H:%M:%S') + " Roboduck Bot started!")
+        print(roboduck.datetime.now().strftime('%Y-%m-%d %H:%M:%S') + " Roboduck Bot started!")
 
     async def on_mention(self, note: Note):
         if not note.author.is_bot:
             text = note.author.action.get_mention() + " "
-            text += create_sentence()
+            text += roboduck.create_sentence()
 
             await note.reply(content=text, cw=contentwarning)  # Reply to a note
 
@@ -56,10 +59,10 @@ class MyBot(commands.Bot):
         await Router(ws).connect_channel(["global", "main"])  # Connect to global and main channels
 
 if __name__ == "__main__":
-    databasepath = Path(__file__).parent.joinpath('roboduck.db')
+    databasepath = roboduck.Path(__file__).parent.joinpath('roboduck.db')
 
-    if not (os.path.exists(databasepath) and os.stat(databasepath).st_size != 0):
+    if not (roboduck.os.path.exists(databasepath) and roboduck.os.stat(databasepath).st_size != 0):
         init_bot()
 
     bot = MyBot()
-    asyncio.run(bot.start(uri, token, timeout=600))
+    asyncio.run(bot.start(url, token))

requirements.txt

@@ -4,3 +4,5 @@ ujson
 requests
 msgpack
 regex
+git+https://github.com/yupix/mipa.git
+git+https://github.com/yupix/mipac.git

roboduck.py

@@ -71,19 +71,24 @@ def get_notes(**kwargs):
     # Read & Sanitize Inputs from Config File
     try:
         include_replies = check_str_to_bool(config.get("markov", "includeReplies"))
-    except (TypeError, ValueError):
+    except (TypeError, ValueError, configparser.NoOptionError):
         include_replies = True
 
     try:
         include_my_renotes = check_str_to_bool(config.get("markov", "includeMyRenotes"))
-    except (TypeError, ValueError):
+    except (TypeError, ValueError, configparser.NoOptionError):
         include_my_renotes = False
 
     try:
         exclude_nsfw = check_str_to_bool(config.get("markov", "excludeNsfw"))
-    except (TypeError, ValueError):
+    except (TypeError, ValueError, configparser.NoOptionError):
         exclude_nsfw = True
 
+    try:
+        exclude_links = check_str_to_bool(config.get("markov", "exclude_links"))
+    except (TypeError, ValueError, configparser.NoOptionError):
+        exclude_links = False
+
     run = True
     oldnote = ""
 
@@ -146,6 +151,9 @@ def get_notes(**kwargs):
         content = content.replace("::", ": :")  # Break long emoji chains
         content = content.replace("@", "@" + chr(8203))
 
+        if exclude_links:
+            content = regex.sub(r"(http|https):\/\/([\w_-]+(?:(?:\.[\w_-]+)+))([\w.,@?^=%&:\/~+#-]*[\w@?^=%&\/~+#-]))", "", content)
+
         note_dict = {"id": element["id"], "text": content, "timestamp": lastTimestamp, "user_id": userid}
         return_list.append(note_dict)