2018-02-13 13:55:17 +00:00
|
|
|
Synapse S3 Storage Provider
|
|
|
|
===========================
|
|
|
|
|
|
|
|
This module can be used by synapse as a storage provider, allowing it to fetch
|
|
|
|
and store media in Amazon S3.
|
|
|
|
|
|
|
|
|
|
|
|
Usage
|
|
|
|
-----
|
|
|
|
|
|
|
|
The `s3_storage_provider.py` should be on the PYTHONPATH when starting
|
|
|
|
synapse.
|
|
|
|
|
|
|
|
Example of entry in synapse config:
|
|
|
|
|
|
|
|
```yaml
|
|
|
|
media_storage_providers:
|
|
|
|
- module: s3_storage_provider.S3StorageProviderBackend
|
|
|
|
store_local: True
|
|
|
|
store_remote: True
|
|
|
|
store_synchronous: True
|
|
|
|
config:
|
|
|
|
bucket: <S3_BUCKET_NAME>
|
2019-03-14 21:56:23 +00:00
|
|
|
# All of the below options are optional, for use with non-AWS S3-like
|
|
|
|
# services, or to specify access tokens here instead of some external method.
|
|
|
|
region_name: <S3_REGION_NAME>
|
|
|
|
endpoint_url: <S3_LIKE_SERVICE_ENDPOINT_URL>
|
|
|
|
access_key_id: <S3_ACCESS_KEY_ID>
|
|
|
|
secret_access_key: <S3_SECRET_ACCESS_KEY>
|
2020-01-27 16:56:44 +00:00
|
|
|
|
2020-03-12 18:35:31 +00:00
|
|
|
# The object storage class used when uploading files to the bucket.
|
|
|
|
# Default is STANDARD.
|
|
|
|
#storage_class: "STANDARD_IA"
|
|
|
|
|
|
|
|
# The maximum number of concurrent threads which will be used to connect
|
|
|
|
# to S3. Each thread manages a single connection. Default is 40.
|
2020-01-27 16:56:44 +00:00
|
|
|
#
|
|
|
|
#threadpool_size: 20
|
2018-02-13 13:55:17 +00:00
|
|
|
```
|
|
|
|
|
|
|
|
This module uses `boto3`, and so the credentials should be specified as
|
|
|
|
described [here](https://boto3.readthedocs.io/en/latest/guide/configuration.html#guide-configuration).
|
|
|
|
|
2019-06-03 14:06:06 +00:00
|
|
|
Regular cleanup job
|
|
|
|
-------------------
|
|
|
|
|
2020-04-27 10:58:27 +00:00
|
|
|
There is additionally a script at `scripts/s3_media_upload` which can be used
|
|
|
|
in a regular job to upload content to s3, then delete that from local disk.
|
|
|
|
This script can be used in combination with configuration for the storage
|
|
|
|
provider to pull media from s3, but upload it asynchronously.
|
2019-06-03 14:06:06 +00:00
|
|
|
|
2019-06-05 09:01:09 +00:00
|
|
|
Once the package is installed, the script should be run somewhat like the
|
2020-04-27 10:58:27 +00:00
|
|
|
following. We suggest using `tmux` or `screen` as these can take a long time
|
|
|
|
on larger servers.
|
2019-06-05 09:01:09 +00:00
|
|
|
|
2020-04-06 14:41:40 +00:00
|
|
|
`database.yaml` should contain the keys that would be passed to psycopg2 to
|
2019-06-05 09:01:09 +00:00
|
|
|
connect to your database. They can be found in the contents of the
|
|
|
|
`database`.`args` parameter in your homeserver.yaml.
|
2019-06-03 14:06:06 +00:00
|
|
|
|
|
|
|
More options are available in the command help.
|
|
|
|
|
|
|
|
```
|
|
|
|
> cd s3_media_upload
|
2019-06-05 09:01:09 +00:00
|
|
|
# cache.db will be created if absent. database.yaml is required to
|
|
|
|
# contain PG credentials
|
2019-06-03 14:06:06 +00:00
|
|
|
> ls
|
|
|
|
cache.db database.yaml
|
2019-06-05 09:01:09 +00:00
|
|
|
# Update cache from /path/to/media/store looking for files not used
|
|
|
|
# within 2 months
|
2019-06-03 14:06:06 +00:00
|
|
|
> s3_media_upload update /path/to/media/store 2m
|
|
|
|
Syncing files that haven't been accessed since: 2018-10-18 11:06:21.520602
|
|
|
|
Synced 0 new rows
|
|
|
|
100%|█████████████████████████████████████████████████████████████| 1074/1074 [00:33<00:00, 25.97files/s]
|
|
|
|
Updated 0 as deleted
|
|
|
|
|
|
|
|
> s3_media_upload upload /path/to/media/store matrix_s3_bucket_name --storage-class STANDARD_IA --delete
|
|
|
|
# prepare to wait a long time
|
|
|
|
```
|
2021-11-22 15:33:10 +00:00
|
|
|
|
|
|
|
Packaging
|
|
|
|
---------
|
|
|
|
|
|
|
|
For maintainers:
|
|
|
|
|
|
|
|
1. Update the `__version__` in setup.py.
|
|
|
|
2. Build, check and upload to PyPI as follows:
|
|
|
|
|
|
|
|
```shell
|
|
|
|
python -m build
|
|
|
|
twine check dist/*
|
|
|
|
twine upload dist/*
|
|
|
|
```
|