Commit graph

22 commits

Author SHA1 Message Date
Ben Banfield-Zanin
acbcc412cc
Allow configuration of where the Synapse homeserver yaml file containing DB settings is loaded from (#111) 2024-03-14 12:10:03 +00:00
Ben Banfield-Zanin
9490124b2a
Support using a prefix of the S3 bucket (#110) 2024-03-14 12:08:38 +00:00
Victor Freire
6f8b3821aa
Add support to read database configuration from homeserver.yaml (#91) 2023-03-14 15:38:51 +00:00
Cody Neiman
c65f9e5ba5
Fix SSE-C extra_arg checks (#87)
Fixes https://github.com/matrix-org/synapse-s3-storage-provider/issues/86
2022-11-15 12:19:19 +00:00
Cody Wyatt Neiman
5957773cf7
Refactor eargs to extra_args 2022-11-04 13:17:00 -04:00
Cody Wyatt Neiman
ff28d0a02f
Remove nargs=? from --sse-customer-algo arg 2022-11-04 13:10:40 -04:00
Cody Wyatt Neiman
d08c2dcf22
Resolve linting errors 2022-10-30 16:52:01 -04:00
Cody Wyatt Neiman
c118826d14
Fix remaining eargs run_upload reference 2022-10-24 20:28:11 -04:00
Cody Wyatt Neiman
4ab0624023
Fix run_upload eargs 2022-10-24 20:00:23 -04:00
Cody Wyatt Neiman
93b41de6b4
Add S3 SSE-C support
https://docs.aws.amazon.com/AmazonS3/latest/userguide/ServerSideEncryptionCustomerKeys.html
2022-10-24 14:20:20 -04:00
Alex Auvolat
8926b4e417
Add support for Synapse running on sqlite3 database in scripts/s3_media_upload (#71)
Signed-off-by: Alex Auvolat <alex@adnab.me>
2022-01-13 18:14:52 +00:00
Sean Quah
04e3d31b40
Add support for thumbnail offload to scripts/s3_media_upload (#61)
Media are now considered deleted only if both the original file and all
thumbnails have been deleted.

`cache.db`s built before this change may incorrectly count media as
deleted while their thumbnails still exist in the local cache. This can
be resolved by either:
 a) deleting `cache.db` and running an `update` to crawl through the
    entire local cache again. This may take an extremely long time for
    large Synapse deployments.
 b) uploading the contents of local_thumbnails/ and remote_thumbnail/
    manually, then deleting the uploaded files. Note that a running
    Synapse instance may write new thumbnails during the process.

    If the S3 storage provider has been installed since the very start
    and configured to store both local and remote media synchronously,
    all thumbnails should already be in S3 and the upload step can be
    skipped.

This commit changes the behavior of the `write` command. Previously,
`write` would only output undeleted file paths. Now the output contains
a mix of file and thumbnail directory paths, which may sometimes already
be deleted / not exist.
2021-09-15 10:18:26 +01:00
Sean
a5b15d644d
Misc improvements to scripts/s3_media_upload and fix CI (#59)
scripts/s3_media_upload:
* Mark `scripts/s3_media_upload` as executable
* Fix `scripts/s3_media_upload` shebang to respect virtual environments
* Format `scripts/s3_media_upload`
* Remove unused imports from `scripts/s3_media_upload`
* Include `scripts/s3_media_upload` in CI checks
* Refactor `s3_media_upload`'s `run_check_delete` to use `to_path` instead of duplicating code

CI:
* Fix branch names in CI config
2021-09-10 11:39:50 +01:00
rkfg
3c3fafd6a2
Support intelligent tiering in s3 upload script (#52) 2021-02-18 18:35:38 +00:00
Erik Johnston
887ee24d76
Merge pull request #48 from matrix-org/michaelkaye/allow_option_not_to_use_upload
Option to not show progressbar.
2021-01-18 10:36:31 +00:00
Erik Johnston
3c8ea6076f
Fix up formatting
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2021-01-18 10:08:42 +00:00
Jonathan de Jong
954e221396 add endpoint url 2020-11-07 14:12:09 +01:00
Michael Kaye
52f2a5bbef Make writes to progress be in global scope. 2020-10-29 14:39:14 +00:00
Michael Kaye
03d21a79df Option to not show progressbar. 2020-10-28 15:59:49 +00:00
Erik Johnston
f1b7711d05 Fix upload script to handle media that has never been accessed
This is done by falling back to the `created_ts` if `last_access_ts` is
null.

Fixes #45.
2020-10-27 16:45:20 +00:00
Michael Kaye
cea4154788 Move to safe_yaml per https://msg.pyyaml.org/load 2019-06-05 10:01:22 +01:00
Michael Kaye
fcfc95d0e6 Add a s3_media_upload script 2019-06-03 15:06:06 +01:00