Cody Wyatt Neiman
8383d2e346
Merge branch 'main' into sse-c
2022-10-30 17:57:17 -04:00
Cody Wyatt Neiman
f39ce16a97
Fix set whitespace
2022-10-30 16:58:08 -04:00
Cody Wyatt Neiman
d08c2dcf22
Resolve linting errors
2022-10-30 16:52:01 -04:00
David Robertson
40c6a5599b
Fix CI and bump actions dependencies to deal with GHA deprecations. ( #83 )
2022-10-25 12:37:48 +01:00
Cody Wyatt Neiman
1e2cec3e56
Fix s3_download_task args order
2022-10-24 21:53:28 -04:00
Cody Wyatt Neiman
c118826d14
Fix remaining eargs run_upload reference
2022-10-24 20:28:11 -04:00
Cody Wyatt Neiman
4ab0624023
Fix run_upload eargs
2022-10-24 20:00:23 -04:00
Cody Wyatt Neiman
3e3f059f6c
Match SSE-C docs closer to other optional configs
2022-10-24 14:58:26 -04:00
Cody Wyatt Neiman
93b41de6b4
Add S3 SSE-C support
...
https://docs.aws.amazon.com/AmazonS3/latest/userguide/ServerSideEncryptionCustomerKeys.html
2022-10-24 14:20:20 -04:00
David Robertson
b0fa9bb140
Bump minimum required version of PyYAML ( #81 )
2022-10-12 12:27:06 +01:00
David Robertson
6d7c60a04a
Correct name of secret in release job
2022-10-12 12:00:17 +01:00
David Robertson
3a312edebd
Bump version to 1.1.1
2022-10-12 11:55:09 +01:00
David Robertson
e762c859ac
Fix dependencies syntax ( #80 )
2022-10-12 11:51:55 +01:00
Erik Johnston
5d734dee80
Merge pull request #77 from kittykat/patch-1
...
Add issue automation for triage
2022-09-02 17:52:31 +02:00
Kat Gerasimova
5a70b9ac8f
Add issue automation for triage
...
Move new issues to https://github.com/orgs/matrix-org/projects/67 for triage
2022-09-02 10:18:45 +01:00
Michael Kaye
ffd3fa4773
Update to v1.0.1 of setup-matrix-synapse ( #73 )
...
Required to be compatible with recently released synapse versions v1.56.0 or higher.
2022-04-06 11:59:54 +01:00
Michael Kaye
10558c8edc
Integration test s3 storage provider against latest synapse release ( #72 )
...
Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
2022-03-25 18:20:02 +01:00
Alex Auvolat
8926b4e417
Add support for Synapse running on sqlite3 database in scripts/s3_media_upload ( #71 )
...
Signed-off-by: Alex Auvolat <alex@adnab.me>
2022-01-13 18:14:52 +00:00
David Robertson
f821cff55e
Make the release action upload to real PyPI ( #70 )
...
* Run `build` and `twine check` in regular CI
Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
2021-12-01 17:28:02 +00:00
David Robertson
acd45b4aec
Bump version to 1.1.1-test
...
I have already uploaded a v1.1 to testpypi (and proper pypi for that
matter) and cannot reupload it.
2021-11-24 13:23:30 +00:00
David Robertson
ab561b53bd
Correct CI's invocation of twine
2021-11-24 13:23:30 +00:00
David Robertson
af3d6c8ab7
Test uploading to PyPI on release ( #69 )
2021-11-23 15:13:39 +00:00
David Robertson
84858af0ca
Update version number and give setup.py some love ( #68 )
...
* Mark version as 1.1
The previous owner of the PyPI package updated to a version 1.1 two days
ago, so this ought to be bumped to 1.1 too. See https://pypi.org/project/synapse-s3-storage-provider/1.1/#description
* Extra options in setup.py
* Packaging instructions
2021-11-22 15:33:10 +00:00
Sean Quah
04d5fb7fe9
Configure @matrix-org/synapse-core to be the code owner for the repo ( #67 )
...
Signed-off-by: Sean Quah <seanq@element.io>
2021-10-25 14:30:03 +01:00
Brendan Abolivier
fd1e12385d
Use the right config section ( #62 )
2021-10-12 12:14:27 +02:00
Sean Quah
04e3d31b40
Add support for thumbnail offload to scripts/s3_media_upload ( #61 )
...
Media are now considered deleted only if both the original file and all
thumbnails have been deleted.
`cache.db`s built before this change may incorrectly count media as
deleted while their thumbnails still exist in the local cache. This can
be resolved by either:
a) deleting `cache.db` and running an `update` to crawl through the
entire local cache again. This may take an extremely long time for
large Synapse deployments.
b) uploading the contents of local_thumbnails/ and remote_thumbnail/
manually, then deleting the uploaded files. Note that a running
Synapse instance may write new thumbnails during the process.
If the S3 storage provider has been installed since the very start
and configured to store both local and remote media synchronously,
all thumbnails should already be in S3 and the upload step can be
skipped.
This commit changes the behavior of the `write` command. Previously,
`write` would only output undeleted file paths. Now the output contains
a mix of file and thumbnail directory paths, which may sometimes already
be deleted / not exist.
2021-09-15 10:18:26 +01:00
Sean
a5b15d644d
Misc improvements to scripts/s3_media_upload and fix CI ( #59 )
...
scripts/s3_media_upload:
* Mark `scripts/s3_media_upload` as executable
* Fix `scripts/s3_media_upload` shebang to respect virtual environments
* Format `scripts/s3_media_upload`
* Remove unused imports from `scripts/s3_media_upload`
* Include `scripts/s3_media_upload` in CI checks
* Refactor `s3_media_upload`'s `run_check_delete` to use `to_path` instead of duplicating code
CI:
* Fix branch names in CI config
2021-09-10 11:39:50 +01:00
rkfg
3c3fafd6a2
Support intelligent tiering in s3 upload script ( #52 )
2021-02-18 18:35:38 +00:00
Patrick Cloke
c0bc3ce95a
Switch CI to GitHub Actions. ( #53 )
2021-02-17 11:22:05 -05:00
Richard van der Hoff
236e0cddb8
Improve the efficiency of the S3 storage provider ( #50 )
...
there are a few separate things here, which I hope will mean that (a) we use less CPU, and (b) what CPU we do use gets traced to the requests that cause it rather than getting lost down the sofa.
2021-01-21 12:22:43 +00:00
Erik Johnston
887ee24d76
Merge pull request #48 from matrix-org/michaelkaye/allow_option_not_to_use_upload
...
Option to not show progressbar.
2021-01-18 10:36:31 +00:00
Erik Johnston
3c8ea6076f
Fix up formatting
...
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2021-01-18 10:08:42 +00:00
Erik Johnston
3d4cc94a0f
Merge pull request #49 from ShadowJonathan/master
...
Add S3 endpoint url
2020-11-19 17:40:00 +00:00
Jonathan de Jong
954e221396
add endpoint url
2020-11-07 14:12:09 +01:00
Michael Kaye
52f2a5bbef
Make writes to progress be in global scope.
2020-10-29 14:39:14 +00:00
Michael Kaye
03d21a79df
Option to not show progressbar.
2020-10-28 15:59:49 +00:00
Erik Johnston
d58c93d7ca
Merge pull request #47 from matrix-org/erikj/fallback_to_created_ts
...
Fix upload script to handle media that has never been accessed
2020-10-28 10:02:05 +00:00
Erik Johnston
f1b7711d05
Fix upload script to handle media that has never been accessed
...
This is done by falling back to the `created_ts` if `last_access_ts` is
null.
Fixes #45 .
2020-10-27 16:45:20 +00:00
Erik Johnston
e1680af511
Merge pull request #43 from matrix-org/erikj/stop_thread_pool
...
Stop ThreadPool on exit
2020-10-19 20:35:10 +01:00
Erik Johnston
9521804529
Stop ThreadPool on exit
...
If we don't do this then we end up delaying shutdown by ~30s.
2020-10-19 17:42:39 +01:00
Patrick Cloke
b4b6525be8
Compatibility with changes to the LoggingContext in Synapse ( #36 )
2020-05-05 07:11:38 -04:00
Andrew Morgan
26ff8d8def
Fix name of s3_media_upload script and other minor README cleanups ( #35 )
2020-04-27 11:58:27 +01:00
fabianunterstell
e4adf02bfd
Update typo in README.md ( #33 )
2020-04-06 15:41:40 +01:00
Kevin Kuehler
a38b15b2a8
doc: Add undocumented storage_class to example ( #32 )
2020-03-12 18:35:31 +00:00
Richard van der Hoff
17a65cf4f7
Use a threadpool for downloading things from S3 ( #30 )
...
This is a good thing because we need to create a new S3 client for each thread, and creating S3 clients is relatively expensive.
2020-01-27 16:56:44 +00:00
Richard van der Hoff
410c4382ab
Black the codebase ( #29 )
...
black ftw
2020-01-23 11:48:59 +00:00
Michael Kaye
fa0194e225
Merge pull request #26 from matrix-org/michaelkaye/move_to_non-binary_psycopg2
...
Move to non-binary psycopg2 dependency
2019-10-18 15:53:23 +01:00
Michael Kaye
916790767b
Move to non-binary setup.py
2019-10-03 17:53:57 +01:00
Michael Kaye
339a72dbb4
Merge pull request #27 from matrix-org/michaelkaye/no_py27
...
Don't test against py2.7
2019-10-03 17:53:21 +01:00
Michael Kaye
675d6c65e9
Ensure we still perform "packaging" test
2019-10-03 17:47:04 +01:00
