mirror_maker/synapse-s3-storage-provider
1
0
Fork 1
mirror of https://github.com/matrix-org/synapse-s3-storage-provider.git synced 2024-10-23 07:29:40 +00:00
Code Issues Projects Releases Packages Wiki Activity
78 commits 16 branches 9 tags 375 KiB
Commit graph

78 commits

This branch
This branch
All branches
Author SHA1 Message Date
Michael Kaye
ffd3fa4773
Update to v1.0.1 of setup-matrix-synapse (#73) 
Required to be compatible with recently released synapse versions v1.56.0 or higher.
 2022-04-06 11:59:54 +01:00
Michael Kaye
10558c8edc
Integration test s3 storage provider against latest synapse release (#72) 
Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
 2022-03-25 18:20:02 +01:00
Alex Auvolat
8926b4e417
Add support for Synapse running on sqlite3 database in scripts/s3_media_upload (#71) 
Signed-off-by: Alex Auvolat <alex@adnab.me>
 2022-01-13 18:14:52 +00:00
David Robertson
f821cff55e
Make the release action upload to real PyPI (#70) 
* Run `build` and `twine check` in regular CI

Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
 2021-12-01 17:28:02 +00:00
David Robertson
acd45b4aec
Bump version to 1.1.1-test 
I have already uploaded a v1.1 to testpypi (and proper pypi for that
matter) and cannot reupload it.
 2021-11-24 13:23:30 +00:00
David Robertson
ab561b53bd
Correct CI's invocation of twine 2021-11-24 13:23:30 +00:00
David Robertson
af3d6c8ab7
Test uploading to PyPI on release (#69) 2021-11-23 15:13:39 +00:00
David Robertson
84858af0ca
Update version number and give setup.py some love (#68) 
* Mark version as 1.1

The previous owner of the PyPI package updated to a version 1.1 two days
ago, so this ought to be bumped to 1.1 too. See https://pypi.org/project/synapse-s3-storage-provider/1.1/#description

* Extra options in setup.py

* Packaging instructions
 2021-11-22 15:33:10 +00:00
Sean Quah
04d5fb7fe9
Configure @matrix-org/synapse-core to be the code owner for the repo (#67) 
Signed-off-by: Sean Quah <seanq@element.io>
 2021-10-25 14:30:03 +01:00
Brendan Abolivier
fd1e12385d
Use the right config section (#62) 2021-10-12 12:14:27 +02:00
Sean Quah
04e3d31b40
Add support for thumbnail offload to scripts/s3_media_upload (#61) 
Media are now considered deleted only if both the original file and all
thumbnails have been deleted.

`cache.db`s built before this change may incorrectly count media as
deleted while their thumbnails still exist in the local cache. This can
be resolved by either:
 a) deleting `cache.db` and running an `update` to crawl through the
    entire local cache again. This may take an extremely long time for
    large Synapse deployments.
 b) uploading the contents of local_thumbnails/ and remote_thumbnail/
    manually, then deleting the uploaded files. Note that a running
    Synapse instance may write new thumbnails during the process.

    If the S3 storage provider has been installed since the very start
    and configured to store both local and remote media synchronously,
    all thumbnails should already be in S3 and the upload step can be
    skipped.

This commit changes the behavior of the `write` command. Previously,
`write` would only output undeleted file paths. Now the output contains
a mix of file and thumbnail directory paths, which may sometimes already
be deleted / not exist.
 2021-09-15 10:18:26 +01:00
Sean
a5b15d644d
Misc improvements to scripts/s3_media_upload and fix CI (#59) 
scripts/s3_media_upload:
* Mark `scripts/s3_media_upload` as executable
* Fix `scripts/s3_media_upload` shebang to respect virtual environments
* Format `scripts/s3_media_upload`
* Remove unused imports from `scripts/s3_media_upload`
* Include `scripts/s3_media_upload` in CI checks
* Refactor `s3_media_upload`'s `run_check_delete` to use `to_path` instead of duplicating code

CI:
* Fix branch names in CI config
 2021-09-10 11:39:50 +01:00
rkfg
3c3fafd6a2
Support intelligent tiering in s3 upload script (#52) 2021-02-18 18:35:38 +00:00
Patrick Cloke
c0bc3ce95a
Switch CI to GitHub Actions. (#53) 2021-02-17 11:22:05 -05:00
Richard van der Hoff
236e0cddb8
Improve the efficiency of the S3 storage provider (#50) 
there are a few separate things here, which I hope will mean that (a) we use less CPU, and (b) what CPU we do use gets traced to the requests that cause it rather than getting lost down the sofa.
 2021-01-21 12:22:43 +00:00
Erik Johnston
887ee24d76
Merge pull request #48 from matrix-org/michaelkaye/allow_option_not_to_use_upload 
Option to not show progressbar.
 2021-01-18 10:36:31 +00:00
Erik Johnston
3c8ea6076f
Fix up formatting 
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
 2021-01-18 10:08:42 +00:00
Erik Johnston
3d4cc94a0f
Merge pull request #49 from ShadowJonathan/master 
Add S3 endpoint url
 2020-11-19 17:40:00 +00:00
Jonathan de Jong
954e221396 add endpoint url 2020-11-07 14:12:09 +01:00
Michael Kaye
52f2a5bbef Make writes to progress be in global scope. 2020-10-29 14:39:14 +00:00
Michael Kaye
03d21a79df Option to not show progressbar. 2020-10-28 15:59:49 +00:00
Erik Johnston
d58c93d7ca
Merge pull request #47 from matrix-org/erikj/fallback_to_created_ts 
Fix upload script to handle media that has never been accessed
 2020-10-28 10:02:05 +00:00
Erik Johnston
f1b7711d05 Fix upload script to handle media that has never been accessed 
This is done by falling back to the `created_ts` if `last_access_ts` is
null.

Fixes #45.
 2020-10-27 16:45:20 +00:00
Erik Johnston
e1680af511
Merge pull request #43 from matrix-org/erikj/stop_thread_pool 
Stop ThreadPool on exit
 2020-10-19 20:35:10 +01:00
Erik Johnston
9521804529 Stop ThreadPool on exit 
If we don't do this then we end up delaying shutdown by ~30s.
 2020-10-19 17:42:39 +01:00
Patrick Cloke
b4b6525be8
Compatibility with changes to the LoggingContext in Synapse (#36) 2020-05-05 07:11:38 -04:00
Andrew Morgan
26ff8d8def
Fix name of s3_media_upload script and other minor README cleanups (#35) 2020-04-27 11:58:27 +01:00
fabianunterstell
e4adf02bfd
Update typo in README.md (#33) 2020-04-06 15:41:40 +01:00
Kevin Kuehler
a38b15b2a8
doc: Add undocumented storage_class to example (#32) 2020-03-12 18:35:31 +00:00
Richard van der Hoff
17a65cf4f7
Use a threadpool for downloading things from S3 (#30) 
This is a good thing because we need to create a new S3 client for each thread, and creating S3 clients is relatively expensive.
 2020-01-27 16:56:44 +00:00
Richard van der Hoff
410c4382ab
Black the codebase (#29) 
black ftw
 2020-01-23 11:48:59 +00:00
Michael Kaye
fa0194e225
Merge pull request #26 from matrix-org/michaelkaye/move_to_non-binary_psycopg2 
Move to non-binary psycopg2 dependency
 2019-10-18 15:53:23 +01:00
Michael Kaye
916790767b Move to non-binary setup.py 2019-10-03 17:53:57 +01:00
Michael Kaye
339a72dbb4
Merge pull request #27 from matrix-org/michaelkaye/no_py27 
Don't test against py2.7
 2019-10-03 17:53:21 +01:00
Michael Kaye
675d6c65e9 Ensure we still perform "packaging" test 2019-10-03 17:47:04 +01:00
Michael Kaye
9f39da09a9 Don't test against py2.7 2019-10-03 17:46:03 +01:00
Michael Kaye
d42a38dea9
Merge pull request #17 from rkfg/patch-1 
Support intelligent tiering
 2019-06-10 12:14:41 +01:00
Michael Kaye
266c2ace4e
Merge pull request #20 from matrix-org/michaelkaye/use_boto3_session_from_localdata 
Use boto3 session from localdata.
 2019-06-10 12:12:03 +01:00
Michael Kaye
e56e98c6df
Merge pull request #19 from matrix-org/michaelkaye/docs_and_deprecations 
Documentation and deprecation warnings
 2019-06-10 12:11:43 +01:00
Michael Kaye
b0c94ff56e Use boto3 session from localdata. 
This prevents issues when the thread-unsafe Session() is
used by multiple threads by allocating one in local_data.
 2019-06-06 09:57:49 +01:00
Michael Kaye
cea4154788 Move to safe_yaml per https://msg.pyyaml.org/load 2019-06-05 10:01:22 +01:00
Michael Kaye
00d3697e63 Add more documentation, correct line endings. 2019-06-05 10:01:09 +01:00
Michael Kaye
ad284721a0
Merge pull request #18 from matrix-org/michaelkaye/add_cleanup_script 
Add a s3_media_upload script
 2019-06-04 14:47:12 +01:00
rkfg
ad979a6bea Support intelligent tiering 2019-06-04 09:38:43 +03:00
Michael Kaye
fcfc95d0e6 Add a s3_media_upload script 2019-06-03 15:06:06 +01:00
Erik Johnston
197709f1f0
Merge pull request #16 from Sharparam/feature/improvements 
Generalize and improve threading
 2019-05-22 14:32:50 +01:00
Adam Hellberg
af9ee9df53
Fix test_s3.py to work with python 2 2019-03-14 23:10:00 +01:00
Adam Hellberg
5fa3b097dc
Update README 2019-03-14 22:56:23 +01:00
Adam Hellberg
dd1d03f955
Fix line too long 2019-03-14 22:54:41 +01:00
Adam Hellberg
f4df1a69eb
Allow configuration of region and access keys 
Adds the option to set more boto3 options: region_name,
aws_access_key_id, and aws_secret_access_key. This makes it easier to
configure without having to be careful about some CLI tool having the
correct configuration. Also allows setting the region name.
 2019-03-14 22:51:16 +01:00