Victor LABORIE
44bac8dbc0
Add support for session_token
2024-10-15 20:56:05 +02:00
Adam Hellberg
2c46a764f7
Remove empty LoggingContext ( #109 )
...
As per the comment on parent repo:
https://github.com/matrix-org/synapse-s3-storage-provider/issues/56#issuecomment-1103873051
Signed-off-by: Adam Hellberg <sharparam@sharparam.com>
2024-03-19 16:59:39 +00:00
Ben Banfield-Zanin
9490124b2a
Support using a prefix of the S3 bucket ( #110 )
2024-03-14 12:08:38 +00:00
Cody Neiman
c65f9e5ba5
Fix SSE-C extra_arg checks ( #87 )
...
Fixes https://github.com/matrix-org/synapse-s3-storage-provider/issues/86
2022-11-15 12:19:19 +00:00
Cody Wyatt Neiman
139e44673c
Add extra_args comment and link to docs
2022-11-04 13:18:53 -04:00
Cody Wyatt Neiman
5957773cf7
Refactor eargs to extra_args
2022-11-04 13:17:00 -04:00
Cody Wyatt Neiman
f39ce16a97
Fix set whitespace
2022-10-30 16:58:08 -04:00
Cody Wyatt Neiman
d08c2dcf22
Resolve linting errors
2022-10-30 16:52:01 -04:00
Cody Wyatt Neiman
1e2cec3e56
Fix s3_download_task args order
2022-10-24 21:53:28 -04:00
Cody Wyatt Neiman
93b41de6b4
Add S3 SSE-C support
...
https://docs.aws.amazon.com/AmazonS3/latest/userguide/ServerSideEncryptionCustomerKeys.html
2022-10-24 14:20:20 -04:00
Brendan Abolivier
fd1e12385d
Use the right config section ( #62 )
2021-10-12 12:14:27 +02:00
Richard van der Hoff
236e0cddb8
Improve the efficiency of the S3 storage provider ( #50 )
...
there are a few separate things here, which I hope will mean that (a) we use less CPU, and (b) what CPU we do use gets traced to the requests that cause it rather than getting lost down the sofa.
2021-01-21 12:22:43 +00:00
Erik Johnston
9521804529
Stop ThreadPool on exit
...
If we don't do this then we end up delaying shutdown by ~30s.
2020-10-19 17:42:39 +01:00
Patrick Cloke
b4b6525be8
Compatibility with changes to the LoggingContext in Synapse ( #36 )
2020-05-05 07:11:38 -04:00
Richard van der Hoff
17a65cf4f7
Use a threadpool for downloading things from S3 ( #30 )
...
This is a good thing because we need to create a new S3 client for each thread, and creating S3 clients is relatively expensive.
2020-01-27 16:56:44 +00:00
Richard van der Hoff
410c4382ab
Black the codebase ( #29 )
...
black ftw
2020-01-23 11:48:59 +00:00
Michael Kaye
d42a38dea9
Merge pull request #17 from rkfg/patch-1
...
Support intelligent tiering
2019-06-10 12:14:41 +01:00
Michael Kaye
b0c94ff56e
Use boto3 session from localdata.
...
This prevents issues when the thread-unsafe Session() is
used by multiple threads by allocating one in local_data.
2019-06-06 09:57:49 +01:00
rkfg
ad979a6bea
Support intelligent tiering
2019-06-04 09:38:43 +03:00
Adam Hellberg
dd1d03f955
Fix line too long
2019-03-14 22:54:41 +01:00
Adam Hellberg
f4df1a69eb
Allow configuration of region and access keys
...
Adds the option to set more boto3 options: region_name,
aws_access_key_id, and aws_secret_access_key. This makes it easier to
configure without having to be careful about some CLI tool having the
correct configuration. Also allows setting the region name.
2019-03-14 22:51:16 +01:00
Adam Hellberg
30c30447e7
Merge remote-tracking branch 'djmaze/master' into feature/improvements
2019-03-14 22:43:28 +01:00
Amber Brown
5df69d7201
Fix a Py3 issue and package & test it better ( #15 )
2018-10-23 20:48:22 +11:00
Martin Honermeyer
05d9bf6bd6
Create a separate boto3 session for each file store thread
...
This prevents threading problems, see
https://boto3.amazonaws.com/v1/documentation/api/latest/guide/resources.html#multithreading-multiprocessing .
2018-09-13 23:37:21 +02:00
Erik Johnston
ba5d7856d1
Fix incorrect arguments
2018-08-23 14:46:46 +01:00
Erik Johnston
005f857917
Factor out generic streaming from s3 specific
2018-08-23 12:52:27 +01:00
Erik Johnston
1314baff14
Convert to be a PushProducer
2018-08-23 11:01:36 +01:00
Saad Rhoulam
503fd7ba38
Generalize for usage in other S3-like services.
2018-08-21 01:20:24 -04:00
Erik Johnston
4a1356c0b2
Correctly catch 404 errors
...
The docs are super unclear on how one should catch a missing file
exception, but empirically this should catch the errors we're seeing.
2018-03-14 12:09:41 +00:00
Erik Johnston
925d1c602b
Add storage class config option
2018-02-13 16:59:28 +00:00
Erik Johnston
49414477a0
Implement uploading of files to s3
2018-02-12 15:06:49 +00:00
Erik Johnston
0e0c652212
Fix up comment
2018-02-12 13:50:47 +00:00
Erik Johnston
231426db5f
Reorder file
2018-02-12 13:50:21 +00:00
Erik Johnston
c1239fbea3
Correctly handle logcontexts
2018-02-12 13:49:43 +00:00
Erik Johnston
b6743e6d7b
Implement downloading files from S3
2018-02-07 11:39:58 +00:00
