Erik Johnston
9521804529
Stop ThreadPool on exit
...
If we don't do this then we end up delaying shutdown by ~30s.
2020-10-19 17:42:39 +01:00
Patrick Cloke
b4b6525be8
Compatibility with changes to the LoggingContext in Synapse ( #36 )
2020-05-05 07:11:38 -04:00
Richard van der Hoff
17a65cf4f7
Use a threadpool for downloading things from S3 ( #30 )
...
This is a good thing because we need to create a new S3 client for each thread, and creating S3 clients is relatively expensive.
2020-01-27 16:56:44 +00:00
Richard van der Hoff
410c4382ab
Black the codebase ( #29 )
...
black ftw
2020-01-23 11:48:59 +00:00
Michael Kaye
d42a38dea9
Merge pull request #17 from rkfg/patch-1
...
Support intelligent tiering
2019-06-10 12:14:41 +01:00
Michael Kaye
b0c94ff56e
Use boto3 session from localdata.
...
This prevents issues when the thread-unsafe Session() is
used by multiple threads by allocating one in local_data.
2019-06-06 09:57:49 +01:00
rkfg
ad979a6bea
Support intelligent tiering
2019-06-04 09:38:43 +03:00
Adam Hellberg
dd1d03f955
Fix line too long
2019-03-14 22:54:41 +01:00
Adam Hellberg
f4df1a69eb
Allow configuration of region and access keys
...
Adds the option to set more boto3 options: region_name,
aws_access_key_id, and aws_secret_access_key. This makes it easier to
configure without having to be careful about some CLI tool having the
correct configuration. Also allows setting the region name.
2019-03-14 22:51:16 +01:00
Adam Hellberg
30c30447e7
Merge remote-tracking branch 'djmaze/master' into feature/improvements
2019-03-14 22:43:28 +01:00
Amber Brown
5df69d7201
Fix a Py3 issue and package & test it better ( #15 )
2018-10-23 20:48:22 +11:00
Martin Honermeyer
05d9bf6bd6
Create a separate boto3 session for each file store thread
...
This prevents threading problems, see
https://boto3.amazonaws.com/v1/documentation/api/latest/guide/resources.html#multithreading-multiprocessing .
2018-09-13 23:37:21 +02:00
Erik Johnston
ba5d7856d1
Fix incorrect arguments
2018-08-23 14:46:46 +01:00
Erik Johnston
005f857917
Factor out generic streaming from s3 specific
2018-08-23 12:52:27 +01:00
Erik Johnston
1314baff14
Convert to be a PushProducer
2018-08-23 11:01:36 +01:00
Saad Rhoulam
503fd7ba38
Generalize for usage in other S3-like services.
2018-08-21 01:20:24 -04:00
Erik Johnston
4a1356c0b2
Correctly catch 404 errors
...
The docs are super unclear on how one should catch a missing file
exception, but empirically this should catch the errors we're seeing.
2018-03-14 12:09:41 +00:00
Erik Johnston
925d1c602b
Add storage class config option
2018-02-13 16:59:28 +00:00
Erik Johnston
49414477a0
Implement uploading of files to s3
2018-02-12 15:06:49 +00:00
Erik Johnston
0e0c652212
Fix up comment
2018-02-12 13:50:47 +00:00
Erik Johnston
231426db5f
Reorder file
2018-02-12 13:50:21 +00:00
Erik Johnston
c1239fbea3
Correctly handle logcontexts
2018-02-12 13:49:43 +00:00
Erik Johnston
b6743e6d7b
Implement downloading files from S3
2018-02-07 11:39:58 +00:00