Richard van der Hoff
236e0cddb8
Improve the efficiency of the S3 storage provider ( #50 )
...
there are a few separate things here, which I hope will mean that (a) we use less CPU, and (b) what CPU we do use gets traced to the requests that cause it rather than getting lost down the sofa.
2021-01-21 12:22:43 +00:00
Erik Johnston
887ee24d76
Merge pull request #48 from matrix-org/michaelkaye/allow_option_not_to_use_upload
...
Option to not show progressbar.
2021-01-18 10:36:31 +00:00
Erik Johnston
3c8ea6076f
Fix up formatting
...
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2021-01-18 10:08:42 +00:00
Erik Johnston
3d4cc94a0f
Merge pull request #49 from ShadowJonathan/master
...
Add S3 endpoint url
2020-11-19 17:40:00 +00:00
Jonathan de Jong
954e221396
add endpoint url
2020-11-07 14:12:09 +01:00
Michael Kaye
52f2a5bbef
Make writes to progress be in global scope.
2020-10-29 14:39:14 +00:00
Michael Kaye
03d21a79df
Option to not show progressbar.
2020-10-28 15:59:49 +00:00
Erik Johnston
d58c93d7ca
Merge pull request #47 from matrix-org/erikj/fallback_to_created_ts
...
Fix upload script to handle media that has never been accessed
2020-10-28 10:02:05 +00:00
Erik Johnston
f1b7711d05
Fix upload script to handle media that has never been accessed
...
This is done by falling back to the `created_ts` if `last_access_ts` is
null.
Fixes #45 .
2020-10-27 16:45:20 +00:00
Erik Johnston
e1680af511
Merge pull request #43 from matrix-org/erikj/stop_thread_pool
...
Stop ThreadPool on exit
2020-10-19 20:35:10 +01:00
Erik Johnston
9521804529
Stop ThreadPool on exit
...
If we don't do this then we end up delaying shutdown by ~30s.
2020-10-19 17:42:39 +01:00
Patrick Cloke
b4b6525be8
Compatibility with changes to the LoggingContext in Synapse ( #36 )
2020-05-05 07:11:38 -04:00
Andrew Morgan
26ff8d8def
Fix name of s3_media_upload script and other minor README cleanups ( #35 )
2020-04-27 11:58:27 +01:00
fabianunterstell
e4adf02bfd
Update typo in README.md ( #33 )
2020-04-06 15:41:40 +01:00
Kevin Kuehler
a38b15b2a8
doc: Add undocumented storage_class to example ( #32 )
2020-03-12 18:35:31 +00:00
Richard van der Hoff
17a65cf4f7
Use a threadpool for downloading things from S3 ( #30 )
...
This is a good thing because we need to create a new S3 client for each thread, and creating S3 clients is relatively expensive.
2020-01-27 16:56:44 +00:00
Richard van der Hoff
410c4382ab
Black the codebase ( #29 )
...
black ftw
2020-01-23 11:48:59 +00:00
Michael Kaye
fa0194e225
Merge pull request #26 from matrix-org/michaelkaye/move_to_non-binary_psycopg2
...
Move to non-binary psycopg2 dependency
2019-10-18 15:53:23 +01:00
Michael Kaye
916790767b
Move to non-binary setup.py
2019-10-03 17:53:57 +01:00
Michael Kaye
339a72dbb4
Merge pull request #27 from matrix-org/michaelkaye/no_py27
...
Don't test against py2.7
2019-10-03 17:53:21 +01:00
Michael Kaye
675d6c65e9
Ensure we still perform "packaging" test
2019-10-03 17:47:04 +01:00
Michael Kaye
9f39da09a9
Don't test against py2.7
2019-10-03 17:46:03 +01:00
Michael Kaye
d42a38dea9
Merge pull request #17 from rkfg/patch-1
...
Support intelligent tiering
2019-06-10 12:14:41 +01:00
Michael Kaye
266c2ace4e
Merge pull request #20 from matrix-org/michaelkaye/use_boto3_session_from_localdata
...
Use boto3 session from localdata.
2019-06-10 12:12:03 +01:00
Michael Kaye
e56e98c6df
Merge pull request #19 from matrix-org/michaelkaye/docs_and_deprecations
...
Documentation and deprecation warnings
2019-06-10 12:11:43 +01:00
Michael Kaye
b0c94ff56e
Use boto3 session from localdata.
...
This prevents issues when the thread-unsafe Session() is
used by multiple threads by allocating one in local_data.
2019-06-06 09:57:49 +01:00
Michael Kaye
cea4154788
Move to safe_yaml per https://msg.pyyaml.org/load
2019-06-05 10:01:22 +01:00
Michael Kaye
00d3697e63
Add more documentation, correct line endings.
2019-06-05 10:01:09 +01:00
Michael Kaye
ad284721a0
Merge pull request #18 from matrix-org/michaelkaye/add_cleanup_script
...
Add a s3_media_upload script
2019-06-04 14:47:12 +01:00
rkfg
ad979a6bea
Support intelligent tiering
2019-06-04 09:38:43 +03:00
Michael Kaye
fcfc95d0e6
Add a s3_media_upload script
2019-06-03 15:06:06 +01:00
Erik Johnston
197709f1f0
Merge pull request #16 from Sharparam/feature/improvements
...
Generalize and improve threading
2019-05-22 14:32:50 +01:00
Adam Hellberg
af9ee9df53
Fix test_s3.py to work with python 2
2019-03-14 23:10:00 +01:00
Adam Hellberg
5fa3b097dc
Update README
2019-03-14 22:56:23 +01:00
Adam Hellberg
dd1d03f955
Fix line too long
2019-03-14 22:54:41 +01:00
Adam Hellberg
f4df1a69eb
Allow configuration of region and access keys
...
Adds the option to set more boto3 options: region_name,
aws_access_key_id, and aws_secret_access_key. This makes it easier to
configure without having to be careful about some CLI tool having the
correct configuration. Also allows setting the region name.
2019-03-14 22:51:16 +01:00
Adam Hellberg
30c30447e7
Merge remote-tracking branch 'djmaze/master' into feature/improvements
2019-03-14 22:43:28 +01:00
Amber Brown
5df69d7201
Fix a Py3 issue and package & test it better ( #15 )
2018-10-23 20:48:22 +11:00
Martin Honermeyer
05d9bf6bd6
Create a separate boto3 session for each file store thread
...
This prevents threading problems, see
https://boto3.amazonaws.com/v1/documentation/api/latest/guide/resources.html#multithreading-multiprocessing .
2018-09-13 23:37:21 +02:00
Erik Johnston
a082503444
Merge pull request #10 from matrix-org/michaelkaye/add_boto3_dependency
...
add boto3 as a dependency
2018-08-23 19:03:13 +01:00
Michael Kaye
d8c6a807f5
add boto3 as a dependency
2018-08-23 18:55:50 +01:00
Erik Johnston
2af74f2e28
Merge pull request #9 from matrix-org/erikj/push_producer
...
Convert to being PushProducer and add tests
2018-08-23 15:50:10 +01:00
Erik Johnston
ba5d7856d1
Fix incorrect arguments
2018-08-23 14:46:46 +01:00
Erik Johnston
507a6d7e42
Add TravisCI
2018-08-23 13:26:11 +01:00
Erik Johnston
98a7c55ea2
Add tests
2018-08-23 13:26:08 +01:00
Erik Johnston
005f857917
Factor out generic streaming from s3 specific
2018-08-23 12:52:27 +01:00
Erik Johnston
1314baff14
Convert to be a PushProducer
2018-08-23 11:01:36 +01:00
Michael Kaye
931fd197cf
Merge pull request #8 from matrix-org/hawkowl/packaging
...
Add some packaging
2018-08-21 16:44:04 +01:00
Amber Brown
6abb9aefb7
add some packaging
2018-08-22 00:30:22 +10:00
Saad Rhoulam
503fd7ba38
Generalize for usage in other S3-like services.
2018-08-21 01:20:24 -04:00
