Erik Johnston
3c8ea6076f
Fix up formatting
...
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2021-01-18 10:08:42 +00:00
Erik Johnston
3d4cc94a0f
Merge pull request #49 from ShadowJonathan/master
...
Add S3 endpoint url
2020-11-19 17:40:00 +00:00
Jonathan de Jong
954e221396
add endpoint url
2020-11-07 14:12:09 +01:00
Michael Kaye
52f2a5bbef
Make writes to progress be in global scope.
2020-10-29 14:39:14 +00:00
Michael Kaye
03d21a79df
Option to not show progressbar.
2020-10-28 15:59:49 +00:00
Erik Johnston
d58c93d7ca
Merge pull request #47 from matrix-org/erikj/fallback_to_created_ts
...
Fix upload script to handle media that has never been accessed
2020-10-28 10:02:05 +00:00
Erik Johnston
f1b7711d05
Fix upload script to handle media that has never been accessed
...
This is done by falling back to the `created_ts` if `last_access_ts` is
null.
Fixes #45 .
2020-10-27 16:45:20 +00:00
Erik Johnston
e1680af511
Merge pull request #43 from matrix-org/erikj/stop_thread_pool
...
Stop ThreadPool on exit
2020-10-19 20:35:10 +01:00
Erik Johnston
9521804529
Stop ThreadPool on exit
...
If we don't do this then we end up delaying shutdown by ~30s.
2020-10-19 17:42:39 +01:00
Patrick Cloke
b4b6525be8
Compatibility with changes to the LoggingContext in Synapse ( #36 )
2020-05-05 07:11:38 -04:00
Andrew Morgan
26ff8d8def
Fix name of s3_media_upload script and other minor README cleanups ( #35 )
2020-04-27 11:58:27 +01:00
fabianunterstell
e4adf02bfd
Update typo in README.md ( #33 )
2020-04-06 15:41:40 +01:00
Kevin Kuehler
a38b15b2a8
doc: Add undocumented storage_class to example ( #32 )
2020-03-12 18:35:31 +00:00
Richard van der Hoff
17a65cf4f7
Use a threadpool for downloading things from S3 ( #30 )
...
This is a good thing because we need to create a new S3 client for each thread, and creating S3 clients is relatively expensive.
2020-01-27 16:56:44 +00:00
Richard van der Hoff
410c4382ab
Black the codebase ( #29 )
...
black ftw
2020-01-23 11:48:59 +00:00
Michael Kaye
fa0194e225
Merge pull request #26 from matrix-org/michaelkaye/move_to_non-binary_psycopg2
...
Move to non-binary psycopg2 dependency
2019-10-18 15:53:23 +01:00
Michael Kaye
916790767b
Move to non-binary setup.py
2019-10-03 17:53:57 +01:00
Michael Kaye
339a72dbb4
Merge pull request #27 from matrix-org/michaelkaye/no_py27
...
Don't test against py2.7
2019-10-03 17:53:21 +01:00
Michael Kaye
675d6c65e9
Ensure we still perform "packaging" test
2019-10-03 17:47:04 +01:00
Michael Kaye
9f39da09a9
Don't test against py2.7
2019-10-03 17:46:03 +01:00
Michael Kaye
d42a38dea9
Merge pull request #17 from rkfg/patch-1
...
Support intelligent tiering
2019-06-10 12:14:41 +01:00
Michael Kaye
266c2ace4e
Merge pull request #20 from matrix-org/michaelkaye/use_boto3_session_from_localdata
...
Use boto3 session from localdata.
2019-06-10 12:12:03 +01:00
Michael Kaye
e56e98c6df
Merge pull request #19 from matrix-org/michaelkaye/docs_and_deprecations
...
Documentation and deprecation warnings
2019-06-10 12:11:43 +01:00
Michael Kaye
b0c94ff56e
Use boto3 session from localdata.
...
This prevents issues when the thread-unsafe Session() is
used by multiple threads by allocating one in local_data.
2019-06-06 09:57:49 +01:00
Michael Kaye
cea4154788
Move to safe_yaml per https://msg.pyyaml.org/load
2019-06-05 10:01:22 +01:00
Michael Kaye
00d3697e63
Add more documentation, correct line endings.
2019-06-05 10:01:09 +01:00
Michael Kaye
ad284721a0
Merge pull request #18 from matrix-org/michaelkaye/add_cleanup_script
...
Add a s3_media_upload script
2019-06-04 14:47:12 +01:00
rkfg
ad979a6bea
Support intelligent tiering
2019-06-04 09:38:43 +03:00
Michael Kaye
fcfc95d0e6
Add a s3_media_upload script
2019-06-03 15:06:06 +01:00
Erik Johnston
197709f1f0
Merge pull request #16 from Sharparam/feature/improvements
...
Generalize and improve threading
2019-05-22 14:32:50 +01:00
Adam Hellberg
af9ee9df53
Fix test_s3.py to work with python 2
2019-03-14 23:10:00 +01:00
Adam Hellberg
5fa3b097dc
Update README
2019-03-14 22:56:23 +01:00
Adam Hellberg
dd1d03f955
Fix line too long
2019-03-14 22:54:41 +01:00
Adam Hellberg
f4df1a69eb
Allow configuration of region and access keys
...
Adds the option to set more boto3 options: region_name,
aws_access_key_id, and aws_secret_access_key. This makes it easier to
configure without having to be careful about some CLI tool having the
correct configuration. Also allows setting the region name.
2019-03-14 22:51:16 +01:00
Adam Hellberg
30c30447e7
Merge remote-tracking branch 'djmaze/master' into feature/improvements
2019-03-14 22:43:28 +01:00
Amber Brown
5df69d7201
Fix a Py3 issue and package & test it better ( #15 )
2018-10-23 20:48:22 +11:00
Martin Honermeyer
05d9bf6bd6
Create a separate boto3 session for each file store thread
...
This prevents threading problems, see
https://boto3.amazonaws.com/v1/documentation/api/latest/guide/resources.html#multithreading-multiprocessing .
2018-09-13 23:37:21 +02:00
Erik Johnston
a082503444
Merge pull request #10 from matrix-org/michaelkaye/add_boto3_dependency
...
add boto3 as a dependency
2018-08-23 19:03:13 +01:00
Michael Kaye
d8c6a807f5
add boto3 as a dependency
2018-08-23 18:55:50 +01:00
Erik Johnston
2af74f2e28
Merge pull request #9 from matrix-org/erikj/push_producer
...
Convert to being PushProducer and add tests
2018-08-23 15:50:10 +01:00
Erik Johnston
ba5d7856d1
Fix incorrect arguments
2018-08-23 14:46:46 +01:00
Erik Johnston
507a6d7e42
Add TravisCI
2018-08-23 13:26:11 +01:00
Erik Johnston
98a7c55ea2
Add tests
2018-08-23 13:26:08 +01:00
Erik Johnston
005f857917
Factor out generic streaming from s3 specific
2018-08-23 12:52:27 +01:00
Erik Johnston
1314baff14
Convert to be a PushProducer
2018-08-23 11:01:36 +01:00
Michael Kaye
931fd197cf
Merge pull request #8 from matrix-org/hawkowl/packaging
...
Add some packaging
2018-08-21 16:44:04 +01:00
Amber Brown
6abb9aefb7
add some packaging
2018-08-22 00:30:22 +10:00
Saad Rhoulam
503fd7ba38
Generalize for usage in other S3-like services.
2018-08-21 01:20:24 -04:00
Erik Johnston
54c58922de
Merge pull request #5 from matrix-org/erikj/404_correctly
...
Correctly catch 404 errors
2018-03-14 13:49:40 +00:00
Erik Johnston
4a1356c0b2
Correctly catch 404 errors
...
The docs are super unclear on how one should catch a missing file
exception, but empirically this should catch the errors we're seeing.
2018-03-14 12:09:41 +00:00
