Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement S3 block storage #46

Open
denismo opened this issue Jul 24, 2014 · 0 comments
Open

Implement S3 block storage #46

denismo opened this issue Jul 24, 2014 · 0 comments

Comments

@denismo
Copy link
Owner

denismo commented Jul 24, 2014

DynamoFS storage is quite expensive, while S3 has reasonable price.
S3 allows parallel uploads which can provide data transfers faster than DynamoFS fastest option.
Also, storing large files (GBs) via 64k blocks is very inefficient.

This feature will store data in S3:

  • 1 S3 file per file-system file, the same name and path (in some bucket)
  • Multiple sequential writes will result in multiple parts of a multi-part upload
  • Multi-part upload closes when the file is closed after write
  • File cannot be written to again after the initial open due to immutable nature of S3 (write-once)
  • While the file is being written to, the size and attributes are updated immediately but the file is kept write-locked until it is closed (because it cannot be read from S3 until all multipart uploads finish)
  • After the write, many reads can be executed as usual
  • Reads will be blocked at storage level while the file is being assembled from parts (S3-eventual-consistency) via spin-lock
@denismo denismo added this to the Performance improvements milestone Jul 24, 2014
@denismo denismo self-assigned this Jul 24, 2014
denismo added a commit that referenced this issue Jul 24, 2014
denismo added a commit that referenced this issue Jul 26, 2014
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant