Support S3 Express One Zone #642
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Monthon Klongklaew <[email protected]>
ListObjectsV2 is unordered on S3 Express. This is an issue for our shadowing semantics in readdir, which does a merge-sort of the local and remote keys, assuming they're both ordered. We are going to give up on readdir being ordered on Express -- it's not required by VFS, and the alternative would be to buffer all directory entries in memory before returning any, which would be too expensive on large directories. So this change introduces a new unordered variant of the readdir implementation. It just returns directory entries verbatim from ListObjectsV2, and then returns all otherwise-unreturned local entries at the end of the iterator. Doing this requires Mountpoint to know whether the bucket is a directory bucket or not, and so I added both a command-line flag and auto-detection for that. Signed-off-by: James Bornholt <[email protected]> Signed-off-by: Monthon Klongklaew <[email protected]>
Signed-off-by: James Bornholt <[email protected]>
Signed-off-by: James Bornholt <[email protected]>
Signed-off-by: Monthon Klongklaew <[email protected]>
monthonk
temporarily deployed
to
PR integration tests
GitHub Actions
Inactive
monthonk
temporarily deployed
to
PR integration tests
GitHub Actions
Inactive
monthonk
temporarily deployed
to
PR integration tests
GitHub Actions
Inactive
monthonk
temporarily deployed
to
PR integration tests
GitHub Actions
Inactive
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description of change
This change configure
mountpoint-s3-clientto use the new SigV4-Express signing algorithm when we detected that the bucket is an S3 Express One Zone bucket.The tests are modified to support testing against S3 Express One Zone buckets, we can enable them by providing environment variables
S3_BUCKET_NAME,S3_EXPRESS_ONE_ZONE_ENDPOINTand run the test withs3express_testsfeature flag. For example. if the S3 Express One Zone bucket name isdoc-example-bucket--use1-az5--x-s3, we can set environment variables with the following valuesS3_BUCKET_NAME=doc-example-bucket--use1-az5--x-s3S3_EXPRESS_ONE_ZONE_ENDPOINT=https://doc-example-bucket--use1-az5--x-s3.s3express-use1-az5.us-east-1.amazonaws.comThis PR also includes @jamesbornholt's change to fix a problem where ListObjectsV2 is unordered on S3 Express. This is an issue for our shadowing semantics in readdir, which does a merge-sort of the local and remote keys, assuming they're both ordered.
We are going to give up on readdir being ordered on Express -- it's not required by VFS, and the alternative would be to buffer all directory entries in memory before returning any, which would be too expensive on large directories. So this change introduces a new unordered variant of the readdir implementation. It just returns directory entries verbatim from ListObjectsV2, and then returns all otherwise-unreturned local entries at the end of the iterator.
Doing this requires Mountpoint to know whether the bucket is a directory bucket or not, and so I added both a command-line flag and auto-detection for that.
Does this change impact existing behavior?
No breaking changes.
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license and I agree to the terms of the Developer Certificate of Origin (DCO).