Improve error messages when the multipart upload part limit is reached. #5406
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ypatia
reviewed
Dec 16, 2024
| uri.c_str() + "' " + outcome_error_message(upload_part_outcome); | ||
| if (ctx.upload_part_num > max_multipart_part_num) { | ||
| msg += | ||
| " This error might be resolved by increasing the value of the " |
There was a problem hiding this comment.
I would phrase it as: Maximum part limit might have been hit, try increasing the value of "'vfs.s3.multipart_part_size' config option
There was a problem hiding this comment.
I think that the fact that a part limit has been reached is effectively conveyed in the Part number must be an integer between 1 and 10000, inclusive message returned by S3.
ypatia
approved these changes
Dec 17, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
SC-54602
If an S3 multipart upload gets too large, it can exceed the limit of 10000 parts, and it will fail with the confusing message of
Failed to upload part of S3 object 's3://my-bucket/my-object[Error Type: 100] [HTTP Response Code: 400] [Exception: InvalidArgument] […] : Unable to parse ExceptionName: InvalidArgument Message: Part number must be an integer between 1 and 10000, inclusive, that does not clearly indicate to the user how to resolve it. With this PR, if an error happened while uploading a part, and the part size is greater than 10000, a message is appended suggesting that the error might be resolved by increasing thevfs.s3.multipart_part_sizeconfig option.The logic is robust against a potential increase of the limit by S3, since we don't cause the error, but enhance an already caused one.
We also do the same for Azure, whose limit is five times higher than S3. GCS' compose model does not place any limits on the number of parts of an object.
TYPE: IMPROVEMENT
DESC: Improved error messages when the S3 or Azure multipart upload part size is too low, and suggest improving it.