S3: Abort failed multipart uploads by default

[neowulf]

An option can be given to retry but the retry logic can be tackled in another ticket (or not, the client can handle the retry logic).

Failed multipart uploads incur S3 costs. A completed multipart upload is the one which cleans up the multiparts.

Reference:

1. http://docs.aws.amazon.com/AmazonS3/latest/dev/mpuoverview.html

[neowulf]

Troubleshoot Note

For those who would like to manually clean up in the meantime, perform the following operations:

aws s3api list-multipart-uploads --bucket bucket_name > list.txt

Manually check within the list.txt if all of them need to be aborted. It seems like we don't know which uploadId has failed or completed by looking at this list. To delete a given uploadId, use:

aws s3api abort-multipart-upload --bucket bucket_name --key key_name --upload-id upload_id

If all of the uploads need to be deleted, perform the following:

cat list.txt | jq '.Uploads[].UploadId' | xargs -t -n1 aws s3api abort-multipart-upload --bucket bucket_name --key key_name --upload-id

Ensure all parts have been deleted by performing list-multipart-uploads.

References:

1. http://docs.aws.amazon.com/cli/latest/reference/s3api/list-multipart-uploads.html
2. http://docs.aws.amazon.com/cli/latest/reference/s3api/abort-multipart-upload.html

[mdedetrich]

Do note that Alpakka S3 now contains both the list multipart uploads and abort multipart uploads in the S3 API, this was merged in #2730. This gives you the ability to programatically clean up any failed S3 multipart uploads.

I think this ticket can now be closed

[ennru]

Thank you.
This will be released as part of Alpakka 3.0.4.