Add retry to S3 connection reset errors #274
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation
Every once in a while the indexing plugin receives a connection rest while attempting to upload a large message to S3. This halts the indexing process as there are some cases where an error in the indexer warrants a reset to the previous block. By default the retry logic from the AWS SDK does not retry on connection resets as it can't know whether the request that was made is idempotent or not.
Explanation of Changes
Based on some reading:
PR trying to 'fix' that the SDK no longer retries connection reset errors: aws/aws-sdk-go#2926
How to determine the error: https://github.com/aws/aws-sdk-go/blob/v1.44.224/aws/request/connection_reset_error.go
Custom retryer example: https://github.com/aws/aws-sdk-go/blob/main/example/aws/request/customRetryer/custom_retryer.go
Since we know that the upload action is idempotent we can safely configure the retry SDK to retry connection reset errors.
Testing
Not entirely sure how to test this. At least it still works for the normal flow.
Related PRs and Issues
Closes: #273