Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add retry to S3 connection reset errors #274

Merged
merged 1 commit into from
May 24, 2024

Conversation

Thomasvdam
Copy link
Member

Motivation

Every once in a while the indexing plugin receives a connection rest while attempting to upload a large message to S3. This halts the indexing process as there are some cases where an error in the indexer warrants a reset to the previous block. By default the retry logic from the AWS SDK does not retry on connection resets as it can't know whether the request that was made is idempotent or not.

Explanation of Changes

Based on some reading:
PR trying to 'fix' that the SDK no longer retries connection reset errors: aws/aws-sdk-go#2926
How to determine the error: https://github.com/aws/aws-sdk-go/blob/v1.44.224/aws/request/connection_reset_error.go
Custom retryer example: https://github.com/aws/aws-sdk-go/blob/main/example/aws/request/customRetryer/custom_retryer.go

Since we know that the upload action is idempotent we can safely configure the retry SDK to retry connection reset errors.

Testing

Not entirely sure how to test this. At least it still works for the normal flow.

Related PRs and Issues

Closes: #273

@Thomasvdam Thomasvdam requested a review from a team May 22, 2024 02:39
@Thomasvdam Thomasvdam merged commit 5ecf113 into main May 24, 2024
8 of 17 checks passed
@Thomasvdam Thomasvdam deleted the fix/plugin-conn-reset-retry branch May 24, 2024 01:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

🐛 S3 connection reset errors do not get retried
2 participants