-
Notifications
You must be signed in to change notification settings - Fork 24.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix Bug in Azure Repo Exception Handling #47968
Fix Bug in Azure Repo Exception Handling #47968
Conversation
We were incorrectly handling `IOExceptions` thrown by the `InputStream` side of the upload operation, resulting in a `ClassCastException` as we expected to never get `IOException` from the Azure SDK code but we do in practice. This PR also sets an assertion on `markSupported` for the streams used by the SDK as adding the test for this scenario revealed that the SDK client would retry uploads for non-mark-supporting streams on `IOException` in the `InputStream`.
Pinging @elastic/es-distributed (:Distributed/Snapshot/Restore) |
@@ -294,6 +296,44 @@ public void testWriteLargeBlob() throws Exception { | |||
assertThat(blocks.isEmpty(), is(true)); | |||
} | |||
|
|||
public void testRetryUntilFail() throws IOException { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This test is admittedly a little random/weird, but I figured it captured the initial issue that showed this bug to some degree at least.
plugins/repository-azure/src/main/java/org/elasticsearch/repositories/azure/SocketAccess.java
Outdated
Show resolved
Hide resolved
if (cause instanceof IOException) { | ||
throw (IOException) e.getCause(); | ||
} else { | ||
throw new IOException(cause); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It feels wrong to wrap this as an IOException. Can we just use Throwables.rethrow(cause)
here?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'd rather not tbh. We could wrap in a different exception but Throwables.rethrow
itself just casts to RunTimeException
doesn't it? In the end if the SDK ever changes or we missed some checked exception in our code, we'll get the same bug we started from here again won't we?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
but Throwables.rethrow itself just casts to RunTimeException doesn't it
no it does not
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Huh, Java works in mysterious ways. Sorry for the noise, my bad seems it works just fine :)
=> used it all the way now.
plugins/repository-azure/src/main/java/org/elasticsearch/repositories/azure/SocketAccess.java
Outdated
Show resolved
Hide resolved
Jenkins run elasticsearch-ci/2 (random ML thing) |
Jenkins run elasticsearch-ci/packaging-sample (Vagrant ...) |
Jenkins run elasticsearch-ci/2 (random ML thing again ) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
Thank Yannick + Tanguy! I'm wondering, should this go to |
We were incorrectly handling `IOExceptions` thrown by the `InputStream` side of the upload operation, resulting in a `ClassCastException` as we expected to never get `IOException` from the Azure SDK code but we do in practice. This PR also sets an assertion on `markSupported` for the streams used by the SDK as adding the test for this scenario revealed that the SDK client would retry uploads for non-mark-supporting streams on `IOException` in the `InputStream`.
We were incorrectly handling `IOExceptions` thrown by the `InputStream` side of the upload operation, resulting in a `ClassCastException` as we expected to never get `IOException` from the Azure SDK code but we do in practice. This PR also sets an assertion on `markSupported` for the streams used by the SDK as adding the test for this scenario revealed that the SDK client would retry uploads for non-mark-supporting streams on `IOException` in the `InputStream`.
yes |
We were incorrectly handling `IOExceptions` thrown by the `InputStream` side of the upload operation, resulting in a `ClassCastException` as we expected to never get `IOException` from the Azure SDK code but we do in practice. This PR also sets an assertion on `markSupported` for the streams used by the SDK as adding the test for this scenario revealed that the SDK client would retry uploads for non-mark-supporting streams on `IOException` in the `InputStream`.
We were incorrectly handling `IOExceptions` thrown by the `InputStream` side of the upload operation, resulting in a `ClassCastException` as we expected to never get `IOException` from the Azure SDK code but we do in practice. This PR also sets an assertion on `markSupported` for the streams used by the SDK as adding the test for this scenario revealed that the SDK client would retry uploads for non-mark-supporting streams on `IOException` in the `InputStream`.
We were incorrectly handling `IOExceptions` thrown by the `InputStream` side of the upload operation, resulting in a `ClassCastException` as we expected to never get `IOException` from the Azure SDK code but we do in practice. This PR also sets an assertion on `markSupported` for the streams used by the SDK as adding the test for this scenario revealed that the SDK client would retry uploads for non-mark-supporting streams on `IOException` in the `InputStream`.
We were incorrectly handling `IOExceptions` thrown by the `InputStream` side of the upload operation, resulting in a `ClassCastException` as we expected to never get `IOException` from the Azure SDK code but we do in practice. This PR also sets an assertion on `markSupported` for the streams used by the SDK as adding the test for this scenario revealed that the SDK client would retry uploads for non-mark-supporting streams on `IOException` in the `InputStream`.
We were incorrectly handling
IOExceptions
thrown bythe
InputStream
side of the upload operation, resultingin a
ClassCastException
as we expected to never getIOException
from the Azure SDK code but we do in practice.This PR also sets an assertion on
markSupported
for thestreams used by the SDK as adding the test for this scenario
revealed that the SDK client would retry uploads for
non-mark-supporting streams on
IOException
in theInputStream
.