Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[CI] S3BlobContainerRetriesTests testReadRetriesAfterMeaningfulProgress failing #115145

Closed
elasticsearchmachine opened this issue Oct 18, 2024 · 2 comments · Fixed by #115177
Closed

[CI] S3BlobContainerRetriesTests testReadRetriesAfterMeaningfulProgress failing #115145

elasticsearchmachine opened this issue Oct 18, 2024 · 2 comments · Fixed by #115177
Assignees
@ywangd
Labels
:Distributed Coordination/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs low-risk An open issue or test failure that is a low risk to future releases Team:Distributed (Obsolete) Meta label for distributed team (obsolete). Replaced by Distributed Indexing/Coordination. >test-failure Triaged test failures from CI

Comments

@elasticsearchmachine
Copy link
Collaborator

elasticsearchmachine commented Oct 18, 2024 •
edited
Loading

Build Scans:

Reproduction Line:

./gradlew ":modules:repository-s3:test" --tests "org.elasticsearch.repositories.s3.S3BlobContainerRetriesTests.testReadDoesNotRetryForRepositoryAnalysis" -Dtests.seed=E48173E91C46D576 -Dtests.locale=ta-MY -Dtests.timezone=Cuba -Druntime.java=23

Applicable branches:
8.x

Reproduces locally?:
N/A

Failure History:
See dashboard

Failure Message:

java.lang.AssertionError: failureCount=1

Issue Reasons:

  • [8.x] 3 failures in test testReadRetriesAfterMeaningfulProgress (0.6% fail rate in 470 executions)
  • [8.x] 3 failures in step openjdk23_checkpart1_java-matrix (23.1% fail rate in 13 executions)
  • [8.x] 3 failures in pipeline elasticsearch-periodic (23.1% fail rate in 13 executions)

Note:
This issue was created using new test triage automation. Please report issues or feedback to es-delivery.
@elasticsearchmachine elasticsearchmachine added :Distributed Coordination/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs >test-failure Triaged test failures from CI labels Oct 18, 2024
@elasticsearchmachine
Copy link
Collaborator Author

Pinging @elastic/es-distributed (Team:Distributed)

@elasticsearchmachine elasticsearchmachine added Team:Distributed (Obsolete) Meta label for distributed team (obsolete). Replaced by Distributed Indexing/Coordination. needs:risk Requires assignment of a risk label (low, medium, blocker) labels Oct 18, 2024
@ywangd ywangd self-assigned this Oct 21, 2024
@ywangd ywangd added low-risk An open issue or test failure that is a low risk to future releases and removed needs:risk Requires assignment of a risk label (low, medium, blocker) labels Oct 21, 2024
@ywangd
Copy link
Member

ywangd commented Oct 21, 2024

The failure is JDK 23 specific. I think it matches this JDK change: https://bugs.openjdk.org/browse/JDK-8331847

The HTTP server no longer immediately sends response headers if chunked mode is selected or if the response has a body.

A few tests failed. But the main problem is with testReadDoesNotRetryForRepositoryAnalysis which fails with an assertion its the HttpHandler. This killed the http server and subsquent tests failed because of that. The test needs the header to be sent so that it opens the stream successfully and then fails at reading time. But if no header is sent, the opening fails with an NoHttpResponseException which leads to retry by s3client and the test does not allow retry.

ywangd added a commit to ywangd/elasticsearch that referenced this issue Oct 21, 2024
@ywangd
[Test] Flush response body for progress
2d1ca07 
In JDK23, response headers are no longer always immediately sent. See
also https://bugs.openjdk.org/browse/JDK-8331847
This PR adds flush call for the response body to make progress.

Resolves: elastic#115145
@ywangd ywangd mentioned this issue Oct 21, 2024
Merged
@pxsalehi pxsalehi mentioned this issue Oct 21, 2024
Closed
ywangd added a commit to ywangd/elasticsearch that referenced this issue Oct 21, 2024
@ywangd
[Test] Flush response body for progress (elastic#115177)
99ce577 
In JDK23, response headers are no longer always immediately sent. See
also https://bugs.openjdk.org/browse/JDK-8331847 This PR adds flush call
for the response body to make progress.

Resolves: elastic#115145 Resolves: elastic#115164
(cherry picked from commit 8c23fd7)
salvatore-campagna pushed a commit to salvatore-campagna/elasticsearch that referenced this issue Oct 21, 2024
@ywangd @salvatore-campagna
[Test] Flush response body for progress (elastic#115177)
97c5956 
In JDK23, response headers are no longer always immediately sent. See
also https://bugs.openjdk.org/browse/JDK-8331847 This PR adds flush call
for the response body to make progress.

Resolves: elastic#115145 Resolves: elastic#115164
elasticsearchmachine pushed a commit that referenced this issue Oct 21, 2024
@ywangd
[Test] Flush response body for progress (#115177) (#115209)
b69d23d 
In JDK23, response headers are no longer always immediately sent. See
also https://bugs.openjdk.org/browse/JDK-8331847 This PR adds flush call
for the response body to make progress.

Resolves: #115145 Resolves: #115164
(cherry picked from commit 8c23fd7)
ywangd added a commit to ywangd/elasticsearch that referenced this issue Oct 25, 2024
@ywangd
[Test] Flush response body for progress (elastic#115177)
f3b4607 
In JDK23, response headers are no longer always immediately sent. See
also https://bugs.openjdk.org/browse/JDK-8331847 This PR adds flush call
for the response body to make progress.

Resolves: elastic#115145 Resolves: elastic#115164
(cherry picked from commit 8c23fd7)
elasticsearchmachine pushed a commit that referenced this issue Oct 25, 2024
@ywangd
[8.16] [Test] Flush response body for progress (#115177) (#115613)
9f0551a 
* [Test] Flush response body for progress (#115177)

In JDK23, response headers are no longer always immediately sent. See
also https://bugs.openjdk.org/browse/JDK-8331847 This PR adds flush call
for the response body to make progress.

Resolves: #115145 Resolves: #115164
(cherry picked from commit 8c23fd7)

* unmute
georgewallace pushed a commit to georgewallace/elasticsearch that referenced this issue Oct 25, 2024
@ywangd @georgewallace
[Test] Flush response body for progress (elastic#115177)
74302c8 
In JDK23, response headers are no longer always immediately sent. See
also https://bugs.openjdk.org/browse/JDK-8331847 This PR adds flush call
for the response body to make progress.

Resolves: elastic#115145 Resolves: elastic#115164
jfreden pushed a commit to jfreden/elasticsearch that referenced this issue Nov 4, 2024
@ywangd @jfreden
[Test] Flush response body for progress (elastic#115177)
f78a7f0 
In JDK23, response headers are no longer always immediately sent. See
also https://bugs.openjdk.org/browse/JDK-8331847 This PR adds flush call
for the response body to make progress.

Resolves: elastic#115145 Resolves: elastic#115164
ywangd added a commit to ywangd/elasticsearch that referenced this issue Nov 10, 2024
@ywangd
[Test] Flush response body for progress (elastic#115177)
30fab0b 
In JDK23, response headers are no longer always immediately sent. See
also https://bugs.openjdk.org/browse/JDK-8331847 This PR adds flush call
for the response body to make progress.

Resolves: elastic#115145 Resolves: elastic#115164
(cherry picked from commit 8c23fd7)
elasticsearchmachine pushed a commit that referenced this issue Nov 11, 2024
@ywangd
[Test] Flush response body for progress (#115177) (#116556)
b0ae9f0 
In JDK23, response headers are no longer always immediately sent. See
also https://bugs.openjdk.org/browse/JDK-8331847 This PR adds flush call
for the response body to make progress.

Resolves: #115145 Resolves: #115164
(cherry picked from commit 8c23fd7)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ywangd ywangd

Labels
:Distributed Coordination/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs low-risk An open issue or test failure that is a low risk to future releases Team:Distributed (Obsolete) Meta label for distributed team (obsolete). Replaced by Distributed Indexing/Coordination. >test-failure Triaged test failures from CI
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[Test] Flush response body for progress ywangd/elasticsearch
2 participants
@ywangd @elasticsearchmachine