Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[CI] RepositoryAzureClientYamlTestSuiteIT class failing #111345

Closed
elasticsearchmachine opened this issue Jul 26, 2024 · 4 comments · Fixed by #111397, #111569 or #111611
Closed

[CI] RepositoryAzureClientYamlTestSuiteIT class failing #111345

elasticsearchmachine opened this issue Jul 26, 2024 · 4 comments · Fixed by #111397, #111569 or #111611
Assignees
@DaveCTurner
Labels
:Distributed Coordination/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs needs:risk Requires assignment of a risk label (low, medium, blocker) Team:Distributed Meta label for distributed team (obsolete) >test-failure Triaged test failures from CI

Comments

@elasticsearchmachine
Copy link
Collaborator

elasticsearchmachine commented Jul 26, 2024 •
edited
Loading

Build Scans:

Reproduction Line:

./gradlew ":modules:repository-azure:managedIdentityYamlRestTest" --tests "org.elasticsearch.repositories.azure.RepositoryAzureClientYamlTestSuiteIT.test {yaml=repository_azure/20_repository/teardown}" -Dtests.seed=E6F121B481805CC8 -Dtests.locale=mt -Dtests.timezone=Europe/Busingen -Druntime.java=17 -Dtests.fips.enabled=true

Applicable branches:
main

Reproduces locally?:
N/A

Failure History:
See dashboard

Failure Message:

java.lang.AssertionError: Failure at [repository_azure/20_repository:6]: expected [2xx] status code but api [snapshot.create_repository] returned [500 Internal Server Error] [{"error":{"root_cause":[{"type":"repository_verification_exception","reason":"[repository] path [_integration_tests] is not accessible on master node","stack_trace":"org.elasticsearch.repositories.RepositoryVerificationException: [repository] path [_integration_tests] is not accessible on master node\nCaused by: java.lang.IllegalArgumentException: Authority host must use https scheme.\n\tat [email protected]/com.azure.identity.implementation.util.ValidationUtil.validateAuthHost(ValidationUtil.java:87)\n\tat [email protected]/com.azure.identity.implementation.IdentityClientOptions.loadFromConfiguration(IdentityClientOptions.java:703)\n\tat [email protected]/com.azure.identity.implementation.IdentityClientOptions.<init>(IdentityClientOptions.java:89)\n\tat [email protected]/com.azure.identity.CredentialBuilderBase.<init>(CredentialBuilderBase.java:47)\n\tat [email protected]/com.azure.identity.DefaultAzureCredentialBuilder.<init>(DefaultAzureCredentialBuilder.java:77)\n\tat [email protected]/org.elasticsearch.repositories.azure.AzureClientProvider.createClient(AzureClientProvider.java:184)\n\tat [email protected]/org.elasticsearch.repositories.azure.AzureStorageService.client(AzureStorageService.java:91)\n\tat [email protected]/org.elasticsearch.repositories.azure.AzureBlobStore.getAzureBlobServiceClientClient(AzureBlobStore.java:633)\n\tat [email protected]/org.elasticsearch.repositories.azure.AzureBlobStore.asyncClient(AzureBlobStore.java:629)\n\tat [email protected]/org.elasticsearch.repositories.azure.AzureBlobStore.lambda$executeSingleUpload$19(AzureBlobStore.java:463)\n\tat [email protected]/org.elasticsearch.repositories.azure.SocketAccess.lambda$doPrivilegedVoidException$0(SocketAccess.java:46)\n\tat java.base/java.security.AccessController.doPrivileged(AccessController.java:569)\n\tat [email protected]/org.elasticsearch.repositories.azure.SocketAccess.doPrivilegedVoidException(SocketAccess.java:45)\n\tat [email protected]/org.elasticsearch.repositories.azure.AzureBlobStore.executeSingleUpload(AzureBlobStore.java:462)\n\tat [email protected]/org.elasticsearch.repositories.azure.AzureBlobStore.writeBlob(AzureBlobStore.java:387)\n\tat [email protected]/org.elasticsearch.repositories.azure.AzureBlobContainer.writeBlob(AzureBlobContainer.java:115)\n\tat [email protected]/org.elasticsearch.repositories.azure.AzureBlobContainer.writeBlobAtomic(AzureBlobContainer.java:110)\n\tat [email protected]/org.elasticsearch.repositories.blobstore.BlobStoreRepository.startVerification(BlobStoreRepository.java:2156)\n\tat [email protected]/org.elasticsearch.repositories.RepositoriesService.lambda$validatePutRepositoryRequest$11(RepositoriesService.java:361)\n\tat [email protected]/org.elasticsearch.action.ActionRunnable$1.doRun(ActionRunnable.java:36)\n\tat [email protected]/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:984)\n\tat [email protected]/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat java.base/java.lang.Thread.run(Thread.java:833)\n"}],"type":"repository_verification_exception","reason":"[repository] path [_integration_tests] is not accessible on master node","caused_by":{"type":"illegal_argument_exception","reason":"Authority host must use https scheme.","stack_trace":"java.lang.IllegalArgumentException: Authority host must use https scheme.\n\tat [email protected]/com.azure.identity.implementation.util.ValidationUtil.validateAuthHost(ValidationUtil.java:87)\n\tat [email protected]/com.azure.identity.implementation.IdentityClientOptions.loadFromConfiguration(IdentityClientOptions.java:703)\n\tat [email protected]/com.azure.identity.implementation.IdentityClientOptions.<init>(IdentityClientOptions.java:89)\n\tat [email protected]/com.azure.identity.CredentialBuilderBase.<init>(CredentialBuilderBase.java:47)\n\tat [email protected]/com.azure.identity.DefaultAzureCredentialBuilder.<init>(DefaultAzureCredentialBuilder.java:77)\n\tat [email protected]/org.elasticsearch.repositories.azure.AzureClientProvider.createClient(AzureClientProvider.java:184)\n\tat [email protected]/org.elasticsearch.repositories.azure.AzureStorageService.client(AzureStorageService.java:91)\n\tat [email protected]/org.elasticsearch.repositories.azure.AzureBlobStore.getAzureBlobServiceClientClient(AzureBlobStore.java:633)\n\tat [email protected]/org.elasticsearch.repositories.azure.AzureBlobStore.asyncClient(AzureBlobStore.java:629)\n\tat [email protected]/org.elasticsearch.repositories.azure.AzureBlobStore.lambda$executeSingleUpload$19(AzureBlobStore.java:463)\n\tat [email protected]/org.elasticsearch.repositories.azure.SocketAccess.lambda$doPrivilegedVoidException$0(SocketAccess.java:46)\n\tat java.base/java.security.AccessController.doPrivileged(AccessController.java:569)\n\tat [email protected]/org.elasticsearch.repositories.azure.SocketAccess.doPrivilegedVoidException(SocketAccess.java:45)\n\tat [email protected]/org.elasticsearch.repositories.azure.AzureBlobStore.executeSingleUpload(AzureBlobStore.java:462)\n\tat [email protected]/org.elasticsearch.repositories.azure.AzureBlobStore.writeBlob(AzureBlobStore.java:387)\n\tat [email protected]/org.elasticsearch.repositories.azure.AzureBlobContainer.writeBlob(AzureBlobContainer.java:115)\n\tat [email protected]/org.elasticsearch.repositories.azure.AzureBlobContainer.writeBlobAtomic(AzureBlobContainer.java:110)\n\tat [email protected]/org.elasticsearch.repositories.blobstore.BlobStoreRepository.startVerification(BlobStoreRepository.java:2156)\n\tat [email protected]/org.elasticsearch.repositories.RepositoriesService.lambda$validatePutRepositoryRequest$11(RepositoriesService.java:361)\n\tat [email protected]/org.elasticsearch.action.ActionRunnable$1.doRun(ActionRunnable.java:36)\n\tat [email protected]/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:984)\n\tat [email protected]/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat java.base/java.lang.Thread.run(Thread.java:833)\n"},"stack_trace":"org.elasticsearch.repositories.RepositoryVerificationException: [repository] path [_integration_tests] is not accessible on master node\nCaused by: java.lang.IllegalArgumentException: Authority host must use https scheme.\n\tat [email protected]/com.azure.identity.implementation.util.ValidationUtil.validateAuthHost(ValidationUtil.java:87)\n\tat [email protected]/com.azure.identity.implementation.IdentityClientOptions.loadFromConfiguration(IdentityClientOptions.java:703)\n\tat [email protected]/com.azure.identity.implementation.IdentityClientOptions.<init>(IdentityClientOptions.java:89)\n\tat [email protected]/com.azure.identity.CredentialBuilderBase.<init>(CredentialBuilderBase.java:47)\n\tat [email protected]/com.azure.identity.DefaultAzureCredentialBuilder.<init>(DefaultAzureCredentialBuilder.java:77)\n\tat [email protected]/org.elasticsearch.repositories.azure.AzureClientProvider.createClient(AzureClientProvider.java:184)\n\tat [email protected]/org.elasticsearch.repositories.azure.AzureStorageService.client(AzureStorageService.java:91)\n\tat [email protected]/org.elasticsearch.repositories.azure.AzureBlobStore.getAzureBlobServiceClientClient(AzureBlobStore.java:633)\n\tat [email protected]/org.elasticsearch.repositories.azure.AzureBlobStore.asyncClient(AzureBlobStore.java:629)\n\tat [email protected]/org.elasticsearch.repositories.azure.AzureBlobStore.lambda$executeSingleUpload$19(AzureBlobStore.java:463)\n\tat [email protected]/org.elasticsearch.repositories.azure.SocketAccess.lambda$doPrivilegedVoidException$0(SocketAccess.java:46)\n\tat java.base/java.security.AccessController.doPrivileged(AccessController.java:569)\n\tat [email protected]/org.elasticsearch.repositories.azure.SocketAccess.doPrivilegedVoidException(SocketAccess.java:45)\n\tat [email protected]/org.elasticsearch.repositories.azure.AzureBlobStore.executeSingleUpload(AzureBlobStore.java:462)\n\tat [email protected]/org.elasticsearch.repositories.azure.AzureBlobStore.writeBlob(AzureBlobStore.java:387)\n\tat [email protected]/org.elasticsearch.repositories.azure.AzureBlobContainer.writeBlob(AzureBlobContainer.java:115)\n\tat [email protected]/org.elasticsearch.repositories.azure.AzureBlobContainer.writeBlobAtomic(AzureBlobContainer.java:110)\n\tat [email protected]/org.elasticsearch.repositories.blobstore.BlobStoreRepository.startVerification(BlobStoreRepository.java:2156)\n\tat [email protected]/org.elasticsearch.repositories.RepositoriesService.lambda$validatePutRepositoryRequest$11(RepositoriesService.java:361)\n\tat [email protected]/org.elasticsearch.action.ActionRunnable$1.doRun(ActionRunnable.java:36)\n\tat [email protected]/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:984)\n\tat [email protected]/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat java.base/java.lang.Thread.run(Thread.java:833)\n"},"status":500}]

Issue Reasons:

  • [main] 2 consecutive failures in class org.elasticsearch.repositories.azure.RepositoryAzureClientYamlTestSuiteIT
  • [main] 2 consecutive failures in step openjdk17_checkpart1_java-fips-matrix
  • [main] 3 consecutive failures in step part-1-fips
  • [main] 5 failures in class org.elasticsearch.repositories.azure.RepositoryAzureClientYamlTestSuiteIT (2.6% fail rate in 194 executions)
  • [main] 2 failures in step openjdk17_checkpart1_java-fips-matrix (100.0% fail rate in 2 executions)
  • [main] 3 failures in step part-1-fips (100.0% fail rate in 3 executions)
  • [main] 2 failures in pipeline elasticsearch-periodic (66.7% fail rate in 3 executions)
  • [main] 3 failures in pipeline elasticsearch-pull-request (9.7% fail rate in 31 executions)

Note:
This issue was created using new test triage automation. Please report issues or feedback to es-delivery.
@elasticsearchmachine elasticsearchmachine added :Distributed Coordination/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs >test-failure Triaged test failures from CI Team:Distributed Meta label for distributed team (obsolete) needs:risk Requires assignment of a risk label (low, medium, blocker) labels Jul 26, 2024
@elasticsearchmachine
Copy link
Collaborator Author

Pinging @elastic/es-distributed (Team:Distributed)

elasticsearchmachine added a commit that referenced this issue Jul 26, 2024
@elasticsearchmachine
Mute org.elasticsearch.repositories.azure.RepositoryAzureClientYamlTe…
0eba29e 
…stSuiteIT org.elasticsearch.repositories.azure.RepositoryAzureClientYamlTestSuiteIT #111345
@elasticsearchmachine
Copy link
Collaborator Author

This has been muted on branch main

Mute Reasons:

  • [main] 7 consecutive failures in step openjdk17_checkpart1_java-matrix
  • [main] 5 consecutive failures in step part-1-fips
  • [main] 6 consecutive failures in step openjdk17_checkpart1_java-fips-matrix
  • [main] 6 consecutive failures in step graalvm-ce17_checkpart1_java-matrix
  • [main] 24 failures in class org.elasticsearch.repositories.azure.RepositoryAzureClientYamlTestSuiteIT (7.5% fail rate in 322 executions)
  • [main] 7 failures in step openjdk17_checkpart1_java-matrix (100.0% fail rate in 7 executions)
  • [main] 5 failures in step part-1-fips (100.0% fail rate in 5 executions)
  • [main] 6 failures in step openjdk17_checkpart1_java-fips-matrix (100.0% fail rate in 6 executions)
  • [main] 6 failures in step graalvm-ce17_checkpart1_java-matrix (100.0% fail rate in 6 executions)
  • [main] 7 failures in pipeline elasticsearch-periodic (100.0% fail rate in 7 executions)
  • [main] 2 failures in pipeline elasticsearch-pull-request (2.7% fail rate in 73 executions)

Build Scans:

DaveCTurner added a commit to DaveCTurner/elasticsearch that referenced this issue Jul 29, 2024
@DaveCTurner
Use HTTP for Azure fixture in FIPS tests
8d83138 
We cannot (easily) override the JVM trust store in FIPS tests. This
commit falls back to using HTTP in these cases.

Closes elastic#111279
Closes elastic#111345
Closes elastic#111307
Closes elastic#111377
Closes elastic#111280
@DaveCTurner DaveCTurner mentioned this issue Jul 29, 2024
Merged
elasticsearchmachine added a commit that referenced this issue Jul 29, 2024
@elasticsearchmachine
Mute org.elasticsearch.repositories.azure.RepositoryAzureClientYamlTe…
86dff99 
…stSuiteIT org.elasticsearch.repositories.azure.RepositoryAzureClientYamlTestSuiteIT #111345
@elasticsearchmachine
Copy link
Collaborator Author

This has been muted on branch main

Mute Reasons:

  • [main] 29 failures in class org.elasticsearch.repositories.azure.RepositoryAzureClientYamlTestSuiteIT (4.5% fail rate in 641 executions)
  • [main] 8 failures in step openjdk17_checkpart1_java-matrix (53.3% fail rate in 15 executions)
  • [main] 7 failures in step openjdk17_checkpart1_java-fips-matrix (50.0% fail rate in 14 executions)
  • [main] 7 failures in step graalvm-ce17_checkpart1_java-matrix (50.0% fail rate in 14 executions)
  • [main] 7 failures in step part-1-fips (77.8% fail rate in 9 executions)
  • [main] 8 failures in pipeline elasticsearch-periodic (53.3% fail rate in 15 executions)
  • [main] 4 failures in pipeline elasticsearch-pull-request (4.2% fail rate in 95 executions)

Build Scans:

weizijun added a commit to weizijun/elasticsearch that referenced this issue Jul 30, 2024
@weizijun
Merge branch 'upstream/main' into add-alibabacloud-inference
f8fa7b9 
* upstream/main: (105 commits)
  Removing the use of watcher stats from WatchAcTests (elastic#111435)
  Mute org.elasticsearch.xpack.restart.FullClusterRestartIT testSingleDoc {cluster=UPGRADED} elastic#111434
  Make `EnrichPolicyRunner` more properly async (elastic#111321)
  Mute org.elasticsearch.xpack.restart.FullClusterRestartIT testSingleDoc {cluster=OLD} elastic#111430
  Mute org.elasticsearch.xpack.esql.expression.function.aggregate.ValuesTests testGroupingAggregate {TestCase=<long unicode KEYWORDs>} elastic#111428
  Mute org.elasticsearch.xpack.esql.expression.function.aggregate.ValuesTests testGroupingAggregate {TestCase=<long unicode TEXTs>} elastic#111429
  Mute org.elasticsearch.xpack.repositories.metering.azure.AzureRepositoriesMeteringIT org.elasticsearch.xpack.repositories.metering.azure.AzureRepositoriesMeteringIT elastic#111307
  Update semantic_text field to support indexing numeric and boolean data types (elastic#111284)
  Mute org.elasticsearch.repositories.blobstore.testkit.AzureSnapshotRepoTestKitIT testRepositoryAnalysis elastic#111280
  Ensure vector similarity correctly limits inner_hits returned for nested kNN (elastic#111363)
  Fix LogsIndexModeFullClusterRestartIT (elastic#111362)
  Remove 4096 bool query max limit from docs (elastic#111421)
  Fix score count validation in reranker response (elastic#111212)
  Integrate data generator in LogsDB mode challenge test (elastic#111303)
  ESQL: Add COUNT and COUNT_DISTINCT aggregation tests (elastic#111409)
  [Service Account] Add AutoOps account (elastic#111316)
  [ML] Fix failing test DetectionRulesTests.testEqualsAndHashcode (elastic#111351)
  [ML] Create and inject APM Inference Metrics (elastic#111293)
  [DOCS] Additional reranking docs updates (elastic#111350)
  Mute org.elasticsearch.repositories.azure.RepositoryAzureClientYamlTestSuiteIT org.elasticsearch.repositories.azure.RepositoryAzureClientYamlTestSuiteIT elastic#111345
  ...

# Conflicts:
#	server/src/main/java/org/elasticsearch/TransportVersions.java
@DaveCTurner DaveCTurner self-assigned this Aug 1, 2024
@albertzaharovits albertzaharovits mentioned this issue Aug 3, 2024
Merged
elasticsearchmachine pushed a commit that referenced this issue Aug 5, 2024
@albertzaharovits
Enforce JKS trustore for Azure IT (#111569)
f352418 
Closes #111279 Closes #111345 Closes #111280 Closes #111307 Closes
#111377
elasticsearchmachine added a commit that referenced this issue Aug 5, 2024
@elasticsearchmachine
Mute org.elasticsearch.repositories.azure.RepositoryAzureClientYamlTe…
2a44184 
…stSuiteIT org.elasticsearch.repositories.azure.RepositoryAzureClientYamlTestSuiteIT #111345
@elasticsearchmachine
Copy link
Collaborator Author

This has been muted on branch main

Mute Reasons:

  • [main] 2 consecutive failures in class org.elasticsearch.repositories.azure.RepositoryAzureClientYamlTestSuiteIT
  • [main] 2 consecutive failures in step openjdk17_checkpart1_java-fips-matrix
  • [main] 3 consecutive failures in step part-1-fips
  • [main] 5 failures in class org.elasticsearch.repositories.azure.RepositoryAzureClientYamlTestSuiteIT (2.6% fail rate in 194 executions)
  • [main] 2 failures in step openjdk17_checkpart1_java-fips-matrix (100.0% fail rate in 2 executions)
  • [main] 3 failures in step part-1-fips (100.0% fail rate in 3 executions)
  • [main] 2 failures in pipeline elasticsearch-periodic (66.7% fail rate in 3 executions)
  • [main] 3 failures in pipeline elasticsearch-pull-request (9.7% fail rate in 31 executions)

Build Scans:

DaveCTurner added a commit to DaveCTurner/elasticsearch that referenced this issue Aug 5, 2024
@DaveCTurner
Fix Azure tests in FIPS mode again
631037b 
Closes elastic#111345
Closes elastic#111607
Closes elastic#111608
DaveCTurner added a commit to DaveCTurner/elasticsearch that referenced this issue Aug 5, 2024
@DaveCTurner
Fix Azure tests in FIPS mode again
714dfd9 
Closes elastic#111345
Closes elastic#111607
Closes elastic#111608
@DaveCTurner DaveCTurner mentioned this issue Aug 5, 2024
Merged
mhl-b pushed a commit that referenced this issue Aug 8, 2024
@albertzaharovits @mhl-b
Enforce JKS trustore for Azure IT (#111569)
f3ad5a8 
Closes #111279 Closes #111345 Closes #111280 Closes #111307 Closes
#111377
mhl-b pushed a commit that referenced this issue Aug 8, 2024
@elasticsearchmachine @mhl-b
Mute org.elasticsearch.repositories.azure.RepositoryAzureClientYamlTe…
d75db96 
…stSuiteIT org.elasticsearch.repositories.azure.RepositoryAzureClientYamlTestSuiteIT #111345
mhl-b pushed a commit that referenced this issue Aug 8, 2024
@DaveCTurner @mhl-b
Fix Azure tests in FIPS mode again (#111611)
50036bc 
Closes #111345 Closes #111607 Closes #111608
cbuescher pushed a commit to cbuescher/elasticsearch that referenced this issue Sep 4, 2024
@albertzaharovits @cbuescher
Enforce JKS trustore for Azure IT (elastic#111569)
7b48447 
Closes elastic#111279 Closes elastic#111345 Closes elastic#111280 Closes elastic#111307 Closes
elastic#111377
cbuescher pushed a commit to cbuescher/elasticsearch that referenced this issue Sep 4, 2024
@elasticsearchmachine @cbuescher
Mute org.elasticsearch.repositories.azure.RepositoryAzureClientYamlTe…
e953ec3 
…stSuiteIT org.elasticsearch.repositories.azure.RepositoryAzureClientYamlTestSuiteIT elastic#111345
cbuescher pushed a commit to cbuescher/elasticsearch that referenced this issue Sep 4, 2024
@DaveCTurner @cbuescher
Fix Azure tests in FIPS mode again (elastic#111611)
7788a12 
Closes elastic#111345 Closes elastic#111607 Closes elastic#111608
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@DaveCTurner DaveCTurner

Labels
:Distributed Coordination/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs needs:risk Requires assignment of a risk label (low, medium, blocker) Team:Distributed Meta label for distributed team (obsolete) >test-failure Triaged test failures from CI
Projects
None yet
Milestone
No milestone
2 participants
@DaveCTurner @elasticsearchmachine