Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[CI] GetStackTracesActionIT testGetStackTracesUnfiltered failing #103809

Closed
DaveCTurner opened this issue Jan 2, 2024 · 3 comments · Fixed by #104366
Closed

[CI] GetStackTracesActionIT testGetStackTracesUnfiltered failing #103809

DaveCTurner opened this issue Jan 2, 2024 · 3 comments · Fixed by #104366
Assignees
@danielmitterdorfer
Labels
low-risk An open issue or test failure that is a low risk to future releases Team:obs-knowledge Meta label for Observability Knowledge team >test-failure Triaged test failures from CI :UniversalProfiling/Application Elastic Universal Profiling REST APIs and infrastructure

Comments

@DaveCTurner
Copy link
Contributor

Looks like we are starting an enormous number of shards in this test and 30s isn't quite long enough to start them all.

Build scan:
https://gradle-enterprise.elastic.co/s/olrlw53ute7ok/tests/:x-pack:plugin:profiling:internalClusterTest/org.elasticsearch.xpack.profiling.GetStackTracesActionIT/testGetStackTracesUnfiltered

Reproduction line:

./gradlew ':x-pack:plugin:profiling:internalClusterTest' --tests "org.elasticsearch.xpack.profiling.GetStackTracesActionIT.testGetStackTracesUnfiltered" -Dtests.seed=F498799AC930C9FA -Dtests.locale=pl-PL -Dtests.timezone=Asia/Dhaka -Druntime.java=21

Applicable branches:
main

Reproduces locally?:
Didn't try

Failure history:
Failure dashboard for org.elasticsearch.xpack.profiling.GetStackTracesActionIT#testGetStackTracesUnfiltered

Failure excerpt:

java.lang.AssertionError: timed out waiting for green state

  at __randomizedtesting.SeedInfo.seed([F498799AC930C9FA:8B722FA57FBF3A80]:0)
  at org.junit.Assert.fail(Assert.java:88)
  at org.elasticsearch.test.ESIntegTestCase.ensureColor(ESIntegTestCase.java:1013)
  at org.elasticsearch.test.ESIntegTestCase.ensureGreen(ESIntegTestCase.java:933)
  at org.elasticsearch.test.ESIntegTestCase.ensureGreen(ESIntegTestCase.java:922)
  at org.elasticsearch.xpack.profiling.ProfilingTestCase.setupData(ProfilingTestCase.java:127)
  at jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
  at java.lang.reflect.Method.invoke(Method.java:580)
  at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:980)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
  at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
  at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
  at java.lang.Thread.run(Thread.java:1583)
@DaveCTurner DaveCTurner added :UniversalProfiling/Application Elastic Universal Profiling REST APIs and infrastructure >test-failure Triaged test failures from CI labels Jan 2, 2024
@elasticsearchmachine elasticsearchmachine added blocker Team:obs-knowledge Meta label for Observability Knowledge team labels Jan 2, 2024
@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/obs-knowledge-team (Team:obs-knowledge)

DaveCTurner added a commit that referenced this issue Jan 2, 2024
@DaveCTurner
AwaitsFix for #103809
5670a63
@DaveCTurner
Copy link
Contributor Author

Since this appears to be a general problem with the abstract ProfilingTestCase I've muted all its concrete subclasses in 5670a63.

@danielmitterdorfer danielmitterdorfer added low-risk An open issue or test failure that is a low risk to future releases and removed blocker labels Jan 4, 2024
@danielmitterdorfer
Copy link
Member

Marking as low-risk as this is an issue with the test setup.

@danielmitterdorfer danielmitterdorfer self-assigned this Jan 4, 2024
danielmitterdorfer added a commit to danielmitterdorfer/elasticsearch that referenced this issue Jan 15, 2024
@danielmitterdorfer
Account for shard count in profiling test timeout
8379b21 
With this commit we increase the timeout to wait for a green cluster
state in the integration tests for the profiling plugin. We do this
because the tests create all profiling-related indices with more shards
than usual and that might take longer than usual until the cluster is
green.

Closes elastic#103809
@danielmitterdorfer danielmitterdorfer mentioned this issue Jan 15, 2024
Merged
danielmitterdorfer added a commit that referenced this issue Jan 16, 2024
@danielmitterdorfer
Account for shard count in profiling test timeout (#104366)
ca919d9 
With this commit we increase the timeout to wait for a green cluster
state in the integration tests for the profiling plugin. We do this
because the tests create all profiling-related indices with more shards
than usual and that might take longer than usual until the cluster is
green.

Closes #103809
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@danielmitterdorfer danielmitterdorfer

Labels
low-risk An open issue or test failure that is a low risk to future releases Team:obs-knowledge Meta label for Observability Knowledge team >test-failure Triaged test failures from CI :UniversalProfiling/Application Elastic Universal Profiling REST APIs and infrastructure
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Account for shard count in profiling test timeout danielmitterdorfer/elasticsearch
3 participants
@danielmitterdorfer @DaveCTurner @elasticsearchmachine