Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[CI] TransformContinuousIT testContinuousEvents failing #88063

Closed
benwtrent opened this issue Jun 27, 2022 · 9 comments
Closed

[CI] TransformContinuousIT testContinuousEvents failing #88063

benwtrent opened this issue Jun 27, 2022 · 9 comments
Assignees
@hendrikmuhs
Labels
:ml/Transform Transform Team:ML Meta label for the ML team >test-failure Triaged test failures from CI

Comments

@benwtrent
Copy link
Member

Build scan:
https://gradle-enterprise.elastic.co/s/d2xk4wogo6gdg/tests/:x-pack:plugin:transform:qa:multi-node-tests:javaRestTest/org.elasticsearch.xpack.transform.integration.continuous.TransformContinuousIT/testContinuousEvents

Reproduction line:
./gradlew ':x-pack:plugin:transform:qa:multi-node-tests:javaRestTest' --tests "org.elasticsearch.xpack.transform.integration.continuous.TransformContinuousIT.testContinuousEvents" -Dtests.seed=9C41A496F649E927 -Dtests.locale=lv-LV -Dtests.timezone=SystemV/YST9YDT -Druntime.java=17

Applicable branches:
master

Reproduces locally?:
Didn't try

Failure history:
https://gradle-enterprise.elastic.co/scans/tests?tests.container=org.elasticsearch.xpack.transform.integration.continuous.TransformContinuousIT&tests.test=testContinuousEvents

Failure excerpt:

java.lang.AssertionError: Error in test case [continuous-terms-on-date-pivot-test].If you want to mute the test, please mute [org.elasticsearch.xpack.transform.integration.continuous.TermsOnDateGroupByIT] only, but _not_ [org.elasticsearch.xpack.transform.integration.continuous.TransformContinuousIT] as a whole.

  at __randomizedtesting.SeedInfo.seed([9C41A496F649E927:64EDD250B2BEC433]:0)
  at org.elasticsearch.xpack.transform.integration.continuous.TransformContinuousIT.testContinuousEvents(TransformContinuousIT.java:277)
  at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(NativeMethodAccessorImpl.java:-2)
  at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
  at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:568)
  at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:44)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
  at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:375)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:824)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:475)
  at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:375)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:831)
  at java.lang.Thread.run(Thread.java:833)

  Caused by: java.lang.AssertionError: Number of buckets did not match, source: 91, expected: 94, iteration: 3
  Expected: <94>
       but: was <91>

    at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
    at org.junit.Assert.assertThat(Assert.java:956)
    at org.elasticsearch.xpack.transform.integration.continuous.TermsOnDateGroupByIT.testIteration(TermsOnDateGroupByIT.java:128)
    at org.elasticsearch.xpack.transform.integration.continuous.TransformContinuousIT.testContinuousEvents(TransformContinuousIT.java:268)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(NativeMethodAccessorImpl.java:-2)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:568)
    at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:44)
    at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
    at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
    at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
    at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:375)
    at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:824)
    at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:475)
    at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
    at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
    at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
    at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
    at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
    at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
    at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
    at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:375)
    at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:831)
    at java.lang.Thread.run(Thread.java:833)
@benwtrent benwtrent added :ml/Transform Transform >test-failure Triaged test failures from CI labels Jun 27, 2022
@elasticmachine elasticmachine added the Team:ML Meta label for the ML team label Jun 27, 2022
@elasticmachine
Copy link
Collaborator

Pinging @elastic/ml-core (Team:ML)

benwtrent added a commit to benwtrent/elasticsearch that referenced this issue Jun 27, 2022
@benwtrent
[ML] mute TermsOnDateGroupBy cont. test elastic#88063
a65c797
@benwtrent benwtrent mentioned this issue Jun 27, 2022
Merged
@benwtrent
Copy link
Member Author

The two failures I have seen in CI so far have only been TermsOnDateGroupBy. But, if one other continuous transform test fails, I am gonna mute the whole continuous transform suite.

This is most likely caused by: 9b3eb69

Which is a fundamental change to how continuous transforms are scheduled and executed

@przemekwitek
Copy link
Contributor

przemekwitek commented Jun 27, 2022 •
edited
Loading

This is most likely caused by: 9b3eb69

Thanks for muting the test.
I've seen this failure being a result of my commit and I'm investigating right now.

@andreidan andreidan mentioned this issue Jun 27, 2022
Merged
elasticsearchmachine pushed a commit that referenced this issue Jun 27, 2022
@benwtrent
[ML] mute TermsOnDateGroupBy cont. test #88063 (#88067)
58e194c 
relates: #88063
@grcevski
Copy link
Contributor

I'm hitting this in my CI job on TermsGroupByIT. https://gradle-enterprise.elastic.co/s/tx5a4pf7swuuo/console-log?task=:x-pack:plugin:transform:qa:multi-node-tests:javaRestTest

@hendrikmuhs
Copy link

A recent build after the mute failed for DateHistogramGroupByIT (this is likey the next in line after TermsOnDateGroupBy):

https://gradle-enterprise.elastic.co/s/nd4ek3u3hebey
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+main+intake+multijob+part2/1260/consoleFull

@luigidellaquila
Copy link
Contributor

Another failure on TermsGroupByIT
https://gradle-enterprise.elastic.co/s/wtx3i3ty3jvia
Muting it as well

luigidellaquila added a commit that referenced this issue Jun 28, 2022
@luigidellaquila
Mute TermsGroupByIT test #88063
ac4d63e
@przemekwitek przemekwitek mentioned this issue Jun 28, 2022
Merged
@przemekwitek
Copy link
Contributor

Thanks for the reports.
I've just merged #88116 which is supposed to fix these issues.
Let's see if they occur after this recent change.

@benwtrent benwtrent mentioned this issue Jun 28, 2022
Merged
@przemekwitek przemekwitek mentioned this issue Jun 29, 2022
Merged
@przemekwitek
Copy link
Contributor

przemekwitek commented Jun 30, 2022 •
edited
Loading

Logs from the failed CI run are uploaded and available via:

curl -L -H 'Authorization: c41df94084685404' -o '16202.tar.bz2' https://upload.elastic.co/d/0778821122f427c0d109455735b3bc3694bb47c124acd58600c59787a4a5a5b0

curl -L -H 'Authorization: 1b646e89e2a21eeb' -o '16295.tar.bz2' https://upload.elastic.co/d/3861872b1c8cd79e6112ea5249ed4582732fcd8cc39148e55c74946a97214369

benwtrent added a commit to benwtrent/elasticsearch that referenced this issue Jun 30, 2022
@benwtrent
Mute test elastic#88063
15c0b03
@benwtrent benwtrent mentioned this issue Jun 30, 2022
Merged
elasticsearchmachine pushed a commit that referenced this issue Jun 30, 2022
@benwtrent
Mute test #88063 (#88207)
0eb35b3 
related #88063
@przemekwitek przemekwitek mentioned this issue Jun 30, 2022
Merged
@hendrikmuhs hendrikmuhs mentioned this issue Jul 4, 2022
Merged
@hendrikmuhs
Copy link

No errors seen since #88242, closing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@hendrikmuhs hendrikmuhs

Labels
:ml/Transform Transform Team:ML Meta label for the ML team >test-failure Triaged test failures from CI
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@luigidellaquila @benwtrent @grcevski @hendrikmuhs @elasticmachine @przemekwitek