Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

org.elasticsearch.xpack.ml.integration.RunDataFrameAnalyticsIT.testStopOutlierDetectionWithEnoughDocumentsToScroll fails on master #46705

Closed
przemekwitek opened this issue Sep 13, 2019 · 3 comments · Fixed by #46721
Closed

org.elasticsearch.xpack.ml.integration.RunDataFrameAnalyticsIT.testStopOutlierDetectionWithEnoughDocumentsToScroll fails on master #46705

przemekwitek opened this issue Sep 13, 2019 · 3 comments · Fixed by #46721
Assignees
@dimitris-athanasiou
Labels
:ml Machine learning >test-failure Triaged test failures from CI

Comments

@przemekwitek
Copy link
Contributor

przemekwitek commented Sep 13, 2019 •
edited by dimitris-athanasiou
Loading

org.elasticsearch.xpack.ml.integration.RunDataFrameAnalyticsIT.testStopOutlierDetectionWithEnoughDocumentsToScroll failed on master:

13:40:39 org.elasticsearch.xpack.ml.integration.RunDataFrameAnalyticsIT > testStopOutlierDetectionWithEnoughDocumentsToScroll FAILED
13:40:39     java.lang.AssertionError: 
13:40:39     Expected: <0L>
13:40:39          but: was <1L>
13:40:39         at __randomizedtesting.SeedInfo.seed([CE9E38CE5E762526:5E6FCDEAD92BA21C]:0)
13:40:39         at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
13:40:39         at org.junit.Assert.assertThat(Assert.java:956)
13:40:39         at org.junit.Assert.assertThat(Assert.java:923)
13:40:39         at org.elasticsearch.xpack.ml.integration.MlNativeDataFrameAnalyticsIntegTestCase.cleanUpAnalytics(MlNativeDataFrameAnalyticsIntegTestCase.java:52)
13:40:39         at org.elasticsearch.xpack.ml.integration.MlNativeDataFrameAnalyticsIntegTestCase.cleanUpResources(MlNativeDataFrameAnalyticsIntegTestCase.java:45)
13:40:39         at org.elasticsearch.xpack.ml.integration.MlNativeIntegTestCase.cleanUp(MlNativeIntegTestCase.java:127)
13:40:39         at org.elasticsearch.xpack.ml.integration.RunDataFrameAnalyticsIT.cleanup(RunDataFrameAnalyticsIT.java:46)

The assertion fails during cleanup:

                assertThat(searchStoredProgress(config.getId()).getHits().getTotalHits().value, equalTo(0L));

REPRODUCE WITH: ./gradlew ':x-pack:plugin:ml:qa:native-multi-node-tests:integTestRunner' --tests "org.elasticsearch.xpack.ml.integration.RunDataFrameAnalyticsIT.testStopOutlierDetectionWithEnoughDocumentsToScroll" \
  -Dtests.seed=CE9E38CE5E762526 \
  -Dtests.security.manager=true \
  -Dtests.locale=en-CC \
  -Dtests.timezone=Asia/Choibalsan \
  -Dcompiler.java=12 \
  -Druntime.java=11

Does not reproduce locally.

Build 20190913073350-B735813E
Log https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+multijob-darwin-compatibility/207/console
Build Scans https://gradle-enterprise.elastic.co/s/z4parnu5jlhoyhttps://gradle-enterprise.elastic.co/s/ti53rgd66ecokhttps://gradle-enterprise.elastic.co/s/dqgzy2f5yitna
@przemekwitek przemekwitek added >test-failure Triaged test failures from CI :ml Machine learning labels Sep 13, 2019
@elasticmachine
Copy link
Collaborator

Pinging @elastic/ml-core

@jrodewig
Copy link
Contributor

Had another instance of this:
https://gradle-enterprise.elastic.co/s/zqmybihsdsoro/failure?openFailures=WzBd&openStackTraces=WzFd#top=0

@DaveCTurner DaveCTurner mentioned this issue Sep 13, 2019
Merged
@mark-vieira
Copy link
Contributor

And another:
https://gradle-enterprise.elastic.co/s/klrshptyonuxk/tests/zcquf3hc3eoda-so3ayypsxka5i

dimitris-athanasiou added a commit to dimitris-athanasiou/elasticsearch that referenced this issue Sep 14, 2019
@dimitris-athanasiou
[ML] Avoid marking data frame analytics task completed twice
b7f41ab 
When the stop API is called while the task is running there is
a chance the task gets marked completed twice. This may cause
undesired side effects, like indexing the progress document a second
time after the stop API has returned (the cause for elastic#46705).

This commit adds a check that the task has not been completed before
proceeding to mark it so. In addition, when we update the task's state
we could get some warnings that the task was missing if the stop API
has been called in the meantime. We now check the errors are
`ResourceNotFoundException` and ignore them if so.

Closes elastic#46705
@dimitris-athanasiou dimitris-athanasiou mentioned this issue Sep 14, 2019
Merged
dimitris-athanasiou added a commit that referenced this issue Sep 15, 2019
@dimitris-athanasiou
[ML] Avoid marking data frame analytics task completed twice (#46721)
3f18cf7 
When the stop API is called while the task is running there is
a chance the task gets marked completed twice. This may cause
undesired side effects, like indexing the progress document a second
time after the stop API has returned (the cause for #46705).

This commit adds a check that the task has not been completed before
proceeding to mark it so. In addition, when we update the task's state
we could get some warnings that the task was missing if the stop API
has been called in the meantime. We now check the errors are
`ResourceNotFoundException` and ignore them if so.

Closes #46705
dimitris-athanasiou added a commit to dimitris-athanasiou/elasticsearch that referenced this issue Sep 15, 2019
@dimitris-athanasiou
[7.x][ML] Avoid marking data frame analytics task completed twice (el…
43e2443 
…astic#46721)

When the stop API is called while the task is running there is
a chance the task gets marked completed twice. This may cause
undesired side effects, like indexing the progress document a second
time after the stop API has returned (the cause for elastic#46705).

This commit adds a check that the task has not been completed before
proceeding to mark it so. In addition, when we update the task's state
we could get some warnings that the task was missing if the stop API
has been called in the meantime. We now check the errors are
`ResourceNotFoundException` and ignore them if so.

Closes elastic#46705

Backports elastic#46721
@dimitris-athanasiou dimitris-athanasiou mentioned this issue Sep 15, 2019
Merged
dimitris-athanasiou added a commit to dimitris-athanasiou/elasticsearch that referenced this issue Sep 15, 2019
@dimitris-athanasiou
[7.4][ML] Avoid marking data frame analytics task completed twice (el…
3b5ad59 
…astic#46721)

When the stop API is called while the task is running there is
a chance the task gets marked completed twice. This may cause
undesired side effects, like indexing the progress document a second
time after the stop API has returned (the cause for elastic#46705).

This commit adds a check that the task has not been completed before
proceeding to mark it so. In addition, when we update the task's state
we could get some warnings that the task was missing if the stop API
has been called in the meantime. We now check the errors are
`ResourceNotFoundException` and ignore them if so.

Closes elastic#46705

Backports elastic#46721
@dimitris-athanasiou dimitris-athanasiou mentioned this issue Sep 15, 2019
Merged
dimitris-athanasiou added a commit that referenced this issue Sep 15, 2019
@dimitris-athanasiou
[7.x][ML] Avoid marking data frame analytics task completed twice (#4…
63eb0d9 
…6721) (#46724)

When the stop API is called while the task is running there is
a chance the task gets marked completed twice. This may cause
undesired side effects, like indexing the progress document a second
time after the stop API has returned (the cause for #46705).

This commit adds a check that the task has not been completed before
proceeding to mark it so. In addition, when we update the task's state
we could get some warnings that the task was missing if the stop API
has been called in the meantime. We now check the errors are
`ResourceNotFoundException` and ignore them if so.

Closes #46705

Backports #46721
dimitris-athanasiou added a commit that referenced this issue Sep 15, 2019
@dimitris-athanasiou
[7.4][ML] Avoid marking data frame analytics task completed twice (#4…
0893e55 
…6721) (#46725)

When the stop API is called while the task is running there is
a chance the task gets marked completed twice. This may cause
undesired side effects, like indexing the progress document a second
time after the stop API has returned (the cause for #46705).

This commit adds a check that the task has not been completed before
proceeding to mark it so. In addition, when we update the task's state
we could get some warnings that the task was missing if the stop API
has been called in the meantime. We now check the errors are
`ResourceNotFoundException` and ignore them if so.

Closes #46705

Backports #46721
@matriv matriv mentioned this issue Sep 20, 2019
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@dimitris-athanasiou dimitris-athanasiou

Labels
:ml Machine learning >test-failure Triaged test failures from CI
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[ML] Avoid marking data frame analytics task completed twice dimitris-athanasiou/elasticsearch
5 participants
@mark-vieira @dimitris-athanasiou @elasticmachine @przemekwitek @jrodewig