Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Meta: Fix Watcher Test Failures #42409

Closed
jakelandis opened this issue May 22, 2019 · 12 comments
Closed

Meta: Fix Watcher Test Failures #42409

jakelandis opened this issue May 22, 2019 · 12 comments
Assignees
@jakelandis
Labels
:Data Management/Watcher >test-failure Triaged test failures from CI

Comments

@jakelandis
Copy link
Contributor

jakelandis commented May 22, 2019 •
edited by davidkyle
Loading

The number of Watcher test failures which have been muted is too high. This is a Meta issue to help track and resolve those test failures.

https://github.com/elastic/elasticsearch/issues?utf8=%E2%9C%93&q=is%3Aopen+label%3A%3ACore%2FFeatures%2FWatcher+label%3A%3Etest-failure

WatcherRestIT

Rolling Upgrade Tests

SmokeTestWatcherWithSecurityIT

WatchBackwardsCompatibilityIT

SingleNodeTests

BasicWatcherTests

SmokeTestWatcherTestSuiteIT

EmailAttachmentTests

ActivateWatchTests

Template to add to test failure issues if/when a test fails.

Un-muted this test on PR #____ to obtain additional logs. 

If (when?) this test fails again please obtain the following information before muting the test:

* Copy of the relevant failure
* Copy of the reproduce line
* The Jenkins build link 
* The Gradle scan link
* The relevant cluster logs from "Google Cloud Storage Upload Report" (link found in Jenkins build)

The tests will only be un-muted on master and if no failures after a certain period of time, will back port the un-mute.
@jakelandis jakelandis added >test-failure Triaged test failures from CI :Data Management/Watcher labels May 22, 2019
@jakelandis jakelandis self-assigned this May 22, 2019
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-core-features

This was referenced May 24, 2019
Closed
Closed
Closed
Closed
Merged
@jakelandis
Copy link
Contributor Author

All tests in SmokeTestWatcherWithSecurityIT failed. It is likely that all share the same root cause.

Test Result (6 failures / +6)
    org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.testSearchInputWithInsufficientPrivileges
    org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.testIndexActionHasPermissions
    org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.testIndexActionInsufficientPrivileges
    org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.testSearchTransformHasPermissions
    org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.testSearchTransformInsufficientPermissions
    org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.testSearchInputHasPermissions
Relevant Failure with repro lines: 
10:45:18 org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT > testSearchInputWithInsufficientPrivileges FAILED
10:45:18     org.elasticsearch.client.ResponseException: method [POST], host [http://[::1]:45978], URI [/.watcher-history-*/_search?rest_total_hits_as_int=true], status line [HTTP/1.1 503 Service Unavailable]
10:45:18     {"error":{"root_cause":[],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[]},"status":503}
10:45:18         at __randomizedtesting.SeedInfo.seed([3AF36076A4C37A27:959B3C83AD12D929]:0)
10:45:18         at org.elasticsearch.client.RestClient.convertResponse(RestClient.java:253)
10:45:18         at org.elasticsearch.client.RestClient.performRequest(RestClient.java:231)
10:45:18         at org.elasticsearch.client.RestClient.performRequest(RestClient.java:205)
10:45:18         at org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.lambda$getWatchHistoryEntry$4(SmokeTestWatcherWithSecurityIT.java:326)
10:45:18         at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:850)
10:45:18         at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:836)
10:45:18         at org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.getWatchHistoryEntry(SmokeTestWatcherWithSecurityIT.java:306)
10:45:18         at org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.getWatchHistoryEntry(SmokeTestWatcherWithSecurityIT.java:301)
10:45:18         at org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.testSearchInputWithInsufficientPrivileges(SmokeTestWatcherWithSecurityIT.java:182)
10:45:29 REPRODUCE WITH: ./gradlew :x-pack:qa:smoke-test-watcher-with-security:integTestRunner --tests "org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.testSearchInputWithInsufficientPrivileges" -Dtests.seed=3AF36076A4C37A27 -Dtests.security.manager=true -Dtests.locale=nl-BQ -Dtests.timezone=Asia/Bishkek -Dcompiler.java=12 -Druntime.java=11
10:45:29 
10:45:29 org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT > testIndexActionHasPermissions FAILED
10:45:29     java.lang.AssertionError: 
10:45:29     Expected: is a value equal to or greater than <1>
10:45:29          but: <0> was less than <1>
10:45:29         at __randomizedtesting.SeedInfo.seed([3AF36076A4C37A27:1F46402E52F9591E]:0)
10:45:29         at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
10:45:29         at org.junit.Assert.assertThat(Assert.java:956)
10:45:29         at org.junit.Assert.assertThat(Assert.java:923)
10:45:29         at org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.lambda$getWatchHistoryEntry$4(SmokeTestWatcherWithSecurityIT.java:329)
10:45:29         at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:862)
10:45:29         at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:836)
10:45:29         at org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.getWatchHistoryEntry(SmokeTestWatcherWithSecurityIT.java:306)
10:45:29         at org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.testIndexActionHasPermissions(SmokeTestWatcherWithSecurityIT.java:259)
10:45:40 REPRODUCE WITH: ./gradlew :x-pack:qa:smoke-test-watcher-with-security:integTestRunner --tests "org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.testIndexActionHasPermissions" -Dtests.seed=3AF36076A4C37A27 -Dtests.security.manager=true -Dtests.locale=nl-BQ -Dtests.timezone=Asia/Bishkek -Dcompiler.java=12 -Druntime.java=11
10:45:40 
10:45:40 org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT > testIndexActionInsufficientPrivileges FAILED
10:45:40     java.lang.AssertionError: 
10:45:40     Expected: is a value equal to or greater than <1>
10:45:40          but: <0> was less than <1>
10:45:40         at __randomizedtesting.SeedInfo.seed([3AF36076A4C37A27:204F346B78395B47]:0)
10:45:40         at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
10:45:40         at org.junit.Assert.assertThat(Assert.java:956)
10:45:40         at org.junit.Assert.assertThat(Assert.java:923)
10:45:40         at org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.lambda$getWatchHistoryEntry$4(SmokeTestWatcherWithSecurityIT.java:329)
10:45:40         at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:862)
10:45:40         at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:836)
10:45:40         at org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.getWatchHistoryEntry(SmokeTestWatcherWithSecurityIT.java:306)
10:45:40         at org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.testIndexActionInsufficientPrivileges(SmokeTestWatcherWithSecurityIT.java:282)
10:45:51 REPRODUCE WITH: ./gradlew :x-pack:qa:smoke-test-watcher-with-security:integTestRunner --tests "org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.testIndexActionInsufficientPrivileges" -Dtests.seed=3AF36076A4C37A27 -Dtests.security.manager=true -Dtests.locale=nl-BQ -Dtests.timezone=Asia/Bishkek -Dcompiler.java=12 -Druntime.java=11
10:45:51 
10:45:51 org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT > testSearchTransformHasPermissions FAILED
10:45:51     java.lang.AssertionError: 
10:45:51     Expected: is a value equal to or greater than <1>
10:45:51          but: <0> was less than <1>
10:45:51         at __randomizedtesting.SeedInfo.seed([3AF36076A4C37A27:138387687DCC68CA]:0)
10:45:51         at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
10:45:51         at org.junit.Assert.assertThat(Assert.java:956)
10:45:51         at org.junit.Assert.assertThat(Assert.java:923)
10:45:51         at org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.lambda$getWatchHistoryEntry$4(SmokeTestWatcherWithSecurityIT.java:329)
10:45:51         at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:862)
10:45:51         at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:836)
10:45:51         at org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.getWatchHistoryEntry(SmokeTestWatcherWithSecurityIT.java:306)
10:45:51         at org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.testSearchTransformHasPermissions(SmokeTestWatcherWithSecurityIT.java:210)
10:46:01 REPRODUCE WITH: ./gradlew :x-pack:qa:smoke-test-watcher-with-security:integTestRunner --tests "org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.testSearchTransformHasPermissions" -Dtests.seed=3AF36076A4C37A27 -Dtests.security.manager=true -Dtests.locale=nl-BQ -Dtests.timezone=Asia/Bishkek -Dcompiler.java=12 -Druntime.java=11
10:46:01 
10:46:01 org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT > testSearchTransformInsufficientPermissions FAILED
10:46:01     java.lang.AssertionError: 
10:46:01     Expected: is a value equal to or greater than <1>
10:46:01          but: <0> was less than <1>
10:46:01         at __randomizedtesting.SeedInfo.seed([3AF36076A4C37A27:A786648885C87DCA]:0)
10:46:01         at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
10:46:01         at org.junit.Assert.assertThat(Assert.java:956)
10:46:01         at org.junit.Assert.assertThat(Assert.java:923)
10:46:01         at org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.lambda$getWatchHistoryEntry$4(SmokeTestWatcherWithSecurityIT.java:329)
10:46:01         at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:862)
10:46:01         at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:836)
10:46:01         at org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.getWatchHistoryEntry(SmokeTestWatcherWithSecurityIT.java:306)
10:46:01         at org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.getWatchHistoryEntry(SmokeTestWatcherWithSecurityIT.java:301)
10:46:01         at org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.testSearchTransformInsufficientPermissions(SmokeTestWatcherWithSecurityIT.java:239)
10:46:12 REPRODUCE WITH: ./gradlew :x-pack:qa:smoke-test-watcher-with-security:integTestRunner --tests "org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.testSearchTransformInsufficientPermissions" -Dtests.seed=3AF36076A4C37A27 -Dtests.security.manager=true -Dtests.locale=nl-BQ -Dtests.timezone=Asia/Bishkek -Dcompiler.java=12 -Druntime.java=11
10:46:12 
10:46:12 org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT > testSearchInputHasPermissions FAILED
10:46:12     java.lang.AssertionError: 
10:46:12     Expected: is a value equal to or greater than <1>
10:46:12          but: <0> was less than <1>
10:46:12         at __randomizedtesting.SeedInfo.seed([3AF36076A4C37A27:4822DFE1A9577852]:0)
10:46:12         at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
10:46:12         at org.junit.Assert.assertThat(Assert.java:956)
10:46:12         at org.junit.Assert.assertThat(Assert.java:923)
10:46:12         at org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.lambda$getWatchHistoryEntry$4(SmokeTestWatcherWithSecurityIT.java:329)
10:46:12         at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:862)
10:46:12         at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:836)
10:46:12         at org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.getWatchHistoryEntry(SmokeTestWatcherWithSecurityIT.java:306)
10:46:12         at org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.testSearchInputHasPermissions(SmokeTestWatcherWithSecurityIT.java:158)
10:46:12 REPRODUCE WITH: ./gradlew :x-pack:qa:smoke-test-watcher-with-security:integTestRunner --tests "org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.testSearchInputHasPermissions" -Dtests.seed=3AF36076A4C37A27 -Dtests.security.manager=true -Dtests.locale=nl-BQ -Dtests.timezone=Asia/Bishkek -Dcompiler.java=12 -Druntime.java=11
10:46:12 
10:46:12 Suite: Test class org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT
10:46:12   1> [2019-06-06T21:45:13,833][INFO ][o.e.s.SmokeTestWatcherWithSecurityIT] [testSearchInputWithInsufficientPrivileges] before test
10:46:12   1> [2019-06-06T21:45:13,838][INFO ][o.e.s.SmokeTestWatcherWithSecurityIT] [testSearchInputWithInsufficientPrivileges] initializing REST clients against [http://[::1]:45978]
10:46:12   1> [2019-06-06T21:45:18,712][INFO ][o.e.s.SmokeTestWatcherWithSecurityIT] [testSearchInputWithInsufficientPrivileges] There are still tasks running after this test that might break subsequent tests [indices:data/write/bulk, indices:data/write/bulk[s]].
10:46:12   1> [2019-06-06T21:45:18,713][INFO ][o.e.s.SmokeTestWatcherWithSecurityIT] [testSearchInputWithInsufficientPrivileges] after test
10:46:12   2> REPRODUCE WITH: ./gradlew :x-pack:qa:smoke-test-watcher-with-security:integTestRunner --tests "org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.testSearchInputWithInsufficientPrivileges" -Dtests.seed=3AF36076A4C37A27 -Dtests.security.manager=true -Dtests.locale=nl-BQ -Dtests.timezone=Asia/Bishkek -Dcompiler.java=12 -Druntime.java=11
10:46:12   2> org.elasticsearch.client.ResponseException: method [POST], host [http://[::1]:45978], URI [/.watcher-history-*/_search?rest_total_hits_as_int=true], status line [HTTP/1.1 503 Service Unavailable]
10:46:12     {"error":{"root_cause":[],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[]},"status":503}
10:46:12         at __randomizedtesting.SeedInfo.seed([3AF36076A4C37A27:959B3C83AD12D929]:0)
10:46:12         at org.elasticsearch.client.RestClient.convertResponse(RestClient.java:253)
10:46:12         at org.elasticsearch.client.RestClient.performRequest(RestClient.java:231)
10:46:12         at org.elasticsearch.client.RestClient.performRequest(RestClient.java:205)
10:46:12         at org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.lambda$getWatchHistoryEntry$4(SmokeTestWatcherWithSecurityIT.java:326)
10:46:12         at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:850)
10:46:12         at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:836)
10:46:12         at org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.getWatchHistoryEntry(SmokeTestWatcherWithSecurityIT.java:306)
10:46:12         at org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.getWatchHistoryEntry(SmokeTestWatcherWithSecurityIT.java:301)
10:46:12         at org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.testSearchInputWithInsufficientPrivileges(SmokeTestWatcherWithSecurityIT.java:182)
10:46:12   1> [2019-06-06T21:45:18,748][INFO ][o.e.s.SmokeTestWatcherWithSecurityIT] [testIndexActionHasPermissions] before test
10:46:12   1> [2019-06-06T21:45:29,504][INFO ][o.e.s.SmokeTestWatcherWithSecurityIT] [testIndexActionHasPermissions] There are still tasks running after this test that might break subsequent tests [indices:data/write/bulk, indices:data/write/bulk[s]].
10:46:12   1> [2019-06-06T21:45:29,505][INFO ][o.e.s.SmokeTestWatcherWithSecurityIT] [testIndexActionHasPermissions] after test
10:46:12   2> REPRODUCE WITH: ./gradlew :x-pack:qa:smoke-test-watcher-with-security:integTestRunner --tests "org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.testIndexActionHasPermissions" -Dtests.seed=3AF36076A4C37A27 -Dtests.security.manager=true -Dtests.locale=nl-BQ -Dtests.timezone=Asia/Bishkek -Dcompiler.java=12 -Druntime.java=11
10:46:12   2> java.lang.AssertionError: 
10:46:12     Expected: is a value equal to or greater than <1>
10:46:12          but: <0> was less than <1>
10:46:12         at __randomizedtesting.SeedInfo.seed([3AF36076A4C37A27:1F46402E52F9591E]:0)
10:46:12         at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
10:46:12         at org.junit.Assert.assertThat(Assert.java:956)
10:46:12         at org.junit.Assert.assertThat(Assert.java:923)
10:46:12         at org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.lambda$getWatchHistoryEntry$4(SmokeTestWatcherWithSecurityIT.java:329)
10:46:12         at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:862)
10:46:12         at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:836)
10:46:12         at org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.getWatchHistoryEntry(SmokeTestWatcherWithSecurityIT.java:306)
10:46:12         at org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.testIndexActionHasPermissions(SmokeTestWatcherWithSecurityIT.java:259)
10:46:12   1> [2019-06-06T21:45:29,532][INFO ][o.e.s.SmokeTestWatcherWithSecurityIT] [testIndexActionInsufficientPrivileges] before test
10:46:12   1> [2019-06-06T21:45:40,309][INFO ][o.e.s.SmokeTestWatcherWithSecurityIT] [testIndexActionInsufficientPrivileges] There are still tasks running after this test that might break subsequent tests [indices:data/write/bulk, indices:data/write/bulk[s]].
10:46:12   1> [2019-06-06T21:45:40,310][INFO ][o.e.s.SmokeTestWatcherWithSecurityIT] [testIndexActionInsufficientPrivileges] after test
10:46:12   2> REPRODUCE WITH: ./gradlew :x-pack:qa:smoke-test-watcher-with-security:integTestRunner --tests "org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.testIndexActionInsufficientPrivileges" -Dtests.seed=3AF36076A4C37A27 -Dtests.security.manager=true -Dtests.locale=nl-BQ -Dtests.timezone=Asia/Bishkek -Dcompiler.java=12 -Druntime.java=11
10:46:12   2> java.lang.AssertionError: 
10:46:12     Expected: is a value equal to or greater than <1>
10:46:12          but: <0> was less than <1>
10:46:12         at __randomizedtesting.SeedInfo.seed([3AF36076A4C37A27:204F346B78395B47]:0)
10:46:12         at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
10:46:12         at org.junit.Assert.assertThat(Assert.java:956)
10:46:12         at org.junit.Assert.assertThat(Assert.java:923)
10:46:12         at org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.lambda$getWatchHistoryEntry$4(SmokeTestWatcherWithSecurityIT.java:329)
10:46:12         at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:862)
10:46:12         at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:836)
10:46:12         at org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.getWatchHistoryEntry(SmokeTestWatcherWithSecurityIT.java:306)
10:46:12         at org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.testIndexActionInsufficientPrivileges(SmokeTestWatcherWithSecurityIT.java:282)
10:46:12   1> [2019-06-06T21:45:40,331][INFO ][o.e.s.SmokeTestWatcherWithSecurityIT] [testSearchTransformHasPermissions] before test
10:46:12   1> [2019-06-06T21:45:51,016][INFO ][o.e.s.SmokeTestWatcherWithSecurityIT] [testSearchTransformHasPermissions] There are still tasks running after this test that might break subsequent tests [indices:data/write/bulk, indices:data/write/bulk[s]].
10:46:12   1> [2019-06-06T21:45:51,017][INFO ][o.e.s.SmokeTestWatcherWithSecurityIT] [testSearchTransformHasPermissions] after test
10:46:12   2> REPRODUCE WITH: ./gradlew :x-pack:qa:smoke-test-watcher-with-security:integTestRunner --tests "org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.testSearchTransformHasPermissions" -Dtests.seed=3AF36076A4C37A27 -Dtests.security.manager=true -Dtests.locale=nl-BQ -Dtests.timezone=Asia/Bishkek -Dcompiler.java=12 -Druntime.java=11
10:46:12   2> java.lang.AssertionError: 
10:46:12     Expected: is a value equal to or greater than <1>
10:46:12          but: <0> was less than <1>
10:46:12         at __randomizedtesting.SeedInfo.seed([3AF36076A4C37A27:138387687DCC68CA]:0)
10:46:12         at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
10:46:12         at org.junit.Assert.assertThat(Assert.java:956)
10:46:12         at org.junit.Assert.assertThat(Assert.java:923)
10:46:12         at org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.lambda$getWatchHistoryEntry$4(SmokeTestWatcherWithSecurityIT.java:329)
10:46:12         at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:862)
10:46:12         at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:836)
10:46:12         at org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.getWatchHistoryEntry(SmokeTestWatcherWithSecurityIT.java:306)
10:46:12         at org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.testSearchTransformHasPermissions(SmokeTestWatcherWithSecurityIT.java:210)
10:46:12   1> [2019-06-06T21:45:51,041][INFO ][o.e.s.SmokeTestWatcherWithSecurityIT] [testSearchTransformInsufficientPermissions] before test
10:46:12   1> [2019-06-06T21:46:01,747][INFO ][o.e.s.SmokeTestWatcherWithSecurityIT] [testSearchTransformInsufficientPermissions] There are still tasks running after this test that might break subsequent tests [indices:data/write/bulk, indices:data/write/bulk[s]].
10:46:12   1> [2019-06-06T21:46:01,747][INFO ][o.e.s.SmokeTestWatcherWithSecurityIT] [testSearchTransformInsufficientPermissions] after test
10:46:12   2> REPRODUCE WITH: ./gradlew :x-pack:qa:smoke-test-watcher-with-security:integTestRunner --tests "org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.testSearchTransformInsufficientPermissions" -Dtests.seed=3AF36076A4C37A27 -Dtests.security.manager=true -Dtests.locale=nl-BQ -Dtests.timezone=Asia/Bishkek -Dcompiler.java=12 -Druntime.java=11
10:46:12   2> java.lang.AssertionError: 
10:46:12     Expected: is a value equal to or greater than <1>
10:46:12          but: <0> was less than <1>
10:46:12         at __randomizedtesting.SeedInfo.seed([3AF36076A4C37A27:A786648885C87DCA]:0)
10:46:12         at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
10:46:12         at org.junit.Assert.assertThat(Assert.java:956)
10:46:12         at org.junit.Assert.assertThat(Assert.java:923)
10:46:12         at org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.lambda$getWatchHistoryEntry$4(SmokeTestWatcherWithSecurityIT.java:329)
10:46:12         at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:862)
10:46:12         at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:836)
10:46:12         at org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.getWatchHistoryEntry(SmokeTestWatcherWithSecurityIT.java:306)
10:46:12         at org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.getWatchHistoryEntry(SmokeTestWatcherWithSecurityIT.java:301)
10:46:12         at org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.testSearchTransformInsufficientPermissions(SmokeTestWatcherWithSecurityIT.java:239)
10:46:12   1> [2019-06-06T21:46:01,768][INFO ][o.e.s.SmokeTestWatcherWithSecurityIT] [testSearchInputHasPermissions] before test
10:46:12   1> [2019-06-06T21:46:12,464][INFO ][o.e.s.SmokeTestWatcherWithSecurityIT] [testSearchInputHasPermissions] There are still tasks running after this test that might break subsequent tests [indices:data/write/bulk, indices:data/write/bulk[s]].
10:46:12   1> [2019-06-06T21:46:12,465][INFO ][o.e.s.SmokeTestWatcherWithSecurityIT] [testSearchInputHasPermissions] after test
10:46:12   2> REPRODUCE WITH: ./gradlew :x-pack:qa:smoke-test-watcher-with-security:integTestRunner --tests "org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.testSearchInputHasPermissions" -Dtests.seed=3AF36076A4C37A27 -Dtests.security.manager=true -Dtests.locale=nl-BQ -Dtests.timezone=Asia/Bishkek -Dcompiler.java=12 -Druntime.java=11
10:46:12   2> java.lang.AssertionError: 
10:46:12     Expected: is a value equal to or greater than <1>
10:46:12          but: <0> was less than <1>
10:46:12         at __randomizedtesting.SeedInfo.seed([3AF36076A4C37A27:4822DFE1A9577852]:0)
10:46:12         at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
10:46:12         at org.junit.Assert.assertThat(Assert.java:956)
10:46:12         at org.junit.Assert.assertThat(Assert.java:923)
10:46:12         at org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.lambda$getWatchHistoryEntry$4(SmokeTestWatcherWithSecurityIT.java:329)
10:46:12         at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:862)
10:46:12         at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:836)
10:46:12         at org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.getWatchHistoryEntry(SmokeTestWatcherWithSecurityIT.java:306)
10:46:12         at org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.testSearchInputHasPermissions(SmokeTestWatcherWithSecurityIT.java:158)
10:46:12   2> NOTE: leaving temporary files on disk at: /var/lib/jenkins/workspace/elastic+elasticsearch+master+intake/x-pack/qa/smoke-test-watcher-with-security/build/testrun/integTestRunner/temp/org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT_3AF36076A4C37A27-001
10:46:12   2> NOTE: test params are: codec=Lucene80, sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@53309619), locale=nl-BQ, timezone=Asia/Bishkek
10:46:12   2> NOTE: Linux 3.10.0-957.12.2.el7.x86_64 amd64/Oracle Corporation 11.0.3 (64-bit)/cpus=16,threads=1,free=426557952,total=536870912
10:46:12   2> NOTE: All tests run in this JVM: [SmokeTestWatcherWithSecurityClientYamlTestSuiteIT, SmokeTestWatcherWithSecurityIT]
10:46:12 
10:46:12 66 tests completed, 6 failed
10:46:12 
10:46:12 Tests with failures:
10:46:12  - org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.testSearchInputWithInsufficientPrivileges
10:46:12  - org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.testIndexActionHasPermissions
10:46:12  - org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.testIndexActionInsufficientPrivileges
10:46:12  - org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.testSearchTransformHasPermissions
10:46:12  - org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.testSearchTransformInsufficientPermissions
10:46:12  - org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.testSearchInputHasPermissions

Jenkins build: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+intake/37
Gradle build: https://scans.gradle.com/s/ab6f6mtnx5wvq
Logs: https://storage.cloud.google.com/elasticsearch-ci-artifacts/jobs/elastic+elasticsearch+master+intake/37/log-files.tar.gz
Relevant logs: relevant_logs.tar.gz

@droberts195
Copy link
Contributor

In https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+intake/110/console just one test failed, but it's the same as one of the failures in #42409 (comment) and failed with the exact same reason as the first failure in #42409 (comment).

The failure is:

org.elasticsearch.client.ResponseException: method [POST], host [http://[::1]:43041], URI [/.watcher-history-*/_search?rest_total_hits_as_int=true], status line [HTTP/1.1 503 Service Unavailable]
{"error":{"root_cause":[],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[]},"status":503}
	at __randomizedtesting.SeedInfo.seed([73CA3AF4F90C540D:69766EE925F6756D]:0)
	at org.elasticsearch.client.RestClient.convertResponse(RestClient.java:253)
	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:231)
	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:205)
	at org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.lambda$getWatchHistoryEntry$4(SmokeTestWatcherWithSecurityIT.java:326)
	at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:850)
	at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:836)
	at org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.getWatchHistoryEntry(SmokeTestWatcherWithSecurityIT.java:306)
	at org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.testIndexActionInsufficientPrivileges(SmokeTestWatcherWithSecurityIT.java:282)

At the time this happened the corresponding error in the server side log was:

[2019-06-11T13:59:08,130][INFO ][o.e.c.m.MetaDataCreateIndexService] [node-0] [my_test_index] creating index, cause [auto(bulk api)], templates [], shards [1]/[1], mappings []
[2019-06-11T13:59:08,593][INFO ][o.e.c.m.MetaDataMappingService] [node-0] [my_test_index/MWObuPVBTcC-nxbj8SRzUQ] create_mapping [_doc]
[2019-06-11T13:59:08,837][INFO ][o.e.c.m.MetaDataCreateIndexService] [node-0] [index_not_allowed_to_read] creating index, cause [auto(bulk api)], templates [], shards [1]/[1], mappings []
[2019-06-11T13:59:08,981][INFO ][o.e.c.m.MetaDataMappingService] [node-0] [index_not_allowed_to_read/t8g7ay5HTrGHyDj0TAgzAg] create_mapping [_doc]
[2019-06-11T13:59:09,232][INFO ][o.e.c.m.MetaDataCreateIndexService] [node-0] [.watches] creating index, cause [auto(bulk api)], templates [.watches], shards [1]/[0], mappings [_doc]
[2019-06-11T13:59:09,370][INFO ][o.e.x.w.WatcherService   ] [node-0] reloading watcher, reason [new local watcher shard allocation ids], cancelled [0] queued tasks
[2019-06-11T13:59:09,392][DEBUG][o.e.x.w.WatcherIndexingListener] [node-0] adding watch [FbxatmpquAmnjMGcegxg] to trigger service
[2019-06-11T13:59:09,473][DEBUG][o.e.x.w.WatcherService   ] [node-0] watch service has been reloaded, reason [new local watcher shard allocation ids]
[2019-06-11T13:59:10,502][DEBUG][o.e.x.w.t.s.e.TickerScheduleTriggerEngine] [node-0] triggered job [FbxatmpquAmnjMGcegxg] at [2019-06-11T13:59:10.502Z] (scheduled time was [2019-06-11T13:59:10.394Z])
[2019-06-11T13:59:10,572][INFO ][o.e.c.m.MetaDataCreateIndexService] [node-0] [.triggered_watches] creating index, cause [auto(bulk api)], templates [.triggered_watches], shards [1]/[1], mappings [_doc]
[2019-06-11T13:59:10,576][INFO ][o.e.c.r.a.AllocationService] [node-0] updating number_of_replicas to [0] for indices [.triggered_watches]
[2019-06-11T13:59:11,560][DEBUG][o.e.x.w.t.s.e.TickerScheduleTriggerEngine] [node-0] triggered job [FbxatmpquAmnjMGcegxg] at [2019-06-11T13:59:11.560Z] (scheduled time was [2019-06-11T13:59:11.394Z])
[2019-06-11T13:59:11,607][DEBUG][o.e.x.w.e.ExecutionService] [node-0] executing watch [FbxatmpquAmnjMGcegxg]
[2019-06-11T13:59:11,643][INFO ][o.e.c.m.MetaDataCreateIndexService] [node-0] [.watcher-history-9-2019.06.11] creating index, cause [auto(bulk api)], templates [.watch-history-no-ilm-9], shards [1]/[0], mappings [doc]
[2019-06-11T13:59:11,630][ERROR][o.e.x.w.a.i.IndexActionFactory] [node-0] failed to execute action [FbxatmpquAmnjMGcegxg/index]
org.elasticsearch.ElasticsearchSecurityException: action [indices:data/write/bulk[s]] is unauthorized for user [watcher_manager]
...
...very long stack trace...
...
[2019-06-11T13:59:11,703][DEBUG][o.e.x.w.WatcherIndexingListener] [node-0] adding watch [FbxatmpquAmnjMGcegxg] to trigger service
[2019-06-11T13:59:11,735][DEBUG][o.e.a.s.TransportSearchAction] [node-0] All shards failed for phase: [query]
[2019-06-11T13:59:11,737][WARN ][r.suppressed             ] [node-0] path: /.watcher-history-*/_search, params: {rest_total_hits_as_int=true, index=.watcher-history-*}
org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed
    at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:296) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
    at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:139) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
    at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:259) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
    at org.elasticsearch.action.search.InitialSearchPhase.onShardFailure(InitialSearchPhase.java:105) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
    at org.elasticsearch.action.search.InitialSearchPhase$2.onFailure(InitialSearchPhase.java:273) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
    at org.elasticsearch.action.search.SearchExecutionStatsCollector.onFailure(SearchExecutionStatsCollector.java:73) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
    at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:59) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
    at org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:441) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
    at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1101) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
    at org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1213) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
    at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1187) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
    at org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:60) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
    at org.elasticsearch.action.support.ChannelActionListener.onFailure(ChannelActionListener.java:56) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
    at org.elasticsearch.action.ActionListener$1.onFailure(ActionListener.java:70) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
    at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:64) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
    at org.elasticsearch.search.SearchService$2.doRun(SearchService.java:1052) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
    at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
    at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
    at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:757) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
    at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
    at java.lang.Thread.run(Thread.java:834) [?:?]
[2019-06-11T13:59:11,802][INFO ][o.e.x.w.WatcherService   ] [node-0] stopping watch service, reason [watcher manually marked to shutdown by cluster state update]
[2019-06-11T13:59:11,856][INFO ][o.e.c.m.MetaDataDeleteIndexService] [node-0] [my_test_index/MWObuPVBTcC-nxbj8SRzUQ] deleting index
[2019-06-11T13:59:11,987][INFO ][o.e.c.m.MetaDataMappingService] [node-0] [.watcher-history-9-2019.06.11/uLPXpq2YSmadsCt7RdBAoQ] update_mapping [doc]

@alpar-t
Copy link
Contributor

alpar-t commented Jun 28, 2019

I'we been working on running the build with more parallelism enabled which has unfortunately caused more (I believe) timing related failures such as this one: https://scans.gradle.com/s/m44sa6oyftyue/tests/6ej7h6toy4af6-6g5rmtb4agtbo

@hendrikmuhs
Copy link

more failures:

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+multijob-unix-compatibility/os=oraclelinux-6/95/consoleFull

jakelandis added a commit to jakelandis/elasticsearch that referenced this issue Jun 29, 2019
@jakelandis
SmokeTestWatcherWithSecurityIT: Retry if failures searching .watcher-…
d57e1a2 
…history

Catch exceptions and fail (which allows the assertBusy to retry)
if an error happens from the search against .watcher-history.

Related: elastic#42409
Related: elastic#39169
@jakelandis jakelandis mentioned this issue Jun 29, 2019
Merged
@jakelandis
Copy link
Contributor Author

I spent some time today looking at these failures on the CI nodes with @atorok pending parallelism. I can see the box is working extremely hard and things are slower, but still could not find any relationship between load and these tests. I did find one issue : #43781 which I think will help with the majority of the problems for SmokeTestWatcherWithSecurityIT (non yaml tests).

jakelandis added a commit that referenced this issue Jul 1, 2019
@jakelandis
SmokeTestWatcherWithSecurityIT: Retry if failures searching (#43781)
8a4a26d 
Retry if failures searching .watcher-history

Catch exceptions and fail (which allows the assertBusy to retry)
if an error happens from the search against .watcher-history.

Related: #42409
Related: #39169
jakelandis added a commit to jakelandis/elasticsearch that referenced this issue Jul 1, 2019
@jakelandis
SmokeTestWatcherWithSecurityIT: Retry if failures searching (elastic#…
cde6204 
…43781)

Retry if failures searching .watcher-history

Catch exceptions and fail (which allows the assertBusy to retry)
if an error happens from the search against .watcher-history.

Related: elastic#42409
Related: elastic#39169
@jakelandis jakelandis mentioned this issue Jul 3, 2019
Closed
@jakelandis
Copy link
Contributor Author

I'm also hopeful that #43888 will address the issue noted in https://scans.gradle.com/s/m44sa6oyftyue/tests/6ej7h6toy4af6-6g5rmtb4agtbo it may also be the cause for other non-reproducible failures too.

This was referenced Jul 4, 2019
Merged
Merged
Merged
Merged
Merged
@jkakavas
Copy link
Member

jkakavas commented Jul 23, 2019 •
edited
Loading

Adding this here instead of opening a new issue:
I got https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.8+bwc-tests/185/console with testWatchCrudApis and testWatcherStats failing from WatchBackwardsCompatibilityIT. Both have been failing quite often
https://build-stats.elastic.co/app/kibana#/discover?_g=(refreshInterval:(pause:!t,value:0),time:(from:now-30d,mode:quick,to:now))&_a=(columns:!(_source),index:e58bf320-7efd-11e8-bf69-63c8ef516157,interval:auto,query:(language:lucene,query:'testWatchCrudApis%20OR%20testWatcherStats'),sort:!(time,desc)) and usually at the same runs in 6.8. Maybe we missed backporting one of the fixes above to 6.8 ?

@dimitris-athanasiou
Copy link
Contributor

I also got testWatchCrudApis failure like Giannis mentioned above in 6.8. I raised #44814. Feel free to close if you prefer tracking it just here.

@jdconrad
Copy link
Contributor

Some more:
https://scans.gradle.com/s/2mvg5zzkvrscw/console-log

@davidkyle
Copy link
Member

I added #43988 (WatcherRestIT failure) to the top comment

jakelandis added a commit that referenced this issue Aug 16, 2019
@jakelandis
Watcher add stopped listener (#43939)
567ad7f 
When Watcher is stopped and there are still outstanding watches running
Watcher will report it self as stopped. In normal cases, this is not problematic.

However, for integration tests Watcher is started and stopped between
each test to help ensure a clean slate for each test. The tests are blocking
only on the stopped state and make an implicit assumption that all watches are
finished if the Watcher is stopped. This is an incorrect assumption since
Stopped really means, "I will not accept any more watches". This can lead to
un-predictable behavior in the tests such as message : "Watch is already queued
in thread pool" and state: "not_executed_already_queued".
This can also change the .watcher-history if watches linger between tests.

This commit changes the semantics of a manual stopping watcher to now mean:
"I will not accept any more watches AND all running watches are complete".
There is now an intermediary step "Stopping" and callback to allow transition
to a "Stopped" state when all Watches have completed.

Additionally since this impacts how long the tests will block waiting for a
"Stopped" state, the timeout has been increased.

Related: #42409
jakelandis added a commit to jakelandis/elasticsearch that referenced this issue Aug 16, 2019
@jakelandis
Watcher add stopped listener (elastic#43939)
e233f7a 
When Watcher is stopped and there are still outstanding watches running
Watcher will report it self as stopped. In normal cases, this is not problematic.

However, for integration tests Watcher is started and stopped between
each test to help ensure a clean slate for each test. The tests are blocking
only on the stopped state and make an implicit assumption that all watches are
finished if the Watcher is stopped. This is an incorrect assumption since
Stopped really means, "I will not accept any more watches". This can lead to
un-predictable behavior in the tests such as message : "Watch is already queued
in thread pool" and state: "not_executed_already_queued".
This can also change the .watcher-history if watches linger between tests.

This commit changes the semantics of a manual stopping watcher to now mean:
"I will not accept any more watches AND all running watches are complete".
There is now an intermediary step "Stopping" and callback to allow transition
to a "Stopped" state when all Watches have completed.

Additionally since this impacts how long the tests will block waiting for a
"Stopped" state, the timeout has been increased.

Related: elastic#42409
@jakelandis jakelandis mentioned this issue Aug 16, 2019
Merged
jakelandis added a commit that referenced this issue Aug 22, 2019
@jakelandis
Watcher add stopped listener (#43939) (#45670)
1dab739 
When Watcher is stopped and there are still outstanding watches running
Watcher will report it self as stopped. In normal cases, this is not problematic.

However, for integration tests Watcher is started and stopped between
each test to help ensure a clean slate for each test. The tests are blocking
only on the stopped state and make an implicit assumption that all watches are
finished if the Watcher is stopped. This is an incorrect assumption since
Stopped really means, "I will not accept any more watches". This can lead to
un-predictable behavior in the tests such as message : "Watch is already queued
in thread pool" and state: "not_executed_already_queued".
This can also change the .watcher-history if watches linger between tests.

This commit changes the semantics of a manual stopping watcher to now mean:
"I will not accept any more watches AND all running watches are complete".
There is now an intermediary step "Stopping" and callback to allow transition
to a "Stopped" state when all Watches have completed.

Additionally since this impacts how long the tests will block waiting for a
"Stopped" state, the timeout has been increased.

Related: #42409
@jakelandis
Copy link
Contributor Author

Closing this issue out since I believe that this meta issue is causing more noise then value and has outlived it's usefulness. We can address the test failures one by one.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jakelandis jakelandis

Labels
:Data Management/Watcher >test-failure Triaged test failures from CI
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
9 participants
@jakelandis @jdconrad @davidkyle @alpar-t @hendrikmuhs @droberts195 @dimitris-athanasiou @jkakavas @elasticmachine