Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[CI] HistoryTemplateIndexActionMappingsTests.testIndexActionFields is failing #65091

Closed
valeriy42 opened this issue Nov 16, 2020 · 6 comments · Fixed by #67404
Closed

[CI] HistoryTemplateIndexActionMappingsTests.testIndexActionFields is failing #65091

valeriy42 opened this issue Nov 16, 2020 · 6 comments · Fixed by #67404
Labels
:Data Management/Watcher Team:Data Management Meta label for data/management team >test-failure Triaged test failures from CI

Comments

@valeriy42
Copy link
Contributor

valeriy42 commented Nov 16, 2020 •
edited
Loading

Build scan: https://gradle-enterprise.elastic.co/s/r4r5gsjm4daqg

Repro line:

./gradlew ':x-pack:plugin:watcher:internalClusterTest' --tests "org.elasticsearch.xpack.watcher.history.HistoryTemplateIndexActionMappingsTests.testIndexActionFields" -Dtests.seed=30DAF85125EF7D32 -Dtests.security.manager=true -Dtests.locale=mt -Dtests.timezone=America/Montevideo -Druntime.java=11

Reproduces locally?: No

Applicable branches: master

Failure history: build stats failed 7 times in the last 7 days

Failure excerpt:

org.elasticsearch.xpack.watcher.history.HistoryTemplateIndexActionMappingsTests > testIndexActionFields FAILED
    java.lang.AssertionError: 
    Expected: is <1L>
         but: was <2L>
        at __randomizedtesting.SeedInfo.seed([30DAF85125EF7D32:969774AB3FFE9711]:0)
        at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
        at org.junit.Assert.assertThat(Assert.java:956)
        at org.junit.Assert.assertThat(Assert.java:923)
        at org.elasticsearch.xpack.watcher.history.HistoryTemplateIndexActionMappingsTests.testIndexActionFields(HistoryTemplateIndexActionMappingsTests.java:55)
REPRODUCE WITH: ./gradlew ':x-pack:plugin:watcher:internalClusterTest' --tests "org.elasticsearch.xpack.watcher.history.HistoryTemplateIndexActionMappingsTests.testIndexActionFields" -Dtests.seed=30DAF85125EF7D32 -Dtests.security.manager=true -Dtests.locale=mt -Dtests.timezone=America/Montevideo -Druntime.java=11

Suite: Test class org.elasticsearch.xpack.watcher.history.HistoryTemplateIndexActionMappingsTests
  1> [2020-11-16T12:31:50,810][INFO ][o.e.x.w.h.HistoryTemplateIndexActionMappingsTests] [testIndexActionFields] before test
  1> [2020-11-16T12:31:50,814][INFO ][o.e.x.w.h.HistoryTemplateIndexActionMappingsTests] [testIndexActionFields] [HistoryTemplateIndexActionMappingsTests#testIndexActionFields]: setting up test
  1> [2020-11-16T12:31:50,815][INFO ][o.e.t.InternalTestCluster] [testIndexActionFields] Setup InternalTestCluster [SUITE-TEST_WORKER_VM=[327]-CLUSTER_SEED=[1106769756033557643]-HASH=[DC617A31AF]-cluster] with seed [F5C093AAC86A48B] using [3] dedicated masters, [3] (data) nodes and [0] coord only nodes (master nodes are [auto-managed])
  1> [2020-11-16T12:31:50,851][INFO ][o.e.n.Node               ] [testIndexActionFields] version[8.0.0-SNAPSHOT], pid[100130], build[unknown/unknown/95a0ed430473f1414ad02ffce71891ade6d89cd7/2020-11-16T15:17:22.794479Z], OS[Linux/4.15.0-1086-gcp/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/11.0.2/11.0.2+7]
  1> [2020-11-16T12:31:50,851][INFO ][o.e.n.Node               ] [testIndexActionFields] JVM home [/var/lib/jenkins/.java/openjdk-11.0.2-linux]
  1> [2020-11-16T12:31:50,852][DEPRECATION][o.e.d.n.Node             ] [testIndexActionFields] data_stream.dataset="deprecation.elasticsearch" data_stream.namespace="default" data_stream.type="logs" ecs.version="1.6" key="no-jdk" message="no-jdk distributions that do not bundle a JDK are deprecated and will be removed in a future release"
  1> [2020-11-16T12:31:50,852][INFO ][o.e.n.Node               ] [testIndexActionFields] JVM arguments [-Dfile.encoding=UTF8, -Des.scripting.update.ctx_in_params=false, -Des.search.rewrite_sort=true, -Des.set.netty.runtime.available.processors=false, -Des.transport.cname_in_publish_address=true, -Dgradle.dist.lib=/var/lib/jenkins/.gradle/wrapper/dists/gradle-6.6.1-all/ejrtlte9hlw8v6ii20a9584rs/gradle-6.6.1/lib, -Dgradle.user.home=/var/lib/jenkins/.gradle, -Dgradle.worker.jar=/var/lib/jenkins/.gradle/caches/6.6.1/workerMain/gradle-worker.jar, -Dio.netty.noKeySetOptimization=true, -Dio.netty.noUnsafe=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Djava.awt.headless=true, -Djava.locale.providers=SPI,COMPAT, -Djna.nosys=true, -Dorg.gradle.native=false, -Dtests.artifact=watcher, -Dtests.gradle=true, -Dtests.logger.level=WARN, -Dtests.security.manager=true, -Dtests.seed=30DAF85125EF7D32, -Dtests.task=:x-pack:plugin:watcher:internalClusterTest, --illegal-access=warn, -XX:+HeapDumpOnOutOfMemoryError, -esa, -XX:HeapDumpPath=/dev/shm/elastic+elasticsearch+master+multijob+fast+part2/x-pack/plugin/watcher/build/heapdump, -Xms512m, -Xmx512m, -Dfile.encoding=UTF-8, -Djava.io.tmpdir=/dev/shm/elastic+elasticsearch+master+multijob+fast+part2/x-pack/plugin/watcher/build/testrun/internalClusterTest/temp, -Duser.country=US, -Duser.language=en, -Duser.variant, -ea]
  1> [2020-11-16T12:31:50,852][WARN ][o.e.n.Node               ] [testIndexActionFields] version [8.0.0-SNAPSHOT] is a pre-release version of Elasticsearch and is not suitable for production
  1> [2020-11-16T12:31:50,854][INFO ][o.e.x.w.t.TimeWarpedWatcher] [testIndexActionFields] using time warped watchers plugin
  1> [2020-11-16T12:31:50,856][INFO ][o.e.p.PluginsService     ] [testIndexActionFields] no modules loaded
  1> [2020-11-16T12:31:50,856][INFO ][o.e.p.PluginsService     ] [testIndexActionFields] loaded plugin [org.elasticsearch.analysis.common.CommonAnalysisPlugin]
  1> [2020-11-16T12:31:50,856][INFO ][o.e.p.PluginsService     ] [testIndexActionFields] loaded plugin [org.elasticsearch.script.MockMustacheScriptEngine$TestPlugin]
  1> [2020-11-16T12:31:50,856][INFO ][o.e.p.PluginsService     ] [testIndexActionFields] loaded plugin [org.elasticsearch.script.MockScriptService$TestPlugin]
  1> [2020-11-16T12:31:50,856][INFO ][o.e.p.PluginsService     ] [testIndexActionFields] loaded plugin [org.elasticsearch.test.ESIntegTestCase$AssertActionNamePlugin]
  1> [2020-11-16T12:31:50,856][INFO ][o.e.p.PluginsService     ] [testIndexActionFields] loaded plugin [org.elasticsearch.test.ESIntegTestCase$TestSeedPlugin]
  1> [2020-11-16T12:31:50,856][INFO ][o.e.p.PluginsService     ] [testIndexActionFields] loaded plugin [org.elasticsearch.test.MockHttpTransport$TestPlugin]
  1> [2020-11-16T12:31:50,856][INFO ][o.e.p.PluginsService     ] [testIndexActionFields] loaded plugin [org.elasticsearch.test.TestGeoShapeFieldMapperPlugin]
  1> [2020-11-16T12:31:50,857][INFO ][o.e.p.PluginsService     ] [testIndexActionFields] loaded plugin [org.elasticsearch.test.store.MockFSIndexStore$TestPlugin]
  1> [2020-11-16T12:31:50,857][INFO ][o.e.p.PluginsService     ] [testIndexActionFields] loaded plugin [org.elasticsearch.transport.nio.MockNioTransportPlugin]
  1> [2020-11-16T12:31:50,857][INFO ][o.e.p.PluginsService     ] [testIndexActionFields] loaded plugin [org.elasticsearch.xpack.datastreams.DataStreamsPlugin]
  1> [2020-11-16T12:31:50,857][INFO ][o.e.p.PluginsService     ] [testIndexActionFields] loaded plugin [org.elasticsearch.xpack.ilm.IndexLifecycle]
  1> [2020-11-16T12:31:50,857][INFO ][o.e.p.PluginsService     ] [testIndexActionFields] loaded plugin [org.elasticsearch.xpack.watcher.test.TimeWarpedWatcher]
  1> [2020-11-16T12:31:50,863][INFO ][o.e.e.NodeEnvironment    ] [testIndexActionFields] using [2] data paths, mounts [[/dev/shm (tmpfs)]], net usable_space [43.4gb], net total_space [47.2gb], types [tmpfs]
  1> [2020-11-16T12:31:50,864][INFO ][o.e.e.NodeEnvironment    ] [testIndexActionFields] heap size [512mb], compressed ordinary object pointers [true]
  1> [2020-11-16T12:31:50,873][INFO ][o.e.n.Node               ] [testIndexActionFields] node name [node_s0], node ID [WLE0cJ77Qxa_r1uKx10uYg], cluster name [SUITE-TEST_WORKER_VM=[327]-CLUSTER_SEED=[1106769756033557643]-HASH=[DC617A31AF]-cluster], roles [ingest, remote_cluster_client, master]
  1> [2020-11-16T12:31:51,053][INFO ][o.e.d.DiscoveryModule    ] [testIndexActionFields] using discovery type [zen] and seed hosts providers [settings, file]
  1> [2020-11-16T12:31:51,209][INFO ][o.e.n.Node               ] [testIndexActionFields] initialized
  1> [2020-11-16T12:31:51,212][INFO ][o.e.n.Node               ] [testIndexActionFields] version[8.0.0-SNAPSHOT], pid[100130], build[unknown/unknown/95a0ed430473f1414ad02ffce71891ade6d89cd7/2020-11-16T15:17:22.794479Z], OS[Linux/4.15.0-1086-gcp/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/11.0.2/11.0.2+7]
  1> [2020-11-16T12:31:51,227][INFO ][o.e.n.Node               ] [testIndexActionFields] JVM home [/var/lib/jenkins/.java/openjdk-11.0.2-linux]
  1> [2020-11-16T12:31:51,228][DEPRECATION][o.e.d.n.Node             ] [testIndexActionFields] data_stream.dataset="deprecation.elasticsearch" data_stream.namespace="default" data_stream.type="logs" ecs.version="1.6" key="no-jdk" message="no-jdk distributions that do not bundle a JDK are deprecated and will be removed in a future release"
  1> [2020-11-16T12:31:51,228][INFO ][o.e.n.Node               ] [testIndexActionFields] JVM arguments [-Dfile.encoding=UTF8, -Des.scripting.update.ctx_in_params=false, -Des.search.rewrite_sort=true, -Des.set.netty.runtime.available.processors=false, -Des.transport.cname_in_publish_address=true, -Dgradle.dist.lib=/var/lib/jenkins/.gradle/wrapper/dists/gradle-6.6.1-all/ejrtlte9hlw8v6ii20a9584rs/gradle-6.6.1/lib, -Dgradle.user.home=/var/lib/jenkins/.gradle, -Dgradle.worker.jar=/var/lib/jenkins/.gradle/caches/6.6.1/workerMain/gradle-worker.jar, -Dio.netty.noKeySetOptimization=true, -Dio.netty.noUnsafe=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Djava.awt.headless=true, -Djava.locale.providers=SPI,COMPAT, -Djna.nosys=true, -Dorg.gradle.native=false, -Dtests.artifact=watcher, -Dtests.gradle=true, -Dtests.logger.level=WARN, -Dtests.security.manager=true, -Dtests.seed=30DAF85125EF7D32, -Dtests.task=:x-pack:plugin:watcher:internalClusterTest, --illegal-access=warn, -XX:+HeapDumpOnOutOfMemoryError, -esa, -XX:HeapDumpPath=/dev/shm/elastic+elasticsearch+master+multijob+fast+part2/x-pack/plugin/watcher/build/heapdump, -Xms512m, -Xmx512m, -Dfile.encoding=UTF-8, -Djava.io.tmpdir=/dev/shm/elastic+elasticsearch+master+multijob+fast+part2/x-pack/plugin/watcher/build/testrun/internalClusterTest/temp, -Duser.country=US, -Duser.language=en, -Duser.variant, -ea]
  1> [2020-11-16T12:31:51,229][WARN ][o.e.n.Node               ] [testIndexActionFields] version [8.0.0-SNAPSHOT] is a pre-release version of Elasticsearch and is not suitable for production
  1> [2020-11-16T12:31:51,229][INFO ][o.e.x.w.t.TimeWarpedWatcher] [testIndexActionFields] using time warped watchers plugin
  1> [2020-11-16T12:31:51,230][INFO ][o.e.p.PluginsService     ] [testIndexActionFields] no modules loaded
  1> [2020-11-16T12:31:51,230][INFO ][o.e.p.PluginsService     ] [testIndexActionFields] loaded plugin [org.elasticsearch.analysis.common.CommonAnalysisPlugin]
  1> [2020-11-16T12:31:51,231][INFO ][o.e.p.PluginsService     ] [testIndexActionFields] loaded plugin [org.elasticsearch.script.MockMustacheScriptEngine$TestPlugin]
  1> [2020-11-16T12:31:51,231][INFO ][o.e.p.PluginsService     ] [testIndexActionFields] loaded plugin [org.elasticsearch.script.MockScriptService$TestPlugin]
  1> [2020-11-16T12:31:51,231][INFO ][o.e.p.PluginsService     ] [testIndexActionFields] loaded plugin [org.elasticsearch.test.ESIntegTestCase$AssertActionNamePlugin]
  1> [2020-11-16T12:31:51,231][INFO ][o.e.p.PluginsService     ] [testIndexActionFields] loaded plugin [org.elasticsearch.test.ESIntegTestCase$TestSeedPlugin]
  1> [2020-11-16T12:31:51,231][INFO ][o.e.p.PluginsService     ] [testIndexActionFields] loaded plugin [org.elasticsearch.test.MockHttpTransport$TestPlugin]
  1> [2020-11-16T12:31:51,231][INFO ][o.e.p.PluginsService     ] [testIndexActionFields] loaded plugin [org.elasticsearch.test.TestGeoShapeFieldMapperPlugin]
  1> [2020-11-16T12:31:51,232][INFO ][o.e.p.PluginsService     ] [testIndexActionFields] loaded plugin [org.elasticsearch.test.store.MockFSIndexStore$TestPlugin]
  1> [2020-11-16T12:31:51,232][INFO ][o.e.p.PluginsService     ] [testIndexActionFields] loaded plugin [org.elasticsearch.transport.nio.MockNioTransportPlugin]
  1> [2020-11-16T12:31:51,232][INFO ][o.e.p.PluginsService     ] [testIndexActionFields] loaded plugin [org.elasticsearch.xpack.datastreams.DataStreamsPlugin]
  1> [2020-11-16T12:31:51,232][INFO ][o.e.p.PluginsService     ] [testIndexActionFields] loaded plugin [org.elasticsearch.xpack.ilm.IndexLifecycle]
  1> [2020-11-16T12:31:51,232][INFO ][o.e.p.PluginsService     ] [testIndexActionFields] loaded plugin [org.elasticsearch.xpack.watcher.test.TimeWarpedWatcher]
  1> [2020-11-16T12:31:51,238][INFO ][o.e.e.NodeEnvironment    ] [testIndexActionFields] using [2] data paths, mounts [[/dev/shm (tmpfs)]], net usable_space [43.4gb], net total_space [47.2gb], types [tmpfs]
  1> [2020-11-16T12:31:51,239][INFO ][o.e.e.NodeEnvironment    ] [testIndexActionFields] heap size [512mb], compressed ordinary object pointers [true]
  1> [2020-11-16T12:31:51,249][INFO ][o.e.n.Node               ] [testIndexActionFields] node name [node_s1], node ID [4f6qJ9JmR12iEkuIvZtdkQ], cluster name [SUITE-TEST_WORKER_VM=[327]-CLUSTER_SEED=[1106769756033557643]-HASH=[DC617A31AF]-cluster], roles [ingest, remote_cluster_client, master]
  1> [2020-11-16T12:31:51,382][INFO ][o.e.d.DiscoveryModule    ] [testIndexActionFields] using discovery type [zen] and seed hosts providers [settings, file]
  1> [2020-11-16T12:31:51,504][INFO ][o.e.n.Node               ] [testIndexActionFields] initialized
  1> [2020-11-16T12:31:51,508][INFO ][o.e.n.Node               ] [testIndexActionFields] version[8.0.0-SNAPSHOT], pid[100130], build[unknown/unknown/95a0ed430473f1414ad02ffce71891ade6d89cd7/2020-11-16T15:17:22.794479Z], OS[Linux/4.15.0-1086-gcp/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/11.0.2/11.0.2+7]
  1> [2020-11-16T12:31:51,509][INFO ][o.e.n.Node               ] [testIndexActionFields] JVM home [/var/lib/jenkins/.java/openjdk-11.0.2-linux]
  1> [2020-11-16T12:31:51,510][DEPRECATION][o.e.d.n.Node             ] [testIndexActionFields] data_stream.dataset="deprecation.elasticsearch" data_stream.namespace="default" data_stream.type="logs" ecs.version="1.6" key="no-jdk" message="no-jdk distributions that do not bundle a JDK are deprecated and will be removed in a future release"
  1> [2020-11-16T12:31:51,510][INFO ][o.e.n.Node               ] [testIndexActionFields] JVM arguments [-Dfile.encoding=UTF8, -Des.scripting.update.ctx_in_params=false, -Des.search.rewrite_sort=true, -Des.set.netty.runtime.available.processors=false, -Des.transport.cname_in_publish_address=true, -Dgradle.dist.lib=/var/lib/jenkins/.gradle/wrapper/dists/gradle-6.6.1-all/ejrtlte9hlw8v6ii20a9584rs/gradle-6.6.1/lib, -Dgradle.user.home=/var/lib/jenkins/.gradle, -Dgradle.worker.jar=/var/lib/jenkins/.gradle/caches/6.6.1/workerMain/gradle-worker.jar, -Dio.netty.noKeySetOptimization=true, -Dio.netty.noUnsafe=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Djava.awt.headless=true, -Djava.locale.providers=SPI,COMPAT, -Djna.nosys=true, -Dorg.gradle.native=false, -Dtests.artifact=watcher, -Dtests.gradle=true, -Dtests.logger.level=WARN, -Dtests.security.manager=true, -Dtests.seed=30DAF85125EF7D32, -Dtests.task=:x-pack:plugin:watcher:internalClusterTest, --illegal-access=warn, -XX:+HeapDumpOnOutOfMemoryError, -esa, -XX:HeapDumpPath=/dev/shm/elastic+elasticsearch+master+multijob+fast+part2/x-pack/plugin/watcher/build/heapdump, -Xms512m, -Xmx512m, -Dfile.encoding=UTF-8, -Djava.io.tmpdir=/dev/shm/elastic+elasticsearch+master+multijob+fast+part2/x-pack/plugin/watcher/build/testrun/internalClusterTest/temp, -Duser.country=US, -Duser.language=en, -Duser.variant, -ea]
  1> [2020-11-16T12:31:51,513][WARN ][o.e.n.Node               ] [testIndexActionFields] version [8.0.0-SNAPSHOT] is a pre-release version of Elasticsearch and is not suitable for production
  1> [2020-11-16T12:31:51,513][INFO ][o.e.x.w.t.TimeWarpedWatcher] [testIndexActionFields] using time warped watchers plugin
  1> [2020-11-16T12:31:51,514][INFO ][o.e.p.PluginsService     ] [testIndexActionFields] no modules loaded
  1> [2020-11-16T12:31:51,515][INFO ][o.e.p.PluginsService     ] [testIndexActionFields] loaded plugin [org.elasticsearch.analysis.common.CommonAnalysisPlugin]
  1> [2020-11-16T12:31:51,517][INFO ][o.e.p.PluginsService     ] [testIndexActionFields] loaded plugin [org.elasticsearch.script.MockMustacheScriptEngine$TestPlugin]
  1> [2020-11-16T12:31:51,517][INFO ][o.e.p.PluginsService     ] [testIndexActionFields] loaded plugin [org.elasticsearch.script.MockScriptService$TestPlugin]
  1> [2020-11-16T12:31:51,517][INFO ][o.e.p.PluginsService     ] [testIndexActionFields] loaded plugin [org.elasticsearch.test.ESIntegTestCase$AssertActionNamePlugin]
  1> [2020-11-16T12:31:51,517][INFO ][o.e.p.PluginsService     ] [testIndexActionFields] loaded plugin [org.elasticsearch.test.ESIntegTestCase$TestSeedPlugin]
  1> [2020-11-16T12:31:51,518][INFO ][o.e.p.PluginsService     ] [testIndexActionFields] loaded plugin [org.elasticsearch.test.MockHttpTransport$TestPlugin]
  1> [2020-11-16T12:31:51,518][INFO ][o.e.p.PluginsService     ] [testIndexActionFields] loaded plugin [org.elasticsearch.test.TestGeoShapeFieldMapperPlugin]
  1> [2020-11-16T12:31:51,518][INFO ][o.e.p.PluginsService     ] [testIndexActionFields] loaded plugin [org.elasticsearch.test.store.MockFSIndexStore$TestPlugin]
  1> [2020-11-16T12:31:51,518][INFO ][o.e.p.PluginsService     ] [testIndexActionFields] loaded plugin [org.elasticsearch.transport.nio.MockNioTransportPlugin]
  1> [2020-11-16T12:31:51,518][INFO ][o.e.p.PluginsService     ] [testIndexActionFields] loaded plugin [org.elasticsearch.xpack.datastreams.DataStreamsPlugin]
  1> [2020-11-16T12:31:51,519][INFO ][o.e.p.PluginsService     ] [testIndexActionFields] loaded plugin [org.elasticsearch.xpack.ilm.IndexLifecycle]
  1> [2020-11-16T12:31:51,535][INFO ][o.e.p.PluginsService     ] [testIndexActionFields] loaded plugin [org.elasticsearch.xpack.watcher.test.TimeWarpedWatcher]
  1> [2020-11-16T12:31:51,543][INFO ][o.e.e.NodeEnvironment    ] [testIndexActionFields] using [2] data paths, mounts [[/dev/shm (tmpfs)]], net usable_space [43.4gb], net total_space [47.2gb], types [tmpfs]
  1> [2020-11-16T12:31:51,544][INFO ][o.e.e.NodeEnvironment    ] [testIndexActionFields] heap size [512mb], compressed ordinary object pointers [true]
  1> [2020-11-16T12:31:51,548][INFO ][o.e.n.Node               ] [testIndexActionFields] node name [node_s2], node ID [t2r9oPnUSti7AAnSY0YkHQ], cluster name [SUITE-TEST_WORKER_VM=[327]-CLUSTER_SEED=[1106769756033557643]-HASH=[DC617A31AF]-cluster], roles [ingest, remote_cluster_client, master]
  1> [2020-11-16T12:31:51,700][INFO ][o.e.d.DiscoveryModule    ] [testIndexActionFields] using discovery type [zen] and seed hosts providers [settings, file]
  1> [2020-11-16T12:31:51,779][INFO ][o.e.n.Node               ] [testIndexActionFields] initialized
  1> [2020-11-16T12:31:51,781][INFO ][o.e.n.Node               ] [testIndexActionFields] version[8.0.0-SNAPSHOT], pid[100130], build[unknown/unknown/95a0ed430473f1414ad02ffce71891ade6d89cd7/2020-11-16T15:17:22.794479Z], OS[Linux/4.15.0-1086-gcp/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/11.0.2/11.0.2+7]
  1> [2020-11-16T12:31:51,782][INFO ][o.e.n.Node               ] [testIndexActionFields] JVM home [/var/lib/jenkins/.java/openjdk-11.0.2-linux]
  1> [2020-11-16T12:31:51,782][DEPRECATION][o.e.d.n.Node             ] [testIndexActionFields] data_stream.dataset="deprecation.elasticsearch" data_stream.namespace="default" data_stream.type="logs" ecs.version="1.6" key="no-jdk" message="no-jdk distributions that do not bundle a JDK are deprecated and will be removed in a future release"
  1> [2020-11-16T12:31:51,783][INFO ][o.e.n.Node               ] [testIndexActionFields] JVM arguments [-Dfile.encoding=UTF8, -Des.scripting.update.ctx_in_params=false, -Des.search.rewrite_sort=true, -Des.set.netty.runtime.available.processors=false, -Des.transport.cname_in_publish_address=true, -Dgradle.dist.lib=/var/lib/jenkins/.gradle/wrapper/dists/gradle-6.6.1-all/ejrtlte9hlw8v6ii20a9584rs/gradle-6.6.1/lib, -Dgradle.user.home=/var/lib/jenkins/.gradle, -Dgradle.worker.jar=/var/lib/jenkins/.gradle/caches/6.6.1/workerMain/gradle-worker.jar, -Dio.netty.noKeySetOptimization=true, -Dio.netty.noUnsafe=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Djava.awt.headless=true, -Djava.locale.providers=SPI,COMPAT, -Djna.nosys=true, -Dorg.gradle.native=false, -Dtests.artifact=watcher, -Dtests.gradle=true, -Dtests.logger.level=WARN, -Dtests.security.manager=true, -Dtests.seed=30DAF85125EF7D32, -Dtests.task=:x-pack:plugin:watcher:internalClusterTest, --illegal-access=warn, -XX:+HeapDumpOnOutOfMemoryError, -esa, -XX:HeapDumpPath=/dev/shm/elastic+elasticsearch+master+multijob+fast+part2/x-pack/plugin/watcher/build/heapdump, -Xms512m, -Xmx512m, -Dfile.encoding=UTF-8, -Djava.io.tmpdir=/dev/shm/elastic+elasticsearch+master+multijob+fast+part2/x-pack/plugin/watcher/build/testrun/internalClusterTest/temp, -Duser.country=US, -Duser.language=en, -Duser.variant, -ea]
  1> [2020-11-16T12:31:51,784][WARN ][o.e.n.Node               ] [testIndexActionFields] version [8.0.0-SNAPSHOT] is a pre-release version of Elasticsearch and is not suitable for production
  1> [2020-11-16T12:31:51,784][INFO ][o.e.x.w.t.TimeWarpedWatcher] [testIndexActionFields] using time warped watchers plugin
  1> [2020-11-16T12:31:51,785][INFO ][o.e.p.PluginsService     ] [testIndexActionFields] no modules loaded
  1> [2020-11-16T12:31:51,785][INFO ][o.e.p.PluginsService     ] [testIndexActionFields] loaded plugin [org.elasticsearch.analysis.common.CommonAnalysisPlugin]
  1> [2020-11-16T12:31:51,786][INFO ][o.e.p.PluginsService     ] [testIndexActionFields] loaded plugin [org.elasticsearch.script.MockMustacheScriptEngine$TestPlugin]
  1> [2020-11-16T12:31:51,787][INFO ][o.e.p.PluginsService     ] [testIndexActionFields] loaded plugin [org.elasticsearch.script.MockScriptService$TestPlugin]
  1> [2020-11-16T12:31:51,787][INFO ][o.e.p.PluginsService     ] [testIndexActionFields] loaded plugin [org.elasticsearch.test.ESIntegTestCase$AssertActionNamePlugin]
  1> [2020-11-16T12:31:51,788][INFO ][o.e.p.PluginsService     ] [testIndexActionFields] loaded plugin [org.elasticsearch.test.ESIntegTestCase$TestSeedPlugin]
  1> [2020-11-16T12:31:51,788][INFO ][o.e.p.PluginsService     ] [testIndexActionFields] loaded plugin [org.elasticsearch.test.MockHttpTransport$TestPlugin]
  1> [2020-11-16T12:31:51,789][INFO ][o.e.p.PluginsService     ] [testIndexActionFields] loaded plugin [org.elasticsearch.test.TestGeoShapeFieldMapperPlugin]
  1> [2020-11-16T12:31:51,789][INFO ][o.e.p.PluginsService     ] [testIndexActionFields] loaded plugin [org.elasticsearch.test.store.MockFSIndexStore$TestPlugin]
  1> [2020-11-16T12:31:51,789][INFO ][o.e.p.PluginsService     ] [testIndexActionFields] loaded plugin [org.elasticsearch.transport.nio.MockNioTransportPlugin]
  1> [2020-11-16T12:31:51,790][INFO ][o.e.p.PluginsService     ] [testIndexActionFields] loaded plugin [org.elasticsearch.xpack.datastreams.DataStreamsPlugin]
  1> [2020-11-16T12:31:51,792][INFO ][o.e.p.PluginsService     ] [testIndexActionFields] loaded plugin [org.elasticsearch.xpack.ilm.IndexLifecycle]
  1> [2020-11-16T12:31:51,792][INFO ][o.e.p.PluginsService     ] [testIndexActionFields] loaded plugin [org.elasticsearch.xpack.watcher.test.TimeWarpedWatcher]
  1> [2020-11-16T12:31:51,799][INFO ][o.e.e.NodeEnvironment    ] [testIndexActionFields] using [2] data paths, mounts [[/dev/shm (tmpfs)]], net usable_space [43.4gb], net total_space [47.2gb], types [tmpfs]
  1> [2020-11-16T12:31:51,802][INFO ][o.e.e.NodeEnvironment    ] [testIndexActionFields] heap size [512mb], compressed ordinary object pointers [true]
  1> [2020-11-16T12:31:51,806][INFO ][o.e.n.Node               ] [testIndexActionFields] node name [node_s3], node ID [Auk9Zaa6Qt-q_nll-qhOVw], cluster name [SUITE-TEST_WORKER_VM=[327]-CLUSTER_SEED=[1106769756033557643]-HASH=[DC617A31AF]-cluster], roles [ingest, remote_cluster_client, data]
  1> [2020-11-16T12:31:51,984][INFO ][o.e.d.DiscoveryModule    ] [testIndexActionFields] using discovery type [zen] and seed hosts providers [settings, file]
  1> [2020-11-16T12:31:52,102][INFO ][o.e.n.Node               ] [testIndexActionFields] initialized
  1> [2020-11-16T12:31:52,105][INFO ][o.e.n.Node               ] [testIndexActionFields] version[8.0.0-SNAPSHOT], pid[100130], build[unknown/unknown/95a0ed430473f1414ad02ffce71891ade6d89cd7/2020-11-16T15:17:22.794479Z], OS[Linux/4.15.0-1086-gcp/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/11.0.2/11.0.2+7]
  1> [2020-11-16T12:31:52,106][INFO ][o.e.n.Node               ] [testIndexActionFields] JVM home [/var/lib/jenkins/.java/openjdk-11.0.2-linux]
  1> [2020-11-16T12:31:52,106][DEPRECATION][o.e.d.n.Node             ] [testIndexActionFields] data_stream.dataset="deprecation.elasticsearch" data_stream.namespace="default" data_stream.type="logs" ecs.version="1.6" key="no-jdk" message="no-jdk distributions that do not bundle a JDK are deprecated and will be removed in a future release"
  1> [2020-11-16T12:31:52,106][INFO ][o.e.n.Node               ] [testIndexActionFields] JVM arguments [-Dfile.encoding=UTF8, -Des.scripting.update.ctx_in_params=false, -Des.search.rewrite_sort=true, -Des.set.netty.runtime.available.processors=false, -Des.transport.cname_in_publish_address=true, -Dgradle.dist.lib=/var/lib/jenkins/.gradle/wrapper/dists/gradle-6.6.1-all/ejrtlte9hlw8v6ii20a9584rs/gradle-6.6.1/lib, -Dgradle.user.home=/var/lib/jenkins/.gradle, -Dgradle.worker.jar=/var/lib/jenkins/.gradle/caches/6.6.1/workerMain/gradle-worker.jar, -Dio.netty.noKeySetOptimization=true, -Dio.netty.noUnsafe=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Djava.awt.headless=true, -Djava.locale.providers=SPI,COMPAT, -Djna.nosys=true, -Dorg.gradle.native=false, -Dtests.artifact=watcher, -Dtests.gradle=true, -Dtests.logger.level=WARN, -Dtests.security.manager=true, -Dtests.seed=30DAF85125EF7D32, -Dtests.task=:x-pack:plugin:watcher:internalClusterTest, --illegal-access=warn, -XX:+HeapDumpOnOutOfMemoryError, -esa, -XX:HeapDumpPath=/dev/shm/elastic+elasticsearch+master+multijob+fast+part2/x-pack/plugin/watcher/build/heapdump, -Xms512m, -Xmx512m, -Dfile.encoding=UTF-8, -Djava.io.tmpdir=/dev/shm/elastic+elasticsearch+master+multijob+fast+part2/x-pack/plugin/watcher/build/testrun/internalClusterTest/temp, -Duser.country=US, -Duser.language=en, -Duser.variant, -ea]
  1> [2020-11-16T12:31:52,107][WARN ][o.e.n.Node               ] [testIndexActionFields] version [8.0.0-SNAPSHOT] is a pre-release version of Elasticsearch and is not suitable for production
  1> [2020-11-16T12:31:52,107][INFO ][o.e.x.w.t.TimeWarpedWatcher] [testIndexActionFields] using time warped watchers plugin
  1> [2020-11-16T12:31:52,114][INFO ][o.e.p.PluginsService     ] [testIndexActionFields] no modules loaded
  1> [2020-11-16T12:31:52,116][INFO ][o.e.p.PluginsService     ] [testIndexActionFields] loaded plugin [org.elasticsearch.analysis.common.CommonAnalysisPlugin]
  1> [2020-11-16T12:31:52,119][INFO ][o.e.p.PluginsService     ] [testIndexActionFields] loaded plugin [org.elasticsearch.script.MockMustacheScriptEngine$TestPlugin]
  1> [2020-11-16T12:31:52,119][INFO ][o.e.p.PluginsService     ] [testIndexActionFields] loaded plugin [org.elasticsearch.script.MockScriptService$TestPlugin]
  1> [2020-11-16T12:31:52,120][INFO ][o.e.p.PluginsService     ] [testIndexActionFields] loaded plugin [org.elasticsearch.test.ESIntegTestCase$AssertActionNamePlugin]
  1> [2020-11-16T12:31:52,120][INFO ][o.e.p.PluginsService     ] [testIndexActionFields] loaded plugin [org.elasticsearch.test.ESIntegTestCase$TestSeedPlugin]
  1> [2020-11-16T12:31:52,120][INFO ][o.e.p.PluginsService     ] [testIndexActionFields] loaded plugin [org.elasticsearch.test.MockHttpTransport$TestPlugin]
  1> [2020-11-16T12:31:52,120][INFO ][o.e.p.PluginsService     ] [testIndexActionFields] loaded plugin [org.elasticsearch.test.TestGeoShapeFieldMapperPlugin]
  1> [2020-11-16T12:31:52,120][INFO ][o.e.p.PluginsService     ] [testIndexActionFields] loaded plugin [org.elasticsearch.test.store.MockFSIndexStore$TestPlugin]
  1> [2020-11-16T12:31:52,120][INFO ][o.e.p.PluginsService     ] [testIndexActionFields] loaded plugin [org.elasticsearch.transport.nio.MockNioTransportPlugin]
  1> [2020-11-16T12:31:52,120][INFO ][o.e.p.PluginsService     ] [testIndexActionFields] loaded plugin [org.elasticsearch.xpack.datastreams.DataStreamsPlugin]
  1> [2020-11-16T12:31:52,120][INFO ][o.e.p.PluginsService     ] [testIndexActionFields] loaded plugin [org.elasticsearch.xpack.ilm.IndexLifecycle]
  1> [2020-11-16T12:31:52,120][INFO ][o.e.p.PluginsService     ] [testIndexActionFields] loaded plugin [org.elasticsearch.xpack.watcher.test.TimeWarpedWatcher]
  1> [2020-11-16T12:31:52,163][INFO ][o.e.e.NodeEnvironment    ] [testIndexActionFields] using [2] data paths, mounts [[/dev/shm (tmpfs)]], net usable_space [43.4gb], net total_space [47.2gb], types [tmpfs]
  1> [2020-11-16T12:31:52,164][INFO ][o.e.e.NodeEnvironment    ] [testIndexActionFields] heap size [512mb], compressed ordinary object pointers [true]
  1> [2020-11-16T12:31:52,168][INFO ][o.e.n.Node               ] [testIndexActionFields] node name [node_s4], node ID [ltFylxUSQ9anDIsodXNq_g], cluster name [SUITE-TEST_WORKER_VM=[327]-CLUSTER_SEED=[1106769756033557643]-HASH=[DC617A31AF]-cluster], roles [ingest, remote_cluster_client, data]
  1> [2020-11-16T12:31:52,361][INFO ][o.e.d.DiscoveryModule    ] [testIndexActionFields] using discovery type [zen] and seed hosts providers [settings, file]
  1> [2020-11-16T12:31:52,417][INFO ][o.e.n.Node               ] [testIndexActionFields] initialized
  1> [2020-11-16T12:31:52,419][INFO ][o.e.n.Node               ] [testIndexActionFields] version[8.0.0-SNAPSHOT], pid[100130], build[unknown/unknown/95a0ed430473f1414ad02ffce71891ade6d89cd7/2020-11-16T15:17:22.794479Z], OS[Linux/4.15.0-1086-gcp/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/11.0.2/11.0.2+7]
  1> [2020-11-16T12:31:52,419][INFO ][o.e.n.Node               ] [testIndexActionFields] JVM home [/var/lib/jenkins/.java/openjdk-11.0.2-linux]
  1> [2020-11-16T12:31:52,420][DEPRECATION][o.e.d.n.Node             ] [testIndexActionFields] data_stream.dataset="deprecation.elasticsearch" data_stream.namespace="default" data_stream.type="logs" ecs.version="1.6" key="no-jdk" message="no-jdk distributions that do not bundle a JDK are deprecated and will be removed in a future release"
  1> [2020-11-16T12:31:52,420][INFO ][o.e.n.Node               ] [testIndexActionFields] JVM arguments [-Dfile.encoding=UTF8, -Des.scripting.update.ctx_in_params=false, -Des.search.rewrite_sort=true, -Des.set.netty.runtime.available.processors=false, -Des.transport.cname_in_publish_address=true, -Dgradle.dist.lib=/var/lib/jenkins/.gradle/wrapper/dists/gradle-6.6.1-all/ejrtlte9hlw8v6ii20a9584rs/gradle-6.6.1/lib, -Dgradle.user.home=/var/lib/jenkins/.gradle, -Dgradle.worker.jar=/var/lib/jenkins/.gradle/caches/6.6.1/workerMain/gradle-worker.jar, -Dio.netty.noKeySetOptimization=true, -Dio.netty.noUnsafe=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Djava.awt.headless=true, -Djava.locale.providers=SPI,COMPAT, -Djna.nosys=true, -Dorg.gradle.native=false, -Dtests.artifact=watcher, -Dtests.gradle=true, -Dtests.logger.level=WARN, -Dtests.security.manager=true, -Dtests.seed=30DAF85125EF7D32, -Dtests.task=:x-pack:plugin:watcher:internalClusterTest, --illegal-access=warn, -XX:+HeapDumpOnOutOfMemoryError, -esa, -XX:HeapDumpPath=/dev/shm/elastic+elasticsearch+master+multijob+fast+part2/x-pack/plugin/watcher/build/heapdump, -Xms512m, -Xmx512m, -Dfile.encoding=UTF-8, -Djava.io.tmpdir=/dev/shm/elastic+elasticsearch+master+multijob+fast+part2/x-pack/plugin/watcher/build/testrun/internalClusterTest/temp, -Duser.country=US, -Duser.language=en, -Duser.variant, -ea]
  1> [2020-11-16T12:31:52,420][WARN ][o.e.n.Node               ] [testIndexActionFields] version [8.0.0-SNAPSHOT] is a pre-release version of Elasticsearch and is not suitable for production
  1> [2020-11-16T12:31:52,420][INFO ][o.e.x.w.t.TimeWarpedWatcher] [testIndexActionFields] using time warped watchers plugin
  1> [2020-11-16T12:31:52,421][INFO ][o.e.p.PluginsService     ] [testIndexActionFields] no modules loaded
  1> [2020-11-16T12:31:52,421][INFO ][o.e.p.PluginsService     ] [testIndexActionFields] loaded plugin [org.elasticsearch.analysis.common.CommonAnalysisPlugin]
  1> [2020-11-16T12:31:52,421][INFO ][o.e.p.PluginsService     ] [testIndexActionFields] loaded plugin [org.elasticsearch.script.MockMustacheScriptEngine$TestPlugin]
  1> [2020-11-16T12:31:52,422][INFO ][o.e.p.PluginsService     ] [testIndexActionFields] loaded plugin [org.elasticsearch.script.MockScriptService$TestPlugin]
  1> [2020-11-16T12:31:52,422][INFO ][o.e.p.PluginsService     ] [testIndexActionFields] loaded plugin [org.elasticsearch.test.ESIntegTestCase$AssertActionNamePlugin]
  1> [2020-11-16T12:31:52,422][INFO ][o.e.p.PluginsService     ] [testIndexActionFields] loaded plugin [org.elasticsearch.test.ESIntegTestCase$TestSeedPlugin]
  1> [2020-11-16T12:31:52,422][INFO ][o.e.p.PluginsService     ] [testIndexActionFields] loaded plugin [org.elasticsearch.test.MockHttpTransport$TestPlugin]
  1> [2020-11-16T12:31:52,422][INFO ][o.e.p.PluginsService     ] [testIndexActionFields] loaded plugin [org.elasticsearch.test.TestGeoShapeFieldMapperPlugin]
  1> [2020-11-16T12:31:52,422][INFO ][o.e.p.PluginsService     ] [testIndexActionFields] loaded plugin [org.elasticsearch.test.store.MockFSIndexStore$TestPlugin]
  1> [2020-11-16T12:31:52,422][INFO ][o.e.p.PluginsService     ] [testIndexActionFields] loaded plugin [org.elasticsearch.transport.nio.MockNioTransportPlugin]
  1> [2020-11-16T12:31:52,422][INFO ][o.e.p.PluginsService     ] [testIndexActionFields] loaded plugin [org.elasticsearch.xpack.datastreams.DataStreamsPlugin]
  1> [2020-11-16T12:31:52,422][INFO ][o.e.p.PluginsService     ] [testIndexActionFields] loaded plugin [org.elasticsearch.xpack.ilm.IndexLifecycle]
  1> [2020-11-16T12:31:52,422][INFO ][o.e.p.PluginsService     ] [testIndexActionFields] loaded plugin [org.elasticsearch.xpack.watcher.test.TimeWarpedWatcher]
  1> [2020-11-16T12:31:52,428][INFO ][o.e.e.NodeEnvironment    ] [testIndexActionFields] using [2] data paths, mounts [[/dev/shm (tmpfs)]], net usable_space [43.4gb], net total_space [47.2gb], types [tmpfs]
  1> [2020-11-16T12:31:52,428][INFO ][o.e.e.NodeEnvironment    ] [testIndexActionFields] heap size [512mb], compressed ordinary object pointers [true]
  1> [2020-11-16T12:31:52,433][INFO ][o.e.n.Node               ] [testIndexActionFields] node name [node_s5], node ID [1YbRtksfQLe27ZFK9sDu7g], cluster name [SUITE-TEST_WORKER_VM=[327]-CLUSTER_SEED=[1106769756033557643]-HASH=[DC617A31AF]-cluster], roles [ingest, remote_cluster_client, data]
  1> [2020-11-16T12:31:52,588][INFO ][o.e.d.DiscoveryModule    ] [testIndexActionFields] using discovery type [zen] and seed hosts providers [settings, file]
  1> [2020-11-16T12:31:52,642][INFO ][o.e.n.Node               ] [testIndexActionFields] initialized
  1> [2020-11-16T12:31:52,664][INFO ][o.e.n.Node               ] [[test_SUITE-TEST_WORKER_VM=[327]-CLUSTER_SEED=[1106769756033557643]-HASH=[DC617A31AF]-cluster[T#1]]] starting ...
  1> [2020-11-16T12:31:52,703][INFO ][o.e.n.Node               ] [[test_SUITE-TEST_WORKER_VM=[327]-CLUSTER_SEED=[1106769756033557643]-HASH=[DC617A31AF]-cluster[T#3]]] starting ...
  1> [2020-11-16T12:31:52,716][INFO ][o.e.n.Node               ] [[test_SUITE-TEST_WORKER_VM=[327]-CLUSTER_SEED=[1106769756033557643]-HASH=[DC617A31AF]-cluster[T#2]]] starting ...
  1> [2020-11-16T12:31:52,718][INFO ][o.e.n.Node               ] [[test_SUITE-TEST_WORKER_VM=[327]-CLUSTER_SEED=[1106769756033557643]-HASH=[DC617A31AF]-cluster[T#5]]] starting ...
  1> [2020-11-16T12:31:52,724][INFO ][o.e.n.Node               ] [[test_SUITE-TEST_WORKER_VM=[327]-CLUSTER_SEED=[1106769756033557643]-HASH=[DC617A31AF]-cluster[T#6]]] starting ...
  1> [2020-11-16T12:31:52,736][INFO ][o.e.n.Node               ] [[test_SUITE-TEST_WORKER_VM=[327]-CLUSTER_SEED=[1106769756033557643]-HASH=[DC617A31AF]-cluster[T#4]]] starting ...
  1> [2020-11-16T12:31:52,954][INFO ][o.e.t.TransportService   ] [[test_SUITE-TEST_WORKER_VM=[327]-CLUSTER_SEED=[1106769756033557643]-HASH=[DC617A31AF]-cluster[T#5]]] publish_address {127.0.0.1:43747}, bound_addresses {[::1]:39863}, {127.0.0.1:43747}
  1> [2020-11-16T12:31:52,955][INFO ][o.e.t.TransportService   ] [[test_SUITE-TEST_WORKER_VM=[327]-CLUSTER_SEED=[1106769756033557643]-HASH=[DC617A31AF]-cluster[T#3]]] publish_address {127.0.0.1:34285}, bound_addresses {[::1]:40305}, {127.0.0.1:34285}
  1> [2020-11-16T12:31:52,967][INFO ][o.e.t.TransportService   ] [[test_SUITE-TEST_WORKER_VM=[327]-CLUSTER_SEED=[1106769756033557643]-HASH=[DC617A31AF]-cluster[T#2]]] publish_address {127.0.0.1:40269}, bound_addresses {[::1]:44539}, {127.0.0.1:40269}
  1> [2020-11-16T12:31:52,972][INFO ][o.e.t.TransportService   ] [[test_SUITE-TEST_WORKER_VM=[327]-CLUSTER_SEED=[1106769756033557643]-HASH=[DC617A31AF]-cluster[T#6]]] publish_address {127.0.0.1:37215}, bound_addresses {[::1]:43239}, {127.0.0.1:37215}
  1> [2020-11-16T12:31:52,973][INFO ][o.e.t.TransportService   ] [[test_SUITE-TEST_WORKER_VM=[327]-CLUSTER_SEED=[1106769756033557643]-HASH=[DC617A31AF]-cluster[T#4]]] publish_address {127.0.0.1:41625}, bound_addresses {[::1]:36327}, {127.0.0.1:41625}
  1> [2020-11-16T12:31:52,991][INFO ][o.e.t.TransportService   ] [[test_SUITE-TEST_WORKER_VM=[327]-CLUSTER_SEED=[1106769756033557643]-HASH=[DC617A31AF]-cluster[T#1]]] publish_address {127.0.0.1:46485}, bound_addresses {[::1]:35399}, {127.0.0.1:46485}
  1> [2020-11-16T12:31:53,668][INFO ][o.e.c.c.Coordinator      ] [node_s0] setting initial configuration to VotingConfiguration{t2r9oPnUSti7AAnSY0YkHQ,4f6qJ9JmR12iEkuIvZtdkQ,WLE0cJ77Qxa_r1uKx10uYg}
  1> [2020-11-16T12:31:53,924][INFO ][o.e.c.s.MasterService    ] [node_s0] elected-as-master ([2] nodes joined)[{node_s1}{4f6qJ9JmR12iEkuIvZtdkQ}{hNeu_e_6SViSOU9lQwi99A}{127.0.0.1}{127.0.0.1:40269}{imr}{xpack.installed=true} elect leader, {node_s0}{WLE0cJ77Qxa_r1uKx10uYg}{zYPCrP9aQB6H7qUdrCQl8w}{127.0.0.1}{127.0.0.1:46485}{imr}{xpack.installed=true} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], node-join[{node_s2}{t2r9oPnUSti7AAnSY0YkHQ}{fIEyoSZPTE6kyZ3rb-AyNg}{127.0.0.1}{127.0.0.1:34285}{imr}{xpack.installed=true} join existing leader], term: 1, version: 1, delta: master node changed {previous [], current [{node_s0}{WLE0cJ77Qxa_r1uKx10uYg}{zYPCrP9aQB6H7qUdrCQl8w}{127.0.0.1}{127.0.0.1:46485}{imr}{xpack.installed=true}]}, added {{node_s1}{4f6qJ9JmR12iEkuIvZtdkQ}{hNeu_e_6SViSOU9lQwi99A}{127.0.0.1}{127.0.0.1:40269}{imr}{xpack.installed=true},{node_s2}{t2r9oPnUSti7AAnSY0YkHQ}{fIEyoSZPTE6kyZ3rb-AyNg}{127.0.0.1}{127.0.0.1:34285}{imr}{xpack.installed=true}}
  1> [2020-11-16T12:31:54,124][INFO ][o.e.c.c.CoordinationState] [node_s0] cluster UUID set to [A4ccD0HzQFigKMo3ngjfmg]
  1> [2020-11-16T12:31:54,125][INFO ][o.e.c.c.CoordinationState] [node_s2] cluster UUID set to [A4ccD0HzQFigKMo3ngjfmg]
  1> [2020-11-16T12:31:54,208][INFO ][o.e.c.s.ClusterApplierService] [node_s2] master node changed {previous [], current [{node_s0}{WLE0cJ77Qxa_r1uKx10uYg}{zYPCrP9aQB6H7qUdrCQl8w}{127.0.0.1}{127.0.0.1:46485}{imr}{xpack.installed=true}]}, added {{node_s1}{4f6qJ9JmR12iEkuIvZtdkQ}{hNeu_e_6SViSOU9lQwi99A}{127.0.0.1}{127.0.0.1:40269}{imr}{xpack.installed=true},{node_s0}{WLE0cJ77Qxa_r1uKx10uYg}{zYPCrP9aQB6H7qUdrCQl8w}{127.0.0.1}{127.0.0.1:46485}{imr}{xpack.installed=true}}, term: 1, version: 1, reason: ApplyCommitRequest{term=1, version=1, sourceNode={node_s0}{WLE0cJ77Qxa_r1uKx10uYg}{zYPCrP9aQB6H7qUdrCQl8w}{127.0.0.1}{127.0.0.1:46485}{imr}{xpack.installed=true}}
  1> [2020-11-16T12:31:54,209][INFO ][o.e.n.Node               ] [[test_SUITE-TEST_WORKER_VM=[327]-CLUSTER_SEED=[1106769756033557643]-HASH=[DC617A31AF]-cluster[T#3]]] started
  1> [2020-11-16T12:31:54,216][INFO ][o.e.c.c.CoordinationState] [node_s1] cluster UUID set to [A4ccD0HzQFigKMo3ngjfmg]
  1> [2020-11-16T12:31:54,306][INFO ][o.e.c.s.ClusterApplierService] [node_s1] master node changed {previous [], current [{node_s0}{WLE0cJ77Qxa_r1uKx10uYg}{zYPCrP9aQB6H7qUdrCQl8w}{127.0.0.1}{127.0.0.1:46485}{imr}{xpack.installed=true}]}, added {{node_s2}{t2r9oPnUSti7AAnSY0YkHQ}{fIEyoSZPTE6kyZ3rb-AyNg}{127.0.0.1}{127.0.0.1:34285}{imr}{xpack.installed=true},{node_s0}{WLE0cJ77Qxa_r1uKx10uYg}{zYPCrP9aQB6H7qUdrCQl8w}{127.0.0.1}{127.0.0.1:46485}{imr}{xpack.installed=true}}, term: 1, version: 1, reason: ApplyCommitRequest{term=1, version=1, sourceNode={node_s0}{WLE0cJ77Qxa_r1uKx10uYg}{zYPCrP9aQB6H7qUdrCQl8w}{127.0.0.1}{127.0.0.1:46485}{imr}{xpack.installed=true}}
  1> [2020-11-16T12:31:54,310][INFO ][o.e.n.Node               ] [[test_SUITE-TEST_WORKER_VM=[327]-CLUSTER_SEED=[1106769756033557643]-HASH=[DC617A31AF]-cluster[T#2]]] started
  1> [2020-11-16T12:31:54,315][INFO ][o.e.c.s.ClusterApplierService] [node_s0] master node changed {previous [], current [{node_s0}{WLE0cJ77Qxa_r1uKx10uYg}{zYPCrP9aQB6H7qUdrCQl8w}{127.0.0.1}{127.0.0.1:46485}{imr}{xpack.installed=true}]}, added {{node_s1}{4f6qJ9JmR12iEkuIvZtdkQ}{hNeu_e_6SViSOU9lQwi99A}{127.0.0.1}{127.0.0.1:40269}{imr}{xpack.installed=true},{node_s2}{t2r9oPnUSti7AAnSY0YkHQ}{fIEyoSZPTE6kyZ3rb-AyNg}{127.0.0.1}{127.0.0.1:34285}{imr}{xpack.installed=true}}, term: 1, version: 1, reason: Publication{term=1, version=1}
  1> [2020-11-16T12:31:54,370][INFO ][o.e.n.Node               ] [[test_SUITE-TEST_WORKER_VM=[327]-CLUSTER_SEED=[1106769756033557643]-HASH=[DC617A31AF]-cluster[T#1]]] started
  1> [2020-11-16T12:31:54,395][INFO ][o.e.c.s.MasterService    ] [node_s0] node-join[{node_s4}{ltFylxUSQ9anDIsodXNq_g}{FW3UP_xOSJ-T6plJPrAZWA}{127.0.0.1}{127.0.0.1:43747}{dir}{xpack.installed=true} join existing leader, {node_s3}{Auk9Zaa6Qt-q_nll-qhOVw}{EuRngn7CTa6QGOVBD1VmRw}{127.0.0.1}{127.0.0.1:41625}{dir}{xpack.installed=true} join existing leader, {node_s5}{1YbRtksfQLe27ZFK9sDu7g}{U-PVUSqpTQO8GwTVscY_ng}{127.0.0.1}{127.0.0.1:37215}{dir}{xpack.installed=true} join existing leader], term: 1, version: 2, delta: added {{node_s3}{Auk9Zaa6Qt-q_nll-qhOVw}{EuRngn7CTa6QGOVBD1VmRw}{127.0.0.1}{127.0.0.1:41625}{dir}{xpack.installed=true},{node_s5}{1YbRtksfQLe27ZFK9sDu7g}{U-PVUSqpTQO8GwTVscY_ng}{127.0.0.1}{127.0.0.1:37215}{dir}{xpack.installed=true},{node_s4}{ltFylxUSQ9anDIsodXNq_g}{FW3UP_xOSJ-T6plJPrAZWA}{127.0.0.1}{127.0.0.1:43747}{dir}{xpack.installed=true}}
  1> [2020-11-16T12:31:54,446][INFO ][o.e.c.s.ClusterApplierService] [node_s2] added {{node_s5}{1YbRtksfQLe27ZFK9sDu7g}{U-PVUSqpTQO8GwTVscY_ng}{127.0.0.1}{127.0.0.1:37215}{dir}{xpack.installed=true},{node_s3}{Auk9Zaa6Qt-q_nll-qhOVw}{EuRngn7CTa6QGOVBD1VmRw}{127.0.0.1}{127.0.0.1:41625}{dir}{xpack.installed=true},{node_s4}{ltFylxUSQ9anDIsodXNq_g}{FW3UP_xOSJ-T6plJPrAZWA}{127.0.0.1}{127.0.0.1:43747}{dir}{xpack.installed=true}}, term: 1, version: 2, reason: ApplyCommitRequest{term=1, version=2, sourceNode={node_s0}{WLE0cJ77Qxa_r1uKx10uYg}{zYPCrP9aQB6H7qUdrCQl8w}{127.0.0.1}{127.0.0.1:46485}{imr}{xpack.installed=true}}
  1> [2020-11-16T12:31:54,450][INFO ][o.e.c.s.ClusterApplierService] [node_s1] added {{node_s5}{1YbRtksfQLe27ZFK9sDu7g}{U-PVUSqpTQO8GwTVscY_ng}{127.0.0.1}{127.0.0.1:37215}{dir}{xpack.installed=true},{node_s3}{Auk9Zaa6Qt-q_nll-qhOVw}{EuRngn7CTa6QGOVBD1VmRw}{127.0.0.1}{127.0.0.1:41625}{dir}{xpack.installed=true},{node_s4}{ltFylxUSQ9anDIsodXNq_g}{FW3UP_xOSJ-T6plJPrAZWA}{127.0.0.1}{127.0.0.1:43747}{dir}{xpack.installed=true}}, term: 1, version: 2, reason: ApplyCommitRequest{term=1, version=2, sourceNode={node_s0}{WLE0cJ77Qxa_r1uKx10uYg}{zYPCrP9aQB6H7qUdrCQl8w}{127.0.0.1}{127.0.0.1:46485}{imr}{xpack.installed=true}}
  1> [2020-11-16T12:31:54,450][INFO ][o.e.c.s.ClusterApplierService] [node_s5] master node changed {previous [], current [{node_s0}{WLE0cJ77Qxa_r1uKx10uYg}{zYPCrP9aQB6H7qUdrCQl8w}{127.0.0.1}{127.0.0.1:46485}{imr}{xpack.installed=true}]}, added {{node_s0}{WLE0cJ77Qxa_r1uKx10uYg}{zYPCrP9aQB6H7qUdrCQl8w}{127.0.0.1}{127.0.0.1:46485}{imr}{xpack.installed=true},{node_s1}{4f6qJ9JmR12iEkuIvZtdkQ}{hNeu_e_6SViSOU9lQwi99A}{127.0.0.1}{127.0.0.1:40269}{imr}{xpack.installed=true},{node_s3}{Auk9Zaa6Qt-q_nll-qhOVw}{EuRngn7CTa6QGOVBD1VmRw}{127.0.0.1}{127.0.0.1:41625}{dir}{xpack.installed=true},{node_s2}{t2r9oPnUSti7AAnSY0YkHQ}{fIEyoSZPTE6kyZ3rb-AyNg}{127.0.0.1}{127.0.0.1:34285}{imr}{xpack.installed=true},{node_s4}{ltFylxUSQ9anDIsodXNq_g}{FW3UP_xOSJ-T6plJPrAZWA}{127.0.0.1}{127.0.0.1:43747}{dir}{xpack.installed=true}}, term: 1, version: 2, reason: ApplyCommitRequest{term=1, version=2, sourceNode={node_s0}{WLE0cJ77Qxa_r1uKx10uYg}{zYPCrP9aQB6H7qUdrCQl8w}{127.0.0.1}{127.0.0.1:46485}{imr}{xpack.installed=true}}
  1> [2020-11-16T12:31:54,450][INFO ][o.e.c.s.ClusterApplierService] [node_s4] master node changed {previous [], current [{node_s0}{WLE0cJ77Qxa_r1uKx10uYg}{zYPCrP9aQB6H7qUdrCQl8w}{127.0.0.1}{127.0.0.1:46485}{imr}{xpack.installed=true}]}, added {{node_s0}{WLE0cJ77Qxa_r1uKx10uYg}{zYPCrP9aQB6H7qUdrCQl8w}{127.0.0.1}{127.0.0.1:46485}{imr}{xpack.installed=true},{node_s5}{1YbRtksfQLe27ZFK9sDu7g}{U-PVUSqpTQO8GwTVscY_ng}{127.0.0.1}{127.0.0.1:37215}{dir}{xpack.installed=true},{node_s1}{4f6qJ9JmR12iEkuIvZtdkQ}{hNeu_e_6SViSOU9lQwi99A}{127.0.0.1}{127.0.0.1:40269}{imr}{xpack.installed=true},{node_s3}{Auk9Zaa6Qt-q_nll-qhOVw}{EuRngn7CTa6QGOVBD1VmRw}{127.0.0.1}{127.0.0.1:41625}{dir}{xpack.installed=true},{node_s2}{t2r9oPnUSti7AAnSY0YkHQ}{fIEyoSZPTE6kyZ3rb-AyNg}{127.0.0.1}{127.0.0.1:34285}{imr}{xpack.installed=true}}, term: 1, version: 2, reason: ApplyCommitRequest{term=1, version=2, sourceNode={node_s0}{WLE0cJ77Qxa_r1uKx10uYg}{zYPCrP9aQB6H7qUdrCQl8w}{127.0.0.1}{127.0.0.1:46485}{imr}{xpack.installed=true}}
  1> [2020-11-16T12:31:54,459][INFO ][o.e.c.s.ClusterApplierService] [node_s3] master node changed {previous [], current [{node_s0}{WLE0cJ77Qxa_r1uKx10uYg}{zYPCrP9aQB6H7qUdrCQl8w}{127.0.0.1}{127.0.0.1:46485}{imr}{xpack.installed=true}]}, added {{node_s0}{WLE0cJ77Qxa_r1uKx10uYg}{zYPCrP9aQB6H7qUdrCQl8w}{127.0.0.1}{127.0.0.1:46485}{imr}{xpack.installed=true},{node_s5}{1YbRtksfQLe27ZFK9sDu7g}{U-PVUSqpTQO8GwTVscY_ng}{127.0.0.1}{127.0.0.1:37215}{dir}{xpack.installed=true},{node_s1}{4f6qJ9JmR12iEkuIvZtdkQ}{hNeu_e_6SViSOU9lQwi99A}{127.0.0.1}{127.0.0.1:40269}{imr}{xpack.installed=true},{node_s2}{t2r9oPnUSti7AAnSY0YkHQ}{fIEyoSZPTE6kyZ3rb-AyNg}{127.0.0.1}{127.0.0.1:34285}{imr}{xpack.installed=true},{node_s4}{ltFylxUSQ9anDIsodXNq_g}{FW3UP_xOSJ-T6plJPrAZWA}{127.0.0.1}{127.0.0.1:43747}{dir}{xpack.installed=true}}, term: 1, version: 2, reason: ApplyCommitRequest{term=1, version=2, sourceNode={node_s0}{WLE0cJ77Qxa_r1uKx10uYg}{zYPCrP9aQB6H7qUdrCQl8w}{127.0.0.1}{127.0.0.1:46485}{imr}{xpack.installed=true}}
  1> [2020-11-16T12:31:54,497][INFO ][o.e.n.Node               ] [[test_SUITE-TEST_WORKER_VM=[327]-CLUSTER_SEED=[1106769756033557643]-HASH=[DC617A31AF]-cluster[T#5]]] started
  1> [2020-11-16T12:31:54,537][INFO ][o.e.n.Node               ] [[test_SUITE-TEST_WORKER_VM=[327]-CLUSTER_SEED=[1106769756033557643]-HASH=[DC617A31AF]-cluster[T#6]]] started
  1> [2020-11-16T12:31:54,557][INFO ][o.e.n.Node               ] [[test_SUITE-TEST_WORKER_VM=[327]-CLUSTER_SEED=[1106769756033557643]-HASH=[DC617A31AF]-cluster[T#4]]] started
  1> [2020-11-16T12:31:54,561][INFO ][o.e.c.s.ClusterApplierService] [node_s0] added {{node_s3}{Auk9Zaa6Qt-q_nll-qhOVw}{EuRngn7CTa6QGOVBD1VmRw}{127.0.0.1}{127.0.0.1:41625}{dir}{xpack.installed=true},{node_s5}{1YbRtksfQLe27ZFK9sDu7g}{U-PVUSqpTQO8GwTVscY_ng}{127.0.0.1}{127.0.0.1:37215}{dir}{xpack.installed=true},{node_s4}{ltFylxUSQ9anDIsodXNq_g}{FW3UP_xOSJ-T6plJPrAZWA}{127.0.0.1}{127.0.0.1:43747}{dir}{xpack.installed=true}}, term: 1, version: 2, reason: Publication{term=1, version=2}
  1> [2020-11-16T12:31:54,698][INFO ][o.e.g.GatewayService     ] [node_s0] recovered [0] indices into cluster_state
  1> [2020-11-16T12:31:54,730][INFO ][o.e.c.m.MetadataIndexTemplateService] [node_s0] adding index template [.triggered_watches] for index patterns [.triggered_watches*]
  1> [2020-11-16T12:31:54,843][INFO ][o.e.c.m.MetadataIndexTemplateService] [node_s0] adding index template [.watches] for index patterns [.watches*]
  1> [2020-11-16T12:31:54,985][INFO ][o.e.c.m.MetadataIndexTemplateService] [node_s0] adding index template [.watch-history-14] for index patterns [.watcher-history-14*]
  1> [2020-11-16T12:31:55,142][INFO ][o.e.c.m.MetadataIndexTemplateService] [node_s0] adding index template [.slm-history] for index patterns [.slm-history-5*]
  1> [2020-11-16T12:31:55,231][INFO ][o.e.x.i.a.TransportPutLifecycleAction] [node_s0] adding index lifecycle policy [watch-history-ilm-policy]
  1> [2020-11-16T12:31:55,315][INFO ][o.e.x.i.a.TransportPutLifecycleAction] [node_s0] adding index lifecycle policy [slm-history-ilm-policy]
  1> [2020-11-16T12:31:55,873][INFO ][o.e.l.LicenseService     ] [node_s1] license [1cee699a-0b11-4cd1-8f3d-957287d66927] mode [trial] - valid
  1> [2020-11-16T12:31:55,888][INFO ][o.e.l.LicenseService     ] [node_s5] license [1cee699a-0b11-4cd1-8f3d-957287d66927] mode [trial] - valid
  1> [2020-11-16T12:31:55,911][INFO ][o.e.l.LicenseService     ] [node_s2] license [1cee699a-0b11-4cd1-8f3d-957287d66927] mode [trial] - valid
  1> [2020-11-16T12:31:55,929][INFO ][o.e.l.LicenseService     ] [node_s3] license [1cee699a-0b11-4cd1-8f3d-957287d66927] mode [trial] - valid
  1> [2020-11-16T12:31:55,954][INFO ][o.e.l.LicenseService     ] [node_s4] license [1cee699a-0b11-4cd1-8f3d-957287d66927] mode [trial] - valid
  1> [2020-11-16T12:31:56,091][INFO ][o.e.l.LicenseService     ] [node_s0] license [1cee699a-0b11-4cd1-8f3d-957287d66927] mode [trial] - valid
  1> [2020-11-16T12:31:56,102][WARN ][o.e.c.m.MetadataIndexTemplateService] [node_s0] legacy template [random_index_template] has index patterns [*] matching patterns from existing composable templates [.triggered_watches,.watch-history-14,.slm-history,.watches] with patterns (.triggered_watches => [.triggered_watches*],.watch-history-14 => [.watcher-history-14*],.slm-history => [.slm-history-5*],.watches => [.watches*]); this template [random_index_template] may be ignored in favor of a composable template at index creation time
  1> [2020-11-16T12:31:56,106][INFO ][o.e.c.m.MetadataIndexTemplateService] [node_s0] adding template [random_index_template] for index patterns [*]
  1> [2020-11-16T12:31:56,188][INFO ][o.e.x.w.h.HistoryTemplateIndexActionMappingsTests] [testIndexActionFields] [HistoryTemplateIndexActionMappingsTests#testIndexActionFields]: all set up test
  1> [2020-11-16T12:31:56,193][INFO ][o.e.x.w.h.HistoryTemplateIndexActionMappingsTests] [testIndexActionFields] [HistoryTemplateIndexActionMappingsTests#testIndexActionFields]: freezing time on nodes
  1> [2020-11-16T12:31:56,213][INFO ][o.e.x.w.h.HistoryTemplateIndexActionMappingsTests] [testIndexActionFields] waiting to stop watcher, current states [Tuple [v1=node_s3 (0), v2=STARTED], Tuple [v1=node_s5 (0), v2=STARTED], Tuple [v1=node_s1 (0), v2=STARTED], Tuple [v1=node_s2 (0), v2=STARTED], Tuple [v1=node_s4 (0), v2=STARTED], Tuple [v1=node_s0 (0), v2=STARTED]]
  1> [2020-11-16T12:31:56,271][INFO ][o.e.x.w.WatcherService   ] [node_s5] stopping watch service, reason [watcher manually marked to shutdown by cluster state update]
  1> [2020-11-16T12:31:56,272][INFO ][o.e.x.w.WatcherService   ] [node_s4] stopping watch service, reason [watcher manually marked to shutdown by cluster state update]
  1> [2020-11-16T12:31:56,272][INFO ][o.e.x.w.WatcherLifeCycleService] [node_s5] watcher has stopped
  1> [2020-11-16T12:31:56,271][INFO ][o.e.x.w.WatcherService   ] [node_s3] stopping watch service, reason [watcher manually marked to shutdown by cluster state update]
  1> [2020-11-16T12:31:56,272][INFO ][o.e.x.w.WatcherLifeCycleService] [node_s4] watcher has stopped
  1> [2020-11-16T12:31:56,273][INFO ][o.e.x.w.WatcherLifeCycleService] [node_s3] watcher has stopped
  1> [2020-11-16T12:31:56,271][INFO ][o.e.x.w.WatcherService   ] [node_s1] stopping watch service, reason [watcher manually marked to shutdown by cluster state update]
  1> [2020-11-16T12:31:56,273][INFO ][o.e.x.w.WatcherLifeCycleService] [node_s1] watcher has stopped
  1> [2020-11-16T12:31:56,276][INFO ][o.e.x.w.WatcherService   ] [node_s2] stopping watch service, reason [watcher manually marked to shutdown by cluster state update]
  1> [2020-11-16T12:31:56,276][INFO ][o.e.x.w.WatcherLifeCycleService] [node_s2] watcher has stopped
  1> [2020-11-16T12:31:56,283][INFO ][o.e.x.w.WatcherService   ] [node_s0] stopping watch service, reason [watcher manually marked to shutdown by cluster state update]
  1> [2020-11-16T12:31:56,284][INFO ][o.e.x.w.WatcherLifeCycleService] [node_s0] watcher has stopped
  1> [2020-11-16T12:31:56,290][INFO ][o.e.x.w.h.HistoryTemplateIndexActionMappingsTests] [testIndexActionFields] waiting to stop watcher, current states [Tuple [v1=node_s3 (0), v2=STOPPED], Tuple [v1=node_s5 (0), v2=STOPPED], Tuple [v1=node_s1 (0), v2=STOPPED], Tuple [v1=node_s2 (0), v2=STOPPED], Tuple [v1=node_s4 (0), v2=STOPPED], Tuple [v1=node_s0 (0), v2=STOPPED]]
  1> [2020-11-16T12:31:56,296][DEPRECATION][o.e.d.c.m.MetadataCreateIndexService] [node_s0] data_stream.dataset="deprecation.elasticsearch" data_stream.namespace="default" data_stream.type="logs" ecs.version="1.6" key="index_name_starts_with_dot" message="index name [.watches] starts with a dot '.', in the next major version, index names starting with a dot are reserved for hidden indices and system indices"
  1> [2020-11-16T12:31:56,304][INFO ][o.e.c.m.MetadataCreateIndexService] [node_s0] [.watches] creating index, cause [api], templates [.watches], shards [1]/[0]
  1> [2020-11-16T12:31:56,306][INFO ][o.e.c.r.a.AllocationService] [node_s0] updating number_of_replicas to [1] for indices [.watches]
  1> [2020-11-16T12:31:56,661][DEPRECATION][o.e.d.c.m.MetadataCreateIndexService] [node_s0] data_stream.dataset="deprecation.elasticsearch" data_stream.namespace="default" data_stream.type="logs" ecs.version="1.6" key="index_name_starts_with_dot" message="index name [.triggered_watches] starts with a dot '.', in the next major version, index names starting with a dot are reserved for hidden indices and system indices"
  1> [2020-11-16T12:31:56,668][INFO ][o.e.c.m.MetadataCreateIndexService] [node_s0] [.triggered_watches] creating index, cause [api], templates [.triggered_watches], shards [1]/[1]
  1> [2020-11-16T12:31:57,231][INFO ][o.e.x.w.h.HistoryTemplateIndexActionMappingsTests] [testIndexActionFields] waiting to start watcher, current states [Tuple [v1=node_s0, v2=STOPPED], Tuple [v1=node_s5, v2=STOPPED], Tuple [v1=node_s1, v2=STOPPED], Tuple [v1=node_s3, v2=STOPPED], Tuple [v1=node_s2, v2=STOPPED], Tuple [v1=node_s4, v2=STOPPED]]
  1> [2020-11-16T12:31:57,597][INFO ][o.e.x.w.h.HistoryTemplateIndexActionMappingsTests] [testIndexActionFields] waiting to start watcher, current states [Tuple [v1=node_s3, v2=STARTING], Tuple [v1=node_s5, v2=STARTED], Tuple [v1=node_s1, v2=STARTING], Tuple [v1=node_s2, v2=STARTING], Tuple [v1=node_s4, v2=STARTED], Tuple [v1=node_s0, v2=STARTING]]
  1> [2020-11-16T12:31:57,602][INFO ][o.e.x.w.h.HistoryTemplateIndexActionMappingsTests] [testIndexActionFields] waiting to start watcher, current states [Tuple [v1=node_s0, v2=STARTING], Tuple [v1=node_s5, v2=STARTED], Tuple [v1=node_s1, v2=STARTING], Tuple [v1=node_s3, v2=STARTING], Tuple [v1=node_s2, v2=STARTING], Tuple [v1=node_s4, v2=STARTED]]
  1> [2020-11-16T12:31:57,644][INFO ][o.e.x.w.h.HistoryTemplateIndexActionMappingsTests] [testIndexActionFields] waiting to start watcher, current states [Tuple [v1=node_s0, v2=STARTED], Tuple [v1=node_s5, v2=STARTED], Tuple [v1=node_s1, v2=STARTED], Tuple [v1=node_s3, v2=STARTING], Tuple [v1=node_s2, v2=STARTED], Tuple [v1=node_s4, v2=STARTED]]
  1> [2020-11-16T12:31:57,659][INFO ][o.e.x.w.h.HistoryTemplateIndexActionMappingsTests] [testIndexActionFields] waiting to start watcher, current states [Tuple [v1=node_s0, v2=STARTED], Tuple [v1=node_s5, v2=STARTED], Tuple [v1=node_s1, v2=STARTED], Tuple [v1=node_s3, v2=STARTING], Tuple [v1=node_s2, v2=STARTED], Tuple [v1=node_s4, v2=STARTED]]
  1> [2020-11-16T12:31:57,677][INFO ][o.e.x.w.h.HistoryTemplateIndexActionMappingsTests] [testIndexActionFields] waiting to start watcher, current states [Tuple [v1=node_s0, v2=STARTED], Tuple [v1=node_s5, v2=STARTED], Tuple [v1=node_s1, v2=STARTED], Tuple [v1=node_s3, v2=STARTED], Tuple [v1=node_s2, v2=STARTED], Tuple [v1=node_s4, v2=STARTED]]
  1> [2020-11-16T12:31:58,019][INFO ][o.e.c.r.a.AllocationService] [node_s0] current.health="GREEN" message="Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[.watches][0]]])." previous.health="YELLOW" reason="shards started [[.watches][0]]"
  1> [2020-11-16T12:31:58,102][INFO ][o.e.x.w.WatcherService   ] [node_s3] reloading watcher, reason [new local watcher shard allocation ids], cancelled [0] queued tasks
  1> [2020-11-16T12:31:58,102][INFO ][o.e.x.w.WatcherService   ] [node_s5] reloading watcher, reason [new local watcher shard allocation ids], cancelled [0] queued tasks
  1> [2020-11-16T12:31:58,161][INFO ][o.e.c.m.MetadataCreateIndexService] [node_s0] [the-index] creating index, cause [auto(bulk api)], templates [random_index_template], shards [3]/[0]
  1> [2020-11-16T12:31:58,627][INFO ][o.e.c.r.a.AllocationService] [node_s0] current.health="GREEN" message="Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[the-index][0], [the-index][1]]])." previous.health="YELLOW" reason="shards started [[the-index][0], [the-index][1]]"
  1> [2020-11-16T12:31:58,727][INFO ][o.e.c.m.MetadataMappingService] [node_s0] [the-index/lrACTu5_TrSGmK_7Qcy5rw] create_mapping
  1> [2020-11-16T12:31:58,935][INFO ][o.e.c.m.MetadataCreateIndexService] [node_s0] [.ds-.watcher-history-14-000001] creating index, cause [initialize_data_stream], templates [.watch-history-14], shards [1]/[0]
  1> [2020-11-16T12:31:58,937][INFO ][o.e.c.r.a.AllocationService] [node_s0] updating number_of_replicas to [1] for indices [.ds-.watcher-history-14-000001]
  1> [2020-11-16T12:31:58,961][INFO ][o.e.c.m.MetadataCreateDataStreamService] [node_s0] adding data stream [.watcher-history-14] with write index [.ds-.watcher-history-14-000001] and backing indices []
  1> [2020-11-16T12:31:59,207][INFO ][o.e.x.i.IndexLifecycleTransition] [node_s0] moving index [.ds-.watcher-history-14-000001] from [null] to [{"phase":"new","action":"complete","name":"complete"}] in policy [watch-history-ilm-policy]
  1> [2020-11-16T12:31:59,562][INFO ][o.e.c.m.MetadataMappingService] [node_s0] [.ds-.watcher-history-14-000001/ECV5RDprTJ-0X9lA42-Q1g] update_mapping [_doc]
  1> [2020-11-16T12:31:59,863][WARN ][o.e.x.w.t.A.TimeWarp     ] [testIndexActionFields] watch was triggered on [2] schedulers, expected [1]
  1> [2020-11-16T12:32:00,665][INFO ][o.e.c.r.a.AllocationService] [node_s0] current.health="GREEN" message="Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[.ds-.watcher-history-14-000001][0]]])." previous.health="YELLOW" reason="shards started [[.ds-.watcher-history-14-000001][0]]"
  1> [2020-11-16T12:32:00,880][INFO ][o.e.x.w.h.HistoryTemplateIndexActionMappingsTests] [testIndexActionFields] [#testIndexActionFields]: clearing watcher state
  1> [2020-11-16T12:32:00,887][INFO ][o.e.x.w.h.HistoryTemplateIndexActionMappingsTests] [testIndexActionFields] waiting to stop watcher, current states [Tuple [v1=node_s0 (0), v2=STARTED], Tuple [v1=node_s5 (0), v2=STARTED], Tuple [v1=node_s1 (0), v2=STARTED], Tuple [v1=node_s3 (0), v2=STARTED], Tuple [v1=node_s2 (0), v2=STARTED], Tuple [v1=node_s4 (0), v2=STARTED]]
  1> [2020-11-16T12:32:00,963][INFO ][o.e.x.w.WatcherService   ] [node_s1] stopping watch service, reason [watcher manually marked to shutdown by cluster state update]
  1> [2020-11-16T12:32:00,964][INFO ][o.e.x.w.WatcherService   ] [node_s3] stopping watch service, reason [watcher manually marked to shutdown by cluster state update]
  1> [2020-11-16T12:32:00,965][INFO ][o.e.x.w.WatcherService   ] [node_s2] stopping watch service, reason [watcher manually marked to shutdown by cluster state update]
  1> [2020-11-16T12:32:00,964][INFO ][o.e.x.w.WatcherLifeCycleService] [node_s1] watcher has stopped
  1> [2020-11-16T12:32:00,965][INFO ][o.e.x.w.WatcherLifeCycleService] [node_s3] watcher has stopped
  1> [2020-11-16T12:32:00,965][INFO ][o.e.x.w.WatcherService   ] [node_s4] stopping watch service, reason [watcher manually marked to shutdown by cluster state update]
  1> [2020-11-16T12:32:00,965][INFO ][o.e.x.w.WatcherLifeCycleService] [node_s2] watcher has stopped
  1> [2020-11-16T12:32:00,964][INFO ][o.e.x.w.WatcherService   ] [node_s5] stopping watch service, reason [watcher manually marked to shutdown by cluster state update]
  1> [2020-11-16T12:32:00,965][INFO ][o.e.x.w.WatcherLifeCycleService] [node_s5] watcher has stopped
  1> [2020-11-16T12:32:00,965][INFO ][o.e.x.w.WatcherLifeCycleService] [node_s4] watcher has stopped
  1> [2020-11-16T12:32:00,972][INFO ][o.e.x.w.WatcherService   ] [node_s0] stopping watch service, reason [watcher manually marked to shutdown by cluster state update]
  1> [2020-11-16T12:32:00,973][INFO ][o.e.x.w.WatcherLifeCycleService] [node_s0] watcher has stopped
  1> [2020-11-16T12:32:00,980][INFO ][o.e.x.w.h.HistoryTemplateIndexActionMappingsTests] [testIndexActionFields] waiting to stop watcher, current states [Tuple [v1=node_s0 (0), v2=STOPPED], Tuple [v1=node_s5 (0), v2=STOPPED], Tuple [v1=node_s1 (0), v2=STOPPED], Tuple [v1=node_s3 (0), v2=STOPPED], Tuple [v1=node_s2 (0), v2=STOPPED], Tuple [v1=node_s4 (0), v2=STOPPED]]
  1> [2020-11-16T12:32:00,982][INFO ][o.e.x.d.a.DeleteDataStreamTransportAction] [node_s0] removing data stream [.watcher-history-14]
  1> [2020-11-16T12:32:00,982][INFO ][o.e.c.m.MetadataDeleteIndexService] [node_s0] [.ds-.watcher-history-14-000001/ECV5RDprTJ-0X9lA42-Q1g] deleting index
  1> [2020-11-16T12:32:01,109][INFO ][o.e.x.w.h.HistoryTemplateIndexActionMappingsTests] [testIndexActionFields] [HistoryTemplateIndexActionMappingsTests#testIndexActionFields]: cleaning up after test
  1> [2020-11-16T12:32:01,109][INFO ][o.e.t.InternalTestCluster] [testIndexActionFields] Clearing active scheme time frozen, expected healing time 0s
  1> [2020-11-16T12:32:01,244][INFO ][o.e.c.m.MetadataDeleteIndexService] [node_s0] [.watches/ftT6F3RSTlCOjjXDfuX8SA] deleting index
  1> [2020-11-16T12:32:01,244][INFO ][o.e.c.m.MetadataDeleteIndexService] [node_s0] [the-index/lrACTu5_TrSGmK_7Qcy5rw] deleting index
  1> [2020-11-16T12:32:01,244][INFO ][o.e.c.m.MetadataDeleteIndexService] [node_s0] [.triggered_watches/II66S6JgTeCck9L_u1U20A] deleting index
  1> [2020-11-16T12:32:01,436][INFO ][o.e.c.m.MetadataIndexTemplateService] [node_s0] removing template [random_index_template]
  1> [2020-11-16T12:32:01,512][INFO ][o.e.x.w.h.HistoryTemplateIndexActionMappingsTests] [testIndexActionFields] [HistoryTemplateIndexActionMappingsTests#testIndexActionFields]: cleaned up after test
  1> [2020-11-16T12:32:01,513][INFO ][o.e.x.w.h.HistoryTemplateIndexActionMappingsTests] [testIndexActionFields] after test
  2> REPRODUCE WITH: ./gradlew ':x-pack:plugin:watcher:internalClusterTest' --tests "org.elasticsearch.xpack.watcher.history.HistoryTemplateIndexActionMappingsTests.testIndexActionFields" -Dtests.seed=30DAF85125EF7D32 -Dtests.security.manager=true -Dtests.locale=mt -Dtests.timezone=America/Montevideo -Druntime.java=11
@valeriy42 valeriy42 added >test-failure Triaged test failures from CI :Data Management/Watcher labels Nov 16, 2020
@elasticmachine elasticmachine added the Team:Data Management Meta label for data/management team label Nov 16, 2020
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-core-features (Team:Core/Features)

@valeriy42 valeriy42 mentioned this issue Nov 16, 2020
Merged
valeriy42 added a commit that referenced this issue Nov 17, 2020
@valeriy42
[TEST] Mute HistoryTemplateIndexActionMappingsTests.testIndexActionFi…
1c32318 
…elds (#65092)

Relates #65091
@probakowski probakowski mentioned this issue Nov 18, 2020
Merged
probakowski added a commit that referenced this issue Nov 18, 2020
@probakowski
Fix trigger assertion error in watcher tests (#65186)
b3a4fc4 
When we trigger watcher in tests we may fail if scheduler is paused.
That can happen when watcher is reloading due to changes in its shards. This is transient state and retry should fix it.
This change adds assertBusy around triggering code so we will try again.

Closes #65183 #65176 #65117 #65091 #65089 #65086 #65064 #65063
@probakowski
Copy link
Contributor

Should be fixed by #65186

@nik9000
Copy link
Member

nik9000 commented Nov 24, 2020

Looks to have come back with https://gradle-enterprise.elastic.co/s/odoghd4keceqo/tests/:x-pack:plugin:watcher:internalClusterTest/org.elasticsearch.xpack.watcher.history.HistoryTemplateIndexActionMappingsTests/testIndexActionFields#1

@nik9000 nik9000 reopened this Nov 24, 2020
@csoulios
Copy link
Contributor

csoulios commented Dec 3, 2020

I got a failure too: https://gradle-enterprise.elastic.co/s/dfdhzbzw2tsiw/tests/:x-pack:plugin:watcher:internalClusterTest/org.elasticsearch.xpack.watcher.history.HistoryTemplateIndexActionMappingsTests/testIndexActionFields#1

https://gradle-enterprise.elastic.co/s/dfdhzbzw2tsiw

@pugnascotia
Copy link
Contributor

Another failure:

https://gradle-enterprise.elastic.co/s/suq6jzmciaasc/tests/:x-pack:plugin:watcher:internalClusterTest/org.elasticsearch.xpack.watcher.history.HistoryTemplateIndexActionMappingsTests/testIndexActionFields#1

@jaymode
Copy link
Member

jaymode commented Dec 3, 2020

Another one as well: https://gradle-enterprise.elastic.co/s/dfdhzbzw2tsiw/tests/:x-pack:plugin:watcher:internalClusterTest/org.elasticsearch.xpack.watcher.history.HistoryTemplateIndexActionMappingsTests/testIndexActionFields#1

jaymode added a commit to jaymode/elasticsearch that referenced this issue Dec 3, 2020
@jaymode
Mute HistoryTemplateIndexActionMappingsTests
e5a632e 
HistoryTemplateIndexActionMappingsTests testIndexActionFields has been
failing in CI. This commit mutes the test.

Relates elastic#65091
@jaymode jaymode mentioned this issue Dec 3, 2020
Merged
jaymode added a commit that referenced this issue Dec 3, 2020
@jaymode
Mute HistoryTemplateIndexActionMappingsTests (#65860)
5034b11 
HistoryTemplateIndexActionMappingsTests testIndexActionFields has been
failing in CI. This commit mutes the test.

Relates #65091
@probakowski probakowski mentioned this issue Jan 12, 2021
Merged
probakowski added a commit that referenced this issue Jan 13, 2021
@probakowski
Relax asserts in watcher tests (#67404)
f4dd731 
When watcher gets reloaded (due to shards routings change) during tests we sometimes get additional watcher history entry (either that the same watch is already queued or that it got throttled because it tried to execute right after previous one).
That trips our assertions which expect exact number of entries.
This change relaxes them to allow exactly-one-more situation and leaves all other checks in place.

Closes #65831
Closes #65176
Closes #65064
Closes #65091
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
:Data Management/Watcher Team:Data Management Meta label for data/management team >test-failure Triaged test failures from CI
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Relax asserts in watcher tests probakowski/elasticsearch
7 participants
@nik9000 @valeriy42 @csoulios @probakowski @jaymode @pugnascotia @elasticmachine