[CI] TransformInternalIndexIT.testUpdateDeletesOldTransformConfig

[andreidan]

Build scan: https://gradle-enterprise.elastic.co/s/qvjg4l7qfqdxq

Repro line:

./gradlew ':x-pack:plugin:transform:internalClusterTest' --tests "org.elasticsearch.xpack.transform.integration.TransformInternalIndexIT.testUpdateDeletesOldTransformConfig" \
  -Dtests.seed=180F5806D3F6A363 \
  -Dtests.security.manager=true \
  -Dtests.locale=hi \
  -Dtests.timezone=Europe/Volgograd \
  -Druntime.java=11 \
  -Dtests.fips.enabled=true

Reproduces locally?: no

Applicable branches: 7.12

Failure history:

Failure excerpt:

java.lang.AssertionError: test leaves indices that were not deleted: .transform-notifications-000002
Expected: []
     but: was [".transform-notifications-000002"]
	at __randomizedtesting.SeedInfo.seed([180F5806D3F6A363:75F6F23A53C81E91]:0)
	at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
	at org.junit.Assert.assertThat(Assert.java:956)
	at org.elasticsearch.test.ESSingleNodeTestCase.tearDown(ESSingleNodeTestCase.java:134)

[elasticmachine]

Pinging @elastic/ml-core (Team:ML)

[hendrikmuhs]

This seems to be a weakness in ESSingleNodeTestCase

When teardown is run, an index is initializing, so the delete call does not include it:

This message shows the shard initialization:

16:28:09   1> [2021-02-16T19:28:08,465][INFO ][o.e.c.r.a.AllocationService] [node_s_0] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[.transform-notifications-000002][0]]]).

but this message is after the test is already deleting indices:

16:28:09   1> [2021-02-16T19:28:08,238][INFO ][o.e.c.m.MetadataDeleteIndexService] [node_s_0] [.transform-internal-001/yG-HhVHuSp-OdgPQskwkIg] deleting index
16:28:09   1> [2021-02-16T19:28:08,239][INFO ][o.e.c.m.MetadataDeleteIndexService] [node_s_0] [transform-index-deletes-old/tIb4lwATTmijfRYNxL73kg] deleting index
16:28:09   1> [2021-02-16T19:28:08,239][INFO ][o.e.c.m.MetadataDeleteIndexService] [node_s_0] [.transform-internal-006/BNu6CXqJSbeFm3nK3uch1w] deleting index

ESRestTestCase has a method called ensureNoInitializingShards for that reason, it is called before running the teardown. I think we should add the same to ESSingleNodeTestCase.

[dimitris-athanasiou]

Another failure: https://gradle-enterprise.elastic.co/s/pvyzesawuecto

[matriv]

Another failure (7.12 branch): https://gradle-enterprise.elastic.co/s/sthnt7j7vleza

[matriv]

7.x failure: https://gradle-enterprise.elastic.co/s/y7g4tjnrvekuw

[henningandersen]

Looks like this still fails with same error in:

https://gradle-enterprise.elastic.co/s/pysgbxwlswsqq

java.lang.AssertionError: test leaves indices that were not deleted: .transform-notifications-000002

based on commit: 1430e52, which includes #69419.

[droberts195]

I have an idea for what this is. In the REST tests we wait for pending tasks before deleting indices:

elasticsearch/x-pack/plugin/src/test/java/org/elasticsearch/xpack/test/rest/AbstractXPackRestTest.java

Line 207 in 26eef89

if (isWaitForPendingTasks()) {

One of the reasons we had to add that was because of the way an async side effect of deleting an ML job is to write a notification to the notifications index, and that could end up creating the index after it had been deleted by the end-of-test cleanup.

I expect the same is true of updating a transform. If so what's happening is that .transform-notifications-000002 is getting deleted by the end-of-test cleanup, but then sometimes gets recreated in between that and the check of whether the test left indices behind.

So when I moved testUpdateDeletesOldTransformConfig from being a REST test to being an internal cluster test (due to the system indices changes) I needed to copy the "wait for pending tasks" bit to the internal cluster test cleanup, but didn't. However, since this single test seems to be the only one that actually needs this functionality, it's probably better to add it as a bespoke @After method for this particular test file, so that it doesn't slow down other tests. I will try that.

[droberts195]

I expect the same is true of updating a transform.

Yes, it's in:

elasticsearch/x-pack/plugin/transform/src/main/java/org/elasticsearch/xpack/transform/action/TransportUpdateTransformAction.java

Line 331 in 9d68193

auditor.info(config.getId(), "updated transform.");