Failing test: X-Pack Reporting API Integration Tests.x-pack/test/reporting_api_integration/reporting_and_security/ilm_migration_apis·ts - Reporting APIs ILM policy migration APIs "after all" hook for "is not available to unauthorized users"

[kibanamachine]

A test failed on a tracked branch

Error: expected 200 "OK", got 404 "Not Found"
    at Test._assertStatus (/opt/local-ssd/buildkite/builds/kb-n2-4-d955647d99aaa0ef/elastic/kibana-on-merge/kibana/node_modules/supertest/lib/test.js:268:12)
    at Test._assertFunction (/opt/local-ssd/buildkite/builds/kb-n2-4-d955647d99aaa0ef/elastic/kibana-on-merge/kibana/node_modules/supertest/lib/test.js:283:11)
    at Test.assert (/opt/local-ssd/buildkite/builds/kb-n2-4-d955647d99aaa0ef/elastic/kibana-on-merge/kibana/node_modules/supertest/lib/test.js:173:18)
    at localAssert (/opt/local-ssd/buildkite/builds/kb-n2-4-d955647d99aaa0ef/elastic/kibana-on-merge/kibana/node_modules/supertest/lib/test.js:131:12)
    at /opt/local-ssd/buildkite/builds/kb-n2-4-d955647d99aaa0ef/elastic/kibana-on-merge/kibana/node_modules/supertest/lib/test.js:128:5
    at Test.Request.callback (/opt/local-ssd/buildkite/builds/kb-n2-4-d955647d99aaa0ef/elastic/kibana-on-merge/kibana/node_modules/superagent/lib/node/index.js:728:3)
    at /opt/local-ssd/buildkite/builds/kb-n2-4-d955647d99aaa0ef/elastic/kibana-on-merge/kibana/node_modules/superagent/lib/node/index.js:916:18
    at Stream.<anonymous> (/opt/local-ssd/buildkite/builds/kb-n2-4-d955647d99aaa0ef/elastic/kibana-on-merge/kibana/node_modules/superagent/lib/node/parsers/json.js:19:7)
    at Stream.emit (node:events:513:28)
    at Unzip.<anonymous> (/opt/local-ssd/buildkite/builds/kb-n2-4-d955647d99aaa0ef/elastic/kibana-on-merge/kibana/node_modules/superagent/lib/node/unzip.js:55:12)
    at Unzip.emit (node:events:513:28)
    at endReadableNT (node:internal/streams/readable:1358:12)
    at processTicksAndRejections (node:internal/process/task_queues:83:21)

First failure: CI Build - 7.17

[elasticmachine]

Pinging @elastic/appex-sharedux (Team:SharedUX)

[petrklapka]

Test in question

[tsullivan]

The failure happened in the 7.17 branch, and the failed assertion happened in the "after" section of a test suite, making this the correct link:
https://github.com/elastic/kibana/blob/7.17/x-pack/test/reporting_api_integration/reporting_and_security/ilm_migration_apis.ts#L86-L89

The 404 that was expected to be a 200 happened when the cleanup phase of the test looked for indices to clean up, and found none. This could have happened here:
https://github.com/elastic/kibana/blob/7.17/x-pack/test/reporting_api_integration/services/scenarios.ts#L191
or here:
https://github.com/elastic/kibana/blob/7.17/x-pack/test/reporting_api_integration/services/scenarios.ts#L232

I don't see anything that looks like a race condition. The cleanup phase didn't delete the reporting indices, just the documents, so it wouldn't cause an issue with the later code that makes the reporting indices unmanaged by a policy. Perhaps there was a bug where an ILM policy that was applied during a test caused the report indices to be deleted, and that interfered with the cleanup tasks. If that is the case, this test could be flaky rather than a one-off factor in the test environment.

As a remedy, we can add logging to find out which request that was supposed to yield a 200 actually yielded a 404. As it is now, the expectations happens using chaining from supertest, so when looking at the stack trace, all we see are the node modules.

[tsullivan]

Related issue: there was an ILM migration test that failed with a similar error, however this edge case will not happen in 7.17: #110813.

[tsullivan]

non-issue: test is not flaky nor is skipped