[Index Management] Filter system indices from cat API request

[alisonelizabeth]

Fixes #104079
Fixes #103924
Fixes #64473

Related ES issue: elastic/elasticsearch#74687

Summary:

In order to fetch a user's indices to display in Index Management, we first make a request to fetch indices via the index API (GET /*?expand_wildcards=hidden,all), then make a request to the cat indices API. We use the response of the cat indices API to provide supplemental information about each index.

ES recently made changes that includes a new system index - .geoip_databases- in the cat indices API. However, this same index is not included in the index API. Per elastic/elasticsearch#74687 (comment), this is working as expected; cat APIs should return system indices for debugging purposes. However, the change caused the "Indices" tab in Index Management to error.

As a fix, I have filtered the cat indices response so that it only includes information on indices we fetched from the index API. In the future, we should migrate away from using the cat API as suggested in #57286.

How to test

1. Remove/comment out the ingest_geoip.downloader.enabled=false setting from cluster.js
2. Run yarn kbn bootstrap
3. Restart ES and Kibana (ingest_geoip.downloader.enabled=true should be the default behavior)
4. The "Indices" tab in Index Management should load successfully 😄

[alisonelizabeth]

Running 50x through flaky test runner: https://kibana-ci.elastic.co/job/kibana+flaky-test-suite-runner/1695/

[alisonelizabeth]

I'm hoping that by asserting against a specific index this test will be more stable. It's possible for an index to also have a date_stream property (which is not included in the expectedKeys), so this may have been occasionally failing because the index return from body[0] had a date stream associated with. Related comment: #64473 (comment).

[elasticmachine]

Pinging @elastic/kibana-stack-management (Team:Stack Management)

[cjcenizal]

Had one perf-related suggestion. Otherwise, code LGTM!

[cjcenizal]

This might be a bit silly, but I'm wondering if performance could be a concern. We know users might have hundreds of thousands (maybe millions) of indices. In this case we are doubling memory consumption by creating the filteredCatHits variable, and also doubling the number of iterations by adding another filter loop. There's a good chance that testing could reveal this to be a non-issue. But we could also just make the code immune to these concerns by sticking with a single loop and doing our undefined check within the loop:

return catHits.reduce((decoratedIndices, hit) => {
  const index = indices[hit.index];
  
  if (typeof index !== 'undefined') {
    const aliases = Object.keys(index.aliases);

    decoratedIndices.push({
      health: hit.health,
      status: hit.status,
      name: hit.index,
      uuid: hit.uuid,
      primary: hit.pri,
      replica: hit.rep,
      documents: hit['docs.count'],
      size: hit['store.size'],
      isFrozen: hit.sth === 'true', // sth value coming back as a string from ES
      aliases: aliases.length ? aliases : 'none',
      hidden: index.settings.index.hidden === 'true',
      data_stream: index.data_stream,
    });
  }
  
  return decoratedIndices;
}, []);

[alisonelizabeth]

👍 great point. Thanks for fixing!

[cjcenizal]

When we merge this we should split out the bit about tests from #103924 into a new issue:

Evaluate current API integration tests and add better coverage to cover this scenario.

EDIT: Never mind, I realized we're unskipping the test that would have caught this problem.

[alisonelizabeth]

Looks like all runs were successful in the flaky test runner:

[kibanamachine]

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request
• Commit: 3681f57
• Storybooks Preview

Metrics [docs]

✅ unchanged

History

• 💚 Build #136107 succeeded a646d2a
• 💚 Build #136061 succeeded b4d1c18

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream