cat/indices no longer shows indices currently being restored

[sophiec20]

Elasticsearch version (bin/elasticsearch --version):
Version: 8.0.0-SNAPSHOT, Build: default/tar/4957cad/2019-03-11T15:48:39.514013Z, JVM: 1.8.0_151

Plugins installed: []
repository_gcs

JVM version (java -version):

OS version (uname -a if on a Unix-like system):

Description of the problem including expected versus actual behavior:

This is a usability nit.

Restoring a large snapshot may take some time. During this period you can no longer see the restoring indices listed in GET _cat/indices therefore you may think that the restore might not have been initiated.

If you re-run the restore command, you get the error below saying that the index already exists.

This is a change from 6.x where the indices would be listed immediately by GET _cat/indices with their health as yellow. This was a convenient way to see that the restoration was in progress, especially for snapshots with multiple indices.

Perhaps the index list behaviour could be returned to previous, or a more useful error message be displayed to explain that the restore is in progress.

{
  "error": {
    "root_cause": [
      {
        "type": "snapshot_restore_exception",
        "reason": "[my_repo:a_large_snapshot/fYI6EIhEQYm0E_tbDiOEgA] cannot restore index [filebeat-2019] because an open index with same name already exists in the cluster. Either close or delete the existing index or restore the index under a different name by providing a rename pattern and replacement name"
      }
    ],
    "type": "snapshot_restore_exception",
    "reason": "[my_repo:a_large_snapshot/fYI6EIhEQYm0E_tbDiOEgA] cannot restore index [filebeat-2019] because an open index with same name already exists in the cluster. Either close or delete the existing index or restore the index under a different name by providing a rename pattern and replacement name"
  },
  "status": 500
}

Found in 3 node cluster.

Steps to reproduce:

Please include a minimal but complete recreation of the problem, including
(e.g.) index creation, mappings, settings, query etc. The easier you make for
us to reproduce it, the more likely that somebody will take the time to look at it.

1. POST _snapshot/my_repo/a_large_snapshot/_restore -- restore a large snapshot
2. GET _cat/indices -- notice indices are not yet listed
3. POST _snapshot/my_repo/a_large_snapshot/_restore -- re-run the restore again, before the first step has had a chance to finish.

Provide logs (if relevant):

[elasticmachine]

Pinging @elastic/es-distributed

[ywelsch]

I think the bug here is that a restoring index is no longer appearing in _cat/indices. @albertzaharovits this looks related to #38824, and in particular the following line https://github.com/elastic/elasticsearch/pull/38824/files#diff-bfeab4b634ba416b081c4d23a01b1fc8R413 where indices with missing index stats are removed from the output. If multiple indices are restored from a snapshot, it's possible that for some indices none of the primary shards are assigned (as we throttle the number of simultaneously recovering shards). If none of the shards of an index are assigned, that index will not turn up in the indices stats, and the above change will therefore filter it out.

[elasticmachine]

Pinging @elastic/es-core-features

[albertzaharovits]

@ywelsch I think you're right. The change introducing this bug hides indices for which the indices stats API returns empty. From what I understand, indices in the process of being restored also don't appear in the indices stats response.

The simple fix, i.e. to include such indices from the cluster state, will expose names that the user is not authorized to see.
I will mull over it, but my guts feeling for a solution would be to expose these in some API (not reading them from cluster state). Would it make sense to expose them in the indices stats output? I realize this is not exactly the scope of the indices stats API, but given the similarity of the outputs between indices stats and _cat/indices failing to show them in the former compared to the latter might be seen like a bug as well.

[ywelsch]

given the similarity of the outputs between indices stats and _cat/indices failing to show them in the former compared to the latter might be seen like a bug as well

I agree that that is confusing. If we want to address that, then it becomes a much larger discussion though.

I wonder if we can instead use a GetIndexRequest instead of a ClusterStateRequest to get the list of indices to consider for the output