[ML] Make aliases on ML hidden indices into hidden aliases

[droberts195]

Since #52423 and #52778 some of the ML internal indices are hidden indices. A problem has been identified that means these indices do not show up correctly in _cat/indices when security is enabled - see #52304. The solution that is being implemented is to introduce the concept of hidden aliases. This work is ongoing in #52547.

Before 7.7 feature freeze ML will need to make all aliases on our hidden indices into hidden aliases. (Or, if this is not possible in time for 7.7, remove the index.hidden setting from the indices we've added it to so far and try again in 7.8.)

[elasticmachine]

Pinging @elastic/ml-core (:ml)

[przemekwitek]

Update: #52547 has been merged to master so I've started working on this issue.

[przemekwitek]

Short analysis of the possible combinations of hidden/non-hidden index with hidden/non-hidden alias.
TL;DR: All four combinations are possible as long as one alias' is_hidden property is consistent across all the indices the alias points at (see validation in AliasOrIndex::computeAndValidateAliasProperties).

Contents of create_non_hidden_index.json:

{
  "settings": {
    "hidden": false
  }
}

Contents of create_hidden_index.json:

{
  "settings": {
    "hidden": true
  }
}

Contents of aliases_actions.json:

{
  "actions": [
    {
      "add": {
        "alias": "non_hidden_alias",
        "index": "non_hidden_index"
      }
    },
    {
      "add": {
        "alias": "non_hidden_alias",
        "index": "hidden_index"
      }
    },
    {
      "add": {
        "alias": "hidden_alias",
        "index": "non_hidden_index",
        "is_hidden": true
      }
    },
    {
      "add": {
        "alias": "hidden_alias",
        "index": "hidden_index",
        "is_hidden": true
      }
    }
  ]
}

The following sequence of commands results in correct state of the system. No errors are thrown.

curl -X PUT localhost:9200/non_hidden_index -H Content-Type:application/json -d @create_non_hidden_index.json | jq

curl -X PUT localhost:9200/hidden_index -H Content-Type:application/json -d @create_hidden_index.json | jq

curl -X POST localhost:9200/_aliases -H Content-Type:application/json -d @aliases_actions.json | jq

The configuration of the indices is:

curl -X GET localhost:9200/non_hidden_index | jq

{
  "non_hidden_index": {
    "aliases": {
      "hidden_alias": {
        "is_hidden": true
      },
      "non_hidden_alias": {}
    },
    "mappings": {},
    "settings": {
      "index": {
        "hidden": "false",
        "number_of_shards": "1",
        "provided_name": "non_hidden_index",
        "creation_date": "1584454818812",
        "number_of_replicas": "1",
        "uuid": "QY5lJ0uHRYebOD-rQB7zKg",
        "version": {
          "created": "8000099"
        }
      }
    }
  }
}

curl -X GET localhost:9200/hidden_index | jq

{
  "hidden_index": {
    "aliases": {
      "hidden_alias": {
        "is_hidden": true
      },
      "non_hidden_alias": {}
    },
    "mappings": {},
    "settings": {
      "index": {
        "hidden": "true",
        "number_of_shards": "1",
        "provided_name": "hidden_index",
        "creation_date": "1584454823325",
        "number_of_replicas": "1",
        "uuid": "F7RoH2GeQjqlTDBLfypkpA",
        "version": {
          "created": "8000099"
        }
      }
    }
  }
}

[przemekwitek]

@gwbrown confirmed that all four combinations should work fine so there should not be any BWC issues while upgrading from 7.6 to 7.7.

[przemekwitek]

The PR #53160 is now merged to master and 7.x.
Closing the issue.