[ILM] Index Priority

[jakelandis]

ILM is introducing a new "set_priority" action for the hot, warm, and cold phases. It will have 1 option that will control the index.priority

For example:

PUT _ilm/policy/my_policy
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "set_priority": {
            "priority": 100
          },
          "rollover": {
            "max_age": "10m"
          }
        }
      },
      "warm": {
        "min_age": "1m",
        "actions": {
          "set_priority": {
            "priority": 50
          }
        }
      }
    }
  }
}

See elastic/elasticsearch#36905

This new action should also be integrated into the ILM UI. (note - elastic/elasticsearch#36905 is not integrated into Elasticsearch yet, but should be soon)

This should be an optional setting, and would suggest to default it on with values that can range from 0-999. With a default of 100 for hot, 50 for warm, and 0 for cold.

I chose 0 as the lower bound since that is the min. acceptable value for the setting.
I chose 999 as an upper bound, since 1000 is the priority of the .security index, and we want to encourage/ensure .security is recovered first.

I chose 100 as hot default because it is a nice number, that is less then 800 and 900 which are the defaults priorities for the watches and triggered watches indexes.
I chose 50 for warm because it is 1/2 of 100 and there is lot of room between 50 and 100 to tweak warm recovery settings.
I chose 0 for cold because the "i never heard of this value before" default is 1, meaning that cold will recovery after indexes not managed by ILM that haven't been explicitly configured.

The suggestions if implemented would result in default recovery in this order: .security, .watches, .triggered_watches, hot indices, warm indices, un-managed/not configured indices, cold indices. Within each bucket the create/dt/tm of the index is the secondary sort.

[jakelandis]

After a brief discussion, we should not enforce a UI upper bounds that is different from the API's upper bound (which is Integer.Max).

[cjcenizal]

Addressed by #29205