Support remote clusters using API keys

[pebrc]

Elasticsearch starting with version 8.10 supports a new security model for remote cluster connections using API keys (currently in beta):
https://www.elastic.co/guide/en/elasticsearch/reference/current/remote-clusters-api-key.html

We should support this in ECK for the fully managed remote cluster connection setup documented here:

https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-remote-clusters.html#k8s-remote-clusters-connect-internal

• dedicated API key API /_security/cross_cluster/api_key
• CA trust is required (as before)
• additional port for the remote connections (default 9443)

There are few open questions:

• creating the API key requires user input to tailor the permissions to the specific use case. Automating a wildcard api key that allows access to everything would be giving a way the main benefit of the new security model

• our current API does not allow to specify the permissions

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: cluster-one
  namespace: ns-one
spec:
  nodeSets:
  - count: 3
    name: default
  remoteClusters:
  - name: cluster-two
    elasticsearchRef:
      name: cluster-two
      namespace: ns-two 
  version: 8.13.4

But we would need to express the cross-cluster API key permission model somehow:

POST /_security/cross_cluster/api_key
{
  "name": "my-cross-cluster-api-key",
  "expiration": "1d",   
  "access": {
    "search": [  
      {
        "names": ["logs*"]
      }
    ],
    "replication": [  
      {
        "names": ["archive*"]
      }
    ]
  },
  "metadata": {
    "description": "phase one",
    "environment": {
       "level": 1,
       "trusted": true,
       "tags": ["dev", "staging"]
    }
  }
}

Do we want to inline the access section into our CRD? Do we want to leave this as a step to be executed by the user (not very attractive if we want to facilitate automatic setup of these remote cluster relationships) Can we leverage stack config policies in a novel way to do this.

Another question is how migration would work (if we support that at all from old world to new world). The stack documentation is to be found here: https://www.elastic.co/guide/en/elasticsearch/reference/current/remote-clusters-migrate.html

• do we want to try to fully automated this?
• do we want to not automate but allow users to manually transition
• do we do nothing and let both mechanisms exist next to each other?