[Fleet] Migrate Agents from saved object to fleet server indices

[nchaulet]

Description

In the effort of moving the Elastic Agent call from Kibana to Fleet Server, we need to stop reading and writing from kibana saved object and use fleet server system indices instead

Prerequiste to complete the migration:

• the kibana system will need to have the permission to read and write to the .fleet-* indices
• ES will need to create the fleet system indices

First step

While we do not have the correct permission and the system indices we can start to work on the migration, by using a custom kibana system user, and a feature flag.
Introduce a feature flag to start using fleet server indice xpack.fleet.agents.fleetServerEnabled: true
If this flag is enabled we will use a different service that read and write for .fleet-* instead of saved object.

Migration

During the plugin start we will transform all of the existing saved object to .fleet-* document, I discussed this kibana platform and it should work for us.

That draft PR show what it will look like #86179

• Kibana should create the indices.

Saved object call that should be migrated:

• enrollment api keys [Fleet] Use fleet server indices for enrollment keys and to list agents with a feature flag #86179
• agent
  • crud [Fleet] Use fleet server indices for enrollment keys and to list agents with a feature flag #86179
  • reassign [Fleet] Support Fleet server system indices #89372
  • unenroll [Fleet] Support Fleet server system indices #89372
  • update [Fleet] Support Fleet server system indices #89372
  • Elastic agent route (enroll, checkin, ack) [Fleet] Support Fleet server system indices #89372
• agent policy
• actions
• Fix the autocomplete UI for agent to work with .fleet-agents
• Ensure Fleet server indices are correctly created
• Create Fleet server default policy

Second step

• remove the feature flag
• Fix the integration tests to use system indices instead of saved object [Fleet] Remove fleetServerEnabled feature flag and use fleet system indices #92422
• Remove the need of xpack.encryptedSavedObjects.encryptionKey and fleet_enroll user

[elasticmachine]

Pinging @elastic/ingest-management (Team:Ingest Management)

[ruflin]

Does Kibana need access to all .fleet-* indices or only a subset?

[ruflin]

@aleksmaus I think this is part of what we talked about yesterday. @nchaulet As soon as it is ready / testable best ping Aleks so he can test it.

[nchaulet]

Does Kibana need access to all .fleet-* indices or only a subset?

Probably all of .fleet-* indicies especially if we want to support the migration

[mostlyjason]

I added this issue to track migration to fleet server from a UX perspective #90156

[simitt]

Please keep @elastic/apm-server in the loop about when a switch to Fleet Server is recommended or even required. We have CI and dockerized dev setup which are relying on current Elastic Agent/Fleet enrollment. (elastic/apm-server#4650 and https://github.com/elastic/apm-integration-testing/blob/master/docker/apm-server/managed/main.go).

[ruflin]

@nchaulet As anything still pending here?

[nchaulet]

No everything is done here