[Fleet] configure default policy for agent and fleet server from kibana config

[nchaulet]

Description

For cloud we need to be able to configure how default policies are created, as the default policy for fleet server need to be hosted and to not allow all the modifications. This could be nice too as he allow end user to configure policy from kibana config.

For this we will configure policies in kibana config (see this reverted commit edc5cf7) than the policies will be created when the user visit the fleet plugin or call the existing /api/fleet/setup

This will replace the hardcoded creation of the default agent and fleet server policy instead we will have a default config that create theses policies but in cloud different agent policies could be provided.

Kibana config

We could use the key xpack.fleet.policies: (open to suggestions)

Existing default policy

we need to migrate the existing default policy to this we probably just have to add a preconfiguration_id

I discussed this with @Zacqary and looks like the preconfigure service will work without problem here.

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[ruflin]

• Config naming: Not a big fan of the fleet prefix but we already have it for the other bits. Naming wise we should use the indentical terms as we have in the preconfigure API: [Fleet] Fixes to preconfigure API #96094

xpack.fleet:
  agent_policies:
    ...
  packages:
    ...

Maybe we should have a prefix? xpack.fleet.preconfigure:. A user using the API should be able to copy paste the json body into the Kibana config (convert it to yaml, put it under the correct prefix) and just use it.

What will happen if a user deletes a policy that was set up through the config. It it automatically be recreated?

@nchaulet You mention it will replace the hardcoded policies. I assume we still have in the code some hardcoded policies but in the above format instead?

[ruflin]

@nchaulet To make this fully functional, I think we also need #96319 But this we need independent of the API or the config part.

[Zacqary]

Is this issue about enabling all functionality of the Preconfiguration API in kibana.yml, or just for the default policies?

What will happen if a user deletes a policy that was set up through the config. It it automatically be recreated?

If we parse the config during plugin setup, then no, it won't be recreated. Is that okay, or do we want to recreate policies that get deleted?

[ruflin]

This issue is about using the kibana.yml. How do you ensure that it is not recreated if it was deleted? If that is already the case, I think it is great. We should allow a user to delete a policy that is not useful, except the ones we manage.

[Zacqary]

Are there any cases when the initial plugin setup would run again, without a fresh reinstall of Kibana? If so then we might need to add some extra stuff to ensure a policy can't be recreated.

[ruflin]

In the Cloud case, the /setup call will be made every a fleet-server is spun up. But not sure if this is the same you mentioned with the initial plugin setup?

[Zacqary]

I'm talking about the setupIngestManager call in setup.ts. In my testing I've only seen that run the very first time the user loads the Fleet UI. Does it ever run again?

[nchaulet]

the setupIngestManager run every time the user load the fleet UI.

[Zacqary]

All right so it turns out I was mistaken, the policy does get recreated when the user reloads the UI. Must have misremembered the way I tested it before.

I'll find a way to make sure we can permanently delete a policy without changing the config.

[jfsiii]

setupIngestManager is called via plugin lifecycle and it calls functions like

kibana/x-pack/plugins/fleet/server/services/setup.ts

Lines 58 to 63 in e6e3b16

	ensureInstalledDefaultPackages(soClient, esClient),
	outputService.ensureDefaultOutput(soClient),
	agentPolicyService.ensureDefaultAgentPolicy(soClient, esClient),
	agentPolicyService.ensureDefaultFleetServerAgentPolicy(soClient, esClient),
	updateFleetRoleIfExists(esClient),
	settingsService.getSettings(soClient).catch((e: any) => {

It's safe to call them repeatedly, but they don't need to do something each time. The ensure prefix is often "create if missing. error if unsuccessful"

[ruflin]

It would be good to write down the expected behaviours in the different scenarios.

For example in the upgrade case: We have a package foo-1.2.3 and the managed policy bar is created. Now package foo is upgraded to foo-1.3.0 and the kibana config got updated with a new value which overwrites the default in the package. On restart of Kibana or calling setup again, should the policy be updated?

[Zacqary]

@ruflin Should we only automatically update package policies for managed policies, or for all preconfigured policies?

[ruflin]

@Zacqary Good question, what would you recommend?

[Zacqary]

What's the current behavior, for policies that aren't created through preconfiguration? Do managed policies automatically upgrade their integrations? Do non-managed policies ask the user to do it manually?

I think we should replicate however policies are supposed to behave already.

[ruflin]

There is currently not difference between managed and not managed. @jen-huang @nchaulet Your view would be great here.

[nchaulet]

what we currently have is kind of confusing we automatically update the defaults package, but we do not update the default policy.

I think we can easily replicate that by adding a flag to the package key to automatically upgrade (when the user visit the fleet plugin), something like

packages:
 - name: system
    auto_upgrade: true

we could potentially add this to the agent policy too but it will be an additional feature different from what we currently have, should we add it to the policy level or per package?

agentPolicies:
 - name: Policy 1
    auto_upgrade: true

[Zacqary]

I can do the auto_upgrade flag but the problem is, for agentPolicies, there isn't actually a button that the user can press to do this in the UI already.

It feels like a bug that the user can go to the package on the Integrations page, see that an update is available, click Update, and have this not also automatically update all the agent policies. Whatever installed version the Integrations page says there is feels like it should be the universal source of truth.

Is there a reason why a user would not want their agent policy to be auto-upgraded, is my thinking?

[Zacqary]

Talked with @ph and it turns out functionality related to auto_upgrade is already tracked here: #92612

I don't think we need to do anything specific for preconfigured policies, as solving this issue would also apply to every other agent policy.

Adding auto_upgrade to packages might be less complicated but I think it still warrants its own issue. I'd prefer to just get the current preconfiguration feature set working in kibana.yml for a first PR.