[Fleet] Add preconfiguration to kibana config

[Zacqary]

Summary

Closes #96370

Enables using the preconfiguration API in kibana.yml.

Also fixes the following bugs:

• Deleting a preconfigured policy will now keep it permanently deleted, by storing a deletion record of its preconfiguration_id. You'll need to change the preconfiguration_id to recreate a policy.
• Integration policies will now use the current installed version of the integration instead of fetching the latest one from the registry.

Testing

Add this to your kibana.dev.yml:

xpack.fleet.packages:
  - name: apache
    version: 0.4.0

xpack.fleet.agentPolicies:
  - name: Preconfigured Policy From Config
    id: 1
    namespace: test
    package_policies:
      - package:
          name: apache
        name: Apache Integration
      - package:
          name: system
        name: System Integration
        inputs:
          - type: system/metrics
            enabled: true
            vars:
              - name: system.hostfs
                value: home/test
            streams:
              - data_stream:
                  dataset: system.core
                enabled: true
                vars:
                  - name: period
                    value: 20s
          - type: winlog
            enabled: false

Checklist

• Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
• Unit or functional tests were updated or added to match the most common scenarios

[Zacqary]

@simitt Basics of this PR are working if you'd like to check out the feature branch and test with cloud stuff. Config format looks like:

xpack.fleet.packages:
  - name: apache
    version: 0.3.4

xpack.fleet.agentPolicies:
  - name: Preconfigured Policy
    id: 1
    namespace: test
    package_policies:
      - package:
          name: system
        name: System Integration
        inputs:
          - type: system/metrics
            enabled: true
            vars:
              - name: system.hostfs
                value: home/test
            streams:
              - data_stream:
                  dataset: system.core
                enabled: true
                vars:
                  - name: period
                    value: 20s
          - type: winlog
            enabled: false

Ping me if you've got questions.

[elasticmachine]

Pinging @elastic/fleet (Feature:Fleet)

[jbudz]

kibana-docker changes LGTM

[nchaulet]

It works as expected 🚀 I have two questions:

1. Should we use saved object to save deleted policies? it come with a lot of things migration, schema, ...

2. I am wondering if we should use this and default values to create the default and default fleet server policy, it will simplify a lot our code, and we will have only one way to configure preconfigured policy, (this can be done in a following PR to keep that one small and readable)

[simitt]

@Zacqary I played around with this a bit and the preconfigured agent policies and packages were set up 🎉 .

I did also run into a couple of issues though and just leaving my observations here:

Encountered Errors

• When providing this configuration:

xpack.fleet.packages:
  - name: fleet-server
    version: 0.2.0

xpack.fleet.agentPolicies:
  - name: Elastic Agent on Cloud policy
    id: 1
    package_policies:
      - package:
          name: fleet-server
        name: Fleet Server 
        inputs:
          - type: fleet-server
            vars:
            - name: port
              value: 8220
            - name: host
              value: 0.0.0.0

server    log   [14:27:15.645] [error][fleet][plugins] Package key parsing failed: package version was not a valid semver

The package name should be fleet_server instead of fleet-server; receiving a semver error on this is odd though. This was discussed in #92621, not certain if that should have been fixed/and is related to this PR.

• When trying out your example I ran into this error:

If this happens - is there any other way for the user to resolve this issue than to change the kibana.yml and restart Kibana? This is not a nice experience. This also is very concerning for a hosted scenario, where the users might not be able to change the kibana.yml. Running into the error makes the whole Fleet UI unusable. This is related to a version question further down.

Questions ans Observations

• is it already supported to configure that the agent policy is managed?
• will the bundled kibana.yml contain a configuration for the Default Policy and the Default Fleet Server policy?
• how to define the data_stream.namespace?
• how to define the policy to be the fleet-server default policy?
• can you change the logic to install the latest available package version if no version is given? The version should not have to be bound to the stackpack version on cloud.
• trying to configure [xpack.fleet].agentPolicies.0.package_policies.0.inputs.0.name results in an error, as the key is not defined; I think we should allow to define a name
• how does this behave on update/overwrite? If the package in the agent policy has been customized by users after loading from the config file, would the user changes be overwritten on update or is the configuration only loaded if the package or the agent policy doesn't exist? What is the path forward when upgrading from one Kibana version to the next one and the bundled configuration has changed?
• I started Kibana with one preconfigured policy in the yml, which was set up accordingly, then I restarted Kibana with another preconfigured policy, but this time the new policy was not added. How could one achieve that this gets set up?

[nchaulet]

@simitt the error you encoutered is because of a typo in the package name fleet-server => fleet_server we should probably have a better handling as the error reported is not the correct one

[simitt]

@nchaulet exactly, I linked that in my comment, but easy to overlook:

The package name should be fleet_server instead of fleet-server; receiving a semver error on this is odd though. This was discussed in #92621, not certain if that should have been fixed/and is related to this PR.

[Zacqary]

The package name should be fleet_server instead of fleet-server; receiving a semver error on this is odd though.

Agree, we should have a better error for this situation. Unfortunately it's happening because the code is creating a package key called fleet-server-0.2.0 and then performing a split('-') to try to get the semver, so it ends up trying to parse server-0.2.0 as a semver. That parser should probably have a check to make sure there's only one - in the package key and throw the correct error.

Out of scope for this PR, I think. We can file a new issue.

If this happens - is there any other way for the user to resolve this issue than to change the kibana.yml and restart Kibana?

No, there's no way to resolve this in the UI. I can add better error handling for this.

is it already supported to configure that the agent policy is managed?

Yes, add the is_managed: true flag to an agent policy.

will the bundled kibana.yml contain a configuration for the Default Policy and the Default Fleet Server policy?
how to define the policy to be the fleet-server default policy?

We didn't plan on that. The defaults are hard-coded but the user can override them in kibana.yml by setting the is_default/is_default_fleet_server flag on a preconfigured policy to true:

xpack.fleet.agentPolicies:
  - name: Default Fleet Server Override
    id: default-fleet-server
    is_default_fleet_server: true

If you think it'd be valuable to include these policies in kibana.yml we can open a Discuss issue around it.

EDIT: I realized there's a bug in this; the plugin setup ensures the default policies before reading from kibana.yml. I'll push a commit to change the order of events so that this works correctly.

how to define the data_stream.namespace?

I believe it's

- data_stream:
    dataset: dataset.name
  namespace: some-namespace
  vars:
    ...

can you change the logic to install the latest available package version if no version is given? The version should not have to be bound to the stackpack version on cloud.

@ruflin Is this an okay change to make? I know we wanted to keep things predictable by requiring the user to specify a version.

trying to configure [xpack.fleet].agentPolicies.0.package_policies.0.inputs.0.name results in an error, as the key is not defined; I think we should allow to define a name

I was under the impression that input names are hard-coded in the integration manifest and can't be changed by the user. Is that incorrect?

how does this behave on update/overwrite? If the package in the agent policy has been customized by users after loading from the config file, would the user changes be overwritten on update or is the configuration only loaded if the package or the agent policy doesn't exist? What is the path forward when upgrading from one Kibana version to the next one and the bundled configuration has changed?

At the moment there's no update or overwrite support, just creating new policies. If this is important we can prioritize it.

I started Kibana with one preconfigured policy in the yml, which was set up accordingly, then I restarted Kibana with another preconfigured policy, but this time the new policy was not added. How could one achieve that this gets set up?

Did you set a new policy id? The id parameter is what the preconfiguration API uses to determine if a policy is "new" or not.

[Zacqary]

@nchaulet

I am wondering if we should use this and default values to create the default and default fleet server policy, it will simplify a lot our code, and we will have only one way to configure preconfigured policy, (this can be done in a following PR to keep that one small and readable)

I'm pushing a fix so that we can successfully use kibana.yml to override the DEFAULT_AGENT_POLICY/DEFAULT_FLEET_SERVER_AGENT_POLICY definitions by passing the is_default/is_default_fleet_server flag to a preconfigured policy. Is there anything more substantive you were looking for? Should we remove the ensureDefaultAgentPolicy functions entirely?

[nchaulet]

I'm pushing a fix so that we can successfully use kibana.yml to override the DEFAULT_AGENT_POLICY/DEFAULT_FLEET_SERVER_AGENT_POLICY definitions by passing the is_default/is_default_fleet_server flag to a preconfigured policy. Is there anything more substantive you were looking for? Should we remove the ensureDefaultAgentPolicy functions entirely?

I think it would make sense yes to have everything defined as the default kibana config value, and if you want to override you can do whatever you want, and this way there is only one way to create predefined config

[Zacqary]

@nchaulet Tracking this here: #97016

[Zacqary]

@simitt Tracking the semver error issue here: #97018

[ruflin]

@simitt I prefer to keep a predefined version and not latest to have it more predictable. It means at the moment we would need to update the Kibana stack pack for each version at least until there is an other solution. Are you ok with that?

[nchaulet]

Looks good to me as soon as the CI pass

[Zacqary]

@simitt Preconfiguration errors will now be thrown as toasts instead of blocking the Fleet UI

[simitt]

@simitt I prefer to keep a predefined version and not latest to have it more predictable. It means at the moment we would need to update the Kibana stack pack for each version at least until there is an other solution. Are you ok with that?

Not with how it is currently working. When configuring fleet_server version 0.2.0 (while the latest available version is 0.2.1) the setup results in an error. This would mean that the cloud managed policy cannot be set up.

Update: since the errrors are not blocking anymore, updated this comment.

[simitt]

@Zacqary thanks for the answers and changes. I tested again, and apart from the version comment above this looks great! Thanks for the quick implementation.

[kibanamachine]

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request
• Commit: cb65cea
• Storybooks Preview

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
fleet	450	451	+1

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
fleet	711.2KB	711.5KB	+327.0B

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
fleet	345.4KB	345.8KB	+425.0B

Saved Objects .kibana field count

Every field in each saved object type adds overhead to Elasticsearch. Kibana needs to keep the total field count below Elasticsearch's default limit of 1000 fields. Only specify field mappings for the fields you wish to search on or query. See https://www.elastic.co/guide/en/kibana/master/development-plugin-saved-objects.html#_mappings

id	before	after	diff
fleet-preconfiguration-deletion-record	-	2	+2

History

• 💔 Build #119400 failed f5f18f7
• 💔 Build #119396 failed 71e3955
• 💔 Build #119325 failed 50b682b
• 💚 Build #119024 succeeded 693e4ab
• 💚 Build #119008 succeeded f1dbd66

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @Zacqary

[ruflin]

It only gives an error if 0.2.1 is already installed I assume? Or already if it is only released? @simitt @Zacqary Please make a call without me on what the best option here is to. keep. it. moving.

[simitt]

I believe with Fleet Server the problem is that it gets installed already for the default policies, before the configured policy gets picked up to be installed. IMO installing outdated packages should always be allowed, especially if one cannot specify a latest version.
For making progress with this, my suggestion is to ensure outdated packages can be installed, and create a new issue where we can discuss changing to latest.
Does that sound reasonable to you @Zacqary ?

[Zacqary]

@simitt Outdated packages can be installed with the force flag:

xpack.fleet.packages:
  - name: fleet-server
    version: 0.2.0
    force: true

I'm going to merge this since that's already an option, but if that's still problematic for cloud users who can't edit their config, we can discuss making force the default on the new issue.