[Fleet] Allow preconfiguration to install latest package instead of a specific version

[Zacqary]

Requiring a package version in the Preconfiguration API can be problematic for cloud users who don't have access to their kibana.yml file. Specifying an outdated package without the force flag could cause preconfiguration to fail.

Acceptance Criteria

We should have an option to always install the latest version of a package.

To make sure the user is doing this intentionally, we should require the package config to look like:

version: latest

instead of just leaving version blank.

Discussion

Should we set force: true by default? If latest is an option, then when a user specifies a particular package version, they probably explicitly want that version even if there's a newer version available.

[elasticmachine]

Pinging @elastic/fleet (Feature:Fleet)

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[Zacqary]

Pinging @simitt on this as the feature requester.

[ruflin]

@simitt What is our expected behaviour in Cloud? My assumption is, that for fleet-server and apm-server package, the version should always be identical to the stack in the future. To not have to update the stack pack each time, I would expect we can use a variable in the config. Assuming apm and fleet-server packages are versioned with the stack, would there even be a use case that a user could upgrade the package manually?

[simitt]

For apm-server we plan to align the package version with the stack version. Ideally users would not have to upgrade manually (or at all, it would be bundled with the Kibana version). If that's feasible short-term (7.14) I agree we wouldn't need the latest package version for this.

[simitt]

I am looking for a solution for 7.13. Following up on @Zacqary 's comment #96588 (comment), I think using force is problematic. It would lead to a force install of the defined fleet-server version every time Kibana is restarted - which is not an edge case on cloud. For example, if users change the logs&metrics settings a restart is triggered. An existing fleet server version should not be overriden by that, users might have manually triggered an update since the last Kibana startup.

As a short term solution - could a declared package just be ignored if it is already installed in a newer version?

[Zacqary]

I think we need this functionality for #97016. We're trying to replace the bespoke functions for installing default packages and policies with the preconfiguration API, and these defaults always have to pull the latest version of the system package.

So in some sense, we're already preconfiguring one package with the latest flag.

IMO as long as we only install the latest package if the user explicitly sets latest and they can't do it by accident, it's predictable enough.

[Zacqary]

As a short term solution - could a declared package just be ignored if it is already installed in a newer version?

So instead of using force, we could:

• Install an outdated version if the user explicitly puts the outdated version in the config file
• Don't reinstall that outdated version if Kibana upgrades to a newer version, even if the config file hasn't changed
• Remove the force flag entirely

I think that makes more sense for our planned use case. force works if you're calling the Preconfiguration API once, but it doesn't seem to make sense for the config file that gets reinitialized every page refresh.

[simitt]

@Zacqary it sounds like we are heading in the same direction. I think a supporting latest version is the most straight forward from a users perspective; but if not feasible for 7.13, then the proposed flow of only installing an older version (even without force flag) if no newer version is already installed sounds good to me.