[Fleet] Support installable but not add-able integrations

[nchaulet]

Description

The Elastic Agent integration (elastic/integrations#462) introduce a new concept of integration that you can install but not add to an agent policy.

Installing the Elastic agent integration will create dashboards for the Elastic Agent, but there is not point to add the integration to an agent policy as monitoring an agent is done through different property on the agent policy.

Should we add a new flag on the package that allow to install and integration but not to add it to an agent policy

Should we treat monitoring as other integration? and add the Elastic Agent integration to the agent policy

[elasticmachine]

Pinging @elastic/ingest-management (Team:Ingest Management)

[ycombinator]

@mtojek An issue to keep an eye on as it might affect the package spec, validation, potentially testing with elastic-package, etc.

[ph]

Adding a note, the Elastic Agent package is a use case. I think we should extend that to more than just that specific integration.
Imagine a world where multiple kind of integration exists, some can be installed only in Kibana (like ML job), some can installed on the Elastic Agent, some are only documentation. This is a bit linked to the integration UI effort.

cc @jen-huang @mostlyjason @ruflin

[mostlyjason]

Can someone share context for what data this dashboard is showing? It looks like CPU and memory which seems similar to stack monitoring data. Is this the data you get when you create an agent policy and select "collect system metrics"?

If so, why not just make it an input like all the others? Then we don't need special-case configuration or logic, it'd just be a regular integration with a regular input. We could make it a default integration similar to the system integration, so its available for all agents unless they remove it from their agent policy.

[ruflin]

From my perspective, it is just a package without any input templates. A package without an input template should be shown in the list, can be installed but should not show up in the "add integrations" view.

@mostlyjason Interesting idea, but I'm not sure it is something we should let users configured like all the other parts as it will become an integral part for Fleet and these metrics / logs must be available. Otherwise a user might remove them and is surprise that some things might then not work anymore.

[mostlyjason]

@ruflin if they uncheck "collect system metrics" what will stop working? i assume its just the logs/metrics on the agent details page, but the agent keeps running and they can still manage it in fleet?

[nchaulet]

if they uncheck "collect system metrics" what will stop working? i assume its just the logs/metrics on the agent details page, but the agent keeps running and they can still manage it in fleet?

Yes the agent will still run we will just not have logs/metrics

[ruflin]

@mostlyjason I expect that in the future coordinator in fleet-server will make some decision on these metrics / logs. But that is also likely going to be in addition shipped through the status so things should keep working.