[Fleet Server] Fleet Server Observability

[mostlyjason]

Let's add a dashboard for Fleet Server operators to help them scale the server when needed and troubleshoot issues

It should show:

• System metrics like CPU, memory usage by host over time. Should be accurate for VMs and containers. Helps to identify infrastructure limits. This should work for agents with system monitoring enabled and hosted agents with only internal monitoring.
• Fleet server process metrics like CPU, memory usage by host over time. Should be accurate for VMs and containers. Identifies capacity usage from Fleet Server compared to other processes.
• Status by host over time. Provides a history of when Fleet Server was offline, updating, unhealthy, etc.
• Log stream component showing errors. Useful for troubleshooting.
• Add a note with a link to the stack monitoring app where users can monitor APM server and standalone FB/MB, which are in the same container.
• Filter on hostname. Lets operators isolate metrics from particular Fleet server hosts.

Stretch:

• Number of active connections by host over time. Lets operators see resource usage as a function of capacity.
• Number of rejected connections by host over time. Lets operators see when limits are reached and the impact on clients.

Related issues:

• Logs, metrics and status info for Fleet Server [Elastic Agent] Logs, metrics and status info for Fleet server beats#24415
• Elastic Agent dashboard https://github.com/elastic/beats/issues/23948
• Enabling metricbeat on ESS/ECE https://github.com/elastic/cloud/issues/74800

Open questions:

1. Should we have a separate dashboard for Fleet Server or combine it with the Elastic Agent dashboard?
   • They should have separate dashboards. They have separate metrics to visualize, like only Fleet Server has connection count. System metrics are particularly useful for Fleet Server because the goal is to maximize utilization and observe when its necessary to scale the infrastructure. The Elastic Agent running on an endpoint should have low utilization so it will be easier to visualize these use cases separately. Also, its a standard pattern to include dashboards for each integration, so it will be more discoverable as part of the Fleet Server integration.
2. Confirm system metrics are enabled on cloud
   • We are not planning to enable them on cloud, but we should reconsider that. I created this issue to discuss it [Fleet] Enable system metrics on Elastic Cloud agent policy kibana#96248
3. How can we filter the fleet server hosts from the other hosts in the dashboards?

[elasticmachine]

Pinging @elastic/agent (Team:Agent)

[simitt]

For which version is this planned?

[ph]

@simitt We haven't added it to the roadmap yet.

[mostlyjason]

@ravikesarwani Can I get your input on the Fleet Server dashboard definition? In particular, the open question about how it relates to the Elastic Agent dashboard?

[ravikesarwani]

@mostlyjason Yes, I would love to help and add my thoughts but don't have a lot of context around the topic of fleet server. Can we setup some time next week to discuss so that you can fill me in and then I can research and add my thoughts?

[michalpristas]

added a 7.14 label, please update if needed for another version

[EricDavisX]

@mostlyjason just pinging - I'm doing follow-ups on open issue for 7.14 - is this still under review / possible merge for 7.14? Assuming not, I'll remove the 7.14 label, please add back if any work was done for this cycle. If this is urgent for the Fleet / Agent GA we may yet have time to finish it out.

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[nimarezainia]

@jlind23 i added this to the data plane board as for some reason GH is not finding the control plane board in this repo. Do you have the same issue?

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[jlind23]

@nimarezainia Nope i don't have this issue. I just added it to the control plane board.

[jlind23]

This one is about exporting this newly created dashboard and pushing it to fleet-server package.

[jlind23]

@nimarezainia Do you have a full list of what are the needed indicators? Could you please work on it before the 8.4 kick off?

[jlind23]

@nimarezainia could you jump in here please?