[Epic] Make Prometheus integration GA

[tetianakravchenko]

Context

Prometheus is a widely adopted and still growing technology for cloud-native users to process system metrics, the growth in adoption of Prometheus makes it a sure candidate to be part of the 20/80 Pareto of cloud-native technologies we want to support to move closer to our objectives by meeting users where they are.

source: CNCF 2021 survey

source: CNCF velocity

To promote our cloud-first strategy we need to support the ingestion of Prometheus metrics into Elastic Cloud in the most typical ways available which are:

• Prometheus endpoint extraction
• Prometheus remote-write
• Querying Prometheus using PromQL

User outcome

Prometheus is mainly used by Kubernetes Cluster Operators because is a powerful tool to collect metrics from infrastructure.

We currently support a module to ingest Prometheus metrics for metricbeat and have a Prometheus integration in technical preview. Both enable users to ingest Prometheus data in the 3 ways mentioned above.

Having a Prometheus integration will enable the following outcomes for the user:

• Leverage the benefits of Elastic agent (new indexing strategy, fleet unified management)
• Out-of-the-box assets which come included with the integration (dashboards, alerts. Some of these to come)

Milestones

The milestones for this work are:

• Find and fix gaps for the Prometheus integration user experience which is available in Technical Preview
• Promote Prometheus integration from technical preview to GA

User experience gaps exploration and fixes
As an outcome of the evaluation issues #3487 #3488 #3489:

• Fix gaps in the Prometheus integration (technical preview) #3662
• [Prometheus integration tutorial] Remove Metricbeat tutorial kibana#135981
• [Prometheus integration] Refactor OOTB dashboard #3668 - ootb dashboard
• Scaling of Prometheus Integration Establish Prometheus Server Integration 'at scale' ingestion limits #3857

Future goals (out of scope)

• Improve and create ootb assets (improve dashboards, create alerts)
• Improve user experience (similar to Improve consistency of Fleet UI deployment to Kubernetes experience  kibana#134747)

[tetianakravchenko]

TODO:

• align http configuration for datasets

[pmeresanu85]

@tetianakravchenko can we consider the Prometheus integration as complete?

[rameshelastic]

@pmeresanu85 , while key tasks are complete, we will be carrying over a couple of items (#3668 etc) to complete in 8.6

[pmeresanu85]

@rameshelastic does this mean that the Prometheus integration is not GA with 8.5?

[tetianakravchenko]

@pmeresanu85 yes, prometheus integration is not yet GA, the only missing part - #3668 as mentioned above

[gizas]

We considered Load test #3857 as an extra work that can be run independently and should not block the promotion

[tetianakravchenko]

TODO: need to be promoted

[gizas]

TODO: need to be promoted

Promotion done today