[prometheus-kube-stack] [v65.0.0] Metrics Scraping Issue: Error on Ingesting Samples with Different Values but Same Timestamp

[Masmiiadm]

Describe the bug a clear and concise description of what the bug is.

Prometheus logs warnings about conflicting samples with the same timestamp, and some metric samples are dropped, leading to incomplete data.
Error on Ingesting Samples with Different Values but Same Timestamp

What's your helm version?

version.BuildInfo{Version:"v3.14.3", GitCommit:"f03cc04caaa8f6d7c3e67cf918929150cf6f3f12", GitTreeState:"clean", GoVersion:"go1.21.7"}

What's your kubectl version?

Client Version: v1.29.3 Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3 Server Version: v1.27.12 WARNING: version difference between client (1.29) and server (1.27) exceeds the supported minor version skew of +/-1

Which chart?

kube-prometheus-stack

What's the chart version?

v65.0.0

What happened?

After upgrading from v51.0.0 to v65.0.0 ( prometheus upgraded fro 2.47 to 2.54 ) . We are encountering an issue where Prometheus is logging warnings about dropping samples due to conflicting values with the same timestamp .
ts=2024-10-14T07:23:44.552Z caller=scrape.go:1744 level=warn component="scrape manager" scrape_pool=consul-appli target=http://x.x.x.x:8081/metrics msg="Error on ingesting samples with different value but same timestamp" num_dropped=864
ts=2024-10-14T07:15:27.184Z caller=scrape.go:1744 level=warn component="scrape manager" scrape_pool=serviceMonitor/monitoring/monitoring-k8s-kube-state-metrics/0 target=http://x.x.x.x::8080/metrics msg="Error on ingesting samples with different value but same timestamp" num_dropped=25
Following the bug fix 14685 , the issue should normally be resolved in version 2.54.1. However, the warnings logs are still appearing

What you expected to happen?

Prometheus should be able to scrape metrics without encountering conflicting values for the same timestamp, or the application/service should emit metrics with unique timestamps.

How to reproduce it?

No response

Enter the changed values of values.yaml?

No particular changes .
CRDs are updated

Enter the command that you execute and failing/misfunctioning.

No particular changes ,

Anything else we need to know?

I downgraded from v65.0.0 to v51.0.0 ( prom-v2.47) and there is no scrapping warnings logs .

[ChristophLoy]

Got the same issue on 65.2.0