Metricbeat Prometheus module can't fetch over 200K event in one shot

[jehutywong]

config:
metricbeat.modules:

• module: prometheus
  period: 10s
  hosts: ["<prometheus_url>"]
  metrics_path: '/federate'
  query:
  'match[]': '{name!=""}'
  namespace: example

response:
"error": {
"message": "Unable to decode response from prometheus endpoint"
}

For confirmed bugs, please report:

• Version: Metricbeat-7.0.0
• Operating System: CentOS 7 x86_64
• Discuss Forum URL: https://discuss.elastic.co/t/metricbeat-prometheus-module-not-support-match-query/177571
• Seen also in Prometheus collector fails against WildFly 18 #14513 (Metricbeat 7.4.2, Ubuntu and Windows)
• Steps to Reproduce:
  When a query in a Prometheus directive returns over roughly 200k events, this error comes out.

[odacremolbap]

I wasn't able to reproduce, any hint on how to replicate the scenario?

[jehutywong]

Well my config is as simple as following. How many events did your Prometheus server return? @odacremolbap
200K is the ceiling of mine. I don;t think this limitation is from Prometheus server, but Metricbeat, as i was able to get all response results from Prometheus using its API with the same query.

`metricbeat.modules:

• module: prometheus
  period: 10s
  hosts: ["<prometheus_url>"]
  metrics_path: '/federate'
  query:
  'match[]': '{name!=""}'
  namespace: example`

[yxyx520]

The reason of your problem is not "Metricbeat Prometheus module can't fetch over 200K event in one shot",please check your prometheus_url is correct.You can use this command and have a test
curl -v -G --data-urlencode 'match[]={name!=""}' http://prometheus_url/federate

[jehutywong]

Well Prometheus API works well with the query
curl -s -G --data-urlencode 'match[]={site=~"n.*"}' http://127.0.0.1:9090/federate | wc -l 535615
But the same query put into metricbeat, i will have the error above

[evandroasjr]

Hi all,
I have this same issue. Any news?

Cheers,
Evandro

[jsoriano]

Since 7.5.0, metricbeat logs more information when this error happens (#13749), could someone with this problem share the error message on 7.5 or greater?

[christian-morin]

I believe I'm seeing the same behavior in a Java Quarkus MicroProfile enabled app (as #14513 referenced here). Running metricbeat 7.5.1 I'm getting the following error:

unable to decode response from prometheus endpoint: decoding of metric family failed: text format parsing error in line 1: expected float as value, got "'t"

The prometheus metrics retrieved from my /metrics endpoint validated correctly using Prometheus promtool but somehow fails when trying to decode the response by metricbeat prometheus collector. (The problem in this case is definitively not a too large response as I only have a few metrics).

I can't understand where it's getting 't from as it's nowhere in the response. Would be great to be able to debug log what data it's actually trying to decode, but even setting debug logging I'm not seeing any information regarding what it's trying to decode? (or even which metric value) only the error above, which is hard to understand where it's coming from.

[jsoriano]

@christian-morin It is possible that for some reason Metricbeat is receiving a different response, if it is an error #15712 will help to investigate it, this change will be included in 7.6.1. In the meantime you could try to run a custom build of the 7.6 branch.

[christian-morin]

@jsoriano ran this with 7.6.1 and noticed a new (and correct) error-description "unexpected status code 406 from server", which made me realise that the request from metricbeat is sent without an Accept header which creates this behavior for some prometheus-implementations which require this header to be set. Noticed an issue has been created for this in #16870

[jsoriano]

@christian-morin thanks for the update! Does the workaround mentioned in #16870 work for you?

[christian-morin]

@jsoriano yes, adding the Accept header in the Prometheus module config as

headers: 
  accept: "text/plain"

Made it start working!

[jsoriano]

Ok, I am going to close this issue as since 7.5 (#13749) it is easier to diagnose what is happening on these cases, and we can continue with specific issues.
For the problem with accept headers we have the more specific #16835.
If someone is having a different problem, with a different error message, please update at least to 7.5.2 and open a new issue with the logged error.

Thanks all!

[ArvindhVenkatraman]

i have the same problem. When the run it as curl could see the events but getting the below error in the module.error.message
unable to decode response from prometheus endpoint: unexpected status code 403 from server. not sure why it is giving me 403 error because it is not forbidden but for some reason it is unable to decode the response. i tried filtering on the match to a simple set but still wont work. Any suggestion ?

[infa-vsakati]

i have the same problem. When the run it as curl could see the events but getting the below error in the module.error.message
unable to decode response from prometheus endpoint: unexpected status code 403 from server. not sure why it is giving me 403 error because it is not forbidden but for some reason it is unable to decode the response. i tried filtering on the match to a simple set but still wont work. Any suggestion ?

did u got solution

[liu-xiao-guo]

I am following the tutorial at https://www.elastic.co/guide/en/observability/8.5/monitor-java-app.html#_what_youll_learn_3, and now I am having the error like:

{"log.level":"error","@timestamp":"2022-11-29T14:29:59.700+0800","log.origin":{"file.name":"module/wrapper.go","file.line":256},"message":"Error fetching data for metricset prometheus.collector: unable to decode response from prometheus endpoint: unexpected status code 500 from server","service.name":"metricbeat","ecs.version":"1.6.0"}

I do not know what caused the problem.