Broken Connection error with prometheus_write

[abhi670329]

We are trying write the metrics from fluent bit to prometheus end point using prometheus_remote_write. We are seeing connection broken errors.

{3:00:26] [debug] [output:prometheus_remote_write:prometheus_remote_write.1] http_post result FLB_RETRY
3:00:26] [error] [output:prometheus_remote_write:prometheus_remote_write.1] ...:9090, HTTP status=0
3:00:26] [error] [http_client] broken connection to ...:9090 ?
3:00:26] [debug] [http_client] not using http_proxy for header
3:00:26] [debug] [upstream] KA connection #108 to ...:9090 is connected
3:00:26] [debug] [output:prometheus_remote_write:prometheus_remote_write.1] final payload size: 44771
3:00:26] [debug] [output:prometheus_remote_write:prometheus_remote_write.1] cmetric_id=0 decoded 0-65532 payload_size=44771
3:00:26] [debug] [task] destroy task=0x7ef84d83af60 (task_id=2)
3:00:26] [debug] [out flush] cb_destroy coro_id=8
3:00:26] [debug] [upstream] KA connection #106 to ...:443 is now available
3:00:26] [debug] [upstream] KA connection #106 to*...*:443 has been assigned (recycled)
3:00:26] [debug] [output:forward:forward.0] send options records=3 chunk=''
3:00:26] [debug] [output:forward:forward.0] request 1825 bytes to flush
3:00:26] [debug] [output:forward:forward.0] task_id=2 assigned to thread #0
3:00:26] [debug] [task] created task=0x7ef84d83af60 id=2 OK
3:00:26] [debug] [output:prometheus_remote_write:prometheus_remote_write.1] cmetrics msgpack size: 65532
3:00:26] [debug] [output:prometheus_remote_write:prometheus_remote_write.1] task_id=1 assigned to thread #0}

Your Environment

• Version used: Fluent Bit v3.0.6
• Configuration:

[INPUT]
Name prometheus_scrape
host 0.0.0.0
port 8080
Tag issue-store
metrics_path /api/metrics
scrape_interval 10s

[OUTPUT]
name prometheus_remote_write
host ...
Port 9090
match issue-store
URI /api/v1/write
log_response_payload true

• Environment name and version (e.g. Kubernetes? What version?):
• Server type and version:
• Operating System and version:Linux
  *PRETTY_NAME="Ubuntu 22.04.4 LTS"
  NAME="Ubuntu"
  VERSION_ID="22.04"
  VERSION="22.04.4 LTS (Jammy Jellyfish)"
• Filters and plugins:

Additional context
We are not able to write metrics to prometheus.

[patrick-stephens]

So are you saying the connection never recovers?
It looks like Prometheus is responding with something - have you checked the logs there? Is remote write enabled?
Also try latest version of Fluent Bit.

[abhi670329]

Yes, The connection never recovers.
There are no logs on prometheus end. Yes, we have enabled remote write.
We are able to remote write metrics from a different application which has less data.