Vector + Prometheus. Prometheus Exporter: TCP bind failed: address already in use (os error 98)

[java-juggled-jazz]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem

I use Docker Compose with Prometheus, Grafana and Vector for monitoring Nginx logs. I've set Prometheus to listen vector:8080/metrics (it's up in Prometheus targets). The port is free. When I start Vector in Docker I see this:

ERROR sink{component_kind="sink" component_id=metrics-to-prometheus component_type=prometheus_exporter component_name=metrics-to-prometheus}: vector::sinks::prometheus::exporter: Failed to start Prometheus exporter: TCP bind failed: Address in use (os error 98).

What should I do?

Configuration

[sources.nginx-logs]
type = "file"
include = ["/var/log/nginx/access.log"]
read_from = "beginning"

[transforms.nginx-metrics]
type = "remap"
inputs = ["nginx-logs"]
source = """
. = parse_regex!(.message, r'^\"(?P<timestamp>.*)\" client=(?P<client>.*) method=(?P<method>.*) request=\"(?P<method_type>.*) (?P<request_path>.*) (?P<http_version>.*)\" request_length=(?P<>
.timestamp = parse_timestamp(.timestamp, "%Y/%m/%d %H:%M:%S %z") ?? now()
.request_length = to_int!(.request_length)
.status = to_int!(.status)
.bytes_sent = to_int!(.bytes_sent)
.body_bytes_sent = to_int!(.body_bytes_sent)
.upstream_status = to_int!(.upstream_status)
.request_time = to_float!(.request_time)
.upstream_response_time = to_float!(.upstream_response_time)
.upstream_connect_time = to_float!(.upstream_connect_time)
.upstream_header_time = to_float!(.upstream_header_time)
.host = "YOUR_HOSTNAME_HERE"
"""

[sinks.metrics-to-console]
type = "console"
inputs = ["nginx-metrics"]
target = "stdout"
encoding.codec = "json"

[sinks.metrics-to-prometheus]
type = "prometheus_exporter"
inputs = [ "nginx-metrics" ]
address = "0.0.0.0:8080"

Version

0.32.2-alpine

Debug Output

No response

Example Data

No response

Additional Context

No response

References

No response

[jszwedko]

Hi @java-juggled-jazz ,

This error indicates that there is a process already listening on port 8080. I'd suggest choosing a different port.

[java-juggled-jazz]

Hi @java-juggled-jazz ,

This error indicates that there is a process already listening on port 8080. I'd suggest choosing a different port.

Logically. But I've tried to change port and nothing happened. netstat -teln on host OS tells that port 8081 not used, same command inside Docker container tells that port 8081 is used and port 8080 not used. Looks like port 8081 is used by Vector.

[java-juggled-jazz]

Just removed Prometheus exporter sink. It works like a charm. Checked ports inside Docker container, netstat tells that Docker resolver address only is used.

[jszwedko]

Just removed Prometheus exporter sink. It works like a charm. Checked ports inside Docker container, netstat tells that Docker resolver address only is used.

That is odd. I tried running your config in Docker and didn't see the same error (it correctly bound to 8080) so I don't think it is something Vector is doing.

[java-juggled-jazz]

That is odd. I tried running your config in Docker and didn't see the same error (it correctly bound to 8080) so I don't think it is something Vector is doing.

Prometheus found address vector:8080/metrics. Looks like Vector works. When I delete prometheus_exporter sink Prometheus cannot find this address. But I can't find Vector metrics starting with vector_ in Prometheus Expressions.

[jszwedko]

But I can't find Vector metrics starting with vector_ in Prometheus Expressions.

If you are using the config you posted above, you need to add the internal_metrics source and connect that to your prometheus_exporter sink.

[java-juggled-jazz]

If you are using the config you posted above, you need to add the internal_metrics source and connect that to your prometheus_exporter sink.

It works. Thanks.