Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

prometheus receiver: more info about scrape error #6900

Closed
morigs opened this issue Dec 17, 2021 · 6 comments
Closed

prometheus receiver: more info about scrape error #6900

morigs opened this issue Dec 17, 2021 · 6 comments
Labels
comp:prometheus Prometheus related issues

Comments

@morigs
Copy link
Contributor

morigs commented Dec 17, 2021

Is your feature request related to a problem? Please describe.
Currently, prometheusreceiver in case of error reports the following in the logs:

warn    internal/metricsbuilder.go:124  Failed to scrape Prometheus endpoint    {"kind": "receiver", "name": "prometheus", "scrape_timestamp": 1639773267689, "target_labels": "map[__name__:up]"}

It would be awesome to have more information about what really went wrong for debugging purposes.

Describe the solution you'd like
Perhaps it's possible to log http request errors. For example, socket hang up, timeout exceeded or response code 503.

@jpkrohling jpkrohling added the comp:prometheus Prometheus related issues label Dec 20, 2021
@jpkrohling
Copy link
Member

cc @Aneurysm9 @dashpole

@dashpole
Copy link
Contributor

See open-telemetry/opentelemetry-collector#2364 (comment). Scraping happens in the prometheus server code, and is a debug log line. You can enable debug logging with --log-level=DEBUG

@morigs
Copy link
Contributor Author

morigs commented Dec 20, 2021

@dashpole Thank you!
It's kinda strange that error is on the debug level

@morigs morigs closed this as completed Dec 20, 2021
@jinxin-fu
Copy link

See open-telemetry/opentelemetry-collector#2364 (comment). Scraping happens in the prometheus server code, and is a debug log line. You can enable debug logging with --log-level=DEBUG

Hi. I am a green hand in this project . I want to ask where to add the debug logging flag --log-level=DEBUG for image "otel/opentelemetry-collector-contrib". Looking forward to your reply Thanks.

@morigs
Copy link
Contributor Author

morigs commented Jan 24, 2022

@jinxin-fu usually, you specify arguments after image name, if you use docker. For exanple,

$ docker run -p 4317:4317 -it --rm -v $(pwd)/otel:/otel --name otel \
  otel/opentelemetry-collector-contrib:0.42.0 --config /otel/config.yaml --log-level=DEBUG

However, it looks like --log-level flag has been deprecated and removed in v0.38.0

From now on you should use config file instead. Use service.telemetry.logs.level

receivers:
  otlp:
    protocols:
      grpc:
      http:

processors:
  batch:

exporters:
  jaeger:
    endpoint: "host.docker.internal:14250"
    tls:
      insecure: true

service:
  telemetry:
    logs:
      level: DEBUG
  extensions: []
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [jaeger]

@jinxin-fu
Copy link

Thanks, I have change the otel-collector-conf with and works normally:

service:
  telemetry:
    logs:
      level: "debug"

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
comp:prometheus Prometheus related issues
Projects
None yet
Development

No branches or pull requests

4 participants