Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unable to scrape HTTP/2-only endpoint #7237

Closed
Daniel15 opened this issue May 11, 2020 · 6 comments · Fixed by #7258
Closed

Unable to scrape HTTP/2-only endpoint #7237

Daniel15 opened this issue May 11, 2020 · 6 comments · Fixed by #7258

Comments

@Daniel15
Copy link
Contributor

What did you do?
Added an endpoint to my Prometheus config that only responds via HTTP/2. It's a gRPC service that I've also added a /metrics HTTP endpoint to, and gRPC services only speak HTTP/2 by default.

What did you expect to see?
Data is properly scraped.

What did you see instead? Under which circumstances?
Prometheus UI says:

Get https://xxxxxxx/metrics: EOF

Server says:

fail: Microsoft.AspNetCore.Server.Kestrel[0]
HTTP/2 over TLS was not negotiated on an HTTP/2-only endpoint.

Environment

  • System information:
Debian "testing"
Linux 5.5.0-2-cloud-amd64 x86_64
  • Prometheus version:
prometheus, version 2.15.2+ds (branch: debian/sid, revision: 2.15.2+ds-2)
  build user:       [email protected]
  build date:       20200113-10:10:54
  go version:       go1.13.6
@roidelapluie
Copy link
Member

See prometheus/common#233

@roidelapluie roidelapluie self-assigned this May 11, 2020
roidelapluie added a commit to roidelapluie/prometheus that referenced this issue May 13, 2020
Enable HTTP/2.
Fixes prometheus#5938
Fixes prometheus#7237

Add support for "0" duration.

Signed-off-by: Julien Pivotto <[email protected]>
roidelapluie added a commit that referenced this issue May 14, 2020
Enable HTTP/2.
Fixes #5938
Fixes #7237

Add support for "0" duration.

Signed-off-by: Julien Pivotto <[email protected]>
kakkoyun pushed a commit to kakkoyun/prometheus that referenced this issue Jun 8, 2020
Enable HTTP/2.
Fixes prometheus#5938
Fixes prometheus#7237

Add support for "0" duration.

Signed-off-by: Julien Pivotto <[email protected]>
@Mario-Hofstaetter
Copy link

Has this been resolved yet?
I am getting the very same error using Prometheus 2.23.0 and ASP.NET Core Kestrel GRPC Service.

Prometheus UI: Get "https://host.docker.internal:5011/metrics?format=prometheus": EOF
Kestrel: [2020-12-03 23:04:58Z ERR] Microsoft.AspNetCore.Server.Kestrel HTTP/2 over TLS was not negotiated on an HTTP/2-only endpoint.

ARGH, I guess it's because of this: prometheus/common#261

@roidelapluie
Copy link
Member

More like #7588 and #8143

@Daniel15
Copy link
Contributor Author

Daniel15 commented Dec 3, 2020

@Mario-Hofstaetter I don't think it's fully fixed yet. For now you'll have to enable HTTP/1.x in your ASP.NET service too, for example by adding this in appsettings.json:

  "Kestrel": {
    "EndpointDefaults": {
      // Temporarily enabling HTTP/1 due to https://github.com/prometheus/prometheus/issues/7237
      "Protocols": "Http1AndHttp2"
    }
  }

This is what I did in my service (https://github.com/Daniel15/dnstools/blob/master/src/DnsTools.Worker/appsettings.json#L9-L14) to allow Prometheus to connect.

@roidelapluie
Copy link
Member

We are looking to see if the Golang HTTP/2 issues are fixed or not, any help is welcome.

@valdemarpavesi
Copy link

We are looking to see if the Golang HTTP/2 issues are fixed or not, any help is welcome.

there is an update here:
#5938

@prometheus prometheus locked as resolved and limited conversation to collaborators Nov 13, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants