Scrape - configure auth header scheme

[dazinator]

Proposal

Allow Authorization header 'scheme' name and value to be configured.

Right now, I can either configure Basic authentication or Bearer authentication. Depending on which I configure, dictates the scheme name prefix in the auth header on the scrape request. For example prometheus can be configured to send either of these headers:

• Authorization: BASIC hwhhejfjjjejdnwje
• Authorization: BEARER hwhhejfjjjejdnwje

Each option essentialy places a constraint on the 'scheme name' portion of the authorization header, whilst allowing a static value / token to be passed as the credentials portion. This means I can configure a static value to be passed for the credentials only limited to "Basic" or "Bearer" schemes.

I have an application that uses a custom authentication scheme called API key authentication. This is also a static value that must be passed in the Authorization header. The difference is it requires a custom scheme name 'ApiKey`. I'd like prometheus to allow me to specify the scheme name, so that I am not limited to Basic or Bearer schemes and can use a custom scheme name.

Sure I could set up a custom proxy but this introduces another moving part and more complexity in the landscape so I am thinking this should be weighed against alternatives. Being able to configure the scheme name (as opposed to being restricted to Basic and Bearer) would provide for a more general purpose implementation that makes prometheus more flexible and covers more use cases without much additional complexity that I can tell..

For example right now the following two auth headers are configurable but the third is not.

• Authorization: BASIC hwhhejfjjjejdnwje
• Authorization: BEARER hwhhejfjjjejdnwje
• Authorization: API-KEY hwhhejfjjjejdnwje

I beleive that allowing an authorization header to be configured - by specifying its components:

• scheme name
• value

.. would make for a more flexibile option to cover more use cases.

“Nice to have” is not a good use case. :)

We have a .net core web api we'd like to scrape that doesn't support basic auth and only supports short lived access tokens using the BEARER authentication scheme. It does support a custom Authorization scheme for api key authentication (which is a long lived value) however prometheus is currently not able to be configured to leverage that.

[roidelapluie]

Hello,

It seems that API-KEY is not valid in HTTP, here are the supported authentication types: https://www.iana.org/assignments/http-authschemes/http-authschemes.xhtml

I do not think that Prometheus should promote non-official HTTP patterns.

As an alternative, you could run a reverse proxy before the application or a http proxy at the client side, WDYT?

[dazinator]

Hello,

It seems that API-KEY is not valid in HTTP, here are the supported authentication types: https://www.iana.org/assignments/http-authschemes/http-authschemes.xhtml

I do not think that Prometheus should promote non-official HTTP patterns.

As an alternative, you could run a reverse proxy before the application or a http proxy at the client side, WDYT?

Firstly, thank you for your feedback on this suggestion.

I don't agree regarding the argument about promoting non standards.. because I am not so sure its non standard or would be promoted!

1. The iana assignments page you linked looks to be a registry of reserved scheme names - it doesn't state that custom scheme names can't be used as part of the http protocol. (I've commentef below with a link to the RFC showing the http protocol does allow for custom scheme names).

2. I don't think prometheus would be "promoting" a non standard by being able to be configured to work with existing apis that already implement custom schemes. The auth scheme / mechanism is already established, the boat has sailed in terms of promotion of that concept. Anything that makes prometheus more flexible and able to authenticate with an existing api increases its chances of being adopted into the landscape, and so should be worth considering if its not adding any great complexity.

Custom schemes are used in the wild. Here is my show case for this, its quite a major player on the interwebs:

https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-auth-using-authorization-header.html

If Amazon are doing it just how non standard can it be?
Look at how mozzila define this header:

https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Authorization

From a development perspective, im not aware of any frameworks that are used to develop web api's that impose a limitation on the scheme names that can be used I this header. In asp.net core for example authentication scheme names are just strings and there is no restriction in the http protocol implementation in asp.net core that says custom scheme names are a violation. There are all kinds of custom authentication middlewares that people create aside from the well known ones like Basic and Bearer. API Key is quite common imho. I imagine nodejs will be the same concept. This inevitably means that developers will be finding use cases for custom scheme names (like I have with my api key scenario).

[dazinator]

Custom scheme names appear perfectly legitimate as per RFC 2617

Also they are actively promoted in various places like this popular resource! https://stackoverflow.com/questions/7802116/custom-http-authorization-header

P.s I understand that at some point prometheus has to draw the line in terms of how it's able to interact and authenticate with an api and hence I understand the pushback to offload this concern to a proxy.. however in this case I don't think this pushback is warranted - sending a http authorization header with some fixed / static values is about as simple and as standards compliant as you can get and should be 100% supported by prometheus if it wants to support a rudimentary attempt at authentication with a api over http via configuration to satisfy the broadest and most general cases.

I think I've made all the cases I can think of now ;-)

P.s thank you for working on and contributing to such an awesome product.

[roidelapluie]

Thank you. This has come up in the past already: prometheus/prometheus#5107 but it is not a frequent request.

I am worried that you signal sigv4, as sigv4 is not just adding a static header. I think that adding a static header is the easy part here. But sigv4 needs to know the entire request to sign it. And if we accept those non-common options, we could have more and more requests for those non-common per-request bearer tokens.

Currently, we have multiple options to change the header: basic_auth, bearer_token, bearer_token file.. and they are mutually exclusive.

I would like to avoid adding a new set of options, so should we change bearer_token with:

authorization_header_type: Bearer
authorization_header_credentials:
authorization_header_credentials_file:

Make a compatibility function, and remove them later, in Prometheus 3.x ? Would that make sense from an end user perspective?

[dazinator]

I am worried that you signal sigv4, as sigv4 is not just adding a static header.

Sorry I am a lowly .net developer - I have no idea what sigv4 is, if I have linked to it - that was coincidence.

I was imagining for this feature that in config we'd be able to set something like (im not trying to suggest actually config keys here just illustration):

• Auth: Custom
• SchemeName: apikey
• Value: hdjwuiwikdkkwk

If this was specified this would equate to a header being sent:

Authorization: apikey hdjwuiwikdkkwk

One side effect of this is that, as this option is more general purpose, it could be used to also configure basic or bearer toke authentication as well - where all you want to do is pass a static value. For basic authentication for example you'd have to configure the value as the properly encoded username and password and the scheme name as basic. For bearer the scheme name would be bearer and the value would be whatever the token is that needs to be passed.

This would just allow purely static values to be passed accross in an auth header with the scheme specified. No additional processing or signing or anything like that.

[roidelapluie]

I have opened the following pull request: #272 to gather feedback from other team members.

[dazinator]

I see the PR has now been merged - thank you for implementing this @roidelapluie

[roidelapluie]

#274 will still be needed before I can push it down to prometheus

[dazinator]

@roidelapluie looks like #274 was merged - can this now be considered closed?
I am new to Prometheus, is there a nightly release I can get from somewhere that would have these changes so I can try it out?

[roidelapluie]

You could test with the docker image prom/prometheus:main or take a binary from ci https://app.circleci.com/pipelines/github/prometheus/prometheus/11049/workflows/c4a4c2d1-2288-429a-afa2-9ad06cb64578/jobs/48406/artifacts

Note: for testing only.

It would be nice to get feedback if it works as expected.

[qmonitoring]

You could test with the docker image prom/prometheus:main or take a binary from ci https://app.circleci.com/pipelines/github/prometheus/prometheus/11049/workflows/c4a4c2d1-2288-429a-afa2-9ad06cb64578/jobs/48406/artifacts

Note: for testing only.

It would be nice to get feedback if it works as expected.

@roidelapluie thank you for your work!

Could you please clarify how to use APIKEY token?

I am trying to scrape an endpoint with an API KEY аuthorization.

prom/prometheus:main is up and running:

docker run -d --network=host --name=prometheus -v /path/prometheus.yml:/etc/prometheus/prometheus.yml prom/prometheus:main --config.file=/etc/prometheus/prometheus.yml --web.listen-address=:9095

with the following prometheus.yml configuration file:

global:
  evaluation_interval: 15s
  scrape_interval: 15s
  scrape_timeout: 10s
  external_labels:
    environment: prometheus_in_docker
	
scrape_configs:

  - job_name: json_exporter_self
    static_configs:
      - targets:
        - <host>:7979 ## Location of the json exporter's real <hostname>:<port>

  ## gather the metrics from third party json sources, via the json exporter
  - job_name: json_exporter_data
    metrics_path: /probe
    authorization:
      credentials: '{my_secret_token}'
      type: apikey	
    static_configs:
      - targets:
        - http://<endpoint_host>:<port>/api/v1/data
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: <host>:7979 ## Location of the json exporter's real <hostname>:<port>

for jsons exporting I use the prometheus-community/json_exporter
with the following config.yml configuration file:

metrics:
- name: example_metric
  path: "{ .data }"
  help: Example 
  labels:
    environment: testdata # static label

Self scraping works fine (http://:7979/metrics), but a json scraping doesn't work (http://:7979/probe), the error is the following:

Failed to fetch JSON response. TARGET: http://<endpoint_host>:/api/v1/data, ERROR: 401 Unauthorized

The endpoint itself is available:
curl -X GET 'http://<endpoint_host>:<port>/api/v1/data' -H 'Authorization: {my_secret_token}'

Is this a misconfiguration on my side? If yes, how can I fix this?

[roidelapluie]

The json_exporter does not pass the authorization to the next target. You should open an issue to support this kind of auth in the json_exporter repository. Maybe we could "pass" the authorization header to the target.