[BUG] Prometheus and OpenMetrics metrics collection always uses Basic auth

[svennis94]

We try to setup our software that has a prometheus endpoint to populate to datadog directly. For prometheus we have build in support for the oAuth token endpoint which works great. When looking at the pometheus / openmetrics configuration example I noticed DataDog having support for this as well. So I tried to set it up, in the example configuration it says the following for the auth_token parameter:

    ##   - type: oauth
    ##     url (required): The token endpoint.
    ##     client_id (required): The client identifier.
    ##     client_secret (required): The client secret.
    ##     basic_auth: Whether the provider expects credentials to be transmitted in
    ##                 an HTTP Basic Auth header. The default is: false
    ##     options: Mapping of additional options to pass to the provider, such as the audience
    ##              or the scope. For example:
    ##                 options:
    ##                   audience: https://example.com
    ##                   scope: read:example

It says basic_auth is by default false. But it sends the oauth request with the basic authentication header Authorization: Basic <base64_enc_client_id_and_client_secret>. This can be seen on every request coming from the datadog agent. Even when adding basic_auth: false to the configuration yaml it still uses basic authentication. I would've expected the client_id and client_secret in the body of the request.

Basic example config:

init_config:

instances:
  - openmetrics_endpoint: 'http://127.0.0.1/stats'
    namespace: 'test-application'
    metrics:
      - .+
    auth_token:
      reader:
        type: oauth
        url: 'http://127.0.0.1/oauth/token'
        client_id: test
        client_secret: test
        basic_auth: false
      writer:
        type: header
        name: Authorization
        value: 'Bearer <TOKEN>'

Agent Environment
Agent is running in docker with version:

datadog-agent        | 2024-09-11 12:43:47 UTC | PROCESS | INFO | (pkg/util/log/log.go:846 in func1) | running on platform: ubuntu
datadog-agent        | 2024-09-11 12:43:47 UTC | PROCESS | INFO | (pkg/util/log/log.go:846 in func1) | running version: 7.57.0

Describe what happened:
oAuth token endpoint is always requested with Basic authentication instead of client_id and client_secret in the x-form-urlencoded format.

Describe what you expected:
Expected to get the client_id and client_secret in the x-form-urlencoded data and the configuration basic_auth to influence whether or not to send the basic_auth.

Steps to reproduce the issue:
Use the above config towards a prometheus endpoint with an oAuth token endpoint.

Additional environment details (Operating System, Cloud provider, etc):
The only additional environment variable set to the datadog agent config is:

OAUTHLIB_INSECURE_TRANSPORT: "true"

Used to allow for local authentication to an internal IP.

[FlorentClarret]

Hi @svennis94 !

Thanks for opening this issue. I'm going to transfer it to integrations-core, the repository where the integrations live. I'll also ping the team about this issue.

[svennis94]

@iliakur I have looked into this issue a bit more and have followed the code path from the openmetrics integration to the base HTTP client in the DataDog integration datadog_checks_base/datadog_checks/base/utils/http.py:801 (AuthTokenOAuthReader). There I saw that the basic authentication header is only generated on the DataDog side if the basic_auth is set to true. But when running the DataDog agent with DD_LOG_LEVEL=debug, there is still the following message:

datadog-agent        | 2024-09-19 07:59:52 UTC | CORE | DEBUG | (pkg/collector/python/datadog_agent.go:146 in LogMessage) | - | (oauth2_session.py:338) | Encoding `client_id` "test" with `client_secret` as Basic auth credentials.

This indicates that somewhere else the basic auth header is being generated and added. As this is not part of the DataDog integrations repository.

When looking further I found that the Requests library for Python is being used for oAuth authentication. Taking a look at the fetch_token method in https://github.com/requests/requests-oauthlib/blob/master/requests_oauthlib/oauth2_session.py shows that the basic authentication header is added when the oAuth library does not get the include_client_id=true option in the arguments.

After reading this and adding the include_client_id: true to the options in the openmetrics yaml configuration, the basic authentication header is no longer present and the metrics are being collected in DataDog.

I think all that needs from DataDog's side is to document, that if you do not want to use the basic auth you also need to set the options with the include_client_id: true.

Working configuration:

init_config:

instances:
  - openmetrics_endpoint: 'http://127.0.0.1/stats'
    namespace: 'test-application'
    metrics:
      - .+
    auth_token:
      reader:
        type: oauth
        url: 'http://127.0.0.1/oauth/token'
        client_id: test
        client_secret: test
        options:
          include_client_id: true    # This will prevent the Basic authentication header and add the client_id and client_secret to the request body instead.
      writer:
        type: header
        name: Authorization
        value: 'Bearer <TOKEN>'