feat: improve configurability of prometheus metrics

[claudio-benfatto]

Related issue

• Prometheus metrics cardinality for the /decisions endpoint #446 discussed with @aeneasr

Proposed changes

This PR introduces a new boolean configuration parameter collapse_request_paths in the prometheus configuration stanza of the oathkeeper.yml file:

serve:
  prometheus:
    port: 9000
    host: localhost
    metrics_path: /metrics
    collapse_request_paths: true

The parameter is optional and if not set it defaults to true.

When set to true it modifies the behaviour of the metrics Middleware so that for all the request metrics that contain the request label (which is equal to the request context path) its value is collapsed to only the first element of the segment. Eg.

curl http://localhost:4456/decisions/service1/users
curl http://localhost:4456/decisions/service1/topics

will result in the following request metrics (trimmed for reading convenience):

# TYPE ory_oathkeeper_request_duration_seconds histogram
ory_oathkeeper_request_duration_seconds_count{method="GET",request="/decisions",service="oathkeeper-api",status_code="404"} 2
# HELP ory_oathkeeper_requests_total Total number of requests
# TYPE ory_oathkeeper_requests_total counter
ory_oathkeeper_requests_total{method="GET",request="/decisions",service="oathkeeper-api",status_code="404"} 2

When collapse_request_paths is set to false the metrics will keep the path previous fine granularity:

# HELP ory_oathkeeper_request_duration_seconds Time spent serving requests.
# TYPE ory_oathkeeper_request_duration_seconds histogram
ory_oathkeeper_request_duration_seconds_count{method="GET",request="/decisions/service1/topics",service="oathkeeper-api",status_code="404"} 1
ory_oathkeeper_request_duration_seconds_count{method="GET",request="/decisions/service1/users",service="oathkeeper-api",status_code="404"} 1
# HELP ory_oathkeeper_requests_total Total number of requests
# TYPE ory_oathkeeper_requests_total counter
ory_oathkeeper_requests_total{method="GET",request="/decisions/service1/topics",service="oathkeeper-api",status_code="404"} 1
ory_oathkeeper_requests_total{method="GET",request="/decisions/service1/users",service="oathkeeper-api",status_code="404"} 1

Checklist

• I have read the contributing guidelines.
• I have read the security policy.
• I confirm that this pull request does not address a security
  vulnerability. If this pull request addresses a security. vulnerability, I
  confirm that I got green light (please contact
  [email protected]) from the maintainers to push
  the changes.
• I have added tests that prove my fix is effective or that my feature
  works.
• I have added or changed the documentation.

Further comments

I'm happy to modify any logic or naming I somehow arbitrarily chosen!

[FredrikAppelros]

Is there anything missing from this PR that blocks it from being reviewed? Anything we can do to help speed it up?

[aeneasr]

No, my bad, I just completely forgot to review it, sorry!

[aeneasr]

Thank you for your hard work!