[OpenMetricsV2] Add an option to send sum and count information when using distribution metrics

[ofek]

Motivation

Resolves:

• [openmetrics] Offer option to send histograms as bucket counters AND distribution metrics #5883
• Add option to OpenMetricsBaseCheck to send distribution counts/sums #7889

[olivielpeau]

The count aggregation on the distribution metric in-app should be fully accurate AFAIK, is there any reason to send it separately?

I understand the sum aggregation on the distribution metric would not be fully accurate though, so it likely makes sense to send it separately as a DD count.

Side note: this will cause merge conflicts with #9276 cc @gh123man

[ofek]

@olivielpeau If we send one, we should send both imo to avoid confusion. The requests also desire both. WDYT?

[olivielpeau]

The main downside I can think of is that for openmetrics integrations that send metrics counted as custom metrics, this would send one more custom metric (count) per distribution bucket even though it's not strictly necessary.

[ofek]

The customers in the linked issue consider both to be useful actually, so I think this is worth it

[ofek]

Added a flag

[olivielpeau]

Haven't reviewed the code in details, but behavior LGTM :)

(the flag should probably be documented?)

[ofek]

Rebased to fix a bunch of ci jobs

[mfpierre]

LGTM, I agree that it shouldn't be the default behavior as the count is duplicate info from what we already have in the distribution metric, but this could be useful for people who really want the accurate sum of values (although percentiles on the distribution metrics could be more useful in an alerting context)

[codecov]

Codecov Report

Merging #9301 (d382a2f) into master (90532d9) will increase coverage by 0.00%.
The diff coverage is 88.88%.

Flag	Coverage Δ
activemq_xml	82.31% <ø> (ø)
amazon_msk	89.53% <ø> (ø)
ambari	86.98% <ø> (ø)
apache	94.90% <ø> (ø)
aspdotnet	93.87% <ø> (ø)
azure_iot_edge	82.01% <ø> (ø)
btrfs	82.91% <ø> (ø)
cacti	84.01% <ø> (ø)
cilium	85.84% <ø> (+1.88%)	⬆️
cisco_aci	95.88% <ø> (ø)
cloud_foundry_api	95.98% <ø> (+0.12%)	⬆️
couchbase	81.45% <ø> (ø)
eks_fargate	94.05% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...ts/openmetrics/test_transformers/test_histogram.py	100.00% <ø> (ø)
...se/checks/openmetrics/v2/transformers/histogram.py	94.73% <88.00%> (ø)
...dog_checks/base/checks/openmetrics/v2/transform.py	85.32% <100.00%> (ø)
zk/tests/conftest.py	92.22% <0.00%> (-2.23%)	⬇️
...adog_checks/cloud_foundry_api/cloud_foundry_api.py	93.70% <0.00%> (+0.34%)	⬆️
spark/datadog_checks/spark/spark.py	86.61% <0.00%> (+0.52%)	⬆️
cilium/tests/conftest.py	78.84% <0.00%> (+3.84%)	⬆️

[mhamrah]

We're sending OpenTelemetry Histograms via a prometheus exporter to Datadog via the agent and the openmetrics check. We know the agent sends these to datadog as distributions, and are seeing very odd counts for these distributions. Does this issue sound familiar, is this PR meant to address this issue on histograms -> distributions -> accurate counts?

TY. For ref, datadog support tickets 565850 and 567929 are related.