feat: datadog evaluation provider support

[sudiptob2]

Fixes #554

[sudiptob2]

I am hoping to get your opinion on DataDog APIs. I'm still getting the hang of things and would appreciate your input on what I've learned so far and where to go next. As a newcomer to DataDog, I'm eager to soak up as much knowledge as possible, so any assistance you can provide would be awesome.

Progress

• I have looked into the dyantrace.go and prometheus.go files and tried to understand EvaluateQuery function.
• Similar to the above implementation the goal datadog.go is to fetch the SLIs from the DataDog API.
• Through this Datadog API we can get the SLOs from datadog.
• From SLO I think we can get SLIs somehow, or there might be other way. But Datadog doesn’t have any API for fetching SLI or something like that. (Need help)
• We can also limit the number of SLO using Limit option.
• If we can determine the SLI value then we can return the value from the EvaluateQuery. (Need help on extracting SLI values from)
• I have created a draft here (experimental)

Ques

• what happens when we have multiple SLOs, how do we consolidate them in a single value?

Thank you guys for your continued support ✅

[sudiptob2]

If you look at the operator/controllers/common/providers/datadog/example_resp.json we have the following data

"query": {
     "denominator": "sum:httpservice.hits{!code:3xx}.as_count()",
     "numerator": "sum:httpservice.hits{code:2xx}.as_count()"
 },

This denominator and numerator is set while creating the SLO. I think we can calculate the SLI values from this using the below formula

(sum:httpservice.hits{code:2xx}.as_count()) / (sum:httpservice.hits{!code:3xx}.as_count())

if this is correct, then I need some help and suggestion on how I can fetch these values from Datadog API using the DD go client. Really appreciate your support from the community.

Edit: I found this API and and studying this code from keptn-sandbox/datadog-service

[codecov]

Codecov Report

Merging #869 (89d8dff) into main (8c4ba83) will decrease coverage by 0.23%.
The diff coverage is 0.00%.

❗ Current head 89d8dff differs from pull request most recent head 97e00a6. Consider uploading reports for the commit 97e00a6 to get more accurate results

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #869      +/-   ##
==========================================
- Coverage   58.60%   58.38%   -0.23%     
==========================================
  Files          97       98       +1     
  Lines        7552     7574      +22     
==========================================
- Hits         4426     4422       -4     
- Misses       2935     2960      +25     
- Partials      191      192       +1     

Impacted Files	Coverage Δ
operator/controllers/common/providers/common.go	100.00% <ø> (ø)
...or/controllers/common/providers/datadog/datadog.go	0.00% <0.00%> (ø)
operator/controllers/common/providers/provider.go	66.66% <0.00%> (-13.34%)	⬇️
...lers/lifecycle/keptnworkloadinstance/controller.go	81.16% <0.00%> (-1.80%)	⬇️

Flag	Coverage Δ
component-tests	41.25% <0.00%> (-0.96%)	⬇️
klt-cert-manager	67.50% <ø> (ø)
scheduler	21.17% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

[sonarqubecloud]

Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
2 Code Smells

No Coverage information
0.0% Duplication

[vadasambar]

@sudiptob2 I am the maintainer of keptn/datadog-service. I am still going through your comments (thank you for all the details). I am working on another project right now. I will try to get to your queries around datadog API client as early as I can (maybe we can schedule a call and get each other up to speed around this; I don't have a good understanding of adding provider to lifecycle-toolkit).

I was supposed to look at this originally but I am happy to see a draft PR and all the effort you are putting into this.

[sudiptob2]

@vadasambar really appreciate your support 🚀

I looked into your implementation and made some significant improvements in the draft. My previous comments are a little bit outdated compared to the current draft. However, those comments I think give you a summary of my progress.
I am still very beginner in all of these and really need your help and support.

[sudiptob2]

Due to a breaking change I am closing this PR and opened #948