feat: retryable http client for datadog metric provider

[kaldorn]

Adds a retryable http.client that automatically handles 429 rate limit responses with exponential backoff.

Not 100% sure this is the best approach for this, open to suggestions.

related to: #2549
closes: #2578

Signed-off-by: kaldorn [[email protected]]

Checklist:

• Either (a) I've created an enhancement proposal and discussed it with the community, (b) this is a bug fix, or (c) this is a chore.
• The title of the PR is (a) conventional with a list of types and scopes found here, (b) states what changed, and (c) suffixes the related issues number. E.g. "fix(controller): Updates such and such. Fixes #1234".
• I've signed my commits with DCO
• I have written unit and/or e2e tests for my change. PRs without these are unlikely to be merged.
• My builds are green. Try syncing with master if they are not.
• My organization is added to USERS.md.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 81.55%. Comparing base (535f244) to head (ec86735).
Report is 634 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #2580      +/-   ##
==========================================
+ Coverage   81.52%   81.55%   +0.02%     
==========================================
  Files         131      131              
  Lines       19587    19596       +9     
==========================================
+ Hits        15969    15981      +12     
+ Misses       2797     2795       -2     
+ Partials      821      820       -1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[github-actions]

Go Published Test Results

1 860 tests   1 860 ✔️  2m 34s ⏱️
   113 suites         0 💤
       1 files           0 ❌

Results for commit ec86735.

♻️ This comment has been updated with latest results.

[github-actions]

E2E Tests Published Test Results

    2 files      2 suites   1h 39m 6s ⏱️
  96 tests   89 ✔️ 3 💤 4 ❌
196 runs  186 ✔️ 6 💤 4 ❌

For more details on these failures, see this check.

Results for commit ec86735.

♻️ This comment has been updated with latest results.

[zachaller]

@kaldorn can I get you to run go mod tidy

[kaldorn]

@zachaller done, i apologize for the delay.

[sonarqubecloud]

Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
1 Code Smell

No Coverage information
39.7% Duplication

[zachaller]

@kaldorn I merged the pr to add datadog v2 to master. I am ok with also having retries if you want to add it will require a rebase and addition to both v1 and v2, if you do not and v2 is enough for your rate limiting/errors issues etc we can close this.

[github-actions]

This PR is stale because it has been open 90 days with no activity.