Remove every default header except User-Agent

[dabcoder]

What does this PR do?

GET requests do not need to include 'Content-Type' in headers (some application firewalls may block them), unless users specifically decide to use custom headers that include such 'Content-Type'.
The goal of this PR is to provide a ('http_method') argument to get different headers depending on the use case (+ ensure backward compatibility).

EDIT:

• remove all default headers but User-Agent.
• re-add 'Content-Type' for checks that use other http methods than GET (the http_check only).

Motivation

Customer reported issue.

Review checklist (to be filled by reviewers)

• PR title must be written as a CHANGELOG entry (see why)
• Files changes must correspond to the primary purpose of the PR as described in the title (small unrelated changes should have their own PR)
• PR must have changelog/ and integration/ labels attached
• Feature or bugfix must have tests
• Git history must be clean
• If PR adds a configuration option, it must be added to the configuration file.

[codecov]

Codecov Report

Merging #3644 into master will decrease coverage by 2.39%.
The diff coverage is 0%.

@@            Coverage Diff             @@
##           master    #3644      +/-   ##
==========================================
- Coverage   86.26%   83.86%    -2.4%     
==========================================
  Files         739       62     -677     
  Lines       38341     4693   -33648     
  Branches     4606      574    -4032     
==========================================
- Hits        33075     3936   -29139     
+ Misses       4000      636    -3364     
+ Partials     1266      121    -1145

[nmuesch]

Thanks for the PR!
Would it make more sense to just make the default here False -

integrations-core/http_check/datadog_checks/http_check/config.py

Line 38 in 943534a

default_headers = is_affirmative(instance.get("include_default_headers", True))

If the user provided this header, we shouldn't remove it

[ofek]

What @nmuesch said 🙂

[dabcoder]

Thanks for the PR!
Would it make more sense to just make the default here False -

integrations-core/http_check/datadog_checks/http_check/config.py

Line 38 in 943534a

default_headers = is_affirmative(instance.get("include_default_headers", True))
If the user provided this header, we shouldn't remove it

🤦‍♂️ I didn't even think of that... Good point.

[masci]

This does more than what's in the title, for example default headers contain User-Agent, do we add it back somehow?

[dabcoder]

Reverting to the original PR title. Editing the headers instead of removing content in individual checks.

[ofek]

@dabcoder We've decided to not ensure backward compatibility. Please remove every default header except User-Agent.

[ofek]

Thanks!