Make HTTP request timeout configurable in prometheus checks #1790

antoinepouille · 2018-06-22T13:17:48Z

What does this PR do?

Adding a prometheus_timeout option available in all prometheus checks, so that it can be set by integration and changed easily by the user.

Motivation

Issues with some checks taking too long to execute (e.g. kubernetes_state)

Review checklist

PR has a meaningful title or PR has the no-changelog label attached
Feature or bugfix has tests
Git history is clean
If PR impacts documentation, docs team has been notified or an issue has been opened on the documentation repo

Additional Notes

Leaving kubelet, gitlab, gitlab_runner and istio integrations aside for now (at least)

rishabh

Just a few things :)

rishabh · 2018-07-02T02:18:25Z

datadog_checks_base/datadog_checks/checks/prometheus/mixins.py

@@ -622,3 +625,13 @@ def _submit_gauges_from_histogram(self, name, metric, send_histograms_buckets=Tr

    def _is_value_valid(self, val):
        return not (isnan(val) or isinf(val))
+
+    def set_prometheus_timeout(self, instance, default_value=10):


Do we need to validate the user's configuration? Requests will automatically trigger a Timeout exception, alerting the user that they need to correct that setting.

It's also possible for the user to set a negative timeout; consider the case when default_instance has a negative timeout and the timeout is not set in the supplied instance.

If requests is handling the negative timeout for us, indeed, no big need to check it.
Removed it 👍

👍
Just tested and I get a ValueError if timeout is less than 0.

Also, I think we can simplify it even further by doing something like this:

def set_prometheus_time(...): self.prometheus_timeout = instance.get('prometheus_timeout', default_value)

rishabh · 2018-07-02T02:28:03Z

datadog_checks_base/tests/test_prometheus.py

@@ -1601,3 +1601,30 @@ def test_health_service_check_failing():
        PrometheusCheck.CRITICAL,
        tags=["endpoint:http://fake.endpoint:10055/metrics"]
    )
+
+def test_set_prometheus_timeout():


We should also be testing for when default_instance has a negative timeout.

👍 Added some tests relative to that, both for PrometheusCheck and GenericPrometheusCheck

Ah, since we're removing the validation part, we don't need to check for negative values, just that the same value that we set is the same as what we get.

Sure, I was a bit confused by your two remarks, so I did both ^^

rishabh

I like that we're not validating anymore which means we can clean up the tests 👍

rishabh · 2018-07-02T15:48:05Z

datadog_checks_base/datadog_checks/checks/prometheus/mixins.py

@@ -622,3 +625,13 @@ def _submit_gauges_from_histogram(self, name, metric, send_histograms_buckets=Tr

    def _is_value_valid(self, val):
        return not (isnan(val) or isinf(val))
+
+    def set_prometheus_timeout(self, instance, default_value=10):


👍
Just tested and I get a ValueError if timeout is less than 0.

Also, I think we can simplify it even further by doing something like this:

def set_prometheus_time(...): self.prometheus_timeout = instance.get('prometheus_timeout', default_value)

rishabh · 2018-07-02T15:51:14Z

datadog_checks_base/tests/test_prometheus.py

@@ -1601,3 +1601,30 @@ def test_health_service_check_failing():
        PrometheusCheck.CRITICAL,
        tags=["endpoint:http://fake.endpoint:10055/metrics"]
    )
+
+def test_set_prometheus_timeout():


Ah, since we're removing the validation part, we don't need to check for negative values, just that the same value that we set is the same as what we get.

rishabh · 2018-07-02T15:52:38Z

datadog_checks_base/tests/test_prometheus_base_check.py

+        'default_namespace': {
+            'prometheus_url': endpoint,
+            'metrics': [{"test_rate": "test.rate"}],
+            'prometheus_timeout': -1,


Let's make it a positive value just to make it cleaner 👍

Thanks for the feedback! 👍

…quests anyway

rishabh · 2018-07-03T14:32:23Z

datadog_checks_base/datadog_checks/checks/prometheus/base_check.py

@@ -159,12 +159,14 @@ def get_scraper(self, instance):
        scraper.extra_headers.update(instance.get("extra_headers", {}))
        # For simple values instance settings overrides optional defaults
        scraper.prometheus_metrics_prefix = instance.get("prometheus_metrics_prefix", default_instance.get("prometheus_metrics_prefix", ''))
-        scraper.label_to_hostname = instance.get("label_to_hostname", default_instance.get("prometheus_url", ""))
+        scraper.label_to_hostname = instance.get("label_to_hostname", default_instance.get("label_to_hostname", ""))


We should fix the tests so we actually catch these issues.

Good practice indeed! I added it, but leaving the other config params untested. Let me know if you think I should add them in the same act :-)

… value

rishabh

LGTM :)

* Prometheus checks: Adding prometheus_timeout options * Adding timeout to GenericPrometheusCheck integrations * better code share for timeout definition * Even better * Adding tests * Adding timeout option to some other PrometheusChecks * Removed changes to gitlab integration * removing gitlab_runner changes * Removing validation for timeout values, leaving them to requests * Adding some other tests for negative values + GenericPrometheusCheck * Simplify logic * Do not check for negative values, they would be set and handled by requests anyway * Adding test for label_to_hostname default instance + changing default value

Related to: #1790

antoinepouille requested review from a team as code owners June 22, 2018 13:17

antoinepouille force-pushed the antoine/make-prometheus-check-timeout-configurable branch 4 times, most recently from 2b1c18c to 0e74b60 Compare June 22, 2018 14:46

antoinepouille added containers integration/prometheus changelog/Added integration/kubernetes_state kind/feature-request labels Jun 22, 2018

antoinepouille added this to the 6.3.1 milestone Jun 22, 2018

masci added integration/kube_dns integration/kube_proxy integration/linkerd and removed kind/enhancement kind/feature-request labels Jun 24, 2018

masci changed the title ~~[prometheus check] make request timeout configurable~~ Make HTTP request timeout configurable Jun 24, 2018

JulienBalestra modified the milestones: 6.3.1, 6.4 Jun 25, 2018

antoinepouille changed the title ~~Make HTTP request timeout configurable~~ Make HTTP request timeout configurable in prometheus checks Jun 25, 2018

masci removed this from the 6.4 milestone Jun 25, 2018

masci mentioned this pull request Jun 26, 2018

kube-state-metrics timing out with agent v6.1.0-6.1.4 / v5.23.0 #1346

Closed

Antoine Pouille added 6 commits June 26, 2018 17:48

Prometheus checks: Adding prometheus_timeout options

14b14d1

Adding timeout to GenericPrometheusCheck integrations

06fe235

better code share for timeout definition

570125b

Even better

92b932b

Adding tests

08db4d9

Adding timeout option to some other PrometheusChecks

f5c67a0

Antoine Pouille added 2 commits June 26, 2018 17:48

Removed changes to gitlab integration

3ff6a96

removing gitlab_runner changes

0edfa1c

antoinepouille force-pushed the antoine/make-prometheus-check-timeout-configurable branch from 7f4c3e9 to 0edfa1c Compare June 26, 2018 15:49

rishabh self-assigned this Jun 29, 2018

rishabh requested changes Jul 2, 2018

View reviewed changes

Antoine Pouille added 2 commits July 2, 2018 15:51

Removing validation for timeout values, leaving them to requests

e5f7d3d

Adding some other tests for negative values + GenericPrometheusCheck

7ba5302

rishabh requested changes Jul 2, 2018

View reviewed changes

Antoine Pouille added 2 commits July 3, 2018 10:59

Simplify logic

1a7c169

Do not check for negative values, they would be set and handled by re…

712ef4d

…quests anyway

rishabh reviewed Jul 3, 2018

View reviewed changes

Adding test for label_to_hostname default instance + changing default…

09b8676

… value

rishabh approved these changes Jul 3, 2018

View reviewed changes

antoinepouille merged commit b026234 into master Jul 3, 2018

antoinepouille deleted the antoine/make-prometheus-check-timeout-configurable branch July 3, 2018 15:32

antoinepouille restored the antoine/make-prometheus-check-timeout-configurable branch July 3, 2018 15:36

antoinepouille deleted the antoine/make-prometheus-check-timeout-configurable branch July 5, 2018 15:58

antoinepouille added this to the 6.4.0 milestone Jul 16, 2018

CharlyF added the test-card-created label Jul 23, 2018

ian28223 added a commit that referenced this pull request Sep 10, 2018

Adding timeout to kubernetes_state integration

474f98b

Related to: #1790

ian28223 mentioned this pull request Sep 10, 2018

Adding timeout to kubernetes_state integration #2198

Merged

4 tasks

hkaj pushed a commit that referenced this pull request Nov 22, 2018

Adding timeout to kubernetes_state integration (#2198)

583f3c3

Related to: #1790

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make HTTP request timeout configurable in prometheus checks #1790

Make HTTP request timeout configurable in prometheus checks #1790

antoinepouille commented Jun 22, 2018 •

edited by masci

Loading

rishabh left a comment

rishabh Jul 2, 2018

rishabh Jul 2, 2018

antoinepouille Jul 2, 2018 •

edited

Loading

rishabh Jul 2, 2018

rishabh Jul 2, 2018

antoinepouille Jul 2, 2018

rishabh Jul 2, 2018

antoinepouille Jul 3, 2018

rishabh left a comment

rishabh Jul 2, 2018

rishabh Jul 2, 2018

rishabh Jul 2, 2018

antoinepouille Jul 3, 2018

rishabh Jul 3, 2018

antoinepouille Jul 3, 2018

rishabh left a comment

Make HTTP request timeout configurable in prometheus checks #1790

Make HTTP request timeout configurable in prometheus checks #1790

Conversation

antoinepouille commented Jun 22, 2018 • edited by masci Loading

What does this PR do?

Motivation

Review checklist

Additional Notes

rishabh left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

antoinepouille Jul 2, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rishabh left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rishabh left a comment

Choose a reason for hiding this comment

antoinepouille commented Jun 22, 2018 •

edited by masci

Loading

antoinepouille Jul 2, 2018 •

edited

Loading