Add external_host_tags wrapper to checks_base #3316

nmuesch · 2019-03-18T14:43:59Z

What does this PR do?

Adds a wrapper to the Datadog Agent's (A6's) set_external_tags function.
This wrapper makes sure the tags are in the expected format (depending on PY2/3) when being sent to the Datadog Agent.

Imports datadog_agent inside the function because it means that we aren't on A5 if we're using the AgentCheck class from here.

Having this allows us to remove some tag normalization from the specific checks. For example, vsphere uses set_external_tags and has to attempt to normalize tags in specific formats depending on the Agent being used (Py2/3) Having this wrapper allows us to let the AgentCheck class do this directly and implicitly, making it less error prone for future checks.

Also added a test

Motivation

Attempting to port a check such as vsphere was proving difficult, as the tags needed to be in different formats depending on whether the check was running in Py 2 or Py 3.

Additional Notes

Anything else we should know when reviewing?

Review checklist (to be filled by reviewers)

PR title must be written as a CHANGELOG entry (see why)
Files changes must correspond to the primary purpose of the PR as described in the title (small unrelated changes should have their own PR)
PR must have changelog/ and integration/ labels attached
Feature or bugfix must have tests
Git history must be clean
If PR adds a configuration option, it must be added to the configuration file.

zippolyte

LGTM

ofek

🔥

datadog_checks_base/datadog_checks/base/checks/base.py

codecov · 2019-03-18T15:03:34Z

Codecov Report

Merging #3316 into master will increase coverage by 4.71%.
The diff coverage is 87.5%.

@@            Coverage Diff             @@
##           master    #3316      +/-   ##
==========================================
+ Coverage   79.54%   84.25%   +4.71%     
==========================================
  Files         152       60      -92     
  Lines        7669     4661    -3008     
  Branches      929      569     -360     
==========================================
- Hits         6100     3927    -2173     
+ Misses       1346      613     -733     
+ Partials      223      121     -102

zippolyte

Actually, not good !! set_external_tags expects a list of tuple like this: ( <hostname>, {<source_type>: tags})

ofek · 2019-03-18T18:24:31Z

datadog_checks_base/datadog_checks/base/checks/base.py

@@ -290,6 +290,19 @@ def service_metadata(self, meta_name, value):
    def check(self, instance):
        raise NotImplementedError

+    def set_external_tags(self, external_tags):
+        from datadog_agent import set_external_tags as _set_external_tags


Why the lazy import?

I figured that we wouldn't need the try/catch at the top of the file if we imported directly in this function. If you're using the AgentCheck class(es) it means you're running on A6, so its gauranteed to be there.

gzussa

Looks good overall 😄

gzussa · 2019-03-19T10:43:33Z

datadog_checks_base/tests/test_agent_check.py

@@ -218,6 +220,21 @@ def test_none_value(self):
        normalized_tags = check._normalize_tags_type(tags, None)
        assert normalized_tags == ['tag:foo']

+    def test_external_host_tag_normalization(self):
+        """
+        Tests that th external_host_tag modifies in place the list of tags in the provided object


🤦‍♂️

gzussa · 2019-03-19T10:44:10Z

datadog_checks_base/datadog_checks/base/checks/base.py

@@ -701,6 +714,19 @@ def convert_to_underscore_separated(self, name):
        metric_name = self.METRIC_REPLACEMENT.sub(br'_', metric_name)
        return self.DOT_UNDERSCORE_CLEANUP.sub(br'.', metric_name).strip(b'_')

+    def set_external_tags(self, external_tags):
+        from datadog_agent import set_external_tags as _set_external_tags
+        # Example of external_tags format


👍 for this comments!!

gzussa · 2019-03-19T10:45:19Z

datadog_checks_base/datadog_checks/base/checks/base.py

+        #     ('hostname2', {'src2_name': ['test2:t3']})
+        # ]
+        for x in external_tags:
+            for src_name, tags in iteritems(x[1]):


Should we validate the structure of x just in case it is not set properly?

I'll wrap in a try/catch around an indexError, what do you think?

Actually, discussed this with @zippolyte The Agent currently emits an error if the format of these tags isn't correct. In that spirit, I've set this to catch an IndexError and log an exception (and then raise that exception) This keeps the same functionality that the Agent was currently performing. (Agent code that does validation - https://github.com/DataDog/datadog-agent/blob/master/pkg/collector/py/datadog_agent.c#L147)

…re into nick/base_ensure

datadog_checks_base/datadog_checks/base/checks/base.py

datadog_checks_base/tests/test_agent_check.py

masci

👍

olivielpeau

one nit, feel free to merge once addressed!

In general, exposing wrapper functions to checks (instead of the builtin datadog_agent module directly) sounds more flexible to me as well 👍

olivielpeau · 2019-03-21T10:53:50Z

datadog_checks_base/tests/test_agent_check.py

@@ -3,6 +3,8 @@
 # (C) Datadog, Inc. 2018
 # All rights reserved
 # Licensed under a 3-clause BSD style license (see LICENSE)
+import sys


is this import needed? (don't see it used anywhere)

Surprised a linter didn't catch this 🤔

It was originally there when I was doing more mocking than necessary, will remove thanks for catching!

nmuesch added 3 commits March 18, 2019 15:12

Move set_external to AgentCheck function

0959bf4

remove vsphere and add more None

6b12e92

remove old line

69ef1df

nmuesch requested review from a team as code owners March 18, 2019 14:43

nmuesch added integration/datadog_checks_base changelog/Added labels Mar 18, 2019

zippolyte approved these changes Mar 18, 2019

View reviewed changes

ofek requested changes Mar 18, 2019

View reviewed changes

datadog_checks_base/datadog_checks/base/checks/base.py Outdated Show resolved Hide resolved

datadog_checks_base/datadog_checks/base/checks/base.py Outdated Show resolved Hide resolved

zippolyte requested changes Mar 18, 2019

View reviewed changes

Rework

f47bfb6

ofek requested changes Mar 18, 2019

View reviewed changes

Add a test

8074149

gzussa reviewed Mar 19, 2019

View reviewed changes

nmuesch added 3 commits March 19, 2019 18:29

Address review

5c1add7

Raise to exception

c1c92d6

Update test_agent_check.py

969321d

zippolyte previously approved these changes Mar 19, 2019

View reviewed changes

nmuesch added 3 commits March 19, 2019 19:44

Use stub datadog_agent

15f2c3b

Merge github.com:DataDog/integrations-core into nick/base_ensure

47d5f00

Merge branch 'nick/base_ensure' of github.com:DataDog/integrations-co…

53cc4a6

…re into nick/base_ensure

nmuesch dismissed zippolyte’s stale review via 53cc4a6 March 19, 2019 18:47

ofek requested changes Mar 19, 2019

View reviewed changes

datadog_checks_base/datadog_checks/base/checks/base.py Outdated Show resolved Hide resolved

datadog_checks_base/datadog_checks/base/checks/base.py Outdated Show resolved Hide resolved

Address review

caac6b7

ofek previously approved these changes Mar 19, 2019

View reviewed changes

zippolyte reviewed Mar 19, 2019

View reviewed changes

datadog_checks_base/tests/test_agent_check.py Outdated Show resolved Hide resolved

Remove extra import

443ba1d

nmuesch dismissed ofek’s stale review via 443ba1d March 20, 2019 09:27

zippolyte previously approved these changes Mar 20, 2019

View reviewed changes

gzussa previously approved these changes Mar 20, 2019

View reviewed changes

masci previously approved these changes Mar 21, 2019

View reviewed changes

olivielpeau previously approved these changes Mar 21, 2019

View reviewed changes

Remove unused import

5139b20

nmuesch dismissed stale reviews from olivielpeau, masci, gzussa, and zippolyte via 5139b20 March 21, 2019 12:05

nmuesch merged commit abf2414 into master Mar 21, 2019

nmuesch deleted the nick/base_ensure branch March 21, 2019 12:16

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add external_host_tags wrapper to checks_base #3316

Add external_host_tags wrapper to checks_base #3316

nmuesch commented Mar 18, 2019 •

edited

Loading

zippolyte left a comment

ofek left a comment

codecov bot commented Mar 18, 2019 •

edited

Loading

zippolyte left a comment •

edited

Loading

ofek Mar 18, 2019

nmuesch Mar 18, 2019

gzussa left a comment

gzussa Mar 19, 2019

nmuesch Mar 19, 2019

gzussa Mar 19, 2019

gzussa Mar 19, 2019

nmuesch Mar 19, 2019

nmuesch Mar 19, 2019 •

edited

Loading

masci left a comment

olivielpeau left a comment

olivielpeau Mar 21, 2019

nmuesch Mar 21, 2019

Add external_host_tags wrapper to checks_base #3316

Add external_host_tags wrapper to checks_base #3316

Conversation

nmuesch commented Mar 18, 2019 • edited Loading

What does this PR do?

Motivation

Additional Notes

Review checklist (to be filled by reviewers)

zippolyte left a comment

Choose a reason for hiding this comment

ofek left a comment

Choose a reason for hiding this comment

codecov bot commented Mar 18, 2019 • edited Loading

Codecov Report

zippolyte left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gzussa left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nmuesch Mar 19, 2019 • edited Loading

Choose a reason for hiding this comment

masci left a comment

Choose a reason for hiding this comment

olivielpeau left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nmuesch commented Mar 18, 2019 •

edited

Loading

codecov bot commented Mar 18, 2019 •

edited

Loading

zippolyte left a comment •

edited

Loading

nmuesch Mar 19, 2019 •

edited

Loading