Support E2E testing #3896

ofek · 2019-06-10T21:00:49Z

What does this PR do?

Adds the ability to test against the real Datadog Agent collector.

Implementation

A new command test is added to the ddev env namespace to facilitate the testing of only e2e tests e.g. any tests decorated with @pytest.mark.e2e. By default, it will assume an environment & Agent has already been spun up via ddev env start ... so you can rapidly iterate by repeated invocations to test. A flag -ne/--new-env will instruct the command to also run ddev env start ... and ddev env stop ... for you. Just like ddev test ..., it accepts an optional list of checks to test, each with optional envs specified. If no checks are selected, it will test what checks have changed in git compared to master (example) and force the behavior of --new-env. You can also use the ddev test ... command directly if you pass the new --e2e flag, though you can only test one environment per invocation.

A new global fixture dd_agent_check is introduced. This fixture yields a function which accepts a config and arbitrary keyword arguments. The function will call ddev env check ... using the configuration passed in as well as any additional args we specified, and feed the results to our standard aggregator stub, which it will then return. Here is a full example:

@pytest.mark.e2e
def test_e2e(dd_agent_check, some_instance):
    aggregator = dd_agent_check(some_instance, rate=True)

Note that it will not call the ddev within the virtualenv, but rather the ddev used to invoke the test run since the CLI requires many more dependencies than we require for testing.

Finally, as we may not want to run e2e tests for all environments (especially due to the increased CI pressure), it is necessary to mark them for e2e test automation in tox.ini. For example:

[tox]
...
envlist =
    py{27,37}-{1.0,5.2,latest}

[testenv]
description =
    py27: e2e ready
...

would mark the envs py27-1.0, py27-5.2, and py27-latest as supporting e2e tests. When the Agent supports Python 3, we will make the Python versions actually influence what the spun up Agents use.

Additional Notes

In order to see for example monotonic counts or rates in the Agent aggregator output you logically need to run the check more than once using the rate flag. However, in doing so you receive more than one submission of other metrics, which may break being able to simply copy/paste things like aggregator.assert_metric(metric_submitted_twice, count=1). In future we should look into how to deduplicate results.
Currently, code coverage of e2e tests is not able to be reported alongside regular tests. Until we implement that we should ignore them from affecting percentages with # no cov.

codecov · 2019-06-10T21:07:42Z

Codecov Report

Merging #3896 into master will decrease coverage by 4.6%.
The diff coverage is 17.64%.

@@            Coverage Diff             @@
##           master    #3896      +/-   ##
==========================================
- Coverage   86.48%   81.88%   -4.61%     
==========================================
  Files         740       96     -644     
  Lines       37835     6204   -31631     
  Branches     4481      731    -3750     
==========================================
- Hits        32723     5080   -27643     
+ Misses       3892      962    -2930     
+ Partials     1220      162    -1058

therve

Looks good, a few questions inside.

.travis.yml

therve · 2019-06-11T13:08:46Z

datadog_checks_dev/datadog_checks/dev/plugin/pytest.py

+    # already been spun up so we prevent another invocation
+    elif e2e_testing():  # no cov
+        # Since the scope is `session` there should only ever be one definition
+        fixture_def = request._fixturemanager._arg2fixturedefs[E2E_FIXTURE_NAME][0]


Ouch. Is there a way not to do that? That looks super fragile.

I really tried, but I found absolutely no other way

datadog_checks_dev/datadog_checks/dev/plugin/pytest.py

therve · 2019-06-11T13:12:02Z

datadog_checks_dev/datadog_checks/dev/tooling/commands/env/test.py

+        tests_ran = True
+
+        for env in envs:
+            if clean:


I would be tempted to not have that clean option. My suggestion would be to do start/test/stop by default. We can keep ddev test --e2e to iterate without setup/teardown of the env, no?

The thing is if one of us or a contributor forgets that flag then the env would be town down with a random agent still up, leaving it in an inconsistent state

I guess... I'm really against using clean though, considering it starts the environment.

I'm not sure I understand what's the suggested workflow here.
The idea is to do ddev env test --clean and that will start, test and stop the env?

I don't understand very well @therve suggested workflow or @ofek's concern here.

Note here: As the ideais not to have to run e2e locally for releases anymore I guess the case of start/test/stop is going to become less frequent for developers than it is now

I guess that's fine. We can always tweak the process later on. I think detecting if the env is running would be a nice improvement.

datadog_checks_dev/datadog_checks/dev/tooling/testing.py

hithwen · 2019-06-14T11:20:58Z

datadog_checks_dev/datadog_checks/dev/tooling/commands/env/test.py

+        'config (agent5, agent6, etc.)'
+    ),
+)
+@click.option('--dev/--prod', help='Whether to use the latest version of a check or what is shipped')


which one is the default?

hithwen · 2019-06-14T11:22:53Z

datadog_checks_dev/datadog_checks/dev/tooling/commands/env/test.py

+        'Ex: -e DD_URL=app.datadoghq.com -e DD_API_KEY=123456'
+    ),
+)
+@click.option('--clean', '-c', is_flag=True, help='Execute setup and tear down actions')


maybe sth like --new-env clean has already a lot of meaning from build scripts and it sounds confusing

+1 for --new-env.

hithwen · 2019-06-14T11:28:40Z

datadog_checks_dev/datadog_checks/dev/tooling/commands/env/test.py

+        tests_ran = True
+
+        for env in envs:
+            if clean:


I'm not sure I understand what's the suggested workflow here.
The idea is to do ddev env test --clean and that will start, test and stop the env?

I don't understand very well @therve suggested workflow or @ofek's concern here.

ofek added dev/testing dev/tooling changelog/Added integration/datadog_checks_dev labels Jun 10, 2019

ofek requested review from a team as code owners June 10, 2019 21:00

Support E2E testing

23b705f

ofek force-pushed the ofek/e2e branch from 8b3336b to 23b705f Compare June 10, 2019 22:38

therve reviewed Jun 11, 2019

View reviewed changes

hithwen reviewed Jun 14, 2019

View reviewed changes

address review

060e4d7

therve approved these changes Jun 14, 2019

View reviewed changes

ofek merged commit 6caa96d into master Jun 14, 2019

ofek deleted the ofek/e2e branch June 14, 2019 17:11

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support E2E testing #3896

Support E2E testing #3896

ofek commented Jun 10, 2019 •

edited

Loading

codecov bot commented Jun 10, 2019 •

edited

Loading

therve left a comment

therve Jun 11, 2019

ofek Jun 11, 2019

therve Jun 11, 2019

ofek Jun 11, 2019

therve Jun 11, 2019

hithwen Jun 14, 2019

hithwen Jun 14, 2019

therve Jun 14, 2019

hithwen Jun 14, 2019

ofek Jun 14, 2019

hithwen Jun 14, 2019

therve Jun 14, 2019

hithwen Jun 14, 2019

Support E2E testing #3896

Support E2E testing #3896

Conversation

ofek commented Jun 10, 2019 • edited Loading

What does this PR do?

Implementation

Additional Notes

codecov bot commented Jun 10, 2019 • edited Loading

Codecov Report

therve left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ofek commented Jun 10, 2019 •

edited

Loading

codecov bot commented Jun 10, 2019 •

edited

Loading