testing/smoke: Add initial smoke tests

[marclop]

Motivation/summary

This patch adds an initial set of smoke tests which use the existing
ec_deployment module for the underlying infrastructure and a bash
script to drive the scripted smoke test cases. The script takes in a
MAJOR.MINOR version combination.

The initial smoke tests are slightly different depending if the major
version is 7 or 8, the former will be test upgrades 7.17.last-1 to
7.17.last, while the latter, will upgrade from the previous minor to
the current minor (8.3.0) will upgrade from 8.2.3 to 8.3.0.

The initial set of assertions is kept to a minimum to enable iterating
over the approach rather than adding all the assertions in the first PR.

Initially, we're testing that:

• Create deployment succeeds
• Data is sent to APM Server
• Each of the events sent to APM Server can be found in Elasticsearch.
• Upgrade to the next version succeeds.
• Data is sent to the APM Server
• Each of the events sent to APM Server can be found in Elasticsearch.

As the last step and if any errors occur during the smoke test, the
underlying infrastructure will be destroyed.

I've chosen to use bash since it doesn't seem too bad with the current
set of assertions, we can always improve it and write it in Go, but it
seems more involved than what we need right now.

I've created a gist with the output of the tested execution.

Checklist

- [ ] Update CHANGELOG.asciidoc
- [ ] Update package changelog.yml (only if changes to apmpackage have been made)

• Documentation has been updated

How to test these changes

1. export your cloud credentials (export EC_API_KEY)
2. make smoketest SMOKETEST_VERSIONS=7.17,latest
3. Verify that the exit status is 0 and that the smoke tests all passed (check for 4 instances of the log message -> Smoke tests passed!)

Related issues

Part of #8303

[apmmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2022-06-24T06:58:43.198+0000

• Duration: 28 min 41 sec

Test stats 🧪

Test	Results
Failed	0
Passed	4108
Skipped	13
Total	4121

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

• /hey-apm : Run the hey-apm benchmark.

• /package : Generate and publish the docker images.

• /test windows : Build & tests on Windows.

• run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

[apmmachine]

🌐 Coverage report

Name	Metrics % (covered/total)	Diff
Packages	100.0% (42/42)	💚
Files	91.96% (183/199)	👍
Classes	93.435% (427/457)	👍
Methods	89.163% (1086/1218)	👍 0.082
Lines	76.845% (13325/17340)	👍 0.035
Conditionals	100.0% (0/0)	💚

[lahsivjar]

LGTM with 1 concern, since we are using local backend, in the event of SIGKILL the trap command will not destroy terraform (cases like runners dying). This might lead to some cases where the infra will continue to exist like a zombie.

[marclop]

@lahsivjar I was thinking about using a cleanup task in jenkins even after the script finishes that runs terraform destroy -auto-approve so that in the event the trap code doesn't fire, the infrastructure is destroyed.

This gives us the advantage of 1. we always try and clean up locally, 2. Jenkins will do it twice, and in the event the infrastructure has already been destroyed, the terraform destroy -auto-approve is a no-op.