smoketest: Improve output, parallelize by test #8638

marclop · 2022-07-18T10:54:44Z

Motivation/summary

Improves the smoke test output by redirecting all terraform commands to
a tf.log file, and changes the way the tests are discovered to allow
each test to run in parallel.

Checklist

~~- [ ] Update CHANGELOG.asciidoc~~
~~- [ ] Update package changelog.yml (only if changes to apmpackage have been made)~~
~~- [ ] Documentation has been updated~~

How to test these changes

make smoketest/all SMOKETEST_VERSION=7.17,latest
Verify the output is easily readable.

Related issues

#8303

Improves the smoke test output by redirecting all terraform commands to a `tf.log` file, and changes the way the tests are discovered to allow each test to run in parallel. Signed-off-by: Marc Lopez Rubio <[email protected]>

marclop · 2022-07-18T10:55:28Z

The output for the latest version now looks like this:

$ make smoketest/all
-> Running /Users/marclop/repos/elastic/apm-server/testing/smoke/legacy-standalone-major-managed smoke tests for version latest...
-> Running /Users/marclop/repos/elastic/apm-server/testing/smoke/legacy-managed smoke tests for version latest...
-> Running /Users/marclop/repos/elastic/apm-server/testing/smoke/basic-upgrade smoke tests for version latest...
-> unspecified version, using 8.3
-> Running basic upgrade smoke test for version 8.3.2
-> Creating / Upgrading deployment to version 8.2.[0-9]?([0-9])$
-> APM Server ready!
-> Sending events to APM Server...
-> Asserting component templates and ILM policies...
-> Asserting ingest pipelines...
-> Asserting logs-apm.error-* contains expected documents documents...
-> Asserted 1 error.id=9876543210abcdeffedcba0123456789 exists
-> Asserting traces-apm-* contains expected documents documents...
-> Asserted 1 span.id=1234567890aaaade exists
-> Asserting traces-apm-* contains expected documents documents...
-> Asserted 1 transaction.id=4340a8e0df1906ecbfa9 exists
-> Asserting metrics-apm.internal-* contains expected documents documents...
-> Asserted 1 transaction.type=request exists
-> Upgrading APM Server from 8.2.3 to 8.3.2
-> Creating / Upgrading deployment to version 8.3.2
-> APM Server ready!
-> Sending events to APM Server...
-> Asserting component templates and ILM policies...
-> Asserting ingest pipelines...
-> Asserting logs-apm.error-* contains expected documents documents...
-> Asserted 1 error.id=9876543210abcdeffedcba0123456789 exists
-> Asserting traces-apm-* contains expected documents documents...
-> Asserted 1 span.id=1234567890aaaade exists
-> Asserting traces-apm-* contains expected documents documents...
-> Asserted 1 transaction.id=4340a8e0df1906ecbfa9 exists
-> Asserting metrics-apm.internal-* contains expected documents documents...
-> Asserted 1 transaction.type=request exists
-> Destroying the underlying infrastructure...

apmmachine · 2022-07-18T11:03:37Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2022-07-25T06:13:09.552+0000
Duration: 24 min 57 sec

Test stats 🧪

Test	Results
Failed	0
Passed	2785
Skipped	9
Total	2794

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/hey-apm : Run the hey-apm benchmark.
/package : Generate and publish the docker images.
/test windows : Build & tests on Windows.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

…-test-case

Signed-off-by: Marc Lopez Rubio <[email protected]>

… of https://github.com/marclop/apm-server into f/improve-smoke-test-output-parallelize-each-test-case

apmmachine · 2022-07-19T03:43:57Z

🌐 Coverage report

Name	Metrics % (`covered/total`)	Diff
Packages	100.0% (`41/41`)	💚
Files	91.96% (`183/199`)	👍
Classes	93.435% (`427/457`)	👍
Methods	89.163% (`1086/1218`)	👍
Lines	76.782% (`13314/17340`)	👎 -0.063
Conditionals	100.0% (`0/0`)	💚

apmmachine · 2022-07-19T03:43:58Z

📚 Go benchmark report

Diff with the main branch

name                                                            old time/op    new time/op    delta
pkg:github.com/elastic/apm-server/internal/agentcfg goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/internal/decoder goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/internal/model/modelindexer goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/internal/processor/stream goos:linux goarch:amd64
RUMV3Processor-12                                                 0.00ns ±18%    0.00ns ± 5%  -32.45%  (p=0.016 n=5+4)
pkg:github.com/elastic/apm-server/internal/publish goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/x-pack/apm-server/aggregation/spanmetrics goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/x-pack/apm-server/aggregation/txmetrics goos:linux goarch:amd64
AggregateTransaction-12                                           71.9ns ± 1%    70.4ns ± 1%   -1.99%  (p=0.008 n=5+5)
pkg:github.com/elastic/apm-server/x-pack/apm-server/sampling goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/x-pack/apm-server/sampling/eventstorage goos:linux goarch:amd64
ShardedWriteTransactionUncontended-12                              753ns ± 3%     808ns ± 8%   +7.36%  (p=0.032 n=5+5)
ShardedWriteTransactionContended-12                               7.56µs ±29%    5.09µs ±11%  -32.68%  (p=0.016 n=5+4)

name                                                            old alloc/op   new alloc/op   delta
pkg:github.com/elastic/apm-server/internal/agentcfg goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/internal/decoder goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/internal/model/modelindexer goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/internal/processor/stream goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/internal/publish goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/x-pack/apm-server/aggregation/spanmetrics goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/x-pack/apm-server/aggregation/txmetrics goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/x-pack/apm-server/sampling goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/x-pack/apm-server/sampling/eventstorage goos:linux goarch:amd64
ReadEvents/json_codec_big_tx/100_events-12                         942kB ± 0%     942kB ± 0%   -0.01%  (p=0.032 n=5+5)

name                                                            old allocs/op  new allocs/op  delta
pkg:github.com/elastic/apm-server/internal/agentcfg goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/internal/decoder goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/internal/model/modelindexer goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/internal/processor/stream goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/internal/publish goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/x-pack/apm-server/aggregation/spanmetrics goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/x-pack/apm-server/aggregation/txmetrics goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/x-pack/apm-server/sampling goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/x-pack/apm-server/sampling/eventstorage goos:linux goarch:amd64

report generated with https://pkg.go.dev/golang.org/x/perf/cmd/benchstat

marclop · 2022-07-19T03:50:26Z

hm, the tests don't seem to be running in parallel as we expected, the Makefile targets result in a "nothing to do for target XYZ". @amannocci Do you know what we could be doing differently here so it runs as intended?

amannocci · 2022-07-19T09:43:49Z

I looked at the issue and it seems to be related to the smoketest/discover target.
It returns a single line with ' ' as a delimiter instead of a '\n' delimiter.

Let's try this :

.PHONY: smoketest/discover
smoketest/discover:
	@echo "$(SMOKETEST_DIRS)"

Instead of this :

.PHONY: smoketest/discover
smoketest/discover:
	@echo $(SMOKETEST_DIRS)

@marclop

Signed-off-by: Marc Lopez Rubio <[email protected]>

amannocci · 2022-07-19T10:17:23Z

Looks better : https://apm-ci.elastic.co/job/apm-server/job/smoke-tests-mbp/view/change-requests/job/PR-8638/3/

simitt · 2022-07-19T11:21:17Z

Results look wrong:

all 3 jobs seem to try running the same test: Running 7.17.5 standalone to 8.3.2 to 8.3.2 managed
only the first job passes, the other two show the error below
the log states that the test failed ❌ but it is reported to have been successful ✅

[2022-07-19T10:13:08.337Z] + make smoketest/run TEST_DIR=/var/lib/jenkins/workspace/m-server_smoke-tests-mbp_PR-8638/src/github.com/elastic/apm-server/testing/smoke/legacy-standalone-major-managed

[2022-07-19T10:13:08.840Z] -> Running /var/lib/jenkins/workspace/m-server_smoke-tests-mbp_PR-8638/src/github.com/elastic/apm-server/testing/smoke/legacy-standalone-major-managed smoke tests for version 7.17...

[2022-07-19T10:13:09.044Z] -> Running 7.17.5 standalone to 8.3.2 to 8.3.2 managed

[2022-07-19T10:13:09.045Z] -> Creating / Upgrading deployment to version 7.17.5

[2022-07-19T10:13:12.557Z] 

[2022-07-19T10:13:12.557Z] Error: Required plugins are not installed

[2022-07-19T10:13:12.557Z] 

[2022-07-19T10:13:12.557Z] The installed provider plugins are not consistent with the packages selected

[2022-07-19T10:13:12.557Z] in the dependency lock file:

[2022-07-19T10:13:12.557Z]   - registry.terraform.io/hashicorp/local: the cached package for registry.terraform.io/hashicorp/local 2.2.3 (in .terraform/providers) does not match any of the checksums recorded in the dependency lock file

[2022-07-19T10:13:12.557Z] 

[2022-07-19T10:13:12.557Z] Terraform uses external plugins to integrate with a variety of different

[2022-07-19T10:13:12.557Z] infrastructure services. You must install the required plugins before running

[2022-07-19T10:13:12.558Z] Terraform operations.

[2022-07-19T10:13:12.558Z] -> Smoke tests FAILED!!

Signed-off-by: Marc Lopez Rubio <[email protected]>

This reverts commit fd19e4c.

marclop · 2022-07-19T12:03:13Z

I think it happened because all three parallel Jenkins jobs are running in the same folder form what I see in the UI, and the terraform locks are preventing the job from running multiple terraform commands concurrently in the same folder (unintended behavior). I think the splitting of the lines isn't working correctly just yet.

In my machine I can run the command just fine and it returns the right output:

$ make smoketest/discover
/Users/marclop/repos/elastic/apm-server/testing/smoke/legacy-standalone-major-managed
/Users/marclop/repos/elastic/apm-server/testing/smoke/legacy-managed
/Users/marclop/repos/elastic/apm-server/testing/smoke/basic-upgrade

…-test-case

simitt · 2022-07-25T06:14:07Z

CI logic was adapted and test runs are parallelized now.

smoketest: Improve output, parallelize by test

fd316ec

Improves the smoke test output by redirecting all terraform commands to a `tf.log` file, and changes the way the tests are discovered to allow each test to run in parallel. Signed-off-by: Marc Lopez Rubio <[email protected]>

marclop added enhancement backport-skip Skip notification from the automated backport with mergify labels Jul 18, 2022

Merge branch 'main' into f/improve-smoke-test-output-parallelize-each…

9a76247

…-test-case

simitt approved these changes Jul 18, 2022

View reviewed changes

marclop added 2 commits July 19, 2022 11:15

smoke folder structure

de0a7ad

Signed-off-by: Marc Lopez Rubio <[email protected]>

Merge branch 'f/improve-smoke-test-output-parallelize-each-test-case'…

ffd11f3

… of https://github.com/marclop/apm-server into f/improve-smoke-test-output-parallelize-each-test-case

Enclose SMOKETEST_DIRS in double quotes

16ddddd

Signed-off-by: Marc Lopez Rubio <[email protected]>

marclop added 2 commits July 19, 2022 19:56

no locks for terraform apply/init/destroy

fd19e4c

Signed-off-by: Marc Lopez Rubio <[email protected]>

Revert "no locks for terraform apply/init/destroy"

9d61cf8

This reverts commit fd19e4c.

amannocci mentioned this pull request Jul 20, 2022

Correct smoketests running in the CI #8663

Merged

simitt added 2 commits July 22, 2022 16:11

Merge branch 'main' into f/improve-smoke-test-output-parallelize-each…

633ca9b

…-test-case

Merge branch 'main' into f/improve-smoke-test-output-parallelize-each…

1fb982e

…-test-case

simitt enabled auto-merge (squash) July 25, 2022 06:15

simitt merged commit 5e6372e into elastic:main Jul 25, 2022

marclop deleted the f/improve-smoke-test-output-parallelize-each-test-case branch August 15, 2022 06:59

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

smoketest: Improve output, parallelize by test #8638

smoketest: Improve output, parallelize by test #8638

marclop commented Jul 18, 2022 •

edited

Loading

marclop commented Jul 18, 2022

apmmachine commented Jul 18, 2022 •

edited

Loading

Build stats

Test stats 🧪

apmmachine commented Jul 19, 2022 •

edited

Loading

apmmachine commented Jul 19, 2022 •

edited

Loading

marclop commented Jul 19, 2022

amannocci commented Jul 19, 2022 •

edited

Loading

amannocci commented Jul 19, 2022

simitt commented Jul 19, 2022

marclop commented Jul 19, 2022

simitt commented Jul 25, 2022

smoketest: Improve output, parallelize by test #8638

smoketest: Improve output, parallelize by test #8638

Conversation

marclop commented Jul 18, 2022 • edited Loading

Motivation/summary

Checklist

How to test these changes

Related issues

marclop commented Jul 18, 2022

apmmachine commented Jul 18, 2022 • edited Loading

💚 Build Succeeded

Build stats

Test stats 🧪

🤖 GitHub comments

apmmachine commented Jul 19, 2022 • edited Loading

🌐 Coverage report

apmmachine commented Jul 19, 2022 • edited Loading

📚 Go benchmark report

marclop commented Jul 19, 2022

amannocci commented Jul 19, 2022 • edited Loading

amannocci commented Jul 19, 2022

simitt commented Jul 19, 2022

marclop commented Jul 19, 2022

simitt commented Jul 25, 2022

marclop commented Jul 18, 2022 •

edited

Loading

apmmachine commented Jul 18, 2022 •

edited

Loading

apmmachine commented Jul 19, 2022 •

edited

Loading

apmmachine commented Jul 19, 2022 •

edited

Loading

amannocci commented Jul 19, 2022 •

edited

Loading