Add beat serverless tests & related support #3258

fearful-symmetry · 2023-08-16T21:37:40Z

What does this PR do?

There's a lot going on here. This PR:

Makes changes to the integration test framework so it can run beats, and not just elastic-agent
Adds a variety of beat tests for test conditions that are effected by serverless
Updates the serverless client code to reflect ongoing changes to the serverless APIs
Adds new elasticsearch/kibana helpers
Adds a few temporary hacks to the magefile so these tests will run in the agent repo

Some important details and caveats also accompany this:

As of right now, these tests will not pass, nor are they expected to, as beats currently does not have serverless support. They should all run and pass against a stateful deployment.
These tests are in the agent repo, and not beats, as we don't have buildkite set up in beats yet.
There's the potential for some false positives with some of the setup tests. The tests check for the presence of things like index templates, dashboards, etc, and thus they can return a false positive if the tests are run against a pre-existing. I'm not sure if it's possible to tag things like dashboards with some kind of metadata so we can check to see if we're testing "our" dashboards.

Why is it important?

We want to get rid of the old python tests, and also we need tests for serverless support.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have added tests that prove my fix is effective or that my feature works
I have added an integration test or an E2E test

How to test this PR locally

pull down
run mage -v integration:TestBeatServerless metricbeat
To test against a stateful deployment, explicitly set STACK_PROVISIONER to ess

elasticmachine · 2023-08-16T21:37:44Z

Pinging @elastic/elastic-agent (Team:Elastic-Agent)

elasticmachine · 2023-08-16T21:43:55Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2023-09-28T16:01:26.384+0000
Duration: 25 min 25 sec

Test stats 🧪

Test	Results
Failed	0
Passed	6313
Skipped	63
Total	6376

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages.
run integration tests : Run the Elastic Agent Integration tests.
run end-to-end tests : Generate the packages and run the E2E Tests.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

fearful-symmetry · 2023-08-16T22:40:36Z

buildkite failures appear to be unrelated: Error: fetchBinaryFromArtifactsApi failed for auditbeat on linux/amd64: the artifact was not found

elasticmachine · 2023-08-16T22:47:33Z

🌐 Coverage report

Name	Metrics % (`covered/total`)	Diff
Packages	98.78% (`81/82`)	👍
Files	65.878% (`195/296`)	👎 -0.223
Classes	65.574% (`360/549`)	👎 -0.12
Methods	52.282% (`1134/2169`)	👎 -0.462
Lines	37.868% (`12866/33976`)	👎 -0.288
Conditionals	100.0% (`0/0`)	💚

pierrehilbert · 2023-08-17T05:47:41Z

@pazone didn't you fix the fetchBinaryFromArtifactsApi issue?

joshdover · 2023-08-17T10:07:15Z

I'm not sure if it's possible to tag things like dashboards with some kind of metadata so we can check to see if we're testing "our" dashboards.

Best option I can think of that wouldn't require changing what Beats does (like injecting a tag) is to have the test code to look at the updated_at field on the Saved Objects that are created. It's not ideal since it relies on clocks but we can probably make the check relatively relaxed - updated_at in the last 5 minutes or something like that.

pazone · 2023-08-17T15:30:14Z

@pierrehilbert It's a bit another issue. Fixing it

fearful-symmetry · 2023-08-17T19:34:06Z

Alright, took @joshdover 's advice, and the dashboard tests will now check the updated time.

pkg/testing/fetcher_local.go

pkg/testing/fixture.go

blakerouse · 2023-08-21T14:46:54Z

pkg/testing/runner/debian.go

@@ -106,6 +106,7 @@ func (DebianRunner) Copy(ctx context.Context, sshClient *ssh.Client, logger Logg
 			"failed to remove agent directory before unziping new one: %w. stdout: %q, stderr: %q",
 			err, stdout, stderr)
 	}
+	_, _, _ = sshRunCommand(ctx, sshClient, "sudo", []string{"rm", "-rf", "beats"}, nil)


Why is this removing a random beats directory? We set the remote path to $HOME/agent I don't see why the same cannot be used for beats?

So, the working directory is the same here, since tests are still being run out of the agent repo, the only difference is that now we have additional beat artifacts that we upload in addition to the agent working directory.

We could refactor this so we don't have /agent hard-coded, but we'd still need to find a way to delete the working directory and any non-working-directory artifacts. I wasn't too sure what the runner was doing, so I didn't want to blow away the entire home directory or something.

I feel like the best solution here is to probably just to make some of the copy code more generic, and just use some kind of heuristic to delete anything that looks like a working directory or artifact.

Alright, moved things around to make the code a little more generic, at least.

blakerouse · 2023-08-21T14:49:19Z

testing/integration/beat_serverless_test.go

+	testbeatName string
+}
+
+func TestMetricbeatSeverless(t *testing.T) {


Why is this being added in the Elastic Agent repository? I get that it might be faster, but probably only a little bit. All the modules you are using for this to work are in pkg/ which I did on purpose so other repositories can use that code. Why not just add this directly to the beats repository?

It is being done here because it is much faster, serverless has a rapidly approaching deadline and we need tests to check whether Beats will work.

Beats hasn't been ported to Buildkite yet, it is the most complex build we have and it would have taken too long to do that before writing the tests.

This should be temporary, I'm hoping the work to port Beats to buildkite will start soon. We may be able to leave these tests on a branch depending on the timing to avoid this living for too long.

fearful-symmetry · 2023-09-13T18:40:30Z

...never seen that error before

[2023-09-13T18:16:37.809Z]         	            	fork/exec C:\Users\jenkins\AppData\Local\Temp\TestRemovePath2272529880\001\subdir\main.exe: Operation did not complete successfully because the file contains a virus or potentially unwanted software.

pierrehilbert · 2023-09-14T07:26:53Z

@fearful-symmetry we got this error in multiple other places, it has been introduced in the journey to remove our own folder on Windows. That's something we need to fix. We have an open PR for that: #3411
cc @blakerouse @cmacknz

lukeelmers · 2023-09-14T17:38:26Z

We had originally intended to make these public, but we recently decided we would keep them internal for serverless until we had a strong reason not to.

However, I don't think we had considered the use case of beats installing dashboards.

@joshdover Just to follow up on this, the core team decided in elastic/kibana#162418 that for now we will stick with our most recent plan of keeping the saved objects import/export APIs internal while we work out our long term vision for these APIs. This means Beats would indeed need to set x-elastic-internal-origin: Beats in these requests.

Kibana won't care if you set this header unnecessarily, so if it's easier for Beats to just globally always use the header when talking to Kibana, that should be fine too. Sorry for the inconvenience!

fearful-symmetry · 2023-09-18T15:55:27Z

This means Beats would indeed need to set x-elastic-internal-origin: Beats

@lukeelmers does kibana care what the value of that header is? Does it have to be beats ?

lukeelmers · 2023-09-18T21:11:39Z

does kibana care what the value of that header is? Does it have to be beats ?

It is convenient to use the application name for tracking/monitoring purposes, however it isn't strictly required -- Kibana doesn't care what value you use.

…s-tests

fearful-symmetry · 2023-09-22T17:13:15Z

So, the actual serverless PR is here: elastic/beats#36649

cmacknz · 2023-09-25T15:07:44Z

The tests here aren't running in CI per the test report. Otherwise the tests would be failing because the Beats don't actually support serverless yet.

While the tests are in the wrong repository, I think we can just move them to a daily schedule so that we get the test coverage without blocking agent PRs (since no agent code is involved in the standalone Beats tests) while we work on allowing these tests to be triggered from the Beats repository. I will create issues for both of these issues.

We should add a test that ensures that Elastic Agent can be installed and enrolled with a Severless Observability project. Probably we can parameterize the monitoring logs integration test to run against both stateful and serverless. This test shouldn't require the DSL support since Fleet takes care of that for us.

fearful-symmetry · 2023-09-25T16:33:47Z

We should add a test that ensures that Elastic Agent can be installed and enrolled with a Severless Observability project.

As part of this PR? @cmacknz

fearful-symmetry · 2023-09-25T16:35:16Z

Alright, need to pause merging/reviewing this for now, based on this conversation: elastic/beats#36649 (comment)

Turns out under serverless, auth via username:pass and api_key results in different access permissions, which is causing some setup tasks to fail under serverless. Need to change how the tests do auth so they can connect to ES via api_key.

…s-tests

elastic-sonarqube · 2023-09-28T16:13:08Z

SonarQube Quality Gate

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
0.0% Duplication

fearful-symmetry · 2023-09-28T17:43:13Z

@cmacknz I know we talked about it a bit yesterday, but what's the next steps for this? We want to break out all the backing library changes into a different PR? What was the goal with that?

cmacknz · 2023-09-28T18:33:25Z

@cmacknz I know we talked about it a bit yesterday, but what's the next steps for this? We want to break out all the backing library changes into a different PR? What was the goal with that?

The Beat tests here can't be merged without #3470 or elastic/beats#36675.

The support for provisioning serverless projects can be merged however, and we should have an integration test in the agent repository to test that Elastic Agent can ship data to Elasticsearch as mentioned in #3258 (comment). I'm comfortable with merging that to main and having it gate PRs. Such an agent test will also test that Beats can ship data to ES, but won't test the setup commands.

My goal is to ensure we have appropriate serverless test coverage in the right places, triggered by the correct things. I want to avoid having tests against Beat snapshot images gate agent PRs, because that doesn't make any sense. So let's strip this PR down to just the Beat specific functionality and create a separate PR testing serverless support for agent. We can make a dedicated issue for this if it helps.

mergify · 2023-10-05T13:40:32Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b serverless-tests upstream/serverless-tests
git merge upstream/main
git push upstream serverless-tests

fearful-symmetry · 2023-10-25T19:21:12Z

Closed, now at #3658

fearful-symmetry added 4 commits August 15, 2023 15:01

cleaning up

c55bbde

Merge remote-tracking branch 'upstream/main' into serverless-tests

12ac235

final bit of cleanup

e939383

fix magefile, cleanup docs

2afe998

fearful-symmetry added Team:Elastic-Agent Label for the Agent team backport-skip skip-changelog labels Aug 16, 2023

fearful-symmetry self-assigned this Aug 16, 2023

fearful-symmetry requested a review from a team as a code owner August 16, 2023 21:37

fearful-symmetry requested review from ycombinator and pchila August 16, 2023 21:37

fearful-symmetry added 3 commits August 16, 2023 14:51

clean up errors, make linter happy

413c0d7

fix headers

a16767d

fix fields in runner config

7e8970a

add dashboard checks

9e26b7c

Merge remote-tracking branch 'upstream/main' into serverless-tests

3aab6e6

blakerouse requested changes Aug 21, 2023

View reviewed changes

pchila removed their request for review August 25, 2023 12:43

clean up, refactor

369f62e

fearful-symmetry requested a review from blakerouse August 25, 2023 23:12

fearful-symmetry added 2 commits August 25, 2023 16:45

clean up

0b7afae

tinker with env vars

ef23b3a

tinker with dashboards

fd88bf2

fearful-symmetry mentioned this pull request Sep 13, 2023

Add checks for serverless support elastic/elastic-agent-libs#150

Merged

4 tasks

fix ilm tests

ec9c8c5

Merge remote-tracking branch 'upstream/main' into serverless-tests

e312d0b

fearful-symmetry mentioned this pull request Sep 13, 2023

Add basic serverless support for elasticsearch, beats index management elastic/beats#36587

Merged

6 tasks

fearful-symmetry added 2 commits September 21, 2023 08:56

Merge remote-tracking branch 'upstream/main' into serverless-tests

27f8814

Merge remote-tracking branch 'origin/serverless-tests' into serverles…

5611ff9

…s-tests

fearful-symmetry requested a review from cmacknz September 22, 2023 17:13

cmacknz mentioned this pull request Sep 25, 2023

Allow running the standalone Beats serverless integration tests on a daily schedule #3470

Open

cmacknz mentioned this pull request Sep 25, 2023

Support using the Elastic Agent integration testing framework to run tests against Serverless projects elastic/beats#36675

Open

fearful-symmetry added 4 commits September 26, 2023 16:31

use API keys for auth

8267e47

Merge remote-tracking branch 'origin/serverless-tests' into serverles…

a733070

…s-tests

add additional integration tests

acff625

Merge remote-tracking branch 'upstream/main' into serverless-tests

3196a39

fearful-symmetry mentioned this pull request Sep 28, 2023

Update serverless support in integration tests, expand test tooling, add support for beats testing #3486

Merged

7 tasks

fearful-symmetry closed this Oct 25, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add beat serverless tests & related support #3258

Add beat serverless tests & related support #3258

fearful-symmetry commented Aug 16, 2023

elasticmachine commented Aug 16, 2023

elasticmachine commented Aug 16, 2023 •

edited

Loading

Build stats

Test stats 🧪

fearful-symmetry commented Aug 16, 2023

elasticmachine commented Aug 16, 2023 •

edited

Loading

pierrehilbert commented Aug 17, 2023

joshdover commented Aug 17, 2023

pazone commented Aug 17, 2023

fearful-symmetry commented Aug 17, 2023

blakerouse Aug 21, 2023

fearful-symmetry Aug 25, 2023 •

edited

Loading

fearful-symmetry Aug 25, 2023

fearful-symmetry Aug 25, 2023

blakerouse Aug 21, 2023

cmacknz Aug 21, 2023 •

edited

Loading

fearful-symmetry commented Sep 13, 2023

pierrehilbert commented Sep 14, 2023 •

edited

Loading

lukeelmers commented Sep 14, 2023

fearful-symmetry commented Sep 18, 2023

lukeelmers commented Sep 18, 2023

fearful-symmetry commented Sep 22, 2023

cmacknz commented Sep 25, 2023 •

edited

Loading

fearful-symmetry commented Sep 25, 2023

fearful-symmetry commented Sep 25, 2023

elastic-sonarqube bot commented Sep 28, 2023

fearful-symmetry commented Sep 28, 2023

cmacknz commented Sep 28, 2023

mergify bot commented Oct 5, 2023

fearful-symmetry commented Oct 25, 2023

Add beat serverless tests & related support #3258

Add beat serverless tests & related support #3258

Conversation

fearful-symmetry commented Aug 16, 2023

What does this PR do?

Why is it important?

Checklist

How to test this PR locally

elasticmachine commented Aug 16, 2023

elasticmachine commented Aug 16, 2023 • edited Loading

💚 Build Succeeded

Build stats

Test stats 🧪

💚 Flaky test report

🤖 GitHub comments

fearful-symmetry commented Aug 16, 2023

elasticmachine commented Aug 16, 2023 • edited Loading

🌐 Coverage report

pierrehilbert commented Aug 17, 2023

joshdover commented Aug 17, 2023

pazone commented Aug 17, 2023

fearful-symmetry commented Aug 17, 2023

blakerouse Aug 21, 2023

Choose a reason for hiding this comment

fearful-symmetry Aug 25, 2023 • edited Loading

Choose a reason for hiding this comment

fearful-symmetry Aug 25, 2023

Choose a reason for hiding this comment

fearful-symmetry Aug 25, 2023

Choose a reason for hiding this comment

blakerouse Aug 21, 2023

Choose a reason for hiding this comment

cmacknz Aug 21, 2023 • edited Loading

Choose a reason for hiding this comment

fearful-symmetry commented Sep 13, 2023

pierrehilbert commented Sep 14, 2023 • edited Loading

lukeelmers commented Sep 14, 2023

fearful-symmetry commented Sep 18, 2023

lukeelmers commented Sep 18, 2023

fearful-symmetry commented Sep 22, 2023

cmacknz commented Sep 25, 2023 • edited Loading

fearful-symmetry commented Sep 25, 2023

fearful-symmetry commented Sep 25, 2023

elastic-sonarqube bot commented Sep 28, 2023

fearful-symmetry commented Sep 28, 2023

cmacknz commented Sep 28, 2023

mergify bot commented Oct 5, 2023

fearful-symmetry commented Oct 25, 2023

elasticmachine commented Aug 16, 2023 •

edited

Loading

elasticmachine commented Aug 16, 2023 •

edited

Loading

fearful-symmetry Aug 25, 2023 •

edited

Loading

cmacknz Aug 21, 2023 •

edited

Loading

pierrehilbert commented Sep 14, 2023 •

edited

Loading

cmacknz commented Sep 25, 2023 •

edited

Loading