Update serverless support in integration tests, expand test tooling, add support for beats testing

[fearful-symmetry]

What does this PR do?

This PR "breaks apart" #3258 into only changes needed to support serverless instances in integration tests, as well as testing with beats, and various expansions to the tooling available to integration tests. Essentially, this is everything except the actual beats serverless tests.

Why is it important?

We want serverless testing support, and need to support #3258

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in ./changelog/fragments using the changelog tool
• I have added an integration test or an E2E test

[cmacknz]

The one Serverless test you are trying to run is actually being skipped, so it isn't testing anything.

=== RUN   TestMonitoringLogsShipped
    monitoring_logs_test.go:40: got namespace: vlrkpe5ocvjnobdqjqhg1ev0wwzhanqx75ocvbjqbfe
    monitoring_logs_test.go:41: Test is flaky; see https://github.com/elastic/elastic-agent/issues/3081
--- SKIP: TestMonitoringLogsShipped (0.19s)

Test report is here

[cmacknz]

I think this was fixed on main with #3616

This test should also run on Windows and not just Linux, you can remove the OS restriction.

[fearful-symmetry]

Alright, so it looks like those windows tests might have been disabled for a reason, because it's failing. Want to see if it's just timing out, or if there's a deeper issue...

[fearful-symmetry]

how ironic

Failed to connect to backoff(elasticsearch(https://c00f724935b3496cba74c717249987eb.es.us-east-1.aws.staging.elastic.cloud:443)): Connection marked as failed because the onConnect callback failed: Elasticsearch is too old. Please upgrade the instance. If you would like to connect to older instances set output.elasticsearch.allow_older_versions to true. ES=8.11.0, Beat=8.12.0

[cmacknz]

Why the additional 10 minutes, is serverless that much slower?

[fearful-symmetry]

This is actually for stateful, I ran into a couple timeouts while testing.

[cmacknz]

Did it fix the problem, or does it just wait longer before failing? Sometimes the stack can fail to come up and waiting doesn't fix it, it just makes you wait longer until finding out.

[cmacknz]

Changes look good to me, good to merge once the tests pass again. Looks like the failure was in the serverless tests this time.

[fearful-symmetry]

Never seen that before.

=== FAIL: testing/integration TestMonitoringLogsShipped (2.87s)
--
 (windows-amd64-2022) Test output (sudo) (stdout): unable to create 
policy: missing authentication credentials for REST request [/_mget]: 
security_exception

[fearful-symmetry]

Sigh, something seems to be breaking the resulting junit files...

Parsing /build/TEST-go-integration.xml
--


/usr/local/lib/ruby/gems/3.1.0/gems/rexml-3.2.5/lib/rexml/parsers/treeparser.rb:96:in
 `rescue in parse': #<RuntimeError: Illegal character "\\u0000" in 
raw string "

[fearful-symmetry]

Downloaded the XML files, they all appear to parse correctly.

[elastic-sonarqube]

SonarQube Quality Gate

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
0.0% Duplication

[fearful-symmetry]

In typical fashion, can't reproduce the failure, which was a setup issue. I'm guessing there was some intermittent timeout or issue setting up auth against the serverless cluster.