Adjust Fleet recipes for default policy change

[juliaElastic]

Fixes #5262
To adjust recipes to the change in kibana (removing default policies), added system, elastic_agent, fleet_server packages to the preconfigured policies for ECK.

[juliaElastic]

@naemono hi, I'm not sure what is wrong here, the build failed: https://devops-ci.elastic.co/blue/organizations/jenkins/cloud-on-k8s-pr/detail/cloud-on-k8s-pr/7918/pipeline

[pebrc]

@juliaElastic looks like we exceeded our disk quota in the GCP project the tests run in

=Insufficient regional quota to satisfy request: resource "LOCAL_SSD_TOTAL_GB": request requires '1125.0' and is short '1125.0'. project has a quota of '30000.0' with '0.0' available

[juliaElastic]

@pebrc @naemono @david-kow
hm, there are 2 failing e2e tests, is there a way to debug locally?
https://devops-ci.elastic.co/job/cloud-on-k8s-pr-full/107/#showFailuresLink

tried these commands locally, but getting errors:
https://github.com/elastic/cloud-on-k8s/blob/main/dev-setup.md#running-e2e-tests

I tested the yaml configs in my local kibana, and seemed to work fine.
I'm not sure how this ES error is related to my change:

    step.go:43: 
        	Error Trace:	utils.go:87
        	Error:      	Received unexpected error:
        	            	elasticsearch client failed for https://elasticsearch-n4nt-es-internal-http.e2e-1e6oh-mercury.svc:9200/_data_stream/logs-elastic_agent.filebeat-default: 404 Not Found: {Status:404 Error:{CausedBy:{Reason: Type:} Reason:no such index [logs-elastic_agent.filebeat-default] Type:index_not_found_exception RootCause:[{Reason:no such index [logs-elastic_agent.filebeat-default] Type:index_not_found_exception}]}}
        	Test:       	TestFleetKubernetesIntegrationRecipe/ES_data_should_pass_validations

[juliaElastic]

run full pr build

[joshdover]

Changes LGTM from a Fleet perspective

[naemono]

@juliaElastic I'm going to try and replicate the failures you're seeing, and I'll get back to you about this.

[naemono]

@juliaElastic I'm going to try and replicate the failures you're seeing, and I'll get back to you about this.

I'm sorry, I ran into a large number of local issues yesterday trying to replicate this, I'm going to try again today.

[juliaElastic]

@naemono hi, any luck with finding out the root cause of this build failure?

[naemono]

@juliaElastic I've actually had terrible luck replicating this, and am trying one more approach today. I'll report back, and sorry for the delays.

[naemono]

Ok, I've finally had some luck replicating this, and I'll see if I can figure out the root cause. Sorry again for the delays.

=== RUN   TestFleetKubernetesIntegrationRecipe/ES_data_should_pass_validations
Retries (30m0s timeout): .............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
    step.go:43:
        	Error Trace:	utils.go:87
        	Error:      	Received unexpected error:
        	            	elasticsearch client failed for https://elasticsearch-ms5n-es-internal-http.e2e-2l2sg-mercury.svc:9200/_data_stream/logs-elastic_agent.filebeat-default: 404 Not Found: {Status:404 Error:{CausedBy:{Reason: Type:} Reason:no such index [logs-elastic_agent.filebeat-default] Type:index_not_found_exception RootCause:[{Reason:no such index [logs-elastic_agent.filebeat-default] Type:index_not_found_exception}]}}
        	Test:       	TestFleetKubernetesIntegrationRecipe/ES_data_should_pass_validations
{"log.level":"error","@timestamp":"2022-01-31T20:42:40.124Z","message":"stopping early","service.version":"0.0.0-SNAPSHOT+00000000","service.type":"eck","ecs.version":"1.4.0","error":"test failure","error.stack_trace":"github.com/elastic/cloud-on-k8s/test/e2e/test.StepList.RunSequential\n\t/go/src/github.com/elastic/cloud-on-k8s/test/e2e/test/step.go:44\ngithub.com/elastic/cloud-on-k8s/test/e2e/test/helper.RunFile\n\t/go/src/github.com/elastic/cloud-on-k8s/test/e2e/test/helper/yaml.go:162\ngithub.com/elastic/cloud-on-k8s/test/e2e/agent.runAgentRecipe\n\t/go/src/github.com/elastic/cloud-on-k8s/test/e2e/agent/recipes_test.go:226\ngithub.com/elastic/cloud-on-k8s/test/e2e/agent.TestFleetKubernetesIntegrationRecipe\n\t/go/src/github.com/elastic/cloud-on-k8s/test/e2e/agent/recipes_test.go:131\ntesting.tRunner\n\t/usr/local/go/src/testing/testing.go:1259"}

[juliaElastic]

run full pr build

[juliaElastic]

I think the missing piece was is_default flag which was needed in elastic-package repo too: elastic/elastic-package#655
Still waiting for the build to finish, hopefully it will pass.

[naemono]

@juliaElastic I just ran a test locally, and saw it pass, so this definitely resolved the issue. Sorry for the amount of time again that it took me to replicate this. I learned some things about e2e testing during this, definitely.

[juliaElastic]

The build is passing, can someone approve to merge?

[thbkrkr]

LGTM

Nit: trim trailing whitespaces.

[juliaElastic]

LGTM

Nit: trim trailing whitespaces.

Thanks, committed your suggestions. Please merge when ready, I don't have access.