Flaky PR build with no matches for kind "Agent"

[thbkrkr]

We are seeing some flaky PR builds with this error:

{
    "log.level": "info",
    "@timestamp": "2022-07-28T12:51:03.391Z",
    "log.logger": "e2e-qrwqu",
    "message": "Deploying monitoring",
    "service.version": "0.0.0-SNAPSHOT+00000000",
    "service.type": "eck",
    "ecs.version": "1.4.0"
}
{
    "log.level": "error",
    "@timestamp": "2022-07-28T12:51:03.829Z",
    "log.logger": "e2e-qrwqu",
    "message": "Command execution failed",
    "service.version": "0.0.0-SNAPSHOT+00000000",
    "service.type": "eck",
    "ecs.version": "1.4.0",
    "command": "kubectl apply -f /tmp/eck-e2e/e2e-qrwqu/monitoring.yaml2621560727",
    "error": "exit status 1, stderr: error: unable to recognize \"/tmp/eck-e2e/e2e-qrwqu/monitoring.yaml2621560727\": no matches for kind \"Agent\" in version \"agent.k8s.elastic.co/v1alpha1\"",
    "error.stack_trace": "github.com/elastic/cloud-on-k8s/v2/test/e2e/cmd/run.(*helper).kubectl
      /go/src/github.com/elastic/cloud-on-k8s/test/e2e/cmd/run/run.go:738
    github.com/elastic/cloud-on-k8s/v2/test/e2e/cmd/run.(*helper).kubectlApplyTemplate
      /go/src/github.com/elastic/cloud-on-k8s/test/e2e/cmd/run/run.go:723
    github.com/elastic/cloud-on-k8s/v2/test/e2e/cmd/run.(*helper).kubectlApplyTemplateWithCleanup
      /go/src/github.com/elastic/cloud-on-k8s/test/e2e/cmd/run/run.go:728
    github.com/elastic/cloud-on-k8s/v2/test/e2e/cmd/run.(*helper).deployMonitoring
      /go/src/github.com/elastic/cloud-on-k8s/test/e2e/cmd/run/run.go:463
    github.com/elastic/cloud-on-k8s/v2/test/e2e/cmd/run.doRun
      /go/src/github.com/elastic/cloud-on-k8s/test/e2e/cmd/run/run.go:90
    github.com/elastic/cloud-on-k8s/v2/test/e2e/cmd/run.Command.func2
      /go/src/github.com/elastic/cloud-on-k8s/test/e2e/cmd/run/command.go:74
    github.com/spf13/cobra.(*Command).execute
      /go/pkg/mod/github.com/spf13/[email protected]/command.go:872
    github.com/spf13/cobra.(*Command).ExecuteC
      /go/pkg/mod/github.com/spf13/[email protected]/command.go:990
    github.com/spf13/cobra.(*Command).Execute
      /go/pkg/mod/github.com/spf13/[email protected]/command.go:918
    main.main
      /go/src/github.com/elastic/cloud-on-k8s/test/e2e/cmd/main.go:32
    runtime.main
      /usr/local/go/src/runtime/proc.go:250"
}

Examples:
https://devops-ci.elastic.co/blue/organizations/jenkins/cloud-on-k8s-pr/detail/cloud-on-k8s-pr/9424/pipeline
https://devops-ci.elastic.co/blue/organizations/jenkins/cloud-on-k8s-pr/detail/cloud-on-k8s-pr/9430/pipeline

It seems to me that the first occurrence happened in PR #5781 (build) and since we merged the PR, we had the same error in other PR builds.

[thbkrkr]

Closing as this never happened again and the CI was revamped when migrating to Buildkite.