[Fleet] Endpoint security integration tests can fail due to saved object errors

[cmacknz]

The endpoint security integration tests in https://github.com/elastic/elastic-agent/blob/main/testing/integration/endpoint_security_test.go can randomly fail with errors related to saved objects and transforms.

The first error I've observed is Saved object [epm-packages/endpoint] not found}:

    endpoint_security_test.go:83: 
        	Error Trace:	/home/ubuntu/agent/testing/integration/endpoint_security_test.go:248
        	            				/home/ubuntu/agent/testing/integration/endpoint_security_test.go:83
        	Error:      	Received unexpected error:
        	            	http error response with code 404: {StatusCode:404 Error:Not Found Message:Saved object [epm-packages/endpoint] not found}
        	Test:       	TestInstallAndCLIUninstallWithEndpointSecurity

There is a second error that frequently follows this one, data_frame_transform_state_and_stats-endpoint.metadata_current-default-8.9.0]: version conflict, document already exists (current version [1])}. I suspect that this problem only occurs when our first attempt to install fails, and might be a symptom of a partially successful package installation being retried.

    endpoint_security_test.go:83: 
        	Error Trace:	/home/ubuntu/agent/testing/integration/endpoint_security_test.go:248
        	            				/home/ubuntu/agent/testing/integration/endpoint_security_test.go:83
        	Error:      	Received unexpected error:
        	            	http error response with code 500: {StatusCode:500 Error:Internal Server Error Message:Error installing endpoint 8.9.0: runtime_exception
        	            		Caused by:
        	            			version_conflict_engine_exception: [data_frame_transform_state_and_stats-endpoint.metadata_current-default-8.9.0]: version conflict, document already exists (current version [1])
        	            		Root causes:
        	            			runtime_exception: Failed to persist transform statistics for transform [endpoint.metadata_current-default-8.9.0]}
        	Test:       	TestInstallAndCLIUninstallWithEndpointSecurity

[elasticmachine]

Pinging @elastic/elastic-agent (Team:Elastic-Agent)

[cmacknz]

@juliaElastic we are using the package policies API in a test to install Elastic Defend, and it sometimes fails with the errors above related to saved objects. This isn't a bug in the agent itself, and is either a possible bug on the Fleet side or we are just using the APIs wrong. Any ideas what can cause these problems? Particularly the Saved object [epm-packages/endpoint] not found error?

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[juliaElastic]

@cmacknz I've seen some of these errors before, not sure about the cause. Can we document the manual steps to reproduce, so we can look into the kibana logs to debug?

This SO error was reported in a discuss issue where the package was tried to be rolled back:
https://discuss.elastic.co/t/kafka-integration-installation-error-limit-of-total-dimension-fields-16-has-been-exceeded/339052/8

Should we move this to kibana repo to investigate?

[cmacknz]

This is reproduced quite regularly but our integration test for whether the agent can run endpoint security. We use the package policies API to install the Elastic Defend package and it fails here.

https://github.com/elastic/elastic-agent/blob/39f3d7783caae3c11d550c63ab14e9c96dc7bf8a/testing/integration/endpoint_security_test.go#L60

This is where Defend is installed: https://github.com/elastic/elastic-agent/blob/39f3d7783caae3c11d550c63ab14e9c96dc7bf8a/testing/integration/endpoint_security_test.go#L233

Instructions for running the tests are in https://github.com/elastic/elastic-agent/blob/main/docs/test-framework-dev-guide.md.

Our tests will provision a new Elastic Stack for each run which might be part of the problem here, but there is nothing in that test that you couldn't easily script on the CLI against a cloud deployment you created yourself.

Something that continuously uninstalls and then installs the Elastic Defend package might reproduce this.

[cmacknz]

Should we move this to kibana repo to investigate?

Done, CC @jlind23. This bug causes the agent endpoint security integration test to fail fairly regularly.

[juliaElastic]

I could reproduce the bug with transform conflict error, it happens if I try to reinstall the endpoint package a few times from the UI.
I found that the reason is that the code tries to create the same transform multiple times, it seems that the same transform is duplicated in transformPaths here
It looks like a bug because the paths come from the package zip, where there is one file per transform.
Something is wrong with how we are caching the content of the archive, it seems to add another duplicate of a path on every reinstall.

[
  'endpoint-8.9.1/elasticsearch/transform/metadata_current/default.json',
  'endpoint-8.9.1/elasticsearch/transform/metadata_current/default.json',
  'endpoint-8.9.1/elasticsearch/transform/metadata_united/default.json',
  'endpoint-8.9.1/elasticsearch/transform/metadata_united/default.json',
]

I think the duplicates are coming from here, looks like there is a bug where we push the same path twice.
https://github.com/nchaulet/kibana/blob/4fe3cce7d7427ceb4a7886b24b6334ef47da238d/x-pack/plugins/fleet/server/services/epm/archive/storage.ts#L213-L218

I'm not sure if the SO not found error is related, we can see if it can be reproduced after the first issue is fixed.