[Cloud Posture] Install Transforms using package assets

[CohenIdo]

Cloud security’s latest findings Transform is installed today using the Kibana plugin.
Recently, Transform was added as an asset to the package-spec.

This task involves deprecating the code in Kibana that installs Transform, and instead using the package asset.

Definition of done

• Transform installation removed from cloud security's Kibana plugin.
• The same Transform logic is installed using cloud security packages assets
• Make sure DLM is supported (see reference to this requirement https://github.com/elastic/security-team/issues/7441#issuecomment-1708011156)

References:

• https://github.com/elastic/package-spec/tree/main/test/packages/good/elasticsearch/transform
• https://github.com/elastic/package-spec/blob/main/spec/integration/elasticsearch/transform/spec.yml

[elasticmachine]

Pinging @elastic/kibana-accessibility (Project:Accessibility)

[elasticmachine]

Pinging @elastic/kibana-cloud-security-posture (Team:Cloud Security)

[CohenIdo]

WIP

when trying to install transform via integration I get the following error:

Error installing cloud_security_posture 1.2.14: index_not_found_exception Root causes: index_not_found_exception: no such 
index [logs-cloud_security_posture.findings_latest-default-1.2.14]

its mean we first need to create the latest index, since the Transform api not support creating index if not exits.

which is weird - the index should be created by the Transform.

open questions:

• how we will handle with users that there transform is already installed via Kibana? how the upgrade process will look?

[CohenIdo]

When trying to add configuration for creating Transform I am getting the error bellow:

Error installing cloud_security_posture 1.2.30: index_not_found_exception Root causes: index_not_found_exception: no 
such index [logs-cloud_security_posture.findings_latest-default]

• Latest findings transform integrations#5475

When debug it, it seems that it fails on the data stream creation:

I thought maybe there is an error with how the index is created, but the API call for creating the index is identical to the one we are using today:
Create index in fleet for the Transform asset VS creating index in cloud security posture plugin.

However, I tried also to use the example that show in the package-spec and get the same error:

Error installing cloud_security_posture 1.2.26: validation_exception Root causes: validation_exception: Validation Failed: 1: 
no such index [kibana_sample_data_ecommerce];

• add-example-transform integrations#5473

[qn895]

The current issue that might be blocking this package is the insufficient permissions for kibana_system user relating to the indices. As of 8.7, Fleet automatically adds the transform/package version to the index name. So for example, if originally it’s logs-cloud_security_posture.findings_latest-default, it will be installed as logs-cloud_security_posture.findings_latest-default-1.2.30. This is to help with upgrade/migration purposes between package versions. As a result, the current permission set for logs-cloud_security_posture.findings_latest-default from Elasticsearch's side is insufficient (but for example, changing it to logs-cloud_security_posture.findings_latest-default* will resolve the issue).

However, for 8.8, we are currently working a much better mechanism allowing transforms to be installed without using kibana_system user and instead using the user's permission to prevent these issues. We recommend postponing this package change until that work is done. With the new mechanism, there might be some change required from the original package-spec.

[kfirpeled]

Thanks @qn895 that information helps a lot.
For now, we will stick with our current workaround to create the transform through our kibana's plugin instead of using the integration.

[CohenIdo]

Thank you @qn895.
Closing this issue for now.

cc: @tehilashn

[tehilashn]

Thanks for the update @CohenIdo .
As Kfir pointed out, let's wait with that for 8.9

[CohenIdo]

so @qn895 shared with me the following update:

Reaching out to let you know a lot of the work for the permissions are now in 8.8. I have a document that showcases the restrictions and recommended best practice for adding transforms to a package here: https://docs.google.com/document/d/1dpqFxdTrX9ytOvMKHSZzXYUecxqqYu7Sb9_xLGKHajM/edit#heading=h.nwxx72r49asj

In case that after #153875 we will decide to continue using Transform, we are good to go with moving the logic to the package assets.

[kfirpeled]

Removed from block - we would like to have that capability regardless of #153875

[CohenIdo]

Last time we tried to work in this we handled permissions issue, @qn895 updated me that it solved:

Reaching out to let you know a lot of the work for the permissions are now in 8.8. I have a document that showcases the restrictions and recommended best practice for adding transforms to a package here: https://docs.google.com/document/d/1dpqFxdTrX9ytOvMKHSZzXYUecxqqYu7Sb9_xLGKHajM/edit#heading=h.nwxx72r49asj

[sophiec20]

Mentioning for visibility. For auto-restart capabilities in transforms, please set unattended: true in the transform config. PUT _transform