[Cloud Security] implement cloud_security_finding data stream for Wiz

[maxcold]

Proposed commit message

Implements a new data stream for Wiz, with Cloud Configuration Finding data.

Fixes:

• https://github.com/elastic/security-team/issues/9662

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

• update Readme
• figure out rule.evidence (not mapped in native)
• do we need preview version?
• decide if we need dashboard
• [ ]

How to test this PR locally

Running tests captured in the docs https://www.elastic.co/guide/en/integrations-developer/current/testing-and-validation.html

To test locally:

1. checkout the branch of this PR
2. build the Wiz package with elastic-package build
3. Run the stack with elastic-package stack up -d -v
4. Access Kibana on https://localhost:5601/ and install version of Wiz defined in the PR. Enable the new Cloud Configuration Finding data stream. For the Wiz credentials reach out to @maxcold

Related issues

• Fixes https://github.com/elastic/security-team/issues/9662
• Blocks https://github.com/elastic/security-team/issues/9664

Screenshots

[elasticmachine]

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

[kcreddy]

@maxcold, would you be able to add a dashboard for this new datastream to improve integration quality and also add to screenshots?

The CI seems to be failing on outdated README.
I usually run: elastic-package format && elastic-package lint && elastic-package check && elastic-package build to update it.

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 2e05cbe

History

• 💚 Build #14971 succeeded 8a2fdb9
• 💚 Build #14968 succeeded 8f27e16
• 💔 Build #14958 failed 944b808
• 💚 Build #14952 succeeded f991a45
• 💔 Build #14890 failed 0e5df75
• 💚 Build #14797 succeeded 75952a4

[elastic-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
96.7% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

[CohenIdo]

Most of the fields here are part of the ECS schema, I saw that also other integration mapped it manually but what is the reason for not using the ECS schema?

[maxcold]

@CohenIdo this is a good question. I indeed took it from the wiz.vulnerability data stream. @kcreddy do you happen to know why to map these fields manually? Is it only because the ecs@mappings were not there when the integration was first introduced?

Btw now when I think about it, we need to make data_stream.namespace the type keyword to support namespaces in our latest transform. Otherwise, the data from different namespaces would fail to be indexed in the latest index as per elastic/kibana#162889 (comment)

[maxcold]

@CohenIdo the fact that we need data_stream.namespace to be just keyword brings up a question for the vulnerability data stream you work on, how to migrate from constant_keyword to just keyword as it will be a breaking change of the field mapping

[maxcold]

actually, I think we should have constant_keyword in the original mapping but add keyword to the latest index mapping when working on the latest data stream. This way we should avoid validation error for the data_stream.namespace here on the original data stream

[kcreddy]

This is because the package expects these required fields to be defined, otherwise it fails in elastic-package build
https://github.com/elastic/package-spec/blob/main/code/go/internal/validator/semantic/validate_required_fields.go#L16-L25

[elasticmachine]

Package wiz - 1.6.0 containing this change is available at https://epr.elastic.co/search?package=wiz