aws.securityhub_findings: Improve support for CDR

[kcreddy]

Proposed commit message

Improve support for CDR.

• Update securityhub_findings data stream's ingest pipeline to support CDR.
• Update securityhub_findings data stream's mappings according to the new fields.
• Update Findings Summary dashboard with 3 new visualisations using newly added fields.
• Add new latest transform to store latest findings inside destination indices used by Cloud Security workflows.
• Update minimum kibana version to 8.16.0 as the transform privileges are added in [Cloud Security] Add privileges required for AWS SecurityHub related to CDR misconfiguration features elasticsearch#112574 merged into 8.16 Stack release.

Fixes: #11040

Note to reviewers: Please DM me for access to the document(s) linked in the issue, it might help in the review.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.

How to test this PR locally

cd packages/aws && elastic-package stack down && elastic-package build && elastic-package stack up --version=8.14.0 -d -v && eval "$(elastic-package stack shellinit)" && elastic-package test pipeline --generate -v --data-streams=securityhub_findings

Related issues

• Closes aws.securityhub_findings: Implement mappings for Cloud Security Workflow #11040

Screenshots

[elastic-vault-github-plugin-prod]

🚀 Benchmarks report

Package aws 👍(8) 💚(7) 💔(4)

Expand to view

Data stream	Previous EPS	New EPS	Diff (%)	Result
s3access	4651.16	3831.42	-819.74 (-17.62%)	💔
apigateway_logs	10989.01	4464.29	-6524.72 (-59.37%)	💔
ec2_metrics	25000	17857.14	-7142.86 (-28.57%)	💔
firewall_logs	3300.33	2645.5	-654.83 (-19.84%)	💔

To see the full report comment with /test benchmark fullreport

[elasticmachine]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

[efd6]

The issue refers to a document upload, but I cannot find it. So I cannot see whether this follows what has been designed. Is there a reason this is not a public document in the issue?

[kcreddy]

/test

[maxcold]

A couple of more comments to fix and good to go. Thanks for the patience!

[andrewkroh]

The three fields being converted to constant_keyword would all benefit from removal from _source.

I recommend setting the static values in the ecs.yml file where they fields are declared instead of the ingest pipeline, and then exchange the three set processors with a single remove processor that has a description field explaining that the fields are defined as constant_keyword and we are removing the fields from _source to gain storage efficiency.

[andrewkroh]

Also, please update the commit message (in the PR description) to specify why the minimum kibana version was changed.

[kcreddy]

@andrewkroh the comments are addressed in 0e44091 and PR commit message is also updated.

[andrewkroh]

I left a few more minor comments.

[shmsr]

Based on other approvals, approving as a CODEOWNER from @elastic/obs-infraobs-integrations

[elastic-sonarqube]

Quality Gate failed

Failed conditions
48.0% Coverage on New Code (required ≥ 80%)

See analysis details on SonarQube

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: aafe9a9

History

• 💚 Build #17728 succeeded 1080998
• 💚 Build #17708 succeeded a0a7e23
• 💔 Build #17700 failed d861580
• 💔 Build #17673 failed 0e44091
• 💚 Build #17658 succeeded 0dfc25d
• 💚 Build #17134 succeeded 1d92a8f

cc @kcreddy

[elastic-vault-github-plugin-prod]

Package aws - 2.31.0 containing this change is available at https://epr.elastic.co/search?package=aws