Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Cloud Security] release Wiz version 2.0.0 #11414

Merged
merged 18 commits into from
Oct 30, 2024
+366 −147
Merged

[Cloud Security] release Wiz version 2.0.0 #11414

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
03a32bd
remove rule.references from wiz cloud_configuration_finding data stream
maxcold Oct 15, 2024
6dde700
remove rule.references from pipeline test
maxcold Oct 15, 2024
5ca6264
add PR link to changelog
maxcold Oct 16, 2024
4d557e6
release Wiz integration from preview
maxcold Oct 16, 2024
82f8834
Update packages/wiz/changelog.yml
maxcold Oct 17, 2024
c5599fa
rely on external ecs for ECS fields
maxcold Oct 17, 2024
b7cc277
add external: ecs to ECS fields where it type is re-defined
maxcold Oct 18, 2024
25632c2
default to id if provider_id is missing for resource.id
maxcold Oct 18, 2024
9f8506d
add cloud.account.name to wiz vulnerability cdr mapping
maxcold Oct 21, 2024
a2cbc19
Update packages/wiz/changelog.yml
maxcold Oct 23, 2024
e086309
remove rule.* ecs explicit mappings from cloud_configuration_finding …
maxcold Oct 23, 2024
1623967
update README
maxcold Oct 23, 2024
1184a6b
Merge branch 'main' into csp-remmove-wiz-rule-references
maxcold Oct 23, 2024
15734c8
Add Cloud Configuration Finding to the list of data streams in README
maxcold Oct 28, 2024
e7616b0
make README change in the _dev
maxcold Oct 28, 2024
93271df
add 1.8.2 to changelog
maxcold Oct 29, 2024
1560059
update wiz description
maxcold Oct 29, 2024
3f11a0f
add contextual flyout screenshot for Wiz integration
maxcold Oct 29, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 4 additions & 9 deletions packages/wiz/changelog.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,26 +1,21 @@
# newer versions go on top
- version: "1.9.0-preview05"
- version: "2.0.0"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this is the correct way to do this; the versions that have been released should continue to exist.

Also, what is it that makes this a 2.0.0? What is breaking here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what would be the correct way to do that? From a user's point of view if they never enabled "show beta integrations", they would upgrade from 1.8.1 (latest released non-preview version) to 2.0.0 and would get all the changes from 1.9.0-preview* with the change, and as a user I would expect to see this as a part of 2.0.0 changelog

Also, what is it that makes this a 2.0.0? What is breaking here?

this is reflected in the changelog

    - description: Rely on external ecs for ESC fields. rule.reference, rule.descipriton and rule.remediation changed from text to keyword.
      type: breaking-change
      link: https://github.com/elastic/integrations/pull/11414
    - description: Remove rule.references field and its mapping. Please use the ECS rule.reference field instead.
      type: breaking-change
      link: https://github.com/elastic/integrations/pull/11414

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The change here inserts a bunch of breaking changes before a bugfix that had already been released as a 1.9.0 preview. If correct chronological order had been used and they were added as a 1.9.0, the full set of 1.9.0 changes would not have been breaking; there would have been a single new version entry looking like this:

- version: "1.9.0"
  changes:
    - description: Add cloud.account.name mapping to latest_cdr_vulnerabilities transform destination.
      type: bugfix
      link: https://github.com/elastic/integrations/pull/<SOMEOTHERPRNUMBER>
    - description: Default to resource id when provider_id is missing for resource.id field in cloud_configuration_finding data stream.
      type: enhancement
      link: https://github.com/elastic/integrations/pull/<SOMEOTHERPRNUMBER>
    - description: Add cloud.account.name mapping to latest_cdr_vulnerabilities transform destination.
      type: enhancement
      link: https://github.com/elastic/integrations/pull/11414

(leaving all the other already published versions in tact)

then you could have added this breaking enhancement:

- version: "2.0.0"
  changes:
    - description: Rely on external ecs for ESC fields. rule.reference, rule.descipriton and rule.remediation changed from text to keyword.
      type: breaking-change
      link: https://github.com/elastic/integrations/pull/11414
    - description: Remove rule.references field and its mapping. Please use the ECS rule.reference field instead.
      type: breaking-change
      link: https://github.com/elastic/integrations/pull/11414

This is not an academic issue; there is a bugfix that is being hidden here from people who may not want to upgrade across a break, and also blocks them from getting non-breaking enhancements. Admittedly, there is a fix in 1.8.something hidden in a backport branch, so the bugfix is less of a concern, but the non-breaking enhancements are still an issue.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The problem is that 1.9.0-preview1 and 1.9.0-preview3 introduce the latest transforms, which we plan to release as part of the CDR initiative. These changes require the 8.16.x version of the stack; that's why we were asked to hold them in an open PR until closer to the 8.16 release date, as the flow with preview is not advised to be used. So, I can't release 1.9.0 just yet. Additionally, there are problems that we found while testing the CDR approach, together with the product folks, which require breaking changes before we can release transforms. That's why I'm skipping the 1.9.0 release and moving to 2.0.0, as all the changes there are bundled under CDR. We don't want users to be able to install the version with the transforms before the breaking changes are made. I can move the breaking change entries to the top of the changelog if you think that this would help.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is deeply dissatisfying. I have removed my block.

changes:
- description: Remove rule.references field and adjust rule.* mappings.
type: enhancement
maxcold marked this conversation as resolved.
Show resolved Hide resolved
link: https://github.com/elastic/integrations/pull/11414
- description: Increase retention on transfroms to 90 days.
type: enhancement
link: https://github.com/elastic/integrations/pull/11393
- version: "1.9.0-preview04"
changes:
- description: Update vulnerabilities mappings and ingest pipeline for better support in CDR.
type: enhancement
link: https://github.com/elastic/integrations/pull/11348
- version: "1.9.0-preview03"
changes:
- description: Add latest Transform to cloud_configuration_finding data stream to support CDR.
type: enhancement
link: https://github.com/elastic/integrations/pull/10965
- version: "1.9.0-preview02"
changes:
- description: Fix potential `got types.Null, expected iterable type` error.
type: bugfix
link: https://github.com/elastic/integrations/pull/11098
- version: "1.9.0-preview01"
changes:
- description: Add latest Transform to vulnerability data stream to support CDR
type: enhancement
link: https://github.com/elastic/integrations/pull/10895
Expand Down
1 change: 0 additions & 1 deletion ...nfiguration_finding/_dev/test/pipeline/test-cloud-configuration-finding.log-expected.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -212,7 +212,6 @@
"id": "VirtualMachines-021",
"name": "Virtual Machine should not be stopped (allocated) for more than a week",
"reference": "https://learn.microsoft.com/en-us/azure/virtual-machines/states-billing",
"references": "https://learn.microsoft.com/en-us/azure/virtual-machines/states-billing",
"remediation": "Perform the following command to deallocate the VM via Azure CLI:\n```\naz vm deallocate \\\n\t--ids {{vmId}}\n```",
"uuid": "56c8890d-ad68-4659-9414-fb0ed7258c31"
},
Expand Down
5 changes: 0 additions & 5 deletions ...ges/wiz/data_stream/cloud_configuration_finding/elasticsearch/ingest_pipeline/default.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -332,11 +332,6 @@ processors:
tag: set_rule_reference
copy_from: wiz.cloud_configuration_finding.evidence.cloud_configuration_link
ignore_empty_value: true
- set:
field: rule.references
tag: set_rule_references
copy_from: wiz.cloud_configuration_finding.evidence.cloud_configuration_link
ignore_empty_value: true
- remove:
field: json
tag: remove_json
Expand Down
2 changes: 0 additions & 2 deletions packages/wiz/data_stream/cloud_configuration_finding/fields/rule.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,5 @@
type: text
- name: remediation
type: text
- name: references
type: text
- name: reference
type: text
Copy link
Member

@andrewkroh andrewkroh Oct 16, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The rule fields are part of ECS. Anywhere we have an ECS field we should depend on the ECS definition through the use of external: ecs to ensure that we always have consistent mappings. This ensures that there are never data type conflicts between data streams for ECS fields.

This is what it would look like to use external: ecs for the ECS fields in wiz -- d3d335c.

However, after doing that clean-up it becomes easier to spot other problems (not new in this PR)

  1. There are two problematic field definitions that are conflicting with ECS.

    packages/wiz/data_stream/cloud_configuration_finding/fields/rule.yml:10:7 rule.description field declared as type text conflicts with the ECS data type keyword (conflict)
packages/wiz/data_stream/cloud_configuration_finding/fields/rule.yml:14:7 rule.reference field declared as type text conflicts with the ECS data type keyword (conflict)

  2. rule.remediation has multiple data types (keyword, text)

    packages/wiz/elasticsearch/transform/latest_cdr_misconfigurations/fields/rule.yml:12:7 keyword (conflict)
packages/wiz/data_stream/cloud_configuration_finding/fields/rule.yml:12:7 text (conflict)

  3. duplicate definitions of the same field in a data stream

    packages/wiz/elasticsearch/transform/latest_cdr_misconfigurations/fields/ecs.yml:1:3 event.created is declared 2 times (duplicate)
  packages/wiz/elasticsearch/transform/latest_cdr_misconfigurations/fields/ecs.yml:19:3 additional definition (duplicate)
packages/wiz/elasticsearch/transform/latest_cdr_vulnerabilities/fields/fields.yml:5:3 package.name is declared 2 times (duplicate)
  packages/wiz/elasticsearch/transform/latest_cdr_vulnerabilities/fields/fields.yml:23:3 additional definition (duplicate)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@andrewkroh thanks for the detailed feedback! I updated the mapping according to your suggestion. A couple of questions:

  1. how do I check for conflicts with ECS? neither lint nor check produce the same output as you have in your comment
  2. I changed rule.description and rule.reference to ecs and that means the mapping changed from text to keyword which is a breaking change as far as I understand. What should be the guidance for users in the changelog in that case?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how do I check for conflicts with ECS?

I don't know that elastic-package has this capability yet. I have an unofficial means that I occasionally use to audit the mappings.

I changed rule.description and rule.reference to ecs and that means the mapping changed from text to keyword which is a breaking change as far as I understand. What should be the guidance for users in the changelog in that case?

It's hard to prescribe one solution because it depends on how the data is being used and what problem they encounter (types of queries or aggregations they are performing). One solution for conflicts mentioned in this blog post is to apply a runtime field to the old index to make the field a keyword.

Copy link
Contributor

@kcreddy kcreddy Oct 23, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since we are adhering to ECS, should we even define the rule fields that are of external: ecs? The logs-* default ecs@mappings component template should take care of them for the datastream.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kcreddy @andrewkroh Looking at the value of the ecs@mappings in my stack I don't see it as a replacement of external: ecs . Actually, I would like to understand the use of ecs@mappings better, as in our team we understood that it is a replacement of manual mapping, but then not everything that is available in ECS also mapped in this component template. Eg. there are no mappings for the rule.* fields, except the ones falling into broader mappings such as *.name

{
  "dynamic_templates": [
    {
      "ecs_timestamp": {
        "mapping": {
          "ignore_malformed": false,
          "type": "date"
        },
        "match": "@timestamp"
      }
    },
    {
      "ecs_message_match_only_text": {
        "path_match": [
          "message",
          "*.message"
        ],
        "mapping": {
          "type": "match_only_text"
        },
        "unmatch_mapping_type": "object"
      }
    },
    {
      "ecs_non_indexed_keyword": {
        "path_match": [
          "*event.original"
        ],
        "mapping": {
          "index": false,
          "type": "keyword",
          "doc_values": false
        }
      }
    },
    {
      "ecs_non_indexed_long": {
        "path_match": [
          "*.x509.public_key_exponent"
        ],
        "mapping": {
          "index": false,
          "type": "long",
          "doc_values": false
        }
      }
    },
    {
      "ecs_ip": {
        "path_match": [
          "ip",
          "*.ip",
          "*_ip"
        ],
        "mapping": {
          "type": "ip"
        },
        "match_mapping_type": "string"
      }
    },
    {
      "ecs_wildcard": {
        "path_match": [
          "*.io.text",
          "*.message_id",
          "*registry.data.strings",
          "*url.path"
        ],
        "mapping": {
          "type": "wildcard"
        },
        "unmatch_mapping_type": "object"
      }
    },
    {
      "ecs_path_match_wildcard_and_match_only_text": {
        "path_match": [
          "*.body.content",
          "*url.full",
          "*url.original"
        ],
        "mapping": {
          "fields": {
            "text": {
              "type": "match_only_text"
            }
          },
          "type": "wildcard"
        },
        "unmatch_mapping_type": "object"
      }
    },
    {
      "ecs_match_wildcard_and_match_only_text": {
        "mapping": {
          "fields": {
            "text": {
              "type": "match_only_text"
            }
          },
          "type": "wildcard"
        },
        "unmatch_mapping_type": "object",
        "match": [
          "*command_line",
          "*stack_trace"
        ]
      }
    },
    {
      "ecs_path_match_keyword_and_match_only_text": {
        "path_match": [
          "*.title",
          "*.executable",
          "*.name",
          "*.working_directory",
          "*.full_name",
          "*file.path",
          "*file.target_path",
          "*os.full",
          "*email.subject",
          "*vulnerability.description",
          "*user_agent.original"
        ],
        "mapping": {
          "fields": {
            "text": {
              "type": "match_only_text"
            }
          },
          "type": "keyword"
        },
        "unmatch_mapping_type": "object"
      }
    },
    {
      "ecs_date": {
        "path_match": [
          "*.timestamp",
          "*_timestamp",
          "*.not_after",
          "*.not_before",
          "*.accessed",
          "created",
          "*.created",
          "*.installed",
          "*.creation_date",
          "*.ctime",
          "*.mtime",
          "ingested",
          "*.ingested",
          "*.start",
          "*.end",
          "*.indicator.first_seen",
          "*.indicator.last_seen",
          "*.indicator.modified_at",
          "*threat.enrichments.matched.occurred"
        ],
        "mapping": {
          "type": "date"
        },
        "unmatch_mapping_type": "object"
      }
    },
    {
      "ecs_path_match_float": {
        "path_match": [
          "*.score.*",
          "*_score*"
        ],
        "mapping": {
          "type": "float"
        },
        "path_unmatch": "*.version",
        "unmatch_mapping_type": "object"
      }
    },
    {
      "ecs_usage_double_scaled_float": {
        "path_match": "*.usage",
        "mapping": {
          "scaling_factor": 1000,
          "type": "scaled_float"
        },
        "match_mapping_type": [
          "double",
          "long",
          "string"
        ]
      }
    },
    {
      "ecs_geo_point": {
        "path_match": [
          "*.geo.location"
        ],
        "mapping": {
          "type": "geo_point"
        }
      }
    },
    {
      "ecs_flattened": {
        "path_match": [
          "*structured_data",
          "*exports",
          "*imports"
        ],
        "mapping": {
          "type": "flattened"
        },
        "match_mapping_type": "object"
      }
    },
    {
      "all_strings_to_keywords": {
        "mapping": {
          "ignore_above": 1024,
          "type": "keyword"
        },
        "match_mapping_type": "string"
      }
    }
  ]
}

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AFAIU the idea of ecs@mappings component template is to have a generic set of dynamic templates that works across all ECS entities, instead of defining mappings for specific entities like rule.*, vulnerability.* etc.

In our case, following are rule.* fields that are defined as external: ecs and the corresponding dynamic template inside ecs@mappings that is handling it:

Field in rule.yml Handled by dynamic template
rule.uuid all_strings_to_keywords
rule.id all_strings_to_keywords
rule.name ecs_path_match_keyword_and_match_only_text
rule.description all_strings_to_keywords
rule.reference all_strings_to_keywords

Comparing the above dynamic mappings with ECS rule fields, it matches correctly.

rule.remediation is anyway manually defined as it is not in ECS.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kcreddy now I see how keyword mapping is being solved in the template, thanks. Then yes, I agree, it makes sense to remove these explicit ecs mappings

1 change: 0 additions & 1 deletion packages/wiz/docs/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -325,7 +325,6 @@ An example event for `cloud_configuration_finding` looks as following:
| rule.id | | keyword |
| rule.name | | keyword |
| rule.reference | | text |
| rule.references | | text |
| rule.remediation | | text |
| rule.uuid | | keyword |
| tags | User defined tags. | keyword |
Expand Down
2 changes: 1 addition & 1 deletion packages/wiz/elasticsearch/transform/latest_cdr_misconfigurations/fields/ecs.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,4 +29,4 @@
- name: event.type
external: ecs
- name: observer.vendor
external: ecs
external: ecs
8 changes: 3 additions & 5 deletions packages/wiz/elasticsearch/transform/latest_cdr_misconfigurations/fields/rule.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,10 +8,8 @@
- name: name
type: keyword
- name: description
type: text
type: keyword
- name: remediation
type: text
- name: references
type: text
type: keyword
- name: reference
type: text
type: keyword
2 changes: 1 addition & 1 deletion packages/wiz/elasticsearch/transform/latest_cdr_misconfigurations/transform.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,4 +27,4 @@ _meta:
managed: true
# Bump this version to delete, reinstall, and restart the transform during package.
# Version bump is needed if there is any code change in transform.
fleet_transform_version: 0.1.0
fleet_transform_version: 0.1.0
2 changes: 1 addition & 1 deletion packages/wiz/manifest.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
format_version: 3.0.2
name: wiz
title: Wiz
version: "1.9.0-preview05"
version: "2.0.0"
description: Collect logs from Wiz with Elastic Agent.
type: integration
categories:
Expand Down