[Meta] 7.14 Planned changes to all packages

[P1llus]

This issue is an overview of planned changes that affect all or most packages to prepare them for GA.

List of tasks:

• Update ECS version to 1.10
  Each package should have an updated ECS version, even if no changes is needed for the specific package, the new fields added in 1.10 is:
  New fields:
  Datastream: https://github.com/elastic/ecs/blob/master/rfcs/text/0009-data_stream-fields.md
  Beta fields:
  Orchestrator: https://github.com/elastic/ecs/blob/master/rfcs/text/0012-orchestrator-field-set.md
  Experimental:
  Threat fields: https://github.com/elastic/ecs/blob/master/rfcs/text/0018-extend-threat-group-software.md

• Make sure the package is synced with any changes done to its related module
  There might still be changes, hotfixes or enhancements added to modules which is not synced over.

• Make sure all packages have pipeline test files.
  Currently not all packages have pipeline test files, this should be added to make sure it has more feature parity with how CI tests are done for pipelines in the beats repo.

• Remove any edge processing that might still exist.
  All edge processing is removed from the security-integrations packges already, but there might be some smaller processors left to convert.

• Implementation of "Preserve Raw Event" option for all packages.
  All packages should follow the principles discussed in an earlier issue: event.original optionality across all packages #777 (comment)
  Each package will have the following changes:

1. Move the original data from its source field (default is the message field), to event.original.
2. All ingest pipeline processors handling values stored in the message field, will instead handle the event.original field. This simplifies reindexing of data by a lot,
3. Preserve raw event defaults to off.
4. Overwrite any existing setting if some packages might already handle event.original, with on/off by default.
5. A menu option is added to preserve raw events for each package:

    vars:
      - name: preserve_original_event
        required: true
        show_user: true
        title: Preserve original event
        description: Preserves a raw copy of the original event, added to the field event.original
        type: bool
        multi: false
        default: false

Which looks like this:

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

[P1llus]

As discussed @masci these are the changes we would like to perform, in which we could always track cross-team.

[P1llus]

Packages to track:

apache
PR: #1068

• Update ECS.
• Sync package if any module changes.
• Add pipeline test files
• Remove edge processing
• Implement Preserve Raw Event

auditd
PR: #1031

• Update ECS.
• Sync package if any module changes.
• Add pipeline test files
• Remove edge processing
• Implement Preserve Raw Event

aws
PR: #1070

• Update ECS.
• Sync package if any module changes.
• Add pipeline test files
• Remove edge processing
• Implement Preserve Raw Event

azure
PR: #1113
PR to sync changes: #1121

• Update ECS.
• Sync package if any module changes.
• Add pipeline test files
• Remove edge processing
• Implement Preserve Raw Event

barracuda
PR: #1042

• Update ECS.
• Sync package if any module changes.
• Add pipeline test files
• Remove edge processing (need to keep decode_cef processor)
• Implement Preserve Raw Event

bluecoat
PR: #1072

• Update ECS.
• Sync package if any module changes.
• Add pipeline test files
• Remove edge processing
• Implement Preserve Raw Event

cef
PR: #1032

• Update ECS.
• Sync package if any module changes.
• Add pipeline test files
• Remove edge processing (need to keep decode_cef processor)
• Implement Preserve Raw Event

checkpoint
PR: #1033

• Update ECS.
• Sync package if any module changes.
• Add pipeline test files
• Remove edge processing
• Implement Preserve Raw Event

cisco
PR: #1035

• Update ECS.
• Sync package if any module changes.
• Add pipeline test files
• Remove edge processing (with the exception of RSA filesets)
• Implement Preserve Raw Event

crowdstrike
PR: #1036

• Update ECS.
• Sync package if any module changes.
• Add pipeline test files
• Remove edge processing
• Implement Preserve Raw Event

cyberark
PR: #1037

• Update ECS.
• Sync package if any module changes.
• Add pipeline test files
• Remove edge processing (did not remove RSA parser components)
• Implement Preserve Raw Event

cyberarkpas
PR: #1039

• Update ECS.
• Sync package if any module changes.
• Add pipeline test files
• Remove edge processing (did not remove RSA parser components)
• Implement Preserve Raw Event

cylance
PR: #1040

• Update ECS.
• Sync package if any module changes.
• Add pipeline test files
• Remove edge processing (did not remove RSA parser components)
• Implement Preserve Raw Event

docker

• Update ECS.
• Sync package if any module changes.
• Add pipeline test files
• Remove edge processing
• Implement Preserve Raw Event

f5
PR: #1041

• Update ECS.
• Sync package if any module changes.
• Add pipeline test files
• Remove edge processing (did not remove RSA parser components)
• Implement Preserve Raw Event

fortinet
PR: #1092

• Update ECS.
• Sync package if any module changes.
• Add pipeline test files
• Remove edge processing (did not remove RSA parser components)
• Implement Preserve Raw Event

gcp
PR: #1045

• Update ECS.
• Sync package if any module changes.
• Add pipeline test files
• Remove edge processing
• Implement Preserve Raw Event

google_workspace
PR: #1046

• Update ECS.
• Sync package if any module changes.
• Add pipeline test files
• Remove edge processing
• Implement Preserve Raw Event

haproxy
PR: #1048

• Update ECS.
• Sync package if any module changes.
• Add pipeline test files
• Remove edge processing
• Implement Preserve Raw Event

iis
PR: #1054

• Update ECS.
• Sync package if any module changes.
• Add pipeline test files
• Remove edge processing
• Implement Preserve Raw Event

imperva
PR: #1055

• Update ECS.
• Sync package if any module changes.
• Add pipeline test files
• Remove edge processing (did not remove RSA parser components)
• Implement Preserve Raw Event

infoblox
PR: #1056

• Update ECS.
• Sync package if any module changes.
• Add pipeline test files
• Remove edge processing (did not remove RSA parser components)
• Implement Preserve Raw Event

iptables
PR: #1057

• Update ECS.
• Sync package if any module changes.
• Add pipeline test files
• Remove edge processing
• Implement Preserve Raw Event

juniper
PR: #1058

• Update ECS.
• Sync package if any module changes.
• Add pipeline test files
• Remove edge processing (did not remove RSA parser components)
• Implement Preserve Raw Event

kafka
PR: #1116

• Update ECS.
• Sync package if any module changes.
• Add pipeline test files
• Remove edge processing
• Implement Preserve Raw Event

kubernetes

• Update ECS.
• Sync package if any module changes.
• Add pipeline test files
• Remove edge processing
• Implement Preserve Raw Event

linux

• Update ECS.
• Sync package if any module changes.
• Add pipeline test files
• Remove edge processing
• Implement Preserve Raw Event

microsoft
PR: #1059

• Update ECS.
• Sync package if any module changes.
• Add pipeline test files
• Remove edge processing
• Implement Preserve Raw Event

mongodb
PR: #1060

• Update ECS.
• Sync package if any module changes.
• Add pipeline test files
• Remove edge processing
• Implement Preserve Raw Event

mysql

• Update ECS.
• Sync package if any module changes.
• Add pipeline test files
• Remove edge processing
• Implement Preserve Raw Event

nats
PR: #1061

• Update ECS.
• Sync package if any module changes.
• Add pipeline test files
• Remove edge processing
• Implement Preserve Raw Event

netflow
PR: #1062

• Update ECS.
• Sync package if any module changes.
• Add pipeline test files
• Remove edge processing
• Implement Preserve Raw Event

netscout
PR: #1063

• Update ECS.
• Sync package if any module changes.
• Add pipeline test files
• Remove edge processing
• Implement Preserve Raw Event

nginx
PR: #1065

• Update ECS.
• Sync package if any module changes.
• Add pipeline test files
• Remove edge processing
• Implement Preserve Raw Event

nginx_ingress_controller
PR: #1066

• Update ECS.
• Sync package if any module changes.
• Add pipeline test files
• Remove edge processing
• Implement Preserve Raw Event

o365
PR: #1117

• Update ECS.
• Sync package if any module changes.
• Add pipeline test files
• Remove edge processing
• Implement Preserve Raw Event

okta
PR: #1067

• Update ECS.
• Sync package if any module changes.
• Add pipeline test files
• Remove edge processing
• Implement Preserve Raw Event

panw
PR: #1093

• Update ECS.
• Sync package if any module changes.
• Add pipeline test files
• Remove edge processing
• Implement Preserve Raw Event

postgresql
PR: #1095

• Update ECS.
• Sync package if any module changes.
• Add pipeline test files
• Remove edge processing
• Implement Preserve Raw Event

proofpoint
PR: #1096

• Update ECS.
• Sync package if any module changes.
• Add pipeline test files
• Remove edge processing
• Implement Preserve Raw Event

rabbitmq
PR: #1097

• Update ECS.
• Sync package if any module changes.
• Add pipeline test files
• Remove edge processing
• Implement Preserve Raw Event

radware
PR: #1098

• Update ECS.
• Sync package if any module changes.
• Add pipeline test files
• Remove edge processing
• Implement Preserve Raw Event

redis
PR: #1099

• Update ECS.
• Sync package if any module changes.
• Add pipeline test files
• Remove edge processing
• Implement Preserve Raw Event

santa
PR: #1100

• Update ECS.
• Sync package if any module changes.
• Add pipeline test files
• Remove edge processing
• Implement Preserve Raw Event

sonicwall
PR: #1101

• Update ECS.
• Sync package if any module changes.
• Add pipeline test files
• Remove edge processing
• Implement Preserve Raw Event

sophos
#1102

• Update ECS.
• Sync package if any module changes.
• Add pipeline test files
• Remove edge processing
• Implement Preserve Raw Event

squid
#1103

• Update ECS.
• Sync package if any module changes.
• Add pipeline test files
• Remove edge processing
• Implement Preserve Raw Event

stan
PR: #1104

• Update ECS.
• Sync package if any module changes.
• Add pipeline test files
• Remove edge processing
• Implement Preserve Raw Event

suricata
PR: #1105

• Update ECS.
• Sync package if any module changes.
• Add pipeline test files
• Remove edge processing
• Implement Preserve Raw Event

system

• Update ECS.
• Sync package if any module changes.
• Add pipeline test files
• Remove edge processing
• Implement Preserve Raw Event

tomcat
PR: #1106

• Update ECS.
• Sync package if any module changes.
• Add pipeline test files
• Remove edge processing
• Implement Preserve Raw Event

traefik
PR: #1107

• Update ECS.
• Sync package if any module changes.
• Add pipeline test files
• Remove edge processing
• Implement Preserve Raw Event

windows

• Update ECS.
• Sync package if any module changes.
• Add pipeline test files
• Remove edge processing
• Implement Preserve Raw Event

zeek
PR: #1109

• Update ECS.
• Sync package if any module changes.
• Add pipeline test files
• Remove edge processing
• Implement Preserve Raw Event

zoom

• Update ECS.
• Sync package if any module changes.
• Add pipeline test files
• Remove edge processing
• Implement Preserve Raw Event

zscaler
PR: #1112

• Update ECS.
• Sync package if any module changes.
• Add pipeline test files
• Remove edge processing
• Implement Preserve Raw Event

[epixa]

I'm going to close this since 7.14 is out. Feel free to reopen/rename if there's still something to track here.