UPGRADE: Elastic 8.3.2

[weslambert]

Elasticsearch

https://www.elastic.co/guide/en/elasticsearch/reference/master/migrating-8.0.html

• Remove indices.query.bool.max_clause_count (7128b04)
• Remove discovery.zen.* settings
• Review xpack security.*ssl settings
• kibana user has been replaced by kibana_system - review code for impact, and make changes as necessary
• Replace search.remote settings with cluster.remote (99430fd, 99430fd)
• Ensure node.roles is used when specifying node roles
• Replace transport.tcp.port with transport.port
• The S3, GCS and Azure repository plugins are now included in Elasticsearch (Remove plugin installation where necessary) (Security-Onion-Solutions/securityonion-image@da2c7a3)
• Review the ES data directory/layout changes for impact
  https://www.elastic.co/guide/en/elasticsearch/reference/master/migrating-8.0.html#breaking_80_packaging_changes
• Review BCs for REST API(s) (SOC/SOCtopus/utility scripts)
  https://www.elastic.co/guide/en/elasticsearch/reference/master/migrating-8.0.html#breaking_80_rest_api_changes

NOTE: Indices created in Elasticsearch 6.x and earlier versions are not supported.

Logstash

https://www.elastic.co/guide/en/logstash/current/breaking-8.0.html

• Review ECS compatibility setting for Logstash pipelines (might consider pipeline.ecs_compatibility: disabled in pipelines.yml or defaults.yaml for disabling globally if impacted)

Kibana

https://www.elastic.co/guide/en/kibana/master/release-notes-8.0.0.html#breaking-changes-8.0.0

• Switch from so_elastic user with superuser role to so_kibana user with kibana_system role (6f294cc)
• Add xpack.reporting.kibanaServerhostname: localhost in defaults.yaml (b2c2680)
• Update kibana config for 8.1.0 (e0374be)

Curator

• Patch Curator to work with Elastic 8 (178e91c04a2edd726a31857627e87f103f68c76c)

Elastalert

• Update to Elastalert 2.4.0 (36783ff96112c647cbd747d0d179d9eed09f6715, 679aba52b2c7d37cbbfef5b2640dc147ebb8c3ed)
• Ensure match_body mapping contains "enabled": "true" for object (b2437ae8a9b0ae0ddd226f37e1c133348614abe1)
• SOUP needs to delete existing Elastalert indices, and ensure Elastalert does not re-create them before the upgrade is finished (Deprecated feature use in 7.x index fails upgrade to 8.x but cannot be downgraded. elastic/elasticsearch#84199)

Filebeat

• Pipeline load workaround (a18b38d)
• Restructure how modules are enabled (a60ef33)
• Switch from log input to filestream input (712a92a)

https://www.elastic.co/guide/en/beats/libbeat/current/release-notes-8.0.0.html

SOC

• Remove usage of _type

[dougburks]

Tested and verified: