Ingest pipelines from cisco modules are causing warning messages in Elasticsearch

[romain-chanu]

1. Problem

1. Users have observed the following warning messages in Elasticsearch:

[2023-08-14T11:09:30,093][WARN ][o.e.i.c.GrokProcessor    ] [ES-NODE] character class has '-' without escape

2. The above warning message reports a problem with a pattern defined in a grok processor where a hyphen character is used as a literal character in a character class [ .... ] but the hyphen character is not escaped.

3. This currently does not seem to be impacting any functionality - it just creates unnecessary logs in the Elasticsearch logs.

2. Investigation

1. Enabling debug logging as suggested in Grok debugging did not help. The problematic ingest pipeline name or grok pattern definition are no logged, even in debug log level.

2. Reviewing the existing list of Elasticsearch ingest pipelines and checking for unescaped hyphen character in the character class, we have been able to find the problematic patterns defined in https://github.com/elastic/beats/blob/main/x-pack/filebeat/module/cisco/shared/ingest/asa-ftd-pipeline.yml

3. There are 5 occurrences of the following pattern where - is not escaped, resulting in the above warning:

HOSTNAME: "\\b(?:[0-9A-Za-z][0-9A-Za-z-_]{0,62})(?:\\.(?:[0-9A-Za-z][0-9A-Za-z-_]{0,62}))*(\\.?|\\b)"

4. The correct pattern should be:

HOSTNAME: "\\b(?:[0-9A-Za-z][0-9A-Za-z\\-_]{0,62})(?:\\.(?:[0-9A-Za-z][0-9A-Za-z\\-_]{0,62}))*(\\.?|\\b)"

Notice the difference: [0-9A-Za-z-_] should be [0-9A-Za-z\\-_]

3. Steps to reproduce

We have been able to reproduce the problem with Filebeat / Elasticsearch version 8.5.0 but the problem has been observed also in latest versions (e.g version 8.9.0).

1. Download Filebeat, configure it to connect to an Elasticsearch cluster and run filebeat setup -e. Wait for all the Filebeat assets (dashboards, ingest pipelines, etc.) to be loaded.

2. Check the Elasticsearch logs and observe the above warning message.

3. Delete all the ingest pipelines related to Filebeat (can be done in batch through Kibana > Stack Management > Ingest Pipelines)

4. Modify the file filebeat/module/cisco/shared/ingest/asa-ftd-pipeline.yml to replace the 5 occurrences:

FROM:

HOSTNAME: "\\b(?:[0-9A-Za-z][0-9A-Za-z-_]{0,62})(?:\\.(?:[0-9A-Za-z][0-9A-Za-z-_]{0,62}))*(\\.?|\\b)"

TO:

HOSTNAME: "\\b(?:[0-9A-Za-z][0-9A-Za-z\\-_]{0,62})(?:\\.(?:[0-9A-Za-z][0-9A-Za-z\\-_]{0,62}))*(\\.?|\\b)"

5. run filebeat setup -e. Wait for all the Filebeat assets (dashboards, ingest pipelines, etc.) to be loaded.

6. Check the Elasticsearch: above warning message is no longer observed.

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)