[Filebeat] Make Suricata module append to destination.domain list

[andrewkroh]

Describe the enhancement:
Enhance the Suricata pipeline to append to the destination.domain field if it already exists (make it into an array).

I'm using a processor setup like this so I can enrich events with my local network's hostnames.

processors:
    - dns:
        type: reverse
        fields:
          source.ip: source.domain
          destination.ip: destination.domain
          # Suricata Module
          json.src_ip: source.domain
          json.dest_ip: destination.domain

But this causes a cascading failure in the pipeline. The pipeline fails at setting the destination.domain field and bails out. I was expecting to be able to see the error in the indexed event as error.message, but what actually happened was an indexing failure because the pipeline's final remove processor had not run which causes {"type":"mapper_parsing_exception","reason":"failed to parse","caused_by":{"type":"illegal_argument_exception","reason":"Cannot write to a field alias [suricata.eve.event_type]."}}.

Describe a specific use case for the enhancement or feature:

To be able to use the dns processor with the suricata module in order to do reverse DNS enrichment on events.

[andrewkroh]

By using append + remove I was able get a good result. Then I added a script processor to deduplicate the domains and convert the value back to a scalar if there's just one.

POST _ingest/pipeline/_simulate
{
  "pipeline": {
    "description": "append",
    "processors": [
      {
        "append": {
          "if": "ctx.suricata?.eve?.http?.hostname != null",
          "value": "{{suricata.eve.http.hostname}}",
          "field": "destination.domain"
        }
      },
      {
        "remove": {
          "field": "suricata.eve.http.hostname",
          "ignore_failure": true
        }
      },
      {
        "script": {
          "source": """
          def domain = ctx.destination?.domain;
          if (domain instanceof Collection) {
            if (domain.length == 1) {
              ctx.destination.domain = domain[0];
            } else {
              ctx.destination.domain = ctx.destination.domain.stream().distinct().collect(Collectors.toList());
            }
          }
""",
          "ignore_failure": true
        }
      }
    ]
  },
  "docs": [
    {
      "_index": "index",
      "_type": "_doc",
      "_id": "id",
      "_source": {
        "suricata": {
          "eve": {
            "http": {
              "hostname": "www.google.com"
            }
          }
        }
      }
    },
    {
      "_index": "index",
      "_type": "_doc",
      "_id": "id",
      "_source": {
        "suricata": {
          "eve": {
            "http": {
              "hostname": "www.google.com"
            }
          }
        },
        "destination": {
          "domain": "4.3.2.1.google.verizon.com"
        }
      }
    },
    {
      "_index": "index",
      "_type": "_doc",
      "_id": "id",
      "_source": {
        "suricata": {
          "eve": {
            "http": {
              "hostname": "www.google.com"
            }
          }
        },
        "destination": {
          "domain": [
            "4.3.2.1.google.verizon.com",
            "www.google.com"
          ]
        }
      }
    }
  ]
}