Fields using wildcard type should not specify ignore_above param

[ebeahan]

Version: 8.0.0-rc1, 7.16.2
Operating System: Seen on Windows and MacOS

Description

Index templates created by Beats are still specifying an ignore_above setting for wildcard fields.

Example from the winlogbeat-8.0.0-rc1 template on the process.command_line field:

            "command_line": {
              "ignore_above": 1024,
              "type": "wildcard",
              "fields": {
                "text": {
                  "type": "match_only_text"
                }
              }

Long field values are not being indexed as expected with wildcard. Kibana confirms:

Keyword fields include an ignore_above param in their ECS field definitions, but wildcard fields do not:

https://github.com/elastic/beats/blob/8.0/libbeat/_meta/fields.ecs.yml#L4443-L4454

Steps to Reproduce:

• Install Beats 8.0.0-rc1 with default settings and run setup to install default index templates.
• Index events with values > 1024 in wildcard fields.

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[adriansr]

@ebeahan

This is caused by the following code in libbeat:

beats/libbeat/template/processor.go

Lines 320 to 326 in b6b4f1d

	switch f.IgnoreAbove {
	case 0: // Use libbeat default
	property["ignore_above"] = defaultIgnoreAbove
	case -1: // Use ES default
	default: // Use user value
	property["ignore_above"] = f.IgnoreAbove
	}

Which is treating a missing (zero) value as a request to use libbeat's default (1024). This behavior was copied from keyword fields.

Also, I see that event.original (keyword) is suffering from the same problem. ECS doesn't specify an ignore_above which causes libbeat to inject the default 1024 value. I understand we may want to fix this one too.

I wonder what the best approach would be here. Removing this behavior for wildcards only impacts one field outside ECS: x-pack/filebeat/module/threatintel/anomalithreatstream/_meta/fields.yml defines one wildcard field for a custom dataset field.

But for keywords it will be much trickier as they are used everywhere and we have relied on the default being injected for a long time.

This could also be fixed in ECS' Beats generator by forcing an ignore_above: -1 when the setting is missing, to ensure libbeat doesn't overwrite it.

/cc @kvch

[ebeahan]

The event.original field isn't indexed already (index: false and doc_values: false), so having an ignore_above value specified isn't necessary.

A big benefit wildcard has over keyword is the indexing of strings larger than the 32,766 byte Lucene max, but users can't gain that benefit if ignore_above is set to ignore string values greater than 1024 chars.

I only suggest we remove ignore_above from type: wildcard. There have been conversations about increasing the ignore_above standard value to higher than 1024, but let's leave that topic out of this discussion.