EQL returning meta fields such as "_ignored" in the "fields" structure

[FrankHassanabad]

Elasticsearch version (bin/elasticsearch --version):
7.14.0

JVM version (java -version):
openjdk version "11.0.2" 2019-01-15
OpenJDK Runtime Environment 18.9 (build 11.0.2+9)
OpenJDK 64-Bit Server VM 18.9 (build 11.0.2+9, mixed mode)

OS version (uname -a if on a Unix-like system):
Darwin MacBook-Pro.local 20.5.0 Darwin Kernel Version 20.5.0: Sat May 8 05:10:33 PDT 2021; root:xnu-7195.121.3~9/RELEASE_X86_64 x86_64

Description of the problem including expected versus actual behavior:
When you have a query against an index where the _ignored meta data field exists, it shows up within the fields section of the response document. Meta fields such as _ignored do not show up in the fields section for regular queries. Also if I do not ask for the fields in the request, EQL still returns the fields index with _ignored.

The impact at the moment for 7.14.0 is that on detections engine within security_solution we parse using the fields API and when we go to re-index our alerting returns errors since it is trying to insert a _ignored field.

Steps to reproduce:

• Go to dev tools and create a mapping with an ignore_above and a low value.

PUT eql-issue-ignore-fields-delme
{
  "mappings" : {
    "dynamic": "strict",
    "properties" : {
      "@timestamp": {
        "type": "date"
      },
      "some_keyword" : {
        "ignore_above": 5, 
        "type" : "keyword"
      },
      "other_keyword" : {
        "ignore_above": 10, 
        "type" : "keyword"
      }
    }
  }
}

• Add a single document with one field that will be truncated and a second that will not.

PUT eql-issue-ignore-fields-delme/_doc/1
{
  "some_keyword": "longer than normal",
  "other_keyword": "normal"
}

• If we query without fields API, it gives us a response with _fields and _ignored.

GET eql-issue-ignore-fields-delme/_eql/search
{
  "query": "any where true"
}

{
  "is_partial" : false,
  "is_running" : false,
  "took" : 2,
  "timed_out" : false,
  "hits" : {
    "total" : {
      "value" : 1,
      "relation" : "eq"
    },
    "events" : [
      {
        "_index" : "eql-issue-ignore-fields-delme",
        "_id" : "1",
        "_source" : {
          "some_keyword" : "longer than normal",
          "other_keyword" : "normal"
        },
        "fields" : { <--- This should not be here
          "_ignored" : [ <--- This should not be here
            "some_keyword"
          ]
        }
      }
    ]
  }
}

• If we query with fields API, it gives it to us as well.

GET eql-issue-ignore-fields-delme/_eql/search
{
  "query": "any where true",
  "fields": [{ "field": "*", "include_unmapped": "true" }]
}

{
  "is_partial" : false,
  "is_running" : false,
  "took" : 1,
  "timed_out" : false,
  "hits" : {
    "total" : {
      "value" : 1,
      "relation" : "eq"
    },
    "events" : [
      {
        "_index" : "eql-issue-ignore-fields-delme",
        "_id" : "1",
        "_source" : {
          "some_keyword" : "longer than normal",
          "other_keyword" : "normal"
        },
        "fields" : {
          "_ignored" : [ <--- This should not be here.
            "some_keyword"
          ],
          "other_keyword" : [
            "normal"
          ]
        }
      }
    ]
  }
}

• Using regular queries even with the fields API never returns meta fields as far as I can tell.

GET eql-issue-ignore-fields-delme/_search
{
  "query": {
    "match_all": {}
  },
  "fields": [{ "field": "*", "include_unmapped": "true" }]
}

{
  "took" : 2,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 1,
      "relation" : "eq"
    },
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "eql-issue-ignore-fields-delme",
        "_id" : "1",
        "_score" : 1.0,
        "_ignored" : [
          "some_keyword"
        ],
        "_source" : {
          "some_keyword" : "longer than normal",
          "other_keyword" : "normal"
        },
        "fields" : {
          "other_keyword" : [ <--- 👍 the only entry we want.
            "normal"
          ]
        }
      }
    ]
  }
}

[elasticmachine]

Pinging @elastic/es-ql (Team:QL)

[astefan]

Depends on #77171.

[astefan]

@FrankHassanabad this is fixed now, with availability in 7.15.1, 7.16.0 and 8.0.0.