New TermsEnum returns 404 on non-existent subtraction indexes e.g. (logs-*,-elastic-cloud-logs-*) instead of 200

[FrankHassanabad]

Description of the problem including expected versus actual behavior:

On security solutions when we use the KQL autocomplete we are getting back errors from Kibana and noticed that Kibana swapped from aggregations to using the new terms enum API here: elastic/kibana#100174

Tracing it down to the NewTermsEnum functionality and PR: #66452, and doc reference here:
https://www.elastic.co/guide/en/elasticsearch/reference/master/search-terms-enum.html

We noticed that the NewTermsEnum returns a 404 error if a subtraction index does not exist where in the aggs _search older behavior it would never return 404 even if the index does not exist if you're trying to subtract it.

Expected behavior:
You can include a subtraction of a non-existent index with an existent index for the new terms API such as POST /logs-*,-*elastic-cloud-logs-*/_terms_enum and it will not return an error if it doesn't exist and have the same behavior of _search

Actual behavior:
terms enum returns an error when subtracting an index which doesn't exist.

Steps to reproduce:

1. Ensure you create an index such as logs-01 in dev tools

PUT logs-01

2. Ensure you do not have an index such as elastic-cloud-logs

Then perform this query in dev tools to see NewTermsEnum return a 404 on the subtraction index:

# New behavior which does NOT allow us to use "-*elastic-cloud-logs-*" if it does not exist
# to subtract. Instead this blows up with a 404 error which causes a cascading 500 internal server error
POST /logs-*,-*elastic-cloud-logs-*/_terms_enum
{
  "field": "host.name",
  "string": "",
  "index_filter": {
    "bool": {
      "must": [
        {
          "terms": {
            "_tier": [
              "data_hot",
              "data_warm",
              "data_content",
              "data_cold"
            ]
          }
        }
      ]
    }
  }
}

Blows up with this message:

{
  "error" : {
    "root_cause" : [
      {
        "type" : "index_not_found_exception",
        "reason" : "no such index [*elastic-cloud-logs-*]",
        "index_uuid" : "_na_",
        "index" : "*elastic-cloud-logs-*"
      }
    ],
    "type" : "index_not_found_exception",
    "reason" : "no such index [*elastic-cloud-logs-*]",
    "index_uuid" : "_na_",
    "index" : "*elastic-cloud-logs-*"
  },
  "status" : 404
}

However, if you use the aggs version it it returns a 200 with results regardless if the elastic-cloud-logs exists or not:

# Older terms_aggs behavior which allows us to use "-*elastic-cloud-logs-*" if it does not exist to subtract.
# However it will not allow non-existent indexes if they are not prefixed with a dash
GET /logs-*,-*elastic-cloud-logs-*/_search
{
  "size": 0,
  "timeout": "1000ms",
  "terminate_after": 100000,
  "query": {
    "bool": {
      "filter": []
    }
  },
  "aggs": {
    "suggestions": {
      "terms": {
        "field": "host.name",
        "include": ".*",
        "execution_hint": "map",
        "shard_size": 10
      }
    }
  }
}

Workaround
Within Kibana on security solutions you can set the aggs version like so in your kiban.yml file so that autocomplete will work with subtraction indexes again

data.autocomplete.valueSuggestions.method: 'terms_agg'

Elasticsearch version (bin/elasticsearch --version):
8.0.0

JVM version (java -version):
11.0.2

OS version (uname -a if on a Unix-like system):
Darwin MacBook-Pro.local 20.5.0 Darwin Kernel Version 20.5.0: Sat May 8 05:10:33 PDT 2021; root:xnu-7195.121.3~9/RELEASE_X86_64 x86_64

[elasticmachine]

Pinging @elastic/es-search (Team:Search)