CCS: terms lookup query doesn't support looking up terms on remote indices

[lucabelluccini]

Elasticsearch version (bin/elasticsearch --version): 7.6.2, 7.9.0

JVM version (java -version): bundled

Description of the problem including expected versus actual behavior:

It is not possible to use the terms lookup query using a remote index. A 404 error is returned.

Steps to reproduce:

1. Setup a cluster called main
2. Setup a cluster called ccs
3. Configure the ccs cluster to connect to the main cluster using the steps provided in our documentation
4. Create the following data on the main cluster (as per example) and test the lookup terms search:

PUT my-index-000001
{
  "mappings": {
    "properties": {
      "color": { "type": "keyword" }
    }
  }
}

PUT my-index-000001/_doc/1
{
  "color":   ["blue", "green"]
}

PUT my-index-000001/_doc/2
{
  "color":   "blue"
}

GET my-index-000001/_search?pretty
{
  "query": {
    "terms": {
        "color" : {
            "index" : "my-index-000001",
            "id" : "2",
            "path" : "color"
        }
    }
  }
}
# Response
{
  "took" : 17,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 2,
      "relation" : "eq"
    },
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "my-index-000001",
        "_type" : "_doc",
        "_id" : "1",
        "_score" : 1.0,
        "_source" : {
          "color" : [
            "blue",
            "green"
          ]
        }
      },
      {
        "_index" : "my-index-000001",
        "_type" : "_doc",
        "_id" : "2",
        "_score" : 1.0,
        "_source" : {
          "color" : "blue"
        }
      }
    ]
  }
}

5. Use the elastic (superuser) to avoid any security role to setup
6. Search from the ccs cluster to the main cluster (works ok) with a match all query

GET main:my-index-000001/_search
# Response
{
  "took": 4,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "_clusters": {
    "total": 1,
    "successful": 1,
    "skipped": 0
  },
  "hits": {
    "total": {
      "value": 2,
      "relation": "eq"
    },
    "max_score": 1,
    "hits": [
      {
        "_index": "main:my-index-000001",
        "_type": "_doc",
        "_id": "1",
        "_score": 1,
        "_source": {
          "color": [
            "blue",
            "green"
          ]
        }
      },
      {
        "_index": "main:my-index-000001",
        "_type": "_doc",
        "_id": "2",
        "_score": 1,
        "_source": {
          "color": "blue"
        }
      }
    ]
  }
}

7. Use a Terms lookup query on a remote index both with and without prefixing it with main (it fails)

POST main:my-index-000001/_search
{
  "query": {
    "terms": {
        "color" : {
            "index" : "main:my-index-000001",
            "path": "color",
            "id": "2"
        }
    }
  }
}
# Response
{
  "error": {
    "root_cause": [
      {
        "type": "index_not_found_exception",
        "reason": "no such index [main:my-index-000001]",
        "resource.type": "index_expression",
        "resource.id": "main:my-index-000001",
        "index_uuid": "_na_",
        "index": "main:my-index-000001"
      }
    ],
    "type": "index_not_found_exception",
    "reason": "no such index [main:my-index-000001]",
    "resource.type": "index_expression",
    "resource.id": "main:my-index-000001",
    "index_uuid": "_na_",
    "index": "main:my-index-000001"
  },
  "status": 404
}

POST main:my-index-000001/_search
{
  "query": {
    "terms": {
        "color" : {
            "index" : "my-index-000001",
            "path": "color",
            "id": "2"
        }
    }
  }
}
# Response
{
  "error": {
    "root_cause": [
      {
        "type": "index_not_found_exception",
        "reason": "no such index [my-index-000001]",
        "resource.type": "index_expression",
        "resource.id": "my-index-000001",
        "index_uuid": "_na_",
        "index": "my-index-000001"
      }
    ],
    "type": "index_not_found_exception",
    "reason": "no such index [my-index-000001]",
    "resource.type": "index_expression",
    "resource.id": "my-index-000001",
    "index_uuid": "_na_",
    "index": "my-index-000001"
  },
  "status": 404
}

[elasticmachine]

Pinging @elastic/es-search (:Search/Search)

[javanna]

This requires knowledge about how the terms query works internally. Terms lookup uses the get API to retrieve the document that contains the ids. Get is not search, hence it has no cross-cluster support, meaning index prefixes don't get resolved, and indices can only be local. The terms query gets rewritten on the coordinating node (CCS cluster when using CCS), and as part of the rewrite phase the document containing the ids is fetched locally.

This can only work if the lookup data is located in the CCS cluster, and no index prefix is used in the terms query definition.

Otherwise, the lookup can also happen on the client side. Terms lookup is concise, but it isn't more optimized than fetching the ids and providing them directly as part of the terms query, that is what happens under the hood anyways.

[javanna]

I wonder if this should be considered a bug, or maybe converted into a doc improvement to document the limitation that I described above.

[javanna]

@lucabelluccini can I get feedback on the explanation I gave above?

[elasticmachine]

Pinging @elastic/es-docs (>docs)