[Monitoring] Opening Monitoring app on non-CCS cluster triggers CCS error

[marius-dr]

Kibana version: 7.15.0 Snapshot

Describe the bug: If you open the Monitoring app on a default cluster(this includes CCS not being configured) (basically no changes in the yml files from the downloaded packages), the following error shows up:

[illegal_argument_exception: [illegal_argument_exception] Reason: Cross-cluster calls are not supported in this context but remote indices were requested: [*:.monitoring-beats-7-*, *:.monitoring-beats-6-*, *:.monitoring-logstash-6-*, *:.monitoring-logstash-7-*, *:metricbeat-*]: illegal_argument_exception: [illegal_argument_exception] Reason: Cross-cluster calls are not supported in this context but remote indices were requested: [*:.monitoring-beats-7-*, *:.monitoring-beats-6-*, *:.monitoring-logstash-6-*, *:.monitoring-logstash-7-*, *:metricbeat-*]]: Cross-cluster calls are not supported in this context but remote indices were requested: [*:.monitoring-beats-7-*, *:.monitoring-beats-6-*, *:.monitoring-logstash-6-*, *:.monitoring-logstash-7-*, *:metricbeat-*]

This also leaves the monitoring page stuck infinitely on "Loading".

Steps to reproduce:

1. Get the ES and Kibana artifacts.
2. Unzip and start both.
3. Open the monitoring app.

Screenshots (if relevant):

[elasticmachine]

Pinging @elastic/stack-monitoring (Team:Monitoring)

[simianhacker]

This works with 7.15.0 build hash bc1ed9cfa397f0b4267ed6e3e3688d372b87b2ba built on 2021-08-18T16:19:56.037637097Z but it does not work with build hash bc278d661f0b0d72ce292aa636fbb1db96427c40 built on 2021-08-17T13:27:20.599610018Z

[simianhacker]

Actually what I posted above is wrong, I was running the newer snapshot with security. It appears that CCS doesn't work unless security is enabled.

With this query

POST *:foo/_search 
{
  "query": {
    "match_all": {}
  }
}

if you have security disabled it returns this response

{
  "error" : {
    "root_cause" : [
      {
        "type" : "illegal_argument_exception",
        "reason" : "Cross-cluster calls are not supported in this context but remote indices were requested: [*:foo]"
      }
    ],
    "type" : "illegal_argument_exception",
    "reason" : "Cross-cluster calls are not supported in this context but remote indices were requested: [*:foo]"
  },
  "status" : 400
}

but with security

{
  "took" : 0,
  "timed_out" : false,
  "_shards" : {
    "total" : 0,
    "successful" : 0,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 0,
      "relation" : "eq"
    },
    "max_score" : 0.0,
    "hits" : [ ]
  }
}

[jtibshirani]

This error is caused by a recent change we made in 7.15 to throw an error when remote index patterns (like *:foo) are used with APIs like 'get' that don't support cross-cluster search: elastic/elasticsearch#74762. I think that this change accidentally started throwing errors in the search API when a remote index pattern is used, but when CCS is not enabled. I am guessing this was unintentional and just a result of a backport issue, since the master branch doesn't contain this stricter behavior, only 7.x. In any case, I think we should fix this in 7.15 to not throw an error from the search API.

[marius-dr]

@jtibshirani it completely breaks user experience entering monitoring that just start ES + Kibana for preview purposes (without security or CCS configured), hence why I marked it a blocker.

[jtibshirani]

I'm sorry for the delay! I ended up opening a revert PR here: elastic/elasticsearch#76798. My local testing shows it will resolve the issue.

[simianhacker]

Just tested this with BC3, it appears to be working after elastic/elasticsearch#76798 was merged.