CCS Experience - If a cluster fails to respond highlight that the data may be incomplete

[Alex3k]

Describe the feature:
A way to alert/notify that a visualisation leveraging CCS may not be complete if one of the CCS clusters fails to respond. This is on a per visualisation request, not to be notified by a popup or anything like that as it's ephemeral. The ideal solution would be an icon or something on the visualisation itself.

Describe a specific use case for the feature:
If I have two clusters, ClusterA and ClusterB. Both clusters have an index called data and I have an index pattern which is *:data,data . When I create a visualisations (using Lens) of that data everything works wonderfully. When I simulate an issue with ClusterB by messing with the networking, full restart or out right killing it I only get the data from ClusterA - which makes absolute sense.

However, the user doesn't seem to be told that the other clusters (in my case ClusterB) is having issues and hasn't returned any results. From an end user perspective, it feels strange that they haven't been alerted to say that "some clusters haven't responded". I feel this way because if someone is doing analytics or investigations across observability, security or business data not having the full picture could lead to inaccurate results/responses.

This is really important as customers are moving towards leveraging 10's of clusters within a dashboard all connected to by CCS. If one to many clusters do not respond this could lead to bad decisions being made as they are not aware that data is missing.

cc: @stacey-gammon

[stacey-gammon]

@jimczi - should ES be returning errors in this situation? If so, and this is not working on master, this is a bug for the ES repo. If ES reports it as a warning, then the issue is with Kibana, as we don't show every warning from ES to the user. However, we'll still need a way for ES to tell us whether or not a warning should be shown to the user. in 7.16 we implemented a change that showed these to the user but it had to be reverted as too many warnings were showing up, and some that stemmed from developer usage, that the end-user has no control over.

[stacey-gammon]

After discussing with Jim, we think this actually should work in Kibana. Can you check what your skip-unavailable-clusters setting in ES is set to?

https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-cross-cluster-search.html#skip-unavailable-clusters

[stacey-gammon]

File #118736 as well as we decided we might want to show warnings even if that setting is on.

[elasticmachine]

Pinging @elastic/kibana-app-services (Team:AppServicesSv)

[stacey-gammon]

Going to close this out as I think #118736 will be sufficient moving forward, assuming that the customer had theskip-unavailable-clusters setting in ES set to true.

[Alex3k]

@stacey-gammon would it not make sense to keep it open until your linked issue is completed? I feel closing this issue implies that it is solved.

[stacey-gammon]

Not closed because it is solved, but closed because it is a duplicate of #118736. I just don't think we should keep duplicate tickets open, or it becomes more difficult to find the right one, imo.

[Alex3k]

Makes sense :)