[Logs UI] Show warning when CCS searches omit data due to version mismatch

[jasonrhodes]

See: #100876

When a cross-cluster search is extended to a cluster that doesn't support the "fields" API, ES will now return a warning so that we can let users know that some data that exists in older clusters may not be present in the current view.

AC:

• Confirm that the warning is received from the ES query
• Display some kind of notification to the user

@elastic/observability-design / @mukeshelastic we will need some guidance on what kind of warning we should display here, and how often

[elasticmachine]

Pinging @elastic/logs-metrics-ui (Team:logs-metrics-ui)

[weltenwort]

💭 There are other situations as well in which the data might be incomplete:

• Because we use async search queries might be in progress but already have partial results.
• A request might have failure only in some shards but return data for other shards.

Maybe we can find a place in the UI to show these warnings/errors in addition to the partial results. I implemented something like this for the log entry fly-out where I render a call-out at the top when there are results and shard failures at the same time (#81710).

[jasonrhodes]

Do the other scenarios you describe have known error messages so we can accurately determine when that scenario occurs? I think it's a great idea to collapse these things but the CCS search will result in a very specific message from ES, so I wonder if we should just "surface ES warnings" somehow, as a baseline.

[weltenwort]

Yes, each ES error has a type property that we can check. An example can be found there:

kibana/x-pack/plugins/infra/common/search_strategies/common/errors.ts

Lines 25 to 33 in c30dc1e

	const shardFailureSearchStrategyErrorRT = rt.type({
	type: rt.literal('shardFailure'),
	shardInfo: rt.type({
	shard: rt.union([rt.number, rt.null]),
	index: rt.union([rt.string, rt.null]),
	node: rt.union([rt.string, rt.null]),
	}),
	message: rt.union([rt.string, rt.null]),
	});