Surface more information for diagnosing shard failures

[spalger]

Just filed elastic/elasticsearch#16135 because we discoverd a way to cause requests to partially fail reliably, but the user is not told what the actual failures are.

When a response from elasticsearch includes a mixture of shard failures and successes the response is considered successful, but a warning is shown to the user. The warning simply states how many shards failed, but does not allow the user to see why they failed, and the warning is automatically hidden after a timeout.

Instead we should probably allow the user to click "more info", see the different causes for the errors, and prevent the warning from hiding if the user has interacted with it.

[cjcenizal]

Per @timroes in #20767 (comment):

One of the most annoying things with shard failures currently is, that you don't get any insights into the actual failures, and that's very often a painpoint in discuss questions, since users need to check the actual browser request, and dig through that JSON. In the long run we definitely should build this notification in a way, that you can get a list of actual errors from the individual charts.

Courier is currently responsible for surfacing this error. Because courier isn't coupled to any particular part of the UI, it has to surface it in a global way, e.g. through toasts or banners. However, this format isn't conducive to long-form content, like stack traces or error messages which require parsing and time to diagnose. I think we have two solutions:

1. Make the consuming UI responsible for surfacing this error. This will remove responsibility for surfacing the error from Courier, and allow the consumer to present the error in a way that makes the most sense for the UI. However this will be labor-intensive and can result in duplicated effort if it turns out that there's a common format which works best for all UIs.

2. Create a new global system for presenting these types of errors, similar to the inspector. It may even be possible to surface this content inside of the inspector, since its role is to aid diagnosis of problems and performance. I lean towards this option personally.

[cjcenizal]

#11178 contains steps to produce this error and valuable user feedback.

[cjcenizal]

From talking with @sherry-ger and @jaijhala, I understand that having access to the ES logs pertinent to the error would really help debugging. This would require the user to have the proper access privileges though.