[RFC] Query Insights Dashboard

[ansjcy]

Is your feature request related to a problem? Please describe.
With the Macro and Micro visibility metrics collected and aggregated by the query insight framework (opensearch-project/OpenSearch#11429), we can create a dedicated dashboard to visualize those metrics to provide an end to end solution to OS users to enable them to view point in time (e.g. last one hour) query insights.

Describe the solution you'd like
The dashboard should have multiple views in terms of different query insight data, and for each type of data, it should provide an overview and drill down view for the search queries. Overview shows the trend (top queries with latency, query categorization) with time, and the drill down view can provide a flow graph to show detailed data (like latency and resource distribution) across different stages of the query.

For example, for the top-n queries feature (opensearch-project/OpenSearch#11186), the query insight dashboard for "overview" should be similar to the below mock:

And if you drill down into this query, you should see the query details and the resource consumption in those important phases/stages/workflows.

Describe alternatives you've considered
N/A

Additional context
Some interesting discussions around this topic in the comments of: opensearch-project/OpenSearch#11186

[wbeckler]

The detail flow looks visually interesting, but if the problem solving exercise is to identify which part or kind of query is causing an issue, I don't understand how the process flow diagram would shed light on it, because the numbers wouldn't be meaningful without context of what is normal for that stage of the process.

[ansjcy]

@wbeckler Thanks for the comment! In this specific "Top N Queries" case shows on the mock UI, we are capturing "the top N queries that have high latency" in a time window. In this case the detail flow in the drill down of a "top query" would enable the OpenSearch User idnetify "what stage of the search query is taking most of the time", so that they can optimize accordingly.

But yeah I totally agree we should provide some baseline information as well so that the user can understand "what is normal" for a certain stage of the query execution.