[Partial Results] Incremental results visualizations UX

[Dosant]

Part of #84051

We need a common way to represent a partial result state.
Probably, it would overlap with the search session UI. It could be visually connected with it.

We will use this state on a dashboard on each panel that supports partial results:

• Lens (all types)
• Visualize aggs based charts
• Maps

---

related: #76521 (we should get rid of or improve this desaturation) and #55408

[elasticmachine]

Pinging @elastic/kibana-app-services (Team:AppServices)

[mdefazio]

Please see this google doc for a list of all the possible scenarios and views that the UI needs to account for and confirm these are all correct

[wylieconlon]

@Dosant Since you're only focused on aggregations, I tested this by looking at the bsearch responses that Lens is getting. It seems like we aren't getting any partial aggregation results on 7.11 where I tested. I'm not sure if this is a bug or expected, but if we can't get partial data, all we can do is show a loading indicator: #94495

[lizozom]

@wylieconlon partial results were exposed on SearchSource only in 7.12, but they were available in lower level search since 7.10 I think.

If you're just looking at the responses, try focusing on really large data sets and long running queries. I'm working with a 30 million document data set (30% of it is in frozen indices) and I get partial results returned.

Let me know if you need more info. 🙏

[wylieconlon]

@lizozom I think I do need more information, since all I saw was that the hits count was increasing. What data is returned during the partial aggregations? Which buckets? Maybe you could post a sample that you generated?

[mdefazio]

Sharing our current thinking on this for feedback:

• Using an (indeterminate) EuiProgress indicator above each panel (when there are many) can be visually very distracting.
• We want to make sure we account for a11y concerns that may come with using flashing animations. So we want to avoid, (or be cautious of) flashing animations and quick shifts between chart states
• We want to make sure we are accounting for various panels states and extra elements that may appear (errors, custom time range)
• And to reiterate, we should avoid modifying the visualization to achieve these goals: there are too many possibilities and scenarios to accommodate and maintaining this would likely be difficult.

Below are two iterations. We have some slight variations on these that I can share, but these seemed to be the most receptive. With both of these we are also doing a slight fade during chart transitions to help avoid flashing/jumping. I'm faking it with this prototype, so please let me know if there is something else we need to be aware of for this.

I will also add these to this doc so we can gather feedback a bit easier.

1. Use a badge with icon only that ties back to the Search Session icon and shows shard progress in a popover (also tried with 'Loading' text). This would not animate.

2. Show a loading indicator within the small badge—hoping to strike a balance between grabbing attention being too distractive. Should also be display after other badges that may be on a panel to avoid those from jumping around when loading completes.

[wylieconlon]

@mdefazio I have some comments on what you're showing that I couldn't find a place in the doc to comment on:

1. It would be really nice if we can update visualizations with partial data like you're showing, but I haven't been able to reproduce this in any actual Lens queries so far. Maybe @lizozom can help us to reproduce.
2. For date-based visualizations, we have enough information to always keep the X axis the same between updates, but for other types of visualizations the results may be totally changed. For bar charts, the number of items might grow or shrink, or change entirely. Since date histograms are a very common chart type, it would be nice to show this more realistically- for example, by having only the data on the left or right of the chart visible.

[mdefazio]

Since date histograms are a very common chart type, it would be nice to show this more realistically- for example, by having only the data on the left or right of the chart visible.

Thanks for the comments @wylieconlon , I'll try and update to show more realistic time-series examples

[henrikno]

I took a look at the API responses and noticed that the search is 99% done (by number of shards), so it's only waiting for the last 1% before displaying the results. In our case it's querying multiple clusters via CCS, so one of those clusters might be slow. It would be great if it could display these partial results, but with a visual indicator that these results are not complete. (the last 1% is probably not going to change the view completely)

{"took":262313,"timed_out":false,"terminated_early":false,"num_reduce_phases":98,"_shards":{"total":39736,"successful":39681,"skipped":33506,"failed":0},...

[elasticmachine]

Pinging @elastic/kibana-data-discovery (Team:DataDiscovery)

[IanLee1521]

Agreed that some functionality to show in progress results would be great. For me the use case is that I occasionally do large queries over long time windows (though not with CCS). As henrikno states, it would be nice if each or some batched number of shards responses could be rendered as they are available before they all complete.