[Lens] User experience of composing aggregations and calculations

[wylieconlon]

Composing aggregations is the most powerful feature of Elasticsearch, and Lens will not be complete without letting users have this power. There are several unresolved questions about the user experience of composing aggregations in Lens, and this issue summarizes them and proposes a solution based on the tradeoffs. One aspect that I'm considering here is also the boundary between aggregations run inside Elasticsearch vs calculations that are run by Kibana on top of Elasticsearch, and how to express that to users.

Examples

To give two examples where we can make a number of different choices about the user experience:

• Simple: Date histogram with a moving average of the sum of sales
• Complicated: Calculating the average network traffic per second across all networks:
  • Terms aggregation for each network
  • then a date histogram using hourly intervals
  • Then a max aggregation on the field
  • Followed by a derivative pipeline aggregation, scaled to 1 second
  • Finally, in the client (TSVB here) there are two more steps:
  • Ignore all the negative derivatives
  • Sum together all of the Terms into one series

Both examples can be solved many ways. To compare the simple example in Visualize and TSVB:

• In visualize, we have chosen to create a one-level pipeline editor which can't be composed any farther:

• In TSVB, we have chosen to built a multi-level editor with the ability to hide data that isn't used, which is a requirement for calculations in Lens

• But we have also chosen to create a shortcut in TSVB for the complicated case:

Analysis of our existing solutions to this problem

None of our existing solutions is perfect, but TSVB is more composable. There are still limits to its power, such as not being able to do math between two different series. Importantly, it has the ability to compose aggregations that use hidden data: this is shown in the examples above. And the approach doesn't have to sacrifice user experience if we build the right shortcuts like the "Positive Rate" example. Another benefit of the TSVB approach is that you can switch visualizations in TSVB without losing information, because all the visualizations are powered the same way.

The solution chosen by Visualize is to couple the aggregation with a specific part of the visualization, such as "Y axis: Moving average of sum". Only predefined functions are supported, so it is not possible to extend the functionality with support for calculations like we want in Lens. For example, it would not be possible to configure "Y axis: Difference between moving average and sum", which is similar to the types of requests we get often.

Proposal

Lens should take the best elements of the TSVB and Visualize approaches, which will let our user experience be both powerful and easy to use. The key to doing both is consistency, so that users are able to learn the system. There are several parts of this proposal:

• Allow the right side configuration panel to have hidden data that can be used by other aggregations or calculations
• Share the right side panel between all visualizations, so that when you switch charts we don't break your data or lose your state
• Build shortcuts for common operations that can be "broken apart" into an editable state. For example, you use the shortcut "positive rate" and then want to change from "Max" to "Count"
• For advanced editing of the queries, we might need to edit the entire query in a modal editor. This is because queries are more coupled than in a SQL-based system.

This proposal is intended to give us the most flexibility to build different user experiences for targeted users: for example, if we want to build a form-based or text-based system, both systems would be built with the same principals.

[elasticmachine]

Pinging @elastic/kibana-app (Team:KibanaApp)

[wylieconlon]

Now that we've been working on some prototypes for this, I think I can clarify the description above using a fake screenshot. By "composing aggregations", we are talking about creating a hierarchy like this:

cc @cchaos

[wylieconlon]

This is pretty closely related to the idea of "exploding" a shortcut function into its parts. Shortcuts are composed of 2 or more functions. #74800

[wylieconlon]

We have had discussions about the problem of composing aggregations, and the current plan is to implement a single level of composability because we can all agree that it's the most important need. This issue is still relevant because we haven't yet agreed on:

• What the use cases for more composability are, and how prevalent this need it
• Assuming that it's a goal, how do we keep it simple for first time users?

Some related work is the technical plan for supporting the new functions at a single level of composability. Once we can support a single level in code, we should also be able to support 2 or more levels using the same model. This will make it easier to prototype new interactions to solve the simplicity issue. #76828

[flash1293]

I think we had offline discussions about this and we have a good enough idea for how this work and is going to work from here, closing the issue. If that's mistaken, please reopen.