[Lens] Math using TinyMath

[wylieconlon]

Because math is one of the most requested calculations, and also hard to express using a form-based system, I am proposing that we get users to type their math using TinyMath. This approach is different from the Painless-based system used in TSVB because of the syntax and different set of supported features.

I am basing this off my spacetime proof of concept.

How is TinyMath used?

TinyMath expects a context object to be provided with each expression, allowing it to execute expressions against named variables. Lens would provide TinyMath with either row-oriented or column-oriented data as context. From the docs:

// Named properties
evaluate('foo + 20', {foo: 5}); // 25

// Arrays
evaluate('bar + 20', {bar: [1, 2, 3]}); // [21, 22, 23]
evaluate('bar + baz', {bar: [1, 2, 3], baz: [4, 5, 6]}); // [5, 7, 9]
evaluate('multiply(bar, baz) / 10', {bar: [1, 2, 3], baz: [4, 5, 6]}); // [0.4, 1, 1.8]

This would let users perform highly-requested tasks like:

• Dividing two numbers in the same row
• Getting the average value of a column

How will users enter TinyMath?

We only have two options here: Typing without autocomplete, and typing with autocomplete. While autocomplete is preferable, it's a lot harder to get right than the simplest approach where we ask the user to type variable names for each input, and then to type their math:

Why not do this using bucket scripts?

The main reason is that bucket scripts can't handle the architecture for calculations in Lens.

[elasticmachine]

Pinging @elastic/kibana-app (Team:KibanaApp)

[nreese]

cc @elastic/kibana-gis

@wylieconlon this is a really interesting. We would also like to expose calculations in maps #28265 with tiny math. Let us which direction you go so we can expose something similar in maps

[flash1293]

@wylieconlon Do you have an idea already what it would take to solve this with auto-complete? It would be really cool to strip the naming step out, but it doesn't seem trivial (changing column labels, special characters that would have to be escaped, ...)

[wylieconlon]

@nreese Might be worth setting up some time to talk about the potential overlap!

@flash1293 Yes, I have thought about it a bit. There are a few parts that I know we would need for a good autocomplete system, and a few that require more work. In terms of the interaction, we can either:

• Run the literal expression the user has typed
• Define a custom grammar that compiles to TinyMath, so that variable references are better handled

I prefer the second option, compiling to TinyMath, because it's beneficial for the user. For example, we can compile the expression average(duración) which is an invalid character in TinyMath, but valid in Lens, to the expression average(u230-zcx1-342b-692y). Here are the parts that we would definitely need:

• Decide if we want to support "renaming" variables like I showed above.
• Start with the grammar definition, but maybe extend the ValidChar
  • The grammar defines a ValidChar as [0-9A-Za-z._@\[\]-], it won't handle the full set of characters we support.
• To build autocomplete on top of the grammar, we need to manually define:
  • The included functions
  • The names of variables that we have in Lens, for example Count of Records is a variable name
• Finally, we need to build any additional features into the editor. For example, imagine that the variable names are highlighted and support some kind of extra interaction

[wylieconlon]

One of the new concepts that we're exploring in code is auto-generating IDs for each configured function, so that they can be referenced easily when typing. This approach is not perfect, but it has a low learning curve and is also fast to implement. Other approaches haven't been prototyped yet because they have a higher cost to implement.

[flash1293]

Implemented by formula #99297