[Lens] Formula editor (#99297)

* 💄 Hack to fix suggestion box

* 🐛 Fix validation messages

* 🐛 Relax operations check for managedReferences

* Change completion params

* 🏷️ Fix missing arg issue

* ✨ Add more tinymath fns

* 🐛 Improved validation around math operations + multiple named arguments

* 🐛 Use new onError feature in math expression

* ♻️ Refactor namedArguments validation

* 🐛 Fix circular dependency issue in tests + minor fixes

* Move formula into a tab

* 🔥 Leftovers from previous merge

* ✨ Move over namedArgs from previous function

* ✅ Add tests for transferable scenarios

* ✅ Fixed broken test

* ✨ Use custom label for axis

* Allow switching back and forth to formula tab

* Add a section for the function reference

* Add modal editor and markdown docs

* Change the way math nodes are validated

* Use custom portal to fix monaco positioning

* Fix model sharing issues

* Provide signature help

* 🐛 Fix small test issue

* 🐛 Mark pow arguments as required

* 🐛 validate on first render only if a formula is present

* 🔥 Remove log10 fn for now

* ✨ Improved math validation + add tests for math functions

* Fix mount/unmount issues with Monaco

* [Lens] Fully unmount React when flyout closes

* Fix bug with editor frame unmounting

* Fix type

* Add tests for monaco providers, add hover provider

* Add test for last_value

* Usability improvements

* Add KQL and Lucene named parameters

* Add kql, lucene completion and validation

* Fix autocomplete on weird characters and properly connect KQL

* Highlight functions that have additional requirements after validating

* Fix type error and move help text to popover

* Fix escape characters inside KQL

* 🐛 Fix dataType issue when moving over to Formula

* Automatically insert single quotes on every named param

* Only insert single quotes when typing kql= or lucene=

* Reorganize help popover

* Fix merge issues

* Update grammar for formulas

* Fix bad merge

* Rough fullscreen mode

* Type updates

* Pass through fullscreen state

* Remove more chrome from full screen mode

* Fix minor bugs in formula typing

* 🐛 Decouple column order of references and output

* 🔧 Fix tests and types

* ✅ Add first functional test

* Fix copying formulas and empty formula

* Trigger suggestion prompt when hitting enter on function or typing kql=

* 🐛 Prevent flyout from closing while interacting with monaco

* refactoring

* move main column generation into parse module

* fix tests

* refactor small formula styles and markup

* documentation

* adjustments in formula footer

* Formula refactoring (#12)

* refactoring

* move main column generation into parse module

* fix tests

* more style and markup tweak for custom formula

* Fix tests

* [Expressions] Use table column ID instead of name when set

* [Lens] Create managedReference type for formulas

* Fix test failures

* Fix i18n types

* fix fullscreen flex issues

* Delete managedReference when replacing

* refactor css and markup; add button placeholders

* [Lens] Formulas

* Tests for formula

Co-authored-by: Marco Liberati <[email protected]>

* added error count placeholder

* Add tooltips

* Refactoring from code review

* Fix some editor issues

* Update ID matching to match by name sometimes

* Improve performance of Monaco, fix formulas with 0, update labels

* Improve performance of full screen toggle

* Fix formula tests

* fix stuff

* Add an extra case to prevent insertion of duplicate column

* Simplify logic and add test for output ID

* add telemetry for Lens formula (#15)

* Respond to review comments

* ✨ Improve the signatures with better documentation and examples

* adjust border styles to account for docs collapse

* refactor docs markup; restructure docs obj; styles

* Fix formula auto reordering (#18)

* fix formula auto reordering

* add unit test

* Fix and improve suggestion experience in Formula (#19)

* ✨ Revisit documentation and suggestions

* 👌 Integrated feedback

* ✨ Add query validation for quotes

* Usability updates & type fixes

* add search to formula

* fix form styles to match designs

* fix text styles; revert to Markdown for control

* 👌 Integrated more feedback

* improve search

* improve suggestions

* improve suggestions even more

* 🐛 Fix i18n issues (#22)

* Persist formula on leave, fix fullscreen and popovers

* Fix documentation tests

* 🏷️ fix type issue

* 🐛 Remove hidden operations from valid functions list

* 🐛 Fix empty string query edge case

* 🐛 Enable more suggestions + extends validation

* Fix tests that depended on setState being called without function

* Error state and text wrapping updates

* ✨ Add new module to CodeEditor for brackets matching (#25)

* Fix type

* show warning

* keep current quick function

* ✨ Improve suggestions within kql query

* 📷 Fix snapshot editor test

* 🐛 Improved suggestion for single quote and refactored debounce

* Fix lodash usage

* Fix tests

* Revert "keep current quick function"

This reverts commit ed47705.

* Improve performance of dispatch by using timeout

* Improve memoization of datapanel

* Fix escape characters

* fix reduced suggestions

* fix responsiveness

* fix unit test

* Fix autocomplete on nested math

* Show errors and warnings on first render

* fix transposing column crash

* Update comment

* 🐛 Fix field error message

* fix test types

* 📝 Fix i18n name

* 💄 Manage wordwrap via react component

* Fix selector for palettes that interferes with quick functions

* Use word wrapping by default

* Errors for managed references are handled at the top level

* 🐛 Move the cursor just next to new inserted text

* ⚗️ First pass for performance

* 🐛 Fix unwanted change

* ⚡ Memoize as many combobox props as possible

* ⚡ More memoization

* Show errors in hover

* Use temporary invalid state when moving away from formula

* Remove setActiveDimension and shouldClose, fixed by async setters

* Fix test dependency

* do not show quick functions tab

* increase documentation popover width

* fix functional test

* Call setActiveDimension when updating visualization

* Simplify handling of flyout with incomplete columns

* Fix test issues

* add description to formula telemetry

* fix schema

* Update from design feedback

* More review comments

* Hide callout border from v7 theme

Co-authored-by: dej611 <[email protected]>
Co-authored-by: Kibana Machine <[email protected]>
Co-authored-by: Joe Reuter <[email protected]>
Co-authored-by: Michael Marcialis <[email protected]>
Co-authored-by: Joe Reuter <[email protected]>
Co-authored-by: Marco Liberati <[email protected]>
Co-authored-by: Marco Liberati <[email protected]>

packages/kbn-monaco/src/monaco_imports.ts

@@ -21,5 +21,6 @@ import 'monaco-editor/esm/vs/editor/contrib/folding/folding.js'; // Needed for f
 import 'monaco-editor/esm/vs/editor/contrib/suggest/suggestController.js'; // Needed for suggestions
 import 'monaco-editor/esm/vs/editor/contrib/hover/hover.js'; // Needed for hover
 import 'monaco-editor/esm/vs/editor/contrib/parameterHints/parameterHints.js'; // Needed for signature
+import 'monaco-editor/esm/vs/editor/contrib/bracketMatching/bracketMatching.js'; // Needed for brackets matching highlight
 
 export { monaco };

packages/kbn-tinymath/grammar/grammar.peggy

@@ -1,16 +1,16 @@
 // tinymath parsing grammar
 
 {
-   function simpleLocation (location) {
-  // Returns an object representing the position of the function within the expression,
-  // demarcated by the position of its first character and last character. We calculate these values
-  // using the offset because the expression could span multiple lines, and we don't want to deal
-  // with column and line values.
-  return {
-   min: location.start.offset,
-   max: location.end.offset
+  function simpleLocation (location) {
+    // Returns an object representing the position of the function within the expression,
+    // demarcated by the position of its first character and last character. We calculate these values
+    // using the offset because the expression could span multiple lines, and we don't want to deal
+    // with column and line values.
+    return {
+      min: location.start.offset,
+      max: location.end.offset
+    }
   }
- }
 }
 
 start
@@ -74,26 +74,34 @@ Expression
   = AddSubtract
 
 AddSubtract
-  = _ left:MultiplyDivide rest:(('+' / '-') MultiplyDivide)* _ {
-    return rest.reduce((acc, curr) => ({
+  = _ left:MultiplyDivide rest:(('+' / '-') MultiplyDivide)+ _ {
+    const topLevel = rest.reduce((acc, curr) => ({
       type: 'function',
       name: curr[0] === '+' ? 'add' : 'subtract',
       args: [acc, curr[1]],
-      location: simpleLocation(location()),
-      text: text()
-    }), left)
+    }), left);
+    if (typeof topLevel === 'object') {
+      topLevel.location = simpleLocation(location());
+      topLevel.text = text();
+    }
+    return topLevel;
   }
+  / MultiplyDivide
 
 MultiplyDivide
   = _ left:Factor rest:(('*' / '/') Factor)* _ {
-    return rest.reduce((acc, curr) => ({
+    const topLevel = rest.reduce((acc, curr) => ({
       type: 'function',
       name: curr[0] === '*' ? 'multiply' : 'divide',
       args: [acc, curr[1]],
-      location: simpleLocation(location()),
-      text: text()
-    }), left)
+    }), left);
+    if (typeof topLevel === 'object') {
+      topLevel.location = simpleLocation(location());
+      topLevel.text = text();
+    }
+    return topLevel;
   }
+  / Factor
 
 Factor
   = Group

packages/kbn-tinymath/index.d.ts

@@ -24,9 +24,11 @@ export interface TinymathLocation {
 export interface TinymathFunction {
   type: 'function';
   name: string;
-  text: string;
   args: TinymathAST[];
-  location: TinymathLocation;
+  // Location is not guaranteed because PEG grammars are not left-recursive
+  location?: TinymathLocation;
+  // Text is not guaranteed because PEG grammars are not left-recursive
+  text?: string;
 }
 
 export interface TinymathVariable {

packages/kbn-tinymath/test/library.test.js

@@ -41,6 +41,35 @@ describe('Parser', () => {
     });
   });
 
+  describe('Math', () => {
+    it('converts basic symbols into left-to-right pairs', () => {
+      expect(parse('a + b + c - d')).toEqual({
+        args: [
+          {
+            name: 'add',
+            type: 'function',
+            args: [
+              {
+                name: 'add',
+                type: 'function',
+                args: [
+                  expect.objectContaining({ location: { min: 0, max: 2 } }),
+                  expect.objectContaining({ location: { min: 3, max: 6 } }),
+                ],
+              },
+              expect.objectContaining({ location: { min: 7, max: 10 } }),
+            ],
+          },
+          expect.objectContaining({ location: { min: 11, max: 13 } }),
+        ],
+        name: 'subtract',
+        type: 'function',
+        text: 'a + b + c - d',
+        location: { min: 0, max: 13 },
+      });
+    });
+  });
+
   describe('Variables', () => {
     it('strings', () => {
       expect(parse('f')).toEqual(variableEqual('f'));
@@ -263,6 +292,8 @@ describe('Evaluate', () => {
     expect(evaluate('5/20')).toEqual(0.25);
     expect(evaluate('1 + 1 + 2 + 3 + 12')).toEqual(19);
     expect(evaluate('100 / 10 / 10')).toEqual(1);
+    expect(evaluate('0 * 1 - 100 / 10 / 10')).toEqual(-1);
+    expect(evaluate('100 / (10 / 10)')).toEqual(100);
   });
 
   it('equations with functions', () => {

src/plugins/dashboard/server/usage/dashboard_telemetry.test.ts

@@ -72,6 +72,22 @@ const lensXYSeriesB = ({
         visualization: {
           preferredSeriesType: 'seriesB',
         },
+        datasourceStates: {
+          indexpattern: {
+            layers: {
+              first: {
+                columns: {
+                  first: {
+                    operationType: 'terms',
+                  },
+                  second: {
+                    operationType: 'formula',
+                  },
+                },
+              },
+            },
+          },
+        },
       },
     },
   },
@@ -144,6 +160,7 @@ describe('dashboard telemetry', () => {
       expect(collectorData.lensByValue.a).toBe(3);
       expect(collectorData.lensByValue.seriesA).toBe(2);
       expect(collectorData.lensByValue.seriesB).toBe(1);
+      expect(collectorData.lensByValue.formula).toBe(1);
     });
 
     it('handles misshapen lens panels', () => {

src/plugins/dashboard/server/usage/dashboard_telemetry.ts

@@ -27,6 +27,16 @@ interface LensPanel extends SavedDashboardPanel730ToLatest {
         visualization?: {
           preferredSeriesType?: string;
         };
+        datasourceStates?: {
+          indexpattern?: {
+            layers: Record<
+              string,
+              {
+                columns: Record<string, { operationType: string }>;
+              }
+            >;
+          };
+        };
       };
     };
   };
@@ -109,6 +119,19 @@ export const collectByValueLensInfo: DashboardCollectorFunction = (panels, colle
       }
 
       collectorData.lensByValue[type] = collectorData.lensByValue[type] + 1;
+
+      const hasFormula = Object.values(
+        lensPanel.embeddableConfig.attributes.state?.datasourceStates?.indexpattern?.layers || {}
+      ).some((layer) =>
+        Object.values(layer.columns).some((column) => column.operationType === 'formula')
+      );
+
+      if (hasFormula && !collectorData.lensByValue.formula) {
+        collectorData.lensByValue.formula = 0;
+      }
+      if (hasFormula) {
+        collectorData.lensByValue.formula++;
+      }
     }
   }
 };

src/plugins/expressions/common/expression_functions/specs/map_column.ts

@@ -10,7 +10,7 @@ import { Observable } from 'rxjs';
 import { take } from 'rxjs/operators';
 import { i18n } from '@kbn/i18n';
 import { ExpressionFunctionDefinition } from '../types';
-import { Datatable, getType } from '../../expression_types';
+import { Datatable, DatatableColumn, getType } from '../../expression_types';
 
 export interface MapColumnArguments {
   id?: string | null;
@@ -110,10 +110,10 @@ export const mapColumn: ExpressionFunctionDefinition<
 
     return Promise.all(rowPromises).then((rows) => {
       const type = rows.length ? getType(rows[0][columnId]) : 'null';
-      const newColumn = {
+      const newColumn: DatatableColumn = {
         id: columnId,
         name: args.name,
-        meta: { type },
+        meta: { type, params: { id: type } },
       };
       if (args.copyMetaFrom) {
         const metaSourceFrom = columns.find(({ id }) => id === args.copyMetaFrom);

src/plugins/expressions/common/expression_functions/specs/tests/map_column.test.ts

@@ -29,7 +29,11 @@ describe('mapColumn', () => {
     expect(result.type).toBe('datatable');
     expect(result.columns).toEqual([
       ...testTable.columns,
-      { id: 'pricePlusTwo', name: 'pricePlusTwo', meta: { type: 'number' } },
+      {
+        id: 'pricePlusTwo',
+        name: 'pricePlusTwo',
+        meta: { type: 'number', params: { id: 'number' } },
+      },
     ]);
     expect(result.columns[result.columns.length - 1]).toHaveProperty('name', 'pricePlusTwo');
     expect(result.rows[arbitraryRowIndex]).toHaveProperty('pricePlusTwo');

src/plugins/kibana_react/public/code_editor/code_editor.tsx

@@ -187,10 +187,16 @@ export class CodeEditor extends React.Component<Props, {}> {
             wordBasedSuggestions: false,
             wordWrap: 'on',
             wrappingIndent: 'indent',
+            matchBrackets: 'never',
             ...options,
           }}
         />
-        <ReactResizeDetector handleWidth handleHeight onResize={this._updateDimensions} />
+        <ReactResizeDetector
+          handleWidth
+          handleHeight
+          onResize={this._updateDimensions}
+          refreshMode="debounce"
+        />
       </>
     );
   }

x-pack/plugins/canvas/shareable_runtime/webpack.config.js

@@ -38,7 +38,7 @@ module.exports = {
         'src/plugins/data/public/expressions/interpreter'
       ),
       'kbn/interpreter': path.resolve(KIBANA_ROOT, 'packages/kbn-interpreter/target/common'),
-      tinymath: path.resolve(KIBANA_ROOT, 'node_modules/tinymath/lib/tinymath.es5.js'),
+      tinymath: path.resolve(KIBANA_ROOT, 'node_modules/tinymath/lib/tinymath.min.js'),
       core_app_image_assets: path.resolve(KIBANA_ROOT, 'src/core/public/core_app/images'),
     },
     extensions: ['.js', '.json', '.ts', '.tsx', '.scss'],