[ML] Kibana API endpoint for histogram chart data #70976

walterra · 2020-07-07T15:40:01Z

Summary

Part of #69235.

Introduces a dedicated Kibana API endpoint as part of ML plugin's suite of data visualizer API endpoints and moves the logic to query and transform the required data from client to server.
Adds support for sampling to retrieve the data for the field histograms. For now this is not configurable by the end user and is hard coded to 5000. This is to have a first iteration of this functionality in for 7.9 and protect users when querying large clusters. The button to enable the histogram charts now includes a tooltip that mentions the sampler.

Todos

Checklist

Unit or functional tests were updated or added to match the most common scenarios

For maintainers

This was checked for breaking API changes and was labeled appropriately

…r all data fetching.

elasticmachine · 2020-07-09T10:50:41Z

Pinging @elastic/ml-ui (:ml)

peteharverson · 2020-07-09T11:24:23Z

x-pack/plugins/ml/server/routes/schemas/data_visualizer_schema.ts

+export const dataVisualizerFieldHistogramsSchema = schema.object({
+  /** Query to match documents in the index. */
+  query: schema.any(),
+  fields: schema.arrayOf(schema.any()),


Nit - can you add some sort of comment for fields too.

Updated in b794e97.

peteharverson · 2020-07-09T13:55:04Z

x-pack/plugins/ml/public/application/services/ml_api_service/index.ts

+        samplerShardSize,
+      });
+
+      return httpService.http<any>({


I am repeatedly seeing this error the first time I try to view the histograms in the Transforms wizard.

The workflow I use is:

Open the ML plugin. Switch to the Data Frame Analytics page.

Starting creating a DFA job, and view the histograms for the source data.

Then switch to the Transform page in Stack Management.

Create a transform, and click the Histogram charts button

Once in the Transforms wizard, if I then do a browser refresh, the charts then start working as expected.

Tested latest edit with 84203df and this fixes the issue with http for me.

@peteharverson It turned out having the endpoint only as part of the ML plugin doesn't guarantee that it's available when the transform plugin loads. To avoid a runtime dependency between the transforms and ML plugin (which would mean adding ML to transforms' kibana.json) I duplicated the API endpoint so each plugin has their own endpoint to get the histogram data. Both plugins still share the code to created the server side aggregations. The update is in
a117e43, please have another look.

…via transforms.

peteharverson

Tested latest edits against a variety of indexes and LGTM.

walterra · 2020-07-09T16:31:19Z

@elasticmachine merge upstream

…l-api-histograms

darnautov · 2020-07-13T10:27:49Z

x-pack/plugins/ml/public/application/components/data_grid/data_grid.tsx

+                          color="text"
+                          onClick={toggleChartVisibility}
+                        >
+                          {i18n.translate('xpack.ml.dataGrid.histogramButtonText', {


shall we use FormattedMessage in templates?

.../ml/public/application/data_frame_analytics/pages/analytics_creation/hooks/use_index_data.ts

.../pages/analytics_exploration/components/exploration_results_table/use_exploration_results.ts

darnautov · 2020-07-13T10:33:18Z

x-pack/plugins/ml/public/application/datavisualizer/index_based/page.tsx

@@ -107,7 +108,7 @@ export const Page: FC = () => {
    autoRefreshSelector: true,
  });

-  const dataLoader = new DataLoader(currentIndexPattern, kibanaConfig);
+  const dataLoader = new DataLoader(currentIndexPattern, getToastNotifications());


Please avoid usting cached getToastNotifications and retriveve toastNotifications from the kibana context.

The rest of the file uses the same approach and I didn't plan to refactor it in this PR.

...ame_analytics/pages/analytics_exploration/components/outlier_exploration/use_outlier_data.ts

x-pack/plugins/ml/public/application/datavisualizer/index_based/data_loader/data_loader.ts

darnautov · 2020-07-13T11:45:37Z

x-pack/plugins/transform/public/app/app.tsx

+  // TODO Temporary fix to make the Data Grid Histograms in the Transforms Wizard work
+  setDependencyCache({
+    http: appDependencies.http,
+  });
+


I believe you can avoid this temp fix without significant refactoring.

Removed in 3306ff8.

x-pack/plugins/transform/public/app/hooks/use_api.ts

walterra · 2020-07-14T10:21:14Z

@darnautov addressed your comments, tests passed, please have another look.

kibanamachine · 2020-07-14T10:46:33Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: 5b2ecdc

Build metrics

@kbn/optimizer bundle module count

id	value	diff	baseline
ml	387	+1	386

History

💚 Build #61275 succeeded 3883477
💔 Build #60731 failed 1b16be3
💔 Build #60323 failed 45857f9
💔 Build #60257 failed 84203df
💔 Build #60183 failed 4371a65

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

darnautov

LGTM

- Introduces dedicated Kibana API endpoints as part of ML and transform plugin API endpoints and moves the logic to query and transform the required data from client to server. - Adds support for sampling to retrieve the data for the field histograms. For now this is not configurable by the end user and is hard coded to 5000. This is to have a first iteration of this functionality in for 7.9 and protect users when querying large clusters. The button to enable the histogram charts now includes a tooltip that mentions the sampler.

- Introduces dedicated Kibana API endpoints as part of ML and transform plugin API endpoints and moves the logic to query and transform the required data from client to server. - Adds support for sampling to retrieve the data for the field histograms. For now this is not configurable by the end user and is hard coded to 5000. This is to have a first iteration of this functionality in for 7.9 and protect users when querying large clusters. The button to enable the histogram charts now includes a tooltip that mentions the sampler. Co-authored-by: Elastic Machine <[email protected]>

* master: (21 commits) [Maps] 7.9 design improvements (elastic#71563) [ML] Changing all calls to ML endpoints to use internal user (elastic#70487) [eventLog] prevent log writing when initialization fails (elastic#71339) [Observability] landing page always being displayed (elastic#71494) [IM] Address data stream copy feedback (elastic#71615) [Logs UI] Anomalies page dataset filtering (elastic#71110) [data.search.aggs] Remove `use_field_mapping` from top hits agg (elastic#71168) [ML] Anomaly swim lane embeddable navigation and filter actions (elastic#71082) Fixes typo in siem_cloudtrail job description (elastic#71569) Require granted API Keys to have a name (elastic#71623) Update getUsageForCollection (elastic#71609) Only fetch saved elements once (elastic#71310) [SecuritySolution][Resolver] Adding siem index and guarding process ancestry (elastic#71570) [APM] Additional data telemetry changes (elastic#71112) [Visualize] Fix export table for table export links (elastic#71249) [Search] Server side search API (elastic#70446) use inclusive language (elastic#71607) [Security Solution] Hide timeline footer when Resolver is open (elastic#71516) [Index template wizard] Remove shadow and use border for components panels (elastic#71606) [ML] Kibana API endpoint for histogram chart data (elastic#70976) ...

walterra self-assigned this Jul 7, 2020

walterra added :ml Feature:Data Frame Analytics ML data frame analytics features Feature:Transforms ML transforms release_note:skip Skip the PR/issue when compiling release notes v7.9.0 v8.0.0 labels Jul 7, 2020

walterra added 7 commits July 9, 2020 12:47

[ML] Introduce ML API server side histograms endpoint.

1743e9e

[ML] Use histogram endpoint for analytics results pages.

b66de70

[ML] Use histogram endpoint for analytics wizard.

e079df8

[ML] Fix DataLoader dependencies and constructor. Use API endpoint fo…

8296524

…r all data fetching.

[ML] Sampler support. Reintroduce KBN_FIELD_TYPES.

b939a87

[ML] Sample info tooltip.

5daa854

[ML] Added jest and API integration tests.

3f45214

walterra force-pushed the ml-api-histograms branch from 038cbc0 to 3f45214 Compare July 9, 2020 10:47

walterra marked this pull request as ready for review July 9, 2020 10:50

walterra requested a review from a team as a code owner July 9, 2020 10:50

walterra requested review from peteharverson, darnautov and qn895 July 9, 2020 10:51

walterra mentioned this pull request Jul 9, 2020

[ML] Data Grid Histograms (7.9) #69235

Closed

19 tasks

peteharverson reviewed Jul 9, 2020

View reviewed changes

walterra added 2 commits July 9, 2020 14:20

[ML] Fix TS.

ea91900

Merge branch 'master' into ml-api-histograms

4371a65

peteharverson reviewed Jul 9, 2020

View reviewed changes

walterra added 2 commits July 9, 2020 16:06

Merge branch 'master' into ml-api-histograms

0984587

[ML] Use setDependencyCache to initialize http when using DataLoader …

84203df

…via transforms.

peteharverson approved these changes Jul 9, 2020

View reviewed changes

Merge branch 'master' into ml-api-histograms

45857f9

qn895 approved these changes Jul 9, 2020

View reviewed changes

walterra added 3 commits July 13, 2020 08:05

[ML] Histogram API endpoint for transforms.

a117e43

Merge branch 'master' into ml-api-histograms

02f1fc6

Merge branch 'ml-api-histograms' of github.com:walterra/kibana into m…

1b16be3

…l-api-histograms