[ML] Moving file data vizualizer to its own plugin #96408

jgowdyelastic · 2021-04-07T13:44:42Z

Moves ML's file data visualizer component into it's own plugin fileDataVisualizer and then imports it back into ML for use in the Data Visualizer section of the app.

Note, very little new code has been added, all UI code in this PR has just been moved or copied from the ML plugin.

The file data visualizer component relies on a decent amount of code shared with the index data visualizer which has had to be duplicated into the new plugin, along with some common ML utility functions and types.

The feature should function as before with some exceptions.

Boolean field types are missing their preview charts as it would require copying a lot of code, this will be fixed in a follow up PR.
Links to ML pages have been remove after a file has been successfully uploaded. If we still want this these links, they will have to be added in a follow up PR. Plugins which use this component will have to supply the results links to be embedded.

Also adds index_exists and time_field_range endpoints to the fileUpload plugin to assist with importing data, moves the analyze_file endpoint from fileUpload to fileDataVisualizer

Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
Documentation was added for features that require explanation or tutorials
Unit or functional tests were updated or added to match the most common scenarios
This was checked for breaking API changes and was labeled appropriately

jgowdyelastic · 2021-04-08T16:12:15Z

@elasticmachine merge upstream

…oad-plugin

jgowdyelastic · 2021-04-09T08:20:44Z

@elasticmachine merge upstream

…oad-plugin

jgowdyelastic · 2021-04-09T10:31:09Z

retest

jgowdyelastic · 2021-04-09T10:54:21Z

@elasticmachine merge upstream

…oad-plugin

…f github.com:jgowdyelastic/kibana into moving-file-data-visualizer-UI-to-file-upload-plugin

jgowdyelastic · 2021-04-12T08:09:29Z

@elasticmachine merge upstream

…oad-plugin

kibanamachine · 2021-04-13T13:02:18Z

⏳ Build in-progress, with failures

continuous-integration/kibana-ci/pull-request
Commit: dee153b
Storybooks not built yet
This comment will update when the build is complete

Failed CI Steps

Check Bundle Limits

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

jgowdyelastic · 2021-04-13T15:14:54Z

@elasticmachine merge upstream

…oad-plugin

kibanamachine · 2021-04-13T15:34:28Z

⏳ Build in-progress, with failures

continuous-integration/kibana-ci/pull-request
Commit: 4d98b7d
Storybooks not built yet
This comment will update when the build is complete

Failed CI Steps

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

kibanamachine · 2021-04-13T16:10:37Z

⏳ Build in-progress, with failures

continuous-integration/kibana-ci/pull-request
Commit: 21f6ce6
Storybooks not built yet
This comment will update when the build is complete

Failed CI Steps

Check Bundle Limits

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

qn895 · 2021-04-18T18:03:31Z

Code LGTM 🎉

peteharverson

Tested and LGTM.

Would be good to get the boolean type preview chart showing up in the collapsed row, but happy for that to be done in a separate PR.

nreese

There are a lot of duplicated components between file data visualizer and index pattern visualizer. Should fileDataVisualizer expose these components so they can then be used in ML for the index pattern visualizer? Or would it make sense to move index pattern visualizer into this plugin as well?

nreese · 2021-04-19T14:13:58Z

...k/plugins/file_data_visualizer/public/application/components/file_contents/file_contents.tsx

@@ -10,7 +10,7 @@ import React, { FC } from 'react';

 import { EuiTitle, EuiSpacer } from '@elastic/eui';

-import { MLJobEditor, ML_EDITOR_MODE } from '../../../../jobs/jobs_list/components/ml_job_editor';
+import { MLJobEditor, ML_EDITOR_MODE } from '../ml_job_editor';


Should these loose the ML prefix?

nreese · 2021-04-19T14:14:14Z

...ugins/file_data_visualizer/public/application/components/field_type_icon/field_type_icon.tsx

+
+import { i18n } from '@kbn/i18n';
+
+import { getMLJobTypeAriaLabel } from '../../util/field_types_utils';


Should getMLJobTypeAriaLabel lose the ML prefix?

nreese · 2021-04-19T14:15:35Z

...isualizer/public/application/components/file_datavisualizer_view/file_datavisualizer_view.js

-    this.maxFileUploadBytes = getFileUpload().getMaxBytes();
+
+    this.savedObjectsClient = props.savedObjectsClient;
+    this.maxFileUploadBytes = this.props.fileUpload.getMaxBytes();


Nit: use 'props' instead of 'this.props' to be consistent with line above

nreese · 2021-04-19T14:22:29Z

...ins/file_data_visualizer/public/application/components/ml_embedded_map/_ml_embedded_map.scss

@@ -0,0 +1,8 @@
+.mlEmbeddedMapContent {


Should ml prefix be removed?

nreese · 2021-04-19T14:23:09Z

...ugins/file_data_visualizer/public/application/components/ml_embedded_map/ml_embedded_map.tsx

+} from '../../../../../../../src/plugins/embeddable/public';
+import { useFileDataVisualizerKibana } from '../../kibana_context';
+
+export function MlEmbeddedMapComponent({


Should this just be called EmbeddedMapComponent?

nreese · 2021-04-19T14:40:15Z

x-pack/plugins/file_upload/common/types.ts

+    statusCode: number;
+    error: string;
+    message: string;
+    attributes?: any;


How about replacing any with a type definition?

nreese · 2021-04-19T14:42:39Z

x-pack/plugins/file_upload/public/api/index.ts

+): Promise<boolean> {
+  const body = JSON.stringify({ index });
+  const fileUploadModules = await lazyLoadModules();
+  return await fileUploadModules.getHttp().fetch<any>({


How about replacing any with a type definition for check upload resposne?

nreese · 2021-04-19T14:42:57Z

x-pack/plugins/file_upload/public/api/index.ts

+  });
+}
+
+export async function getTimeFieldRange(index: string, query: any, timeFieldName?: string) {


Can query be unknown instead of any?

nreese · 2021-04-19T14:44:46Z

x-pack/plugins/file_upload/server/get_time_field_range.ts

+  client: IScopedClusterClient,
+  index: string[] | string,
+  timeFieldName: string,
+  query: any


Can query be unknown instead of any?

nreese · 2021-04-19T14:46:55Z

x-pack/plugins/file_upload/server/get_time_field_range.ts

+ * 2.0.
+ */
+import { IScopedClusterClient } from 'kibana/server';
+export async function getTimeFieldRange(


Is getTimeFieldRange endpoint even needed? Why not just use already exposed ES search endpoint and have a client function that contains this logic?

yes i agree, the original ML version of this function takes more arguments and so I think justifies having it's own endpoint. i've stripped most of these out for the file_upload version as they're not needed.
I'll move this search to the client side in a follow up PR.

jgowdyelastic · 2021-04-19T15:36:46Z

@nreese

Or would it make sense to move index pattern visualizer into this plugin as well?

Seeing as this new plugin is decoupled from the file upload process, it might make sense to move the index data visualizer in here too.
There are plans on ML's road map to move it out so it can be embedded into discover.
I think for this first version it would be ok to have these components duplicated, If we decide to move the index data visualizer into this plugin (and rename plugin) we can revert some of the changes i've made to separate them out.
If we don't decide to move it in here, I agree that making the components shared between plugins would be good.

jgowdyelastic · 2021-04-20T13:46:02Z

@elasticmachine merge upstream

…oad-plugin

…sualizer-UI-to-file-upload-plugin

nreese

Thanks for moving file data visualizer into its own plugin. This is a great first step to integrating in other Kibana plugins.

LGTM

kibanamachine · 2021-04-20T21:09:49Z

💚 Build Succeeded

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
`fileDataVisualizer`	-	178	+178
`ml`	1803	1732	-71
total			+107

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`fileDataVisualizer`	-	1.0MB	⚠️ +1.0MB
`fileUpload`	789.1KB	789.1KB	+5.0B
`ml`	6.0MB	5.9MB	-155.6KB
total			+881.5KB

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
`fileDataVisualizer`	-	11.8KB	+11.8KB
`fileUpload`	14.0KB	15.0KB	+1014.0B
`ml`	68.9KB	68.5KB	-409.0B
total			+12.4KB

Unknown metric groups

API count

id	before	after	diff
`fileDataVisualizer`	-	19	+19
`fileUpload`	180	115	-65
total			-46

API count missing comments

id	before	after	diff
`fileDataVisualizer`	-	19	+19
`fileUpload`	180	115	-65
total			-46

API count with any type

id	before	after	diff
`fileUpload`	6	4	-2

async chunk count

id	before	after	diff
`fileDataVisualizer`	-	1	+1

History

💔 Build #121041 failed f94eb16
💔 Build #120971 failed 0788b3a
💚 Build #120357 succeeded dd35676
💛 Build #120354 was flaky 1a4eec2
💔 Build #120355 failed 1721143

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @jgowdyelastic

* [ML] Moving file data vizualizer to file upload plugin * removing maps plug dependency * fixing imports * small refactor * adding missing endpoints * fixing translations * fxing table controls * fixing types and disabling geo point test * actually disabling geo point test * making endpoints internal * moving UI code to separate plugin * enabling maps integration * cleaning up dependencies * fixing translation ids * moving analyze file endpoint out of file upload plugin * fixing transtations issues * refactor for lazy loading of component * updating limits * updating plugin asciidoc * code clean up * further clean up * adding comment * fixing really obvious CI error * removing commented out include * reenabling geo point test * fixing incorrectly changed import * removing ml from labels and identifiers * renaming function * moving analyse file endpoint to file upload plugin * reverting import path changes * adding esUiShared back in * fixing navigation tabs alignment in basic license * adding key to tab wrapper * reverting test label * further removal of ml references * removing ml label from more identifiers * fixing tests Co-authored-by: Kibana Machine <[email protected]>

kibanamachine · 2021-04-20T21:20:06Z

💚 Backport successful

Status	Branch	Result
✅	7.x

This backport PR will be merged automatically after passing CI.

* [ML] Moving file data vizualizer to file upload plugin * removing maps plug dependency * fixing imports * small refactor * adding missing endpoints * fixing translations * fxing table controls * fixing types and disabling geo point test * actually disabling geo point test * making endpoints internal * moving UI code to separate plugin * enabling maps integration * cleaning up dependencies * fixing translation ids * moving analyze file endpoint out of file upload plugin * fixing transtations issues * refactor for lazy loading of component * updating limits * updating plugin asciidoc * code clean up * further clean up * adding comment * fixing really obvious CI error * removing commented out include * reenabling geo point test * fixing incorrectly changed import * removing ml from labels and identifiers * renaming function * moving analyse file endpoint to file upload plugin * reverting import path changes * adding esUiShared back in * fixing navigation tabs alignment in basic license * adding key to tab wrapper * reverting test label * further removal of ml references * removing ml label from more identifiers * fixing tests Co-authored-by: Kibana Machine <[email protected]> Co-authored-by: James Gowdy <[email protected]>

Adds versioning to all of the file upload APIs. Versions are added to the server side routes and to the client side functions which call the routes. Updates API tests to add the API version to the request headers. All of the APIs are internal and have been given the version '1'. Also renames `/internal/file_data_visualizer/analyze_file` to `/internal/file_upload/analyze_file` It appears this was a mistake from when the route was moved from the data visualiser plugin midway through development on [this PR](#96408). **Internal APIs** `/internal/file_upload/analyze_file` `/internal/file_upload/has_import_permission` `/internal/file_upload/index_exists` `/internal/file_upload/time_field_range` --------- Co-authored-by: kibanamachine <[email protected]>

jgowdyelastic added 7 commits April 7, 2021 14:43

[ML] Moving file data vizualizer to file upload plugin

7c367e4

removing maps plug dependency

96900b7

fixing imports

fa9059f

small refactor

d1b4f86

adding missing endpoints

8db222b

fixing translations

c0e46ad

fxing table controls

2f45468

Merge branch 'master' into moving-file-data-visualizer-UI-to-file-upl…

176c891

…oad-plugin

kibanamachine and others added 2 commits April 9, 2021 04:20

Merge branch 'master' into moving-file-data-visualizer-UI-to-file-upl…

6a3af38

…oad-plugin

fixing types and disabling geo point test

56e73f5

kibanamachine and others added 3 commits April 9, 2021 06:54

Merge branch 'master' into moving-file-data-visualizer-UI-to-file-upl…

4182f02

…oad-plugin

actually disabling geo point test

d874090

Merge branch 'moving-file-data-visualizer-UI-to-file-upload-plugin' o…

d022521

…f github.com:jgowdyelastic/kibana into moving-file-data-visualizer-UI-to-file-upload-plugin

kibanamachine and others added 4 commits April 12, 2021 04:09

Merge branch 'master' into moving-file-data-visualizer-UI-to-file-upl…

e0d2d2f

…oad-plugin

making endpoints internal

0aa4952

moving UI code to separate plugin

6842e98

enabling maps integration

dee153b

cleaning up dependencies

66c6840

Merge branch 'master' into moving-file-data-visualizer-UI-to-file-upl…

4d98b7d

…oad-plugin

fixing translation ids

21f6ce6

moving analyze file endpoint out of file upload plugin

63f6d4f

peteharverson approved these changes Apr 19, 2021

View reviewed changes

nreese reviewed Apr 19, 2021

View reviewed changes

further removal of ml references

51cba88

kibanamachine and others added 4 commits April 20, 2021 09:46

Merge branch 'master' into moving-file-data-visualizer-UI-to-file-upl…

0788b3a

…oad-plugin

removing ml label from more identifiers

f94eb16

fixing tests

e36a733

Merge remote-tracking branch 'origin/master' into moving-file-data-vi…

ee2f66f

…sualizer-UI-to-file-upload-plugin

nreese approved these changes Apr 20, 2021

View reviewed changes

jgowdyelastic added the auto-backport Deprecated - use backport:version if exact versions are needed label Apr 20, 2021

jgowdyelastic changed the title ~~[ML] Moving file data vizualizer to file upload plugin~~ [ML] Moving file data vizualizer to its own plugin Apr 20, 2021

jgowdyelastic merged commit 088a618 into elastic:master Apr 20, 2021

jgowdyelastic deleted the moving-file-data-visualizer-UI-to-file-upload-plugin branch April 20, 2021 21:17

kibanamachine mentioned this pull request Apr 20, 2021

[7.x] [ML] Moving file data vizualizer to its own plugin (#96408) #97752

Merged

This was referenced Apr 26, 2021

[File Data Visualizer] Fixing missing css imports for file stats table #98312

Merged

[File data visualizer] Embed component into Add Data page in home plugin #98652

Closed

peteharverson mentioned this pull request Apr 30, 2021

[ML] Move the index data visualizer into a separate plugin #98939

Closed

10 tasks

qn895 mentioned this pull request May 28, 2021

[ML] Move Index Data Visualizer into separate plugin (Part 1) #100922

Merged

17 tasks

qn895 mentioned this pull request Jun 4, 2021

[Meta] Move Data Visualizer to its own plugin #101435

Closed

25 tasks

jgowdyelastic mentioned this pull request Jul 1, 2021

[ML] Fixing missing data visualizer links #103932

Merged

3 tasks

jgowdyelastic mentioned this pull request May 23, 2023

[ML] Versioning file upload APIs #158265

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ML] Moving file data vizualizer to its own plugin #96408

[ML] Moving file data vizualizer to its own plugin #96408

jgowdyelastic commented Apr 7, 2021 •

edited

Loading

jgowdyelastic commented Apr 8, 2021

jgowdyelastic commented Apr 9, 2021

jgowdyelastic commented Apr 9, 2021

jgowdyelastic commented Apr 9, 2021

jgowdyelastic commented Apr 12, 2021

kibanamachine commented Apr 13, 2021

jgowdyelastic commented Apr 13, 2021

kibanamachine commented Apr 13, 2021

kibanamachine commented Apr 13, 2021

qn895 commented Apr 18, 2021

peteharverson left a comment

nreese left a comment

nreese Apr 19, 2021

nreese Apr 19, 2021

nreese Apr 19, 2021

nreese Apr 19, 2021

nreese Apr 19, 2021

nreese Apr 19, 2021

nreese Apr 19, 2021

nreese Apr 19, 2021

nreese Apr 19, 2021

nreese Apr 19, 2021

jgowdyelastic Apr 20, 2021

jgowdyelastic commented Apr 19, 2021

jgowdyelastic commented Apr 20, 2021

nreese left a comment

kibanamachine commented Apr 20, 2021

API count

API count missing comments

API count with any type

async chunk count

kibanamachine commented Apr 20, 2021


		import { i18n } from '@kbn/i18n';

		import { getMLJobTypeAriaLabel } from '../../util/field_types_utils';

[ML] Moving file data vizualizer to its own plugin #96408

[ML] Moving file data vizualizer to its own plugin #96408

Conversation

jgowdyelastic commented Apr 7, 2021 • edited Loading

jgowdyelastic commented Apr 8, 2021

jgowdyelastic commented Apr 9, 2021

jgowdyelastic commented Apr 9, 2021

jgowdyelastic commented Apr 9, 2021

jgowdyelastic commented Apr 12, 2021

kibanamachine commented Apr 13, 2021

⏳ Build in-progress, with failures

Failed CI Steps

History

jgowdyelastic commented Apr 13, 2021

kibanamachine commented Apr 13, 2021

⏳ Build in-progress, with failures

Failed CI Steps

History

kibanamachine commented Apr 13, 2021

⏳ Build in-progress, with failures

Failed CI Steps

History

qn895 commented Apr 18, 2021

peteharverson left a comment

Choose a reason for hiding this comment

nreese left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jgowdyelastic commented Apr 19, 2021

jgowdyelastic commented Apr 20, 2021

nreese left a comment

Choose a reason for hiding this comment

kibanamachine commented Apr 20, 2021

💚 Build Succeeded

Metrics [docs]

Module Count

Async chunks

Page load bundle

API count

API count missing comments

API count with any type

async chunk count

History

kibanamachine commented Apr 20, 2021

💚 Backport successful

jgowdyelastic commented Apr 7, 2021 •

edited

Loading