[Meta][ML] Data Visualizer redesign & plugins integration

[qn895]

Phase 1: Redesign & Integration

Purpose: Redesign index data visualizer and file data visualizer to table/grid format and add better integrations with other plugins (Lens, Maps, Discover).

• Index-based data visualizer [ML] Redesign index-based Data Visualizer #85726
  • Functional tests [ML] Fix functional tests for index based Data Visualizer #86071
  • Expand map embed to use for geolocation fields [ML] Add embedded map to geo_point fields for Data Visualizer #88880
  • Match the labels for distribution previews with formatter for that field in the index pattern [ML] Add embedded map to geo_point fields for Data Visualizer #88880
  • Add link to view the selected index in Discover [ML] Add Lens and Discover integration to index based Data Visualizer #89471
  • Expand actions menu to integrate with Lens [ML] Add Lens and Discover integration to index based Data Visualizer #89471
  • Add action for creating Data Frame Analytics jobs [ML] Add Create Data Frame Analytics card to Data Visualizer #91011
  • Explore charting options for date fields (such as https://github.com/walterra/d3-milestones)
• File data visualizer [ML] Redesign file-based Data Visualizer #87598
  • Functional tests
  • Show in the collapsed row to indicate the range of min / median / max for number types (e.g. box plot?)
• Misc. fixes
  • Fix the spacing of the top values keywords between long vs short texts
  • When using a long time range, the event rate chart can appear empty [ML] Fix Data Visualizer event rate chart empty for some indices when using long time range #97655
  • Provide way for clearing field name filter selection [ML] Data viz does not allow you to reset field name picker #110019
• Additional stats
  • Histogram fields [ML] Fix Index Data Visualizer error if index pattern has histogram field type #104553
  • Metric field percentile [ML] Add metric field percentile info to file structure endpoint elasticsearch#37357

---

Phase 2: Migration To Independent Plugins #101435

Purpose: Move all components (the API endpoints as well the file and index based visualizers) of the Data Visualizer from the ml plugin into its own separate plugins file_upload and data_visualizer.

• Move file upload endpoints into its own plugin
• Move File Data Visualizer into its own plugin
• Move Index Data Visualizer into file_data_visualizer plugin [ML] Move the index data visualizer into a separate plugin #98939
  • Add in data recogniser links [ML] Add recognized modules links for Index data visualizer #131342
• Add Index Pattern editor flyout to DV [ML] Add index pattern management to index Data Visualizer #101316 (v7.14)
• Use React.lazy rather than these custom lazy loaders (original comment)

---

Phase 3: Discover Integration, Bug Fixes & Potential Enhancements

• Embed Index Data Visualizer table as a view in Discover
  • Move data visualizer related functional tests out of ml
• Index Data Visualizer
  • Support filters in KQL query bar
  • Ensure the data visualizer performs at scale, for example against indices containing 100s+ of fields
  • Open geo fields in maps [ML] Add Maps ui action to Index data visualizer/Discover's Field statistics #120846
  • Use Elastic charts multi-line time axis [ML] Add multilayer time axis style to Data visualizer doc count chart #117398
  • Increase width of shard size dropdown to fit 100000 option
  • Preserve time picker auto-refresh interval (e.g. if set to Off for large indexes / lots of fields)
  • Add loading spinner when results are being fetched
  • Adjust distribution chart for the case that there is only one value to display in the 0th-90th percentile
    
    
  • Allow drag selection on the document count chart to apply the selected time range to the view (as in Discover)
  • Use runtime field editor for editing runtime fields in index pattern [ML] Add index pattern management to index Data Visualizer #101316 (7.14)
  • Support display of fields for which a custom label has been set [ML] Add index pattern management to index Data Visualizer #101316 (7.14)
  • Add doc count/% for non-aggregatable fields like text fields
  • For geo fields, we should de-duplicate examples. I think this already happens for non-aggregatable text fields [ML] Data Visualizer: Remove duplicated geo examples, support 'version' type, add filters for boolean fields, and add sticky header to Discover #136236:
    
  • For boolean fields, add filter plus / minus icons next to the true and false values in the expanded row summary info [ML] Data Visualizer: Remove duplicated geo examples, support 'version' type, add filters for boolean fields, and add sticky header to Discover #136236
  • Indicate version type fields with icon, as used in Discover [ML] Data Visualizer: Remove duplicated geo examples, support 'version' type, add filters for boolean fields, and add sticky header to Discover #136236
• Pin header row of Field Statistics table when scrolling [Discover] Pin header row in Field Statistics when scrolling #136299
• Add field types in-product help [ML] Add field types in-product help #131577
• Use the display name of the data view instead of the index pattern if display name is available

[elasticmachine]

Pinging @elastic/ml-ui (:ml)

[peteharverson]

Added this item to the list of issues:
For geo fields, we should de-duplicate examples. I think this already happens for non-aggregatable text fields:

[qn895]

Closing this meta issue as most of the items have been done 🎉