Support Dataframes in Histogram and Scatter Plot Visualizations (#1377)

CHANGELOG.md

@@ -16,6 +16,10 @@
   query to display its results in a table. In addition, the SQL Query
   visualization allows the user to see the query that is going to be run against
   the database.
+- [Histogram and Scatter Plot now support Dataframes.][1377] The `Table` and
+  `Column` datatypes are properly visualized. Scatter Plot can display points of
+  different color, shape and size, all as defined by the data within the
+  `Table`.
 
 <br/>![Bug Fixes](/docs/assets/tags/bug_fixes.svg)
 
@@ -100,6 +104,7 @@ you can find their release notes
 [1385]: https://github.com/enso-org/ide/pull/1385
 [1393]: https://github.com/enso-org/ide/pull/1393
 [1392]: https://github.com/enso-org/ide/pull/1392
+[1377]: https://github.com/enso-org/ide/pull/1377
 
 <br/>
 

docs/product/visualizations.md

@@ -226,16 +226,16 @@ In particular:
   - #### Field `theme`
     The IDE's current color theme. Exposes the following methods.
     - ##### Method `getColorForType`
-      Takes a qualified type name and returns the color that is used in the GUI to represent that
-      type.
+      Takes a qualified type name and returns the color that is used in the GUI
+      to represent that type.
     - ##### Method `getForegroundColorForType`
-      Takes a qualified type name and returns the color that should be used for foreground elements
-      (e.g. text) that are shown on top of the background color returned by `getColorForType`.
+      Takes a qualified type name and returns the color that should be used for
+      foreground elements (e.g. text) that are shown on top of the background
+      color returned by `getColorForType`.
     - ##### Method `get`
-      Takes a style sheet path as string and returns the corresponding value from the theme. For
-      example, `get("graph_editor.node.error.panic")` returns the orange color that is used to mark
-      nodes in an error state.
-
+      Takes a style sheet path as string and returns the corresponding value
+      from the theme. For example, `get("graph_editor.node.error.panic")`
+      returns the orange color that is used to mark nodes in an error state.
 
 - ### [Optional] Field `label`
 
@@ -319,3 +319,54 @@ data types have defined conversion to JSON by default. If the visualization
 input is defined as JSON input, the binary stream will be converted to JSON by
 the GUI engine before passing to visualization. It is up to the visualization
 author to handle the textual or binary form.
+
+## Builtin Visualizations
+
+IDE comes with a set of predefined visualizations, including charts.
+
+### Dataframes Support
+
+Some of the predefined visualizations have some special support for `Table` from
+Enso Dataframes library.
+
+#### Histogram
+
+When using `Histogram` visualization on a `Table` value it will first look for a
+column named `value`. If present, it will be used as a data source. Otherwise,
+`Histogram` will use the first numerical column.
+
+#### Scatter Plot
+
+The `Scatter Plot` visualization has several properties for each point. If a
+column of a matching name is present in the `Table` it will be used.
+
+- `x` — position on horizontal axis. If not present, the index column will be
+  used. If there is no index set, the row indices will be used. If this column
+  has a missing value, the point will be omitted.
+- `y` — position on vertical axis. If not present, first numerical column of
+  unrecognized name will be used. If not present, first numerical column will be
+  used. If this column has a missing value, the point will be omitted.
+- `color` — color of the point. The default color is `black` and will be used if
+  column is not present or for its missing values. `color` should be a `Text`
+  column with elements being in a
+  [CSS colors format](https://www.w3schools.com/cssref/css_colors_legal.asp):
+  - Hexadecimal formats, like `#RGB`, `#RRGGBB` and `#RRGGBBAA`.
+  - RGB function-like syntax, e.g. `rgb(255,0,128)` or `rgba(255,0,128,0.5)`.
+  - HSL function-like syntax, e.g. `hsl(120, 100%, 50%)` or
+    `hsla(120, 100%, 50%, 0.3)`.
+  - name of one of
+    [predefined colors](https://www.w3schools.com/colors/colors_names.asp), e.g.
+    `red` or `SteelBlue`.
+- `label` — text to be displayed next to the point.
+- `shape` — shape of the point. Supported shapes are:
+
+  - `cross`;
+  - `diamond`;
+  - `square`;
+  - `star`;
+  - `triangle`.
+
+  The default shape is a circle.
+
+- `size` — size of the point as a (possible floating point) number. Default size
+  of the point is `1.0`.

src/rust/ide/src/model/project/synchronized.rs

@@ -297,7 +297,7 @@ impl Project {
                     //  see https://github.com/enso-org/ide/issues/145
                 }
                 Event::Error(error) => {
-                    error!(logger,"Error emitted by the binary data connection: {error}.");
+                    error!(logger,"Error emitted by the JSON-RPC data connection: {error}.");
                 }
                 _ => {}
             }

src/rust/ide/view/graph-editor/src/builtin/visualization/java_script.rs

@@ -33,17 +33,19 @@ pub fn sql_visualization() -> visualization::java_script::FallibleDefinition {
 /// Return a `JavaScript` Scatter plot visualization.
 pub fn scatter_plot_visualization() -> visualization::java_script::FallibleDefinition {
     let loading_scripts = include_str!("java_script/helpers/loading.js");
+    let number          = include_str!("java_script/helpers/number.js");
     let source          = include_str!("java_script/scatterPlot.js");
-    let source          = format!("{}{}",loading_scripts,source);
+    let source          = format!("{}{}{}",loading_scripts,number,source);
 
     visualization::java_script::Definition::new_builtin(source)
 }
 
 /// Return a `JavaScript` Histogram visualization.
 pub fn histogram_visualization() -> visualization::java_script::FallibleDefinition {
     let loading_scripts = include_str!("java_script/helpers/loading.js");
+    let number          = include_str!("java_script/helpers/number.js");
     let source          = include_str!("java_script/histogram.js");
-    let source          = format!("{}{}",loading_scripts,source);
+    let source          = format!("{}{}{}",loading_scripts,number,source);
 
     visualization::java_script::Definition::new_builtin(source)
 }

src/rust/ide/view/graph-editor/src/builtin/visualization/java_script/helpers/number.js

@@ -0,0 +1,6 @@
+/**
+ * Check that value is a number other than NaN.
+ */
+function isValidNumber(value) {
+    return typeof value === 'number' && Number.isNaN(value) === false
+}

src/rust/ide/view/graph-editor/src/builtin/visualization/java_script/histogram.js

@@ -17,7 +17,7 @@ const ANIMATION_DURATION = 1000
 const LINEAR_SCALE = 'linear'
 const LIGHT_PLOT_COLOR = '#00E890'
 const DARK_PLOT_COLOR = '#E0A63B'
-const DEFAULT_NUMBER_OF_BINS = 10
+const DEFAULT_NUMBER_OF_BINS = 50
 const BUTTON_HEIGHT = 25
 
 /**
@@ -42,6 +42,11 @@ class Histogram extends Visualization {
     static inputType = 'Any'
     static label = 'Histogram'
 
+    constructor(data) {
+        super(data)
+        this.setPreprocessor('here.process_to_json_text', 'Standard.Visualization.Histogram')
+    }
+
     onDataReceived(data) {
         const { parsedData, isUpdate } = this.parseData(data)
         if (!ok(parsedData)) {
@@ -67,7 +72,7 @@ class Histogram extends Visualization {
         } else {
             parsedData = data
         }
-        const isUpdate = parsedData.update === 'diff'
+        const isUpdate = parsedData?.update === 'diff'
         return { parsedData, isUpdate }
     }
 
@@ -84,26 +89,27 @@ class Histogram extends Visualization {
      */
     updateState(data, isUpdate) {
         if (isUpdate) {
-            this._axisSpec = ok(data.axis) ? data.axis : this._axisSpec
-            this._focus = ok(data.focus) ? data.focus : this._focus
-            this._dataValues = ok(data.data.values) ? data.data.values : this.data
-            this._bins = ok(data.bins) ? data.bins : this._bins
+            this._axisSpec = data.axis ?? this._axisSpec
+            this._focus = data.focus ?? this._focus
+            this._dataValues = this.extractValues(data.data.values ?? this.data)
+            this._bins = data.bins ?? this._bins
         } else {
             this._axisSpec = data.axis
             this._focus = data.focus
             this._dataValues = this.extractValues(data)
             this._bins = data.bins
         }
+
+        this._dataValues = this._dataValues.filter(value => isValidNumber(value))
     }
 
     extractValues(rawData) {
         /// Note this is a workaround for #1006, we allow raw arrays and JSON objects to be consumed.
-        if (ok(rawData.data)) {
-            return rawData.data.values
-        } else if (Array.isArray(rawData)) {
+        if (Array.isArray(rawData)) {
             return rawData
+        } else {
+            return rawData.data.values ?? []
         }
-        return []
     }
 
     /**
@@ -118,7 +124,7 @@ class Histogram extends Visualization {
      */
     axisSpec() {
         return (
-            this._axisSpec || {
+            this._axisSpec ?? {
                 x: { scale: LINEAR_SCALE },
                 y: { scale: LINEAR_SCALE },
             }
@@ -129,18 +135,14 @@ class Histogram extends Visualization {
      * Return vales to plot.
      */
     data() {
-        return this._dataValues || {}
+        return this._dataValues ?? {}
     }
 
     /**
      * Return the number of bins to use for the histogram.
      */
     binCount() {
-        if (!ok(this._bins)) {
-            return DEFAULT_NUMBER_OF_BINS
-        } else {
-            return Math.max(1, self._bins)
-        }
+        return Math.max(1, self._bins ?? DEFAULT_NUMBER_OF_BINS)
     }
 
     /**
@@ -432,10 +434,7 @@ class Histogram extends Visualization {
      */
     rescale(scale, withAnimation) {
         const duration = withAnimation ? ANIMATION_DURATION : 0.0
-        this.xAxis
-            .transition()
-            .duration(duration)
-            .call(d3.axisBottom(scale.x).ticks(this.binCount()))
+        this.xAxis.transition().duration(duration).call(d3.axisBottom(scale.x))
         this.yAxis.transition().duration(duration).call(d3.axisLeft(scale.y))
         this.plot
             .selectAll('rect')
@@ -545,14 +544,14 @@ class Histogram extends Visualization {
         const canvas = this.canvas
 
         const fontStyle = '10px DejaVuSansMonoBook'
-        if (axis.x.label !== undefined) {
+        if (axis?.x?.label !== undefined) {
             this.xAxisLabel
                 .attr('y', canvas.inner.height + canvas.margin.bottom - X_AXIS_LABEL_WIDTH / 2.0)
                 .attr('x', canvas.inner.width / 2.0 + this.textWidth(axis.x.label, fontStyle) / 2)
                 .text(axis.x.label)
         }
         // Note: y axis is rotated by 90 degrees, so x/y is switched.
-        if (axis.y.label !== undefined) {
+        if (axis?.y?.label !== undefined) {
             this.yAxisLabel
                 .attr('y', -canvas.margin.left + Y_AXIS_LABEL_WIDTH)
                 .attr('x', -canvas.inner.height / 2 + this.textWidth(axis.y.label, fontStyle) / 2)
@@ -581,8 +580,8 @@ class Histogram extends Visualization {
      */
     extremesAndDeltas() {
         const dataPoints = this.data()
-        let xMin = dataPoints[0]
-        let xMax = dataPoints[0]
+        let xMin = dataPoints[0] ?? 0
+        let xMax = dataPoints[0] ?? 0
 
         dataPoints.forEach(value => {
             if (value < xMin) {
@@ -604,34 +603,15 @@ class Histogram extends Visualization {
      */
     margins() {
         const axis = this.axisSpec()
-        const noXAxis = axis.x.label === undefined
-        const noYAxis = axis.y.label === undefined
+        const noXAxis = axis?.x?.label === undefined
+        const noYAxis = axis?.y?.label === undefined
 
         const top = MARGIN / 2.0
         const right = MARGIN / 2.0
-        if (noXAxis && noYAxis) {
-            return { top, right, bottom: MARGIN, left: MARGIN }
-        } else if (noYAxis) {
-            return {
-                top,
-                right,
-                bottom: MARGIN + X_AXIS_LABEL_WIDTH,
-                left: MARGIN,
-            }
-        } else if (noXAxis) {
-            return {
-                top,
-                right,
-                bottom: MARGIN,
-                left: MARGIN + Y_AXIS_LABEL_WIDTH,
-            }
-        }
-        return {
-            top,
-            right,
-            bottom: MARGIN + X_AXIS_LABEL_WIDTH,
-            left: MARGIN + Y_AXIS_LABEL_WIDTH,
-        }
+        const bottom = MARGIN + (noXAxis ? 0 : X_AXIS_LABEL_WIDTH)
+        const left = MARGIN + (noYAxis ? 0 : Y_AXIS_LABEL_WIDTH)
+
+        return { top, right, bottom, left }
     }
 
     /**