[Explore] Streamlined metric definitions for SQLA and Druid #4663

gabe-lyons · 2018-03-22T01:05:04Z

This PR streamlines the definition of AGGREGATE( column )-style metrics in the explore view. It circumvents the fab view entirely for a much improved user experience.

The metric is stored as a simple object of form {column, aggregate, label} in form data. When a query is run, the adhoc metric objects are sent along with saved metrics to the flask backend, where they are manually parsed in SQLA/Druid expressions.

Here are some quick videos demoing what interactions look like:

Adding and editing metrics:

Applying labels to metrics:

Guide to reviewers:

These streamlined metrics I refer to as adhocMetrics. Traditional metrics I refer to as savedMetrics
The bulk of the frontend logic lives in MetricsControl and AdhocMetricEditPopover. MetricsControl handles data massaging and the dropdown. AdhocMetricEditPopover handles most of the log for editing
The bulk of the flask logic is in connectors/druid/models.py and connectors/sqla/models.py. In those files I handle the parsing of adhoc metrics in sqla/druid aggregations.

Open questions:

There is custom parsing logic all over viz.py. Since this PR changes the way we post form data, we may not be able to enable this feature on all viz types. I made a fix for line charts, but I fear there may be more such issues lurking. (See viz.py change for details)
How should we deal with displaying saved metrics in the dropdown? In this PR, the default order of options in the metrics dropdown is [...columns, ...aggregates, ...saved metrics]. I prefer this ordering because there are many saved metrics that this feature will replace. However I can see it being confusing initially to users to have this dropdown default to unexpected content.
Should we prevent aggregate/column combinations based on column type? Right now I just let the users do as they please.
How should we think about deprecating saved metrics that are just a column+aggregate?

reviewers:
@michellethomas @john-bodley @williaster @graceguo-supercat @mistercrunch

(after putting up this PR I'll continue testing locally so there may be some bug fixes to come, but this is the done deal!)

mistercrunch · 2018-03-22T04:33:48Z

This is super neat and innovative. Can we make the metric (same as metrics but only allows to define one metric) control in-scope for this PR to make sure both metrics and metric are building upon the same building blocks nicely. From what I see it should be pretty trivial, maybe by adding a multiple boolean prop to MetricsControl ?

mistercrunch

Had a few minor comments on the code but overall this looks great. Surprisingly easier to use than I thought and much better than my MetricsControl attempt that I never merged. Glad I held off on it as it just didn't feel right. I think this does.

Few usability ideas:

it's not a common or expected flow, maybe provide hints as people type or when the Select element has focus? Maybe like a small yellow info bubble Pick an aggregate function or a column to aggregate?
a click to edit the metric popover when hovering a button-option ?

I wonder if it's eventually in-scope to allow free form SUM(CASE WHEN...) or APPROX_DISTINCT_COUNT... in that context. Seemed possible to extend into this if that becomes a priority somehow, but it's probably ok to send people to the old flow for those 10-20% use-cases, at least for now. Maybe by then I'll have rewritten the "Table Editor" into something much more usable.

My MetricsControl had a big of logic around handling aggregation types based on column type, but that doesn't seem super necessary for MVP here. Could be nice to have eventually.

mistercrunch · 2018-03-22T04:41:36Z

superset/assets/javascripts/explore/components/AdhocMetricEditPopover.jsx

+        </FormGroup>
+        <Button
+          disabled={!stateIsValid}
+          bsStyle={(hasUnsavedChanges || !stateIsValid) ? 'default' : 'primary'}


NIT: we've veen using bsSize="small" throughout the app

mistercrunch · 2018-03-22T04:42:09Z

superset/assets/javascripts/explore/components/AdhocMetricEditPopover.jsx

+        >
+          Save
+        </Button>
+        <Button onClick={this.props.onClose}>Close</Button>


NIT: bsSize="small"

mistercrunch · 2018-03-22T04:45:16Z

superset/assets/javascripts/explore/components/AdhocMetricOption.jsx

+          overlay={overlay}
+          rootClose
+        >
+          <Label style={{ cursor: 'pointer' }}>


I was told to avoid double curlies whenever possible since they generate new objectids at every render and will trigger PureComponents to rerender when they don't need to, just declare labelStyle in module scope and reference it here.

mistercrunch · 2018-03-22T04:50:17Z

superset/assets/javascripts/explore/components/AggregateOption.jsx

@@ -0,0 +1,20 @@
+import React from 'react';


I think it's ok to declare more than one component in a module if the component is only used once in that one module. Point being to avoid juggling with lots of small files when possible, the components can be refactored out to its own module when it grows to be used in more than one component.

I've been guilty of the other extreme (very long modules) so I'm ok with this if you prefer.

I do prefer more files in general.. I think in this instance it makes sense so all the *Option components are declared in the same way.

++ small files. it simplifies everything vastly if everything has it's own file.

xrmx · 2018-03-22T17:15:30Z

superset/connectors/sqla/models.py

+        label = metric.get('label')
+        if aggregate == 'COUNT_DISTINCT':
+            sa_metric = sa.func.COUNT(sa.distinct(column(column_name)))
+        if aggregate == 'COUNT':


You can use a bunch of elif here to save some comparisons

gabe-lyons · 2018-03-22T17:17:48Z

@mistercrunch thanks for the comments. I'll add it to the metric control as well- forgot about that one.

Also will fix python tests that are failing and add a couple more.

gabe-lyons · 2018-03-22T17:57:51Z

Also like the idea of a click to edit the metric popover when hovering a button-option- I think that would be helpful

codecov-io · 2018-03-23T00:09:12Z

Codecov Report

Merging #4663 into master will increase coverage by 0.31%.
The diff coverage is 83.76%.

@@            Coverage Diff             @@
##           master    #4663      +/-   ##
==========================================
+ Coverage   71.56%   71.87%   +0.31%     
==========================================
  Files         191      204      +13     
  Lines       14958    15323     +365     
  Branches     1100     1177      +77     
==========================================
+ Hits        10705    11014     +309     
- Misses       4250     4306      +56     
  Partials        3        3

Impacted Files	Coverage Δ
...rset/assets/javascripts/explore/stores/visTypes.js	`70.58% <ø> (ø)`	⬆️
...et/assets/javascripts/components/OnPasteSelect.jsx	`93.18% <0%> (ø)`	⬆️
...javascripts/explore/components/AggregateOption.jsx	`100% <100%> (ø)`
...s/javascripts/explore/components/controls/index.js	`100% <100%> (ø)`	⬆️
superset/assets/javascripts/dashboard/reducers.js	`50% <100%> (+0.43%)`	⬆️
...s/javascripts/explore/propTypes/adhocMetricType.js	`100% <100%> (ø)`
...s/javascripts/explore/propTypes/savedMetricType.js	`100% <100%> (ø)`
superset/assets/javascripts/explore/AdhocMetric.js	`100% <100%> (ø)`
superset/connectors/druid/models.py	`79.25% <100%> (+0.39%)`	⬆️
superset/assets/javascripts/explore/constants.js	`100% <100%> (ø)`
... and 26 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 485b0c2...7d59385. Read the comment docs.

michellethomas · 2018-03-23T00:44:22Z

superset/assets/javascripts/explore/propTypes/legacyMetricType.js

@@ -0,0 +1,6 @@
+import PropTypes from 'prop-types';


I don't see this being used anywhere, what's this for?

Oh, good catch! I actually don't use this file anywhere at all! I will delete it.

michellethomas · 2018-03-23T00:48:04Z

superset/assets/javascripts/explore/propTypes/savedMetricType.js

+export default PropTypes.shape({
+  metric_name: PropTypes.string.isRequired,
+  expression: PropTypes.string.isRequired,
+});


Just curious, why have this in a separate file instead of defined as a shape in the file it's used? This isn't being used in multiple places and it doesn't have a complicated structure (with imports). Is it a best practice?

Hmm yeah I could move it into MetricsControl- I agree this doesn't help much.

michellethomas · 2018-03-23T01:07:19Z

superset/assets/javascripts/explore/AdhocMetric.js

+  }
+
+  getDefaultLabel() {
+    return `${this.aggregate || ''}(${(this.column && this.column.column_name) || ''})`;


When would this.aggregate be null?

It wouldn't be often, but in the case that someone clears the aggregate input in the edit popover, and then goes to edit the title, the default title they would see without this logic would be like null(column_name), which seems like a bad user experience. Adding this logic would make the default title appear as (column_name)

hughhhh · 2018-03-23T07:21:19Z

superset/connectors/sqla/models.py

+        column_name = metric.get('column').get('column_name')
+        aggregate = metric.get('aggregate')
+        label = metric.get('label')
+        if aggregate == 'COUNT_DISTINCT':


nit: I would make this a dict() mapping for instance:

aggregate_dict = { 'SUM' : sa.func.COUNT(column(column_name)) ... } sa_metric = aggregate_dict.get(aggregate)

I think this is much cleaner and would allow others to add more metrics easily later.

PS: This is so dope! i can't wait till this lands so we can had some geo functions!

gabe-lyons · 2018-03-27T21:25:29Z

@mistercrunch @michellethomas thanks for the reviews! I addressed all comments.

For Max's more in depth comments:

enabled adhoc metrics on the metric controller. Required adding some additional case statements in the MetricsControl component where I assume value will be an array. Also required some small style updates.
for making it more clear that the tokens are editable, I had the AdhocMetricEditPopover appear by default when a new metric is created. I also updated the default text in the select component.

I also added another feature that HIDES any auto generated metric that is just an aggregate + column.

Please take another look when you have the chance!

mistercrunch · 2018-03-27T22:56:52Z

I'm going to pull the branch personally but would be nice to have gifs in the PR.

mistercrunch · 2018-03-27T22:57:43Z

Removed the [Request for comments] in the title as it's clear we're moving forward with this.

williaster

Left a few mostly minor comments, great addition!

williaster · 2018-03-27T22:35:10Z

superset/assets/.eslintrc

@@ -38,5 +38,8 @@
    "react/no-unescaped-entities": 0,
    "react/no-unused-prop-types": 0,
    "react/no-string-refs": 0,
+    "indent": 0,


I assume these are due to the new eslint version which flagged them?

yeah exactly

williaster · 2018-03-27T22:35:53Z

superset/assets/javascripts/components/ColumnTypeLabel.jsx

-  if (type === '' || type === 'expression') {
+  if (!type) {
+    stringIcon = '?';
+  } else if (type === '' || type === 'expression') {


type === '' is not possible given the first if statement

good catch I'll clarify the first statement

williaster · 2018-03-27T22:38:19Z

superset/assets/javascripts/explore/AdhocMetric.js

+    this.hasCustomLabel = !!(adhocMetric.hasCustomLabel && adhocMetric.label);
+    this.fromFormData = !!adhocMetric.optionName;
+
+    if (this.hasCustomLabel) {


you could consolidate with
this.label = this.hasCustomLabel ? adhocMetric.label : this.getDefaultLabel();

williaster · 2018-03-27T22:38:50Z

superset/assets/javascripts/explore/AdhocMetric.js

+    }
+
+    this.optionName = adhocMetric.optionName ||
+      `metric_${Math.random().toString(36).substring(2, 15)}_${Math.random().toString(36).substring(2, 15)}`;


is there a util func to generate ids that we could use?

this is what my internet research suggested i do 🤕

williaster · 2018-03-27T22:41:13Z

superset/assets/javascripts/explore/components/AdhocMetricEditPopover.jsx

+      <Popover
+        id="metrics-edit-popover"
+        title={popoverTitle}
+        {...rest}


I'd call this ...popoverProps instead of rest

williaster · 2018-03-27T22:51:33Z

superset/assets/javascripts/explore/components/controls/MetricsControl.jsx

+  return option.column_name && (option.column_name.indexOf(valueAfterAggregate) >= 0);
+}
+
+const autoGeneratedMetricRegex = /^(LONG|DOUBLE|FLOAT)?(SUM|AVG|MAX|MIN|COUNT)\([A-Z_][A-Z0-9_]*\)$/i;


nice to have tests for this sort of thing esp if people introduce changes in other files in the future.

williaster · 2018-03-27T22:54:00Z

superset/assets/javascripts/explore/components/controls/MetricsControl.jsx

+          optionRenderer={VirtualizedRendererWrap(option => (
+            <MetricDefinitionOption option={option} />
+          ))}
+          valueRenderer={option => (


again lots of render functions

williaster · 2018-03-27T22:54:53Z

superset/assets/javascripts/explore/main.css

@@ -134,3 +138,7 @@
 .datasource-container {
  overflow: auto;
 }
+
+.edit-adhoc-metric-save-btn {


can you use the existing m-r-5 class? (I think this is the name)

Oh cool! I didn't realize that existed

williaster · 2018-03-27T23:00:20Z

superset/assets/javascripts/components/OnPasteSelect.jsx

@@ -49,8 +49,8 @@ export default class OnPasteSelect extends React.Component {
  render() {
    const SelectComponent = this.props.selectWrap;
    const refFunc = (ref) => {
-      if (this.props.ref) {
-        this.props.ref(ref);
+      if (this.props.refFunc) {


yay for readability!

williaster · 2018-03-27T23:01:08Z

superset/assets/javascripts/explore/AdhocMetric.js

@@ -0,0 +1,37 @@
+export default class AdhocMetric {


dig the class approach!

gabe-lyons · 2018-03-28T17:01:40Z

@williaster thanks for all the comments- they should be addressed now.

Good callout on the PropTypes.object stragglers- I feel better now that they're cleaned up.

gabe-lyons · 2018-03-28T22:47:32Z

@mistercrunch @michellethomas @williaster I believe this PR is ready to merge--- any objectors?

williaster · 2018-03-29T00:20:32Z

lgtm, will let @mistercrunch pull the trigger tho 📈

mistercrunch · 2018-03-29T00:43:16Z

CasperLiu · 2018-03-29T05:36:11Z

This PR looks brilliant, great work!

Thanks!

Which release/version would this functionilty go into?
No meaning to push, just get the rough schedule in mind.

Jiaxu

mistercrunch · 2018-04-03T04:44:22Z

It should be in 0.25.0, no clear timeline but should be a few weeks at most.

CasperLiu · 2018-04-04T08:23:36Z

mistercrunch，

Thanks for your explanation!

Regards,
Jiaxu
刘佳旭

) * adding streamlined metric editing * addressing lint issues on new metrics control * enabling druid

congdanh8391 · 2018-09-12T06:32:39Z

@mistercrunch @michellethomas thanks for the reviews! I addressed all comments.

For Max's more in depth comments:

enabled adhoc metrics on the metric controller. Required adding some additional case statements in the MetricsControl component where I assume value will be an array. Also required some small style updates.

for making it more clear that the tokens are editable, I had the AdhocMetricEditPopover appear by default when a new metric is created. I also updated the default text in the select component.

I also added another feature that HIDES any auto generated metric that is just an aggregate + column.

Please take another look when you have the chance!

Hey,
Adhoc metric is good but in some cases, it's a problem. In tableview, if you define an adhoc metric, it has no format as similar as auto general metric. Please don not hide auto general metric anymore!

Thank

) * adding streamlined metric editing * addressing lint issues on new metrics control * enabling druid

gabe-lyons force-pushed the gabe_adding_metric_definition_modal branch from 0b0dc07 to b6f2246 Compare March 22, 2018 01:08

mistercrunch reviewed Mar 22, 2018

View reviewed changes

xrmx reviewed Mar 22, 2018

View reviewed changes

mistercrunch mentioned this pull request Mar 22, 2018

[WiP][explore] metrics as popovers #3202

Closed

gabe-lyons force-pushed the gabe_adding_metric_definition_modal branch from b6f2246 to 934312c Compare March 22, 2018 23:54

michellethomas reviewed Mar 23, 2018

View reviewed changes

hughhhh reviewed Mar 23, 2018

View reviewed changes

gabe-lyons force-pushed the gabe_adding_metric_definition_modal branch 3 times, most recently from 1aad37e to 0dc1954 Compare March 27, 2018 21:22

mistercrunch changed the title ~~[Request For Comments][Explore] Streamlined metric definitions for SQLA and Druid~~ [Explore] Streamlined metric definitions for SQLA and Druid Mar 27, 2018

williaster reviewed Mar 27, 2018

View reviewed changes

gabe-lyons force-pushed the gabe_adding_metric_definition_modal branch from 0dc1954 to e316bda Compare March 28, 2018 16:54

GabeLoins added 2 commits March 28, 2018 09:56

adding streamlined metric editing

fa610eb

addressing lint issues on new metrics control

7419a94

gabe-lyons force-pushed the gabe_adding_metric_definition_modal branch from e316bda to 3b8c519 Compare March 28, 2018 16:56

gabe-lyons force-pushed the gabe_adding_metric_definition_modal branch from 3b8c519 to 7d59385 Compare March 28, 2018 19:01

enabling druid

7d59385

mistercrunch mentioned this pull request Mar 28, 2018

Automatically assign "Verbose Name" for a table's fields #4701

Closed

3 tasks

mistercrunch merged commit 68dec24 into apache:master Mar 29, 2018

This was referenced Mar 29, 2018

[Explore] including auto generated avg metrics in druid #4718

Merged

[Explore] Adding custom expressions to adhoc metrics #4736

Merged

mistercrunch mentioned this pull request Apr 4, 2018

delete sum and avg field but not deleted. #4757

Closed

gabe-lyons mentioned this pull request Apr 30, 2018

[Explore] Adding Adhoc Filters #4909

Merged

gabe-lyons mentioned this pull request May 21, 2018

Can't select the table's other metrics when explore the chart expect the count(*) metric #5045

Closed

3 tasks

michellethomas pushed a commit to michellethomas/panoramix that referenced this pull request May 24, 2018

[Explore] Streamlined metric definitions for SQLA and Druid (apache#4663

df5720f

) * adding streamlined metric editing * addressing lint issues on new metrics control * enabling druid

wenchma pushed a commit to wenchma/incubator-superset that referenced this pull request Nov 16, 2018

[Explore] Streamlined metric definitions for SQLA and Druid (apache#4663

612634f

) * adding streamlined metric editing * addressing lint issues on new metrics control * enabling druid

mistercrunch added 🏷️ bot A label used by `supersetbot` to keep track of which PR where auto-tagged with release labels 🚢 0.25.0 labels Feb 27, 2024

[Explore] Streamlined metric definitions for SQLA and Druid #4663

[Explore] Streamlined metric definitions for SQLA and Druid #4663

Conversation

gabe-lyons commented Mar 22, 2018

mistercrunch commented Mar 22, 2018

mistercrunch left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gabe-lyons commented Mar 22, 2018

gabe-lyons commented Mar 22, 2018

codecov-io commented Mar 23, 2018 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hughhhh Mar 23, 2018 • edited Loading

Choose a reason for hiding this comment

gabe-lyons commented Mar 27, 2018 • edited Loading

mistercrunch commented Mar 27, 2018

mistercrunch commented Mar 27, 2018

williaster left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gabe-lyons commented Mar 28, 2018

gabe-lyons commented Mar 28, 2018

williaster commented Mar 29, 2018

mistercrunch commented Mar 29, 2018

CasperLiu commented Mar 29, 2018

mistercrunch commented Apr 3, 2018

CasperLiu commented Apr 4, 2018

congdanh8391 commented Sep 12, 2018

codecov-io commented Mar 23, 2018 •

edited

Loading

hughhhh Mar 23, 2018 •

edited

Loading

gabe-lyons commented Mar 27, 2018 •

edited

Loading