feat: Add post processing to QueryObject

[villebro]

CATEGORY

Choose one

• Bug Fix
• Enhancement (new features, refinement)
• Refactor
• Add tests
• Build / Development Environment
• Documentation

SUMMARY

Currently the /api/v1/query endpoint doesn't support post-SQL data processing. This functionality is necessary for decoupling the backend from the frontend, as many of the data operations necessary for advanced visualizations often require data processing either not readily available in SQL, the JavaScript ecosystem, or are unfeasible due to network/computational expense.

This PR adds post-query data processing functionality to Superset necessary for deprecating viz.py, namely

• aggregate (same as SQL GROUP BY)
• pivot (grouping by into column values and aggregation by cell value)
• sort (same as ORDER BY)
• rolling (e.g. moving sums, averages)

This is done by leveraging functionality readily available in Pandas and Numpy. To leverage this functionality, post processing operations can be defined as part of the queries attribute in the QueryContext object. Below is an example from the unit tests, where the mean and 1st quantile are computed on an already aggregated query, which is lastly sorted by the 1st quantile value (descending) and state (ascending by default):

{
    "queries": [
        {
            "granularity": "ds",
            "groupby": ["name", "state"],
            "metrics": [{"label": "sum__num"}],
            "filters": [],
            "row_limit": 100,
            "post_processing": [
                {
                    "operation": "aggregate",
                    "options": {
                        "groupby": ["state"],
                        "aggregates": {
                            "q1": {
                                "operator": "percentile",
                                "column": "sum__num",
                                "options": {"q": 25},
                            },
                            "median": {
                                "operator": "median",
                                "column": "sum__num",
                            },
                        },
                    },
                },
                {
                    "operation": "sort",
                    "options": {
                        "by": ["q1", "state"],
                        "ascending": {"q1": False},
                    },
                },
            ],
        }
    ],
}

This feature should be seen as experimental at this stage. Furthermore, additional unit tests and documentation will be added later, probably in the form of OpenAPI specs.

TEST PLAN

CI + local tests

ADDITIONAL INFORMATION

• Has associated issue: [SIP-38] Visualization plugin refactoring #9187
• Changes UI
• Requires DB Migration.
• Confirm DB Migration upgrade and downgrade tested.
• Introduces new feature or API
• Removes existing feature or API

REVIEWERS

@rusackas @suddjian @kristw @john-bodley @etr2460

[villebro]

This feature requires functionality that was only recently introduced in Pandas 1.0.

[john-bodley]

cc @ktmud

[ktmud]

It would be more useful if

1. Rolling results are added as a new column to the original df
2. Users can select which columns to roll
3. There is the ability to compute rolling diffs

How about an API like this:

[
    {
        "operation": "rolling",
        "options": {
            "columns": ["abc", "cde"],
            # 7 day moving average
            "output_columns": ["abc MA7", "cde MA7"],
            "rolling_type": "mean",
            "win_type": None,
            "window": 7,
            "min_periods": 7,
        },
    },
    {
        "operation": "rolling",
        "options": {
            "columns": ["abc", "cde"],
            # MoM growth computed with 30 MA
            "output_columns": ["abc MoM", "cde MoM"],
            "rolling_type": "mean",
            "win_type": None,
            "window": 30,
            "min_periods": 30,
            # apply `df.diff(...)` if `diff_periods is specified`
            "diff_periods": 30,
        },
    },
    {
        "operation": "rolling",
        "options": {
            "columns": ["abc", "cde"],
            # YoY growth in percentages
            "output_columns": ["abc YoY (%)", "cde YoY (%)"],
            "rolling_type": "mean",
            "win_type": None,
            "window": 365,
            "min_periods": 365,
            # apply `df.pct_change(...)`
            "pct_change_periods": 365,
        },
    },
    # `diff_periods` and `pct_change_periods` are shortcuts
    # to applying additional `diff` and `pct_change` operators
    # on the output columns.
    {
        "operation": "diff",
        "options": {
            "columns": ["abc", "cde"],
            "output_columns": ["abc MoM", "cde MoM"],
            "periods": 7,
        },
    },
]

[villebro]

Thanks @ktmud , these are great suggestions. I think there's room for adding lots of functionality here, but I would prefer to keep the initial PR as compact as possible. How about I make sure all operations support adding new columns (where applicable) and which columns to roll, possibly even introducing the rolling diff operator, but we leave the new diff operation to a later PR?

[ktmud]

Sgtm

[ktmud]

This is great work, btw! Really looking forward to use these API in the charts.

[villebro]

@ktmud I did some cleaning up, and added two new operations, namely select (can be used to drop unneeded columns, rename and reorder columns) and diff. I think this is feature complete for basic operations, let me know what you think.

[dpgaspar]

Great work! Left a few comments

[ktmud]

Thanks for the updates and awesome work! Added a couple of comments.

[codecov-io]

Codecov Report

Merging #9427 into master will not change coverage by %.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #9427   +/-   ##
=======================================
  Coverage   58.76%   58.76%           
=======================================
  Files         385      385           
  Lines       12240    12240           
  Branches     3022     3022           
=======================================
  Hits         7193     7193           
  Misses       4863     4863           
  Partials      184      184           

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5ec0192...644ab9a. Read the comment docs.

[dpgaspar]

LGTM, just a final lint

[villebro]

@ktmud this should be ready to go, any comments before we hit merge?

[ktmud]

all look good! Thanks again for the work.

[ktmud]

@villebro can we change this to rename first, then select? More of the time, rename maps existing columns to a more readable name, and it'd be more convenient to select from the more desirable output column name.