feat: Scheduling queries from SQL Lab

[betodealmeida]

SUMMARY

This PR introduces a lightweight way of scheduling queries in SQL Lab. If the feature flag SCHEDULED_QUERIES is enabled with proper configuration, a button called "Schedule Query" will show up in SQL Lab. The button allows queries to be saved with extra metadata that allows an external scheduler to run it periodically by polling the /savedqueryviewapi/api/read endpoint.

The sample configuration can be changed or expanded to support different metadata needed, depending on the scheduler. We tested it with Apache Airflow at Lyft successfully.

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

This will generate the following payload in savedqueryviewapi/api/read:

{
  ...
  "result": [
    ...
    {
      "description": null, 
      "extra_json": "{\"schedule_info\":{\"output_table\":\"beto_output_table\",\"start_date\":\"2019-04-30T16:00:00.000Z\",\"end_date\":\"2019-05-30T16:00:00.000Z\",\"schedule_interval\":\"@daily\",\"dependencies\":[\"hive://schema.table\"]}}", 
      "id": 4, 
      "label": "Untitled Query", 
      "schema": "null", 
      "sql": "SELECT *\nFROM\nWHERE", 
      "sqlalchemy_uri": "sqlite:////Users/bdealmeida/.superset/unittests.db", 
      "user_email": "[email protected]"
    }
  ],
}

TEST PLAN

Tested end-to-end at Lyft with a Hive query.

ADDITIONAL INFORMATION

• Has associated issue:
• Changes UI
• Requires DB Migration.
• Confirm DB Migration upgrade and downgrade tested.
• Introduces new feature or API
• Removes existing feature or API

REVIEWERS

@mistercrunch @datability-io @DiggidyDave @khtruong @ArgentFalcon

[DiggidyDave]

point of curiosity: Is this URL something we have control over? (I don't see it specified anywhere, so I assume generated from a base class). It feels like a more idiomatic "RESTful" URL for a platform-y backend would be GET /api/savedqueries/

[betodealmeida]

It's automatically generated by FAB. There's a view called SavedQueryViewApi, and FAB automatically adds the .../api/* endpoints for interacting with it.

[codecov-io]

Codecov Report

Merging #7416 into lyft-release-sp8 will decrease coverage by 0.06%.
The diff coverage is 22.85%.

@@                 Coverage Diff                  @@
##           lyft-release-sp8    #7416      +/-   ##
====================================================
- Coverage                65%   64.93%   -0.07%     
====================================================
  Files                   429      430       +1     
  Lines                 21081    21114      +33     
  Branches               2337     2340       +3     
====================================================
+ Hits                  13704    13711       +7     
- Misses                 7253     7279      +26     
  Partials                124      124

Impacted Files	Coverage Δ
superset/assets/src/featureFlags.ts	88.88% <100%> (+13.88%)	⬆️
superset/views/sql_lab.py	94.64% <100%> (ø)	⬆️
...sets/src/SqlLab/components/ScheduleQueryButton.jsx	13.33% <13.33%> (ø)
...uperset/assets/src/SqlLab/components/SqlEditor.jsx	59.8% <50%> (-0.2%)	⬇️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update bfd685e...a3ed2cb. Read the comment docs.

[betodealmeida]

I'm currently writing the SIP for this.

Edit: #7425

[khtruong]

Should we provide default values for the other props?

[betodealmeida]

Ah, good point. I'll mark the other ones as required (sql, schema, dbid).

[khtruong]

👍

[vnourdin]

Hello!

I'm working at M6 (french TV group) and we're working on a job that look a bit like scheduled queries, so I was wondering if we could exchange words on how things are planned in SuperSet and see if we could help each other.

Our job is an "Aggregator" which basically run aggregation queries on daily basis. We structured it similarly you did with scheduled queries: a config file (currently hand made), an Airflow dag which create other dags based on config files, and a Scala/Spark job to run the query (and monitor number of lines to Grafana).
We currently work with Hive as db, what is the target for SuperSet?

You talked about Airflow integration, how did you managed that part at Lyft? Our need seems more specific but I think we can help.

On the config part, there is some info missing for us (full/incremental query, partitions of input tables to put sensors on it, partitions of output table), is this part "adjustable"?

Thanks for your time, I hope we'll can help and share our work!

Have a nice day,
Valentin

[betodealmeida]

Hi, @vnourdin!

On the Superset side, I tried to make this as agnostic as possible. The example config is meant for Airflow, but by changing the config you can add any metadata you need to successfully schedule a query.

At Lyft we're prototyping this with Hive, and later we'll add support to Presto. We're also planning to add an option to upload the data to our Druid database. My co-worker @ArgentFalcon wrote the Airflow pipeline, and I'll see if he can share it with you.

For our proof-of-concept we expect from the user a query filtered by ds and a table name. The table will be created if it not exists, partitioned by ds. Everyday we run the query and insert the results into the corresponding partition, incrementally building the results. For now, this is the only supported workflow.

We also support (or will support) depending on a daily partition of a Hive table or on another Airflow DAG. The way we do that is by adding dependencies (see the modal) like hive://schema.table or airflow://dag_id.

It should be easy to support different workflows. You could add a checkbox asking the user if this is a full or incremental query, for example, and the Airflow DAG would work differently depending on the value.

Hope this helps!

[mistercrunch]

Curious about how you handle different schedule intervals. Since Airflow supports only one interval per DAG, I'm assuming you generate different DAGs for each schedule_interval? Say one for @daily and one for @hourly.

I'd be cool to see the code used for the DAG-generation for reference, especially around the URI-processing into deps, though I'm guessing there are some Lyft-specific operators in the mix.

[vnourdin]

Thanks, that's helping.

We're really interested in exchanging with @ArgentFalcon about the AirFlow part! As our need urge a bit, we'll finish it the way we began, but we aim to switch later and integrate our work with SuperSet scheduled queries.

We also have a ds partition, but not only. This workflow is our main goal, but we know we will have other case.
Indeed, it seems easy to adapt the tool to our specific need.

[Mar-k]

@betodealmeida is your co-worker still planning on sharing his Airflow pipeline? We're also working on scheduling queries from Superset and using Airflow to schedule them. So I am really interested in how you guys built your Airflow pipeline :)

[mistercrunch]

I just pinged @ArgentFalcon on Slack

[ArgentFalcon]

Thanks for you patience, I'm trying to figure out how to paste the information in this DAG without getting it too bogged down by Lyft specific stuff.

[Mar-k]

Thanks @ArgentFalcon. I think the whole DAG is interesting for us. But the most interesting parts would be how you guys check the re-scheduling of different queries (do you guys dynamically create a DAG for every query returned by the API and use that scheduling data to create a DAG) and how you are planning to handle different databases in the future?

But if you figure out how to paste most of the information from this DAG here, that would be great :)

[aurelsaillant]

@ArgentFalcon thanks for your contribution, did you manage to fix the issues with your DAG?

[ArgentFalcon]

Hey, @aurelsaillant Thanks for you patience. We did manage to fix the issue. I'm currently double-checking that I can release the code snippet just so I don't get in trouble.

[aurelsaillant]

Hello @ArgentFalcon, how did your checking go? Thanks again for your contribution

[mistercrunch]

@ArgentFalcon friendly 🏓

[ktmud]

Bumped into an error today where the SQL Lab page fails to render because somehow I set FEATURE_FLAGS['SCHEDULED_QUERIES'] to True in my local configs.

It is a surprise to me that complex objects are added into "flags". I thought it's only about on/off toggles. This is quite confusing and would make it difficult to extend the FEATURE_FLAGS feature structurally (say, moving the management of it to the database or an external API).

Can we maybe introduce a new config value SCHEDULED_QUERIES_SCHEMA and keep FEATURE_FLAGS simple?