[Qubole] - Adding support to process Quantum query types. #4066

sandeepV2 · 2019-08-13T13:29:16Z

Quantum is a server-less interactive service that offers
direct SQL access to user's data lake. Changes are made
to accept quantum query type from user which makes
Cluster Label as optional.

issue: #4065

Quantum is a serverless interactive service that offers direct SQL access to user's data lake. Changes are made to accept `quantum` query type from user which makes `Cluster Label` as optional.

sandeepV2 · 2019-08-14T09:55:31Z

Merging qubole_quantum to master

sandeepV2 · 2019-08-14T10:09:43Z

@arikfr Can you please review this.

arikfr

Thanks, @sandeepV2! Quantum sounds interesting and supporting it in Redash will be great.

See comments.

redash/query_runner/qubole.py

-Dictionary safe access to connection parmeters

sandeepV2 · 2019-08-20T07:29:31Z

I have updated requirements_all_ds.txt with recent release version of SDK (v1.13.0). Checks with failure (build: importlib-metadata>=0.18 and percy) does not seem to relate with my changes.

@arikfr Can we merge it?

sandeepV2 · 2019-08-26T11:39:21Z

@arikfr can i get write access to re-run failed checks? and any hints to get over build issue?

Thanks in advance,
Sandeep

arikfr

The importlib-metadata is something we fixed. Did you rebase with latest master?

redash/query_runner/qubole.py

sandeepV2 · 2019-08-28T06:55:08Z

The importlib-metadata is something we fixed. Did you rebase with latest master?

Yes i did, It is up to date with master.

$git rebase master
Current branch qubole_quantum is up to date.

arikfr · 2019-08-28T12:49:30Z

Did you update your local master branch with the remote one? Otherwise it's probably the same master you branched out from. Looking at your tree I can see that requirements_bundle.txt is not the same as the current master, which explains this issue.

While it's probably safe to merge regardless, it's better to be on the safe side.

sandeepV2 · 2019-08-29T06:16:48Z

Did you update your local master branch with the remote one? Otherwise it's probably the same master you branched out from. Looking at your tree I can see that requirements_bundle.txt is not the same as the current master, which explains this issue.

While it's probably safe to merge regardless, it's better to be on the safe side.

Done ! When can we expect it to be merged to master?

arikfr · 2019-08-29T16:22:59Z

Merged. Thanks :)

* Migrate Parameters component to React (#4006) * Start Parameters Migration * Add dirtyCount * Use workaround with setState * Apply Changes * Add EditSettingsDialog * Add Cmd/Ctrl + Enter behavior * Remove isApplying * Delete Angular version of parameters * Update tests * Remove angular stuff * Update jest * Drag placeholder * Update events * Use old button styling and move css * Reviewing code * Add parameter rearrange test * Add Parameter Settings title change test * Update Parameter Settings button styling * Move parameter url logic back to Parameters * Disable url update when query is new * Styling changes (#4019) * Ran's title width styling * Update drag test * Improve sizing for Number inputs Co-Authored-By: Ran Byron <[email protected]> * Fix issue with dragged parameter wrapping Co-Authored-By: Ran Byron <[email protected]> * Don't reevaluate dirtyParamCount * Allow multiple values :) * Fix parameter alignments * Fix Select width on search * Update client/app/components/Parameters.less Co-Authored-By: Ran Byron <[email protected]> * Humanize param.name * Make sure angular updates Execute disabled status * Add more flake8 tests and fail build if any test fails (#4055) * Add more flake8 tests and fail build if any test fails Run all flake8 E9xx + F63x + F7xx + F82x tests. * long = long in Python 2 * Fix: MySQL connections without SSL are failing (#4090) * Move connection logic into a single method & make sure not to pass ssl value if not used. * Remove wildcard import and format file. * Removed redash-newstyle.less (#4017) * Make sure we always pass a list to _get_column_lists (#4095) (some data sources might return None as the columns list) * [Data Sources] Add: Azure Data Explorer (Kusto) query runner (#4091) * [Data Sources] Add: Azure Data Explorer (Kusto) query runner * CodeClimate fixes * Remove TODO * Fixed configuration properties names for Azure Kusto * Azure Kusto: get_schema in one query * azure-kusto-data update to 0.0.32 * Add Kusto to the default query runners list * [Qubole] - Adding support to process Quantum query types. (#4066) * [Qubole] - Adding support to process Quantum query types. Quantum is a serverless interactive service that offers direct SQL access to user's data lake. Changes are made to accept `quantum` query type from user which makes `Cluster Label` as optional. * -Making quantum as defult query. -Dictionary safe access to connection parmeters * keeping pep8 standards * Maintainig pep8 std * Use latest version of qds-sdk * Use qds-sdk v1.13.0 * Use qds-sdk v1.12.0 * Use qds-sdk v1.13.0 * Updating SDK with verified version * hive as default query type * qds-sdk : Locking most recent release version * qds-sdk : Locking recent release version * falling back to original version of qds-sdk * Dashboard: when updating parameters, run only relevant queries (#3804) * refresh only affected queries in dashboard when parameters are changed * rename pendingParameters to updatedParameters * select which widgets to update according to their mapping as a dashboard-level parameter * use lodash's include * Migrate with SQL statements. (#4105) * Update badge in README.md to link to CircleCI (#4104) * Update README.md * Update README.md * Update README.md Co-Authored-By: Ran Byron <[email protected]> * Update README.md * Fix widget bottom element alignment (#4110) * Fix: allow users with view only acces to use the queries in Query Results (#4112) * Fix: allow users with view only acces to access the queries * Add tests * Update error message * Update error message. Take 2 * Fix Dropdown parameter options appearing behind Dialog (#4109) * Add ability to use Ant's Table loading property when using ItemsTable (#4117) * Display data source icon in query editor (#4119) * Move annotation logic into Query Runner (#4113) * Code formatting * Move annotation logic into query runner, so it can be overriden in the query runner. * Add mixin to __all__ * Switch to flag instead of mixin * Feature (Redshift): option to set query group for adhoc/scheduled queries (#4114) * Add scheduled status to query job metadata. * Add: option to set query group for adhoc/scheduled Redshift queries * Scheduled might not be set for already enqueued queries. * Fix number param value normlization (#4116) * Use ng-src for data source icons (#4123) * hive_ds: show a user friendly error message when possible (#4121) * Update botocore, to get pass pip warning (#4122) * Widget table scroll-x visible (#4101) * Table viz horizontal scroll made visible * Added tests * Fixed snapshot pre-condition * Perhaps this would trigger percy * Upgrade Sentry-SDK and enable additional integratoins (#4127) * Update sentry-sdk version * Add additional Sentry integrations * Migrate Counter visualization to React (#4106) * Migrate Counter to React: Renderer * Migrate Counter to React: Editor * Cleanup * Review and fix rows indexing algorithm * Counter not properly scaled in editor * Fix wrong label for/input id pair * Tests * Tests * Fix vendor prefixes * Remove unnecessary useEffect dependencies * Update tests * Fix Percy snapshot names * Sync botocor eversions across requirements files. (#4128) * Decrease size of widget pagination (#4120) * Added tests * Perhaps this would trigger percy * Decrease size of widget pagination * Removed unused attr * Updated tests * Allow the user to decide how to handle null values in charts (#4071) * #2629 Refactor Chart visualization, add option for handling NULL values (keep/convert to 0.0) * Handle null values in line/area stacking code; some cleanup * Handle edge case: line/area stacking when last value of one of series is missing * Mjnor update to line/area stacking code * Fix line/area normalize to percents feature * Unit tests * Refine tests; add tests for prepareLayout function * Tests for prepareData (heatmap) function * Tests for prepareData (pie) function * Tests for prepareData (bar, line, area) function * Tests for prepareData (scatter, bubble) function * Tests for prepareData (box) function * Remove unused file * Alerts: Add more condition comparison options (#4134) * #4132 Add more condition comparison options * Add arguments to fallback lambda * Remove duplicate messages method (#4131) * Migrate Chart visualization to React Part 1: Renderer (#4130) * Migrate Chart visualization: Renderer * Refine PlotlyChart component; move stylesheets to visualization's folder * Migrate Custom JS Chart to React * Cleanup * Add jsconfig settings with '@' webpack alias (#4135) * Counter Editor: move components to own files (#4138) * Allow users to share aggregated usage information with us (#4108) * Initial commit of BeaconConsent component * Add comment about being able to change setting * Use <Text> correctly * Final version of consent screen * Show beacon consent message on homepage only if it wasn't enabled already. * Add consent setting to organization settings screen. * Add support for custom message in OrgSetting.save. * Implmenet consent saving. * If consent given, send extra data * Add HelpTrigger * Make CodeClimate happy * Wrap everything with DynamicComponent * CHANGELOG for V8-beta. (#4057) * CHANGELOG for V8-beta. * Update CHANGELOG.md * Update CHANGELOG.md * Update CHANGELOG.md * Bug fix: Query view doesn't sync parameters when selecting and deleting (#4146) * Query Snippets: Use onClick instead of link for 'Click here' option (#4144) * Snippets: Don't change url when not needed * Revert "Snippets: Don't change url when not needed" This reverts commit 2f346f3. * Query Snippets: use onClick instead of link * Color picker component (#4136) * Widget filters overlapped by visualization (#4137) * Fix: widget filters overlapped by visualization * Fix tests * Fix tests * CHANGELOG for v8.0.0-beta.2 (#4145) * Stop building tarballs. * Update version reference. * CHANGELOG for 8.0.0-beta.2

* [Qubole] - Adding support to process Quantum query types. Quantum is a serverless interactive service that offers direct SQL access to user's data lake. Changes are made to accept `quantum` query type from user which makes `Cluster Label` as optional. * -Making quantum as defult query. -Dictionary safe access to connection parmeters * keeping pep8 standards * Maintainig pep8 std * Use latest version of qds-sdk * Use qds-sdk v1.13.0 * Use qds-sdk v1.12.0 * Use qds-sdk v1.13.0 * Updating SDK with verified version * hive as default query type * qds-sdk : Locking most recent release version * qds-sdk : Locking recent release version * falling back to original version of qds-sdk

…4066) * [Qubole] - Adding support to process Quantum query types. Quantum is a serverless interactive service that offers direct SQL access to user's data lake. Changes are made to accept `quantum` query type from user which makes `Cluster Label` as optional. * -Making quantum as defult query. -Dictionary safe access to connection parmeters * keeping pep8 standards * Maintainig pep8 std * Use latest version of qds-sdk * Use qds-sdk v1.13.0 * Use qds-sdk v1.12.0 * Use qds-sdk v1.13.0 * Updating SDK with verified version * hive as default query type * qds-sdk : Locking most recent release version * qds-sdk : Locking recent release version * falling back to original version of qds-sdk

[Qubole] - Adding support to process Quantum query types.

9cabdd1

Quantum is a serverless interactive service that offers direct SQL access to user's data lake. Changes are made to accept `quantum` query type from user which makes `Cluster Label` as optional.

sandeepV2 closed this Aug 14, 2019

sandeepV2 reopened this Aug 14, 2019

arikfr requested changes Aug 14, 2019

View reviewed changes

redash/query_runner/qubole.py Show resolved Hide resolved

redash/query_runner/qubole.py Outdated Show resolved Hide resolved

weekly-digest bot mentioned this pull request Aug 19, 2019

Weekly Digest (12 August, 2019 - 19 August, 2019) #4082

Closed

sandeepV2 added 7 commits August 19, 2019 13:25

-Making quantum as defult query.

6e6af57

-Dictionary safe access to connection parmeters

keeping pep8 standards

7a38a45

Maintainig pep8 std

2789a2b

Use latest version of qds-sdk

e5d0015

Use qds-sdk v1.13.0

42b704c

Use qds-sdk v1.12.0

ad98cb4

Use qds-sdk v1.13.0

44058cf

Updating SDK with verified version

b5dce0f

arikfr reviewed Aug 27, 2019

View reviewed changes

redash/query_runner/qubole.py Outdated Show resolved Hide resolved

redash/query_runner/qubole.py Outdated Show resolved Hide resolved

hive as default query type

2144c02

sandeepV2 added 4 commits August 29, 2019 05:11

Merge branch 'master' into qubole_quantum

c7333b4

qds-sdk : Locking most recent release version

8f552b4

qds-sdk : Locking recent release version

964c089

falling back to original version of qds-sdk

e68aa49

arikfr merged commit 4430544 into getredash:master Aug 29, 2019

weekly-digest bot mentioned this pull request Sep 2, 2019

Weekly Digest (26 August, 2019 - 2 September, 2019) #4118

Closed

arikfr mentioned this pull request Sep 28, 2019

[Qubole] Add support to process Quantum query type. #4065

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Qubole] - Adding support to process Quantum query types. #4066

[Qubole] - Adding support to process Quantum query types. #4066

sandeepV2 commented Aug 13, 2019

sandeepV2 commented Aug 14, 2019

sandeepV2 commented Aug 14, 2019

arikfr left a comment

sandeepV2 commented Aug 20, 2019

sandeepV2 commented Aug 26, 2019

arikfr left a comment

sandeepV2 commented Aug 28, 2019

arikfr commented Aug 28, 2019

sandeepV2 commented Aug 29, 2019

arikfr commented Aug 29, 2019

[Qubole] - Adding support to process Quantum query types. #4066

[Qubole] - Adding support to process Quantum query types. #4066

Conversation

sandeepV2 commented Aug 13, 2019

sandeepV2 commented Aug 14, 2019

sandeepV2 commented Aug 14, 2019

arikfr left a comment

Choose a reason for hiding this comment

sandeepV2 commented Aug 20, 2019

sandeepV2 commented Aug 26, 2019

arikfr left a comment

Choose a reason for hiding this comment

sandeepV2 commented Aug 28, 2019

arikfr commented Aug 28, 2019

sandeepV2 commented Aug 29, 2019

arikfr commented Aug 29, 2019