Create Data sources plugin with Manage Datasources Flow #1035

derek-ho · 2023-09-20T21:00:30Z

Description

[Describe what this change achieves]

Issues Resolved

[List any issues this PR will resolve]

Check List

New functionality includes testing.
- All tests pass, including unit test, integration test and doctest
New functionality has been documented.
- New functionality has javadoc added
- New functionality has user manual doc added
Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

* fix name change bug and modify test to test behavior Signed-off-by: Derek Ho <[email protected]> * get rid of lint Signed-off-by: Derek Ho <[email protected]> * test for flyout Signed-off-by: Derek Ho <[email protected]> * flyout to medium size Signed-off-by: Derek Ho <[email protected]> * make accelerate extensible Signed-off-by: Derek Ho <[email protected]> * get datasources and hook up to pplservice Signed-off-by: Derek Ho <[email protected]> * get flint working Signed-off-by: Derek Ho <[email protected]> * add datasource page with steps and buttons on bottom bar Signed-off-by: Derek Ho <[email protected]> * datasources as a new plugin and mostly working Signed-off-by: Derek Ho <[email protected]> * hook up manage to show datasources call Signed-off-by: Derek Ho <[email protected]> * update two tables with descriptions Signed-off-by: Derek Ho <[email protected]> * make some updates to the page Signed-off-by: Derek Ho <[email protected]> * cleanup unused files for data connections Signed-off-by: Derek Ho <[email protected]> * cleanup and add overview panel columns Signed-off-by: Derek Ho <[email protected]> * render tabs Signed-off-by: Derek Ho <[email protected]> * add unit tests Signed-off-by: Derek Ho <[email protected]> * update data test subj and snapshot Signed-off-by: Derek Ho <[email protected]> * Add datasources to management overview Signed-off-by: Derek Ho <[email protected]> * remove spark logo and update snapshot Signed-off-by: Derek Ho <[email protected]> * refactor routes out Signed-off-by: Derek Ho <[email protected]> * separate out the roles Signed-off-by: Derek Ho <[email protected]> * bump version back to 3.0 Signed-off-by: Derek Ho <[email protected]> --------- Signed-off-by: Derek Ho <[email protected]>

* add acceleration management UI skeleton Signed-off-by: Shenoy Pratik <[email protected]> * Create new documentation link for acc Signed-off-by: Shenoy Pratik <[email protected]> * fix typos and minor bugs Signed-off-by: Shenoy Pratik <[email protected]> * update snapshot Signed-off-by: Shenoy Pratik <[email protected]> * update window location to hash Signed-off-by: Shenoy Pratik <[email protected]> * remove unused headers Signed-off-by: Shenoy Pratik <[email protected]> --------- Signed-off-by: Shenoy Pratik <[email protected]>

* rename data sources to data connections Signed-off-by: Derek Ho <[email protected]> * final cleanup Signed-off-by: Derek Ho <[email protected]> * update acceleration breadcrumb Signed-off-by: Derek Ho <[email protected]> * fix API call for data connection page Signed-off-by: Derek Ho <[email protected]> * fix integ test and data test subj and snapshot Signed-off-by: Derek Ho <[email protected]> --------- Signed-off-by: Derek Ho <[email protected]>

…#1008) * add fallback ui for manage and view datasources Signed-off-by: Derek Ho <[email protected]> * always show datasources via pplservice Signed-off-by: Derek Ho <[email protected]> --------- Signed-off-by: Derek Ho <[email protected]>

* basic rendering for the access control tab Signed-off-by: Derek Ho <[email protected]> * hook up basic radio groups and euicombo boxes for query and acceleration permissions Signed-off-by: Derek Ho <[email protected]> * refactor and clean up unuseed inports Signed-off-by: Derek Ho <[email protected]> * remove unused import Signed-off-by: Derek Ho <[email protected]> * fix import and snapshot Signed-off-by: Derek Ho <[email protected]> * fix test Signed-off-by: Derek Ho <[email protected]> * Address PR comments Signed-off-by: Derek Ho <[email protected]> * Address PR comments Signed-off-by: Derek Ho <[email protected]> * Remove unused files and variables Signed-off-by: Derek Ho <[email protected]> --------- Signed-off-by: Derek Ho <[email protected]>

* Address previous PR comments and implement rudimentary delete Signed-off-by: Derek Ho <[email protected]> * Implement modal and instant delete showing up in list Signed-off-by: Derek Ho <[email protected]> * Refactor save or cancel to a shared component, implement hard coded datasource configurations tab Signed-off-by: Derek Ho <[email protected]> * Update test with mock role data Signed-off-by: Derek Ho <[email protected]> * Add most functionality of edit connectiondetails Signed-off-by: Derek Ho <[email protected]> * Address PR comments Signed-off-by: Derek Ho <[email protected]> --------- Signed-off-by: Derek Ho <[email protected]>

codecov · 2023-09-20T21:24:17Z

Codecov Report

Merging #1035 (ad153ed) into main (941eb60) will increase coverage by 1.76%.
Report is 16 commits behind head on main.
The diff coverage is n/a.

@@            Coverage Diff             @@
##             main    #1035      +/-   ##
==========================================
+ Coverage   43.71%   45.47%   +1.76%     
==========================================
  Files         313      329      +16     
  Lines       18714    19968    +1254     
  Branches     4562     4863     +301     
==========================================
+ Hits         8180     9080     +900     
+ Misses      10492    10312     -180     
- Partials       42      576     +534

Flag	Coverage Δ
dashboards-observability	`45.46% <ø> (+1.75%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

see 109 files with indirect coverage changes

Signed-off-by: Shenoy Pratik <[email protected]>

ps48

Looks good. Minor: It would be better if we have couple of folders inside data_connections/components -> configuration_details and connection_details.

derek-ho · 2023-09-22T20:22:01Z

Looks good. Minor: It would be better if we have couple of folders inside data_connections/components -> configuration_details and connection_details.

Yes good idea - I am addressing in this follow up PR: #1046, it was getting quite confusing so folders should help

* Manage datasources (#967) * fix name change bug and modify test to test behavior Signed-off-by: Derek Ho <[email protected]> * get rid of lint Signed-off-by: Derek Ho <[email protected]> * test for flyout Signed-off-by: Derek Ho <[email protected]> * flyout to medium size Signed-off-by: Derek Ho <[email protected]> * make accelerate extensible Signed-off-by: Derek Ho <[email protected]> * get datasources and hook up to pplservice Signed-off-by: Derek Ho <[email protected]> * get flint working Signed-off-by: Derek Ho <[email protected]> * add datasource page with steps and buttons on bottom bar Signed-off-by: Derek Ho <[email protected]> * datasources as a new plugin and mostly working Signed-off-by: Derek Ho <[email protected]> * hook up manage to show datasources call Signed-off-by: Derek Ho <[email protected]> * update two tables with descriptions Signed-off-by: Derek Ho <[email protected]> * make some updates to the page Signed-off-by: Derek Ho <[email protected]> * cleanup unused files for data connections Signed-off-by: Derek Ho <[email protected]> * cleanup and add overview panel columns Signed-off-by: Derek Ho <[email protected]> * render tabs Signed-off-by: Derek Ho <[email protected]> * add unit tests Signed-off-by: Derek Ho <[email protected]> * update data test subj and snapshot Signed-off-by: Derek Ho <[email protected]> * Add datasources to management overview Signed-off-by: Derek Ho <[email protected]> * remove spark logo and update snapshot Signed-off-by: Derek Ho <[email protected]> * refactor routes out Signed-off-by: Derek Ho <[email protected]> * separate out the roles Signed-off-by: Derek Ho <[email protected]> * bump version back to 3.0 Signed-off-by: Derek Ho <[email protected]> --------- Signed-off-by: Derek Ho <[email protected]> * Add acceleration management UI (#989) * add acceleration management UI skeleton Signed-off-by: Shenoy Pratik <[email protected]> * Create new documentation link for acc Signed-off-by: Shenoy Pratik <[email protected]> * fix typos and minor bugs Signed-off-by: Shenoy Pratik <[email protected]> * update snapshot Signed-off-by: Shenoy Pratik <[email protected]> * update window location to hash Signed-off-by: Shenoy Pratik <[email protected]> * remove unused headers Signed-off-by: Shenoy Pratik <[email protected]> --------- Signed-off-by: Shenoy Pratik <[email protected]> * Rename data sources to data connections (#1004) * rename data sources to data connections Signed-off-by: Derek Ho <[email protected]> * final cleanup Signed-off-by: Derek Ho <[email protected]> * update acceleration breadcrumb Signed-off-by: Derek Ho <[email protected]> * fix API call for data connection page Signed-off-by: Derek Ho <[email protected]> * fix integ test and data test subj and snapshot Signed-off-by: Derek Ho <[email protected]> --------- Signed-off-by: Derek Ho <[email protected]> * Add fallback to show if user does not have datasource API permissions (#1008) * add fallback ui for manage and view datasources Signed-off-by: Derek Ho <[email protected]> * always show datasources via pplservice Signed-off-by: Derek Ho <[email protected]> --------- Signed-off-by: Derek Ho <[email protected]> * Add access control tab content (#992) * basic rendering for the access control tab Signed-off-by: Derek Ho <[email protected]> * hook up basic radio groups and euicombo boxes for query and acceleration permissions Signed-off-by: Derek Ho <[email protected]> * refactor and clean up unuseed inports Signed-off-by: Derek Ho <[email protected]> * remove unused import Signed-off-by: Derek Ho <[email protected]> * fix import and snapshot Signed-off-by: Derek Ho <[email protected]> * fix test Signed-off-by: Derek Ho <[email protected]> * Address PR comments Signed-off-by: Derek Ho <[email protected]> * Address PR comments Signed-off-by: Derek Ho <[email protected]> * Remove unused files and variables Signed-off-by: Derek Ho <[email protected]> --------- Signed-off-by: Derek Ho <[email protected]> * Delete datasource and Connection Configuration Tab (#1024) * Address previous PR comments and implement rudimentary delete Signed-off-by: Derek Ho <[email protected]> * Implement modal and instant delete showing up in list Signed-off-by: Derek Ho <[email protected]> * Refactor save or cancel to a shared component, implement hard coded datasource configurations tab Signed-off-by: Derek Ho <[email protected]> * Update test with mock role data Signed-off-by: Derek Ho <[email protected]> * Add most functionality of edit connectiondetails Signed-off-by: Derek Ho <[email protected]> * Address PR comments Signed-off-by: Derek Ho <[email protected]> --------- Signed-off-by: Derek Ho <[email protected]> * remove acceleration components Signed-off-by: Shenoy Pratik <[email protected]> --------- Signed-off-by: Derek Ho <[email protected]> Signed-off-by: Shenoy Pratik <[email protected]> Co-authored-by: Shenoy Pratik <[email protected]>

…roject#1035) * Manage datasources (opensearch-project#967) * fix name change bug and modify test to test behavior Signed-off-by: Derek Ho <[email protected]> * get rid of lint Signed-off-by: Derek Ho <[email protected]> * test for flyout Signed-off-by: Derek Ho <[email protected]> * flyout to medium size Signed-off-by: Derek Ho <[email protected]> * make accelerate extensible Signed-off-by: Derek Ho <[email protected]> * get datasources and hook up to pplservice Signed-off-by: Derek Ho <[email protected]> * get flint working Signed-off-by: Derek Ho <[email protected]> * add datasource page with steps and buttons on bottom bar Signed-off-by: Derek Ho <[email protected]> * datasources as a new plugin and mostly working Signed-off-by: Derek Ho <[email protected]> * hook up manage to show datasources call Signed-off-by: Derek Ho <[email protected]> * update two tables with descriptions Signed-off-by: Derek Ho <[email protected]> * make some updates to the page Signed-off-by: Derek Ho <[email protected]> * cleanup unused files for data connections Signed-off-by: Derek Ho <[email protected]> * cleanup and add overview panel columns Signed-off-by: Derek Ho <[email protected]> * render tabs Signed-off-by: Derek Ho <[email protected]> * add unit tests Signed-off-by: Derek Ho <[email protected]> * update data test subj and snapshot Signed-off-by: Derek Ho <[email protected]> * Add datasources to management overview Signed-off-by: Derek Ho <[email protected]> * remove spark logo and update snapshot Signed-off-by: Derek Ho <[email protected]> * refactor routes out Signed-off-by: Derek Ho <[email protected]> * separate out the roles Signed-off-by: Derek Ho <[email protected]> * bump version back to 3.0 Signed-off-by: Derek Ho <[email protected]> --------- Signed-off-by: Derek Ho <[email protected]> * Add acceleration management UI (opensearch-project#989) * add acceleration management UI skeleton Signed-off-by: Shenoy Pratik <[email protected]> * Create new documentation link for acc Signed-off-by: Shenoy Pratik <[email protected]> * fix typos and minor bugs Signed-off-by: Shenoy Pratik <[email protected]> * update snapshot Signed-off-by: Shenoy Pratik <[email protected]> * update window location to hash Signed-off-by: Shenoy Pratik <[email protected]> * remove unused headers Signed-off-by: Shenoy Pratik <[email protected]> --------- Signed-off-by: Shenoy Pratik <[email protected]> * Rename data sources to data connections (opensearch-project#1004) * rename data sources to data connections Signed-off-by: Derek Ho <[email protected]> * final cleanup Signed-off-by: Derek Ho <[email protected]> * update acceleration breadcrumb Signed-off-by: Derek Ho <[email protected]> * fix API call for data connection page Signed-off-by: Derek Ho <[email protected]> * fix integ test and data test subj and snapshot Signed-off-by: Derek Ho <[email protected]> --------- Signed-off-by: Derek Ho <[email protected]> * Add fallback to show if user does not have datasource API permissions (opensearch-project#1008) * add fallback ui for manage and view datasources Signed-off-by: Derek Ho <[email protected]> * always show datasources via pplservice Signed-off-by: Derek Ho <[email protected]> --------- Signed-off-by: Derek Ho <[email protected]> * Add access control tab content (opensearch-project#992) * basic rendering for the access control tab Signed-off-by: Derek Ho <[email protected]> * hook up basic radio groups and euicombo boxes for query and acceleration permissions Signed-off-by: Derek Ho <[email protected]> * refactor and clean up unuseed inports Signed-off-by: Derek Ho <[email protected]> * remove unused import Signed-off-by: Derek Ho <[email protected]> * fix import and snapshot Signed-off-by: Derek Ho <[email protected]> * fix test Signed-off-by: Derek Ho <[email protected]> * Address PR comments Signed-off-by: Derek Ho <[email protected]> * Address PR comments Signed-off-by: Derek Ho <[email protected]> * Remove unused files and variables Signed-off-by: Derek Ho <[email protected]> --------- Signed-off-by: Derek Ho <[email protected]> * Delete datasource and Connection Configuration Tab (opensearch-project#1024) * Address previous PR comments and implement rudimentary delete Signed-off-by: Derek Ho <[email protected]> * Implement modal and instant delete showing up in list Signed-off-by: Derek Ho <[email protected]> * Refactor save or cancel to a shared component, implement hard coded datasource configurations tab Signed-off-by: Derek Ho <[email protected]> * Update test with mock role data Signed-off-by: Derek Ho <[email protected]> * Add most functionality of edit connectiondetails Signed-off-by: Derek Ho <[email protected]> * Address PR comments Signed-off-by: Derek Ho <[email protected]> --------- Signed-off-by: Derek Ho <[email protected]> * remove acceleration components Signed-off-by: Shenoy Pratik <[email protected]> --------- Signed-off-by: Derek Ho <[email protected]> Signed-off-by: Shenoy Pratik <[email protected]> Co-authored-by: Shenoy Pratik <[email protected]>

* Update mocha (#960) * added code for the change of design of the sidebar, fixed overriding … (#928) * Refactor add_integration_flyout to break out nested methods (#956) * [Log Explorer] Remove top level tabs (#970) * Explorer's timeheader & color change (#959) * Update caching (#965) * Move query_utils to /public (#983) * Remove Zeppelin Parser from Notebook (#985) * Explorer empty bucket (#990) * refactored classnames for sidebar (#933) * Refactor Integrations Backend to abstract IO (#947) * Create React component for Integrations Setup UI (#1009) * Create Data sources plugin with Manage Datasources Flow (#1035) * Manage datasources (#967) * Add acceleration management UI (#989) * Rename data sources to data connections (#1004) * Add fallback to show if user does not have datasource API permissions (#1008) * Add access control tab content (#992) * Delete datasource and Connection Configuration Tab (#1024) * adjust explorer chart color and spacing (#1051) * Configure S3 datasource flow (#1049) * Manage datasources (#967) * Add acceleration management UI (#989) * Rename data sources to data connections (#1004) * Add fallback to show if user does not have datasource API permissions (#1008) * Add access control tab content (#992) * Delete datasource and Connection Configuration Tab (#1024) * modified explorer data grid to follow discover look and feel (#1041) * Create prometheus datasource flow (#1054) * Manage datasources (#967) * Add acceleration management UI (#989) * Rename data sources to data connections (#1004) * Add fallback to show if user does not have datasource API permissions (#1008) * Add access control tab content (#992) * Delete datasource and Connection Configuration Tab (#1024) * added design changes for sidebar (#1061) * updated snapshots (#1062) * Update UI for integrations setup (#1052) * Data sources UI improvements (#1059) * adding redirection to datasources (#1063) * Move convertDateTime, refactor query_utils (#1064) * Setup S3 connection with integrations (#1057) * Use approved svg from UX (#1066) * Fix missing import 'moment' on query_utils. (#1067) --------- Signed-off-by: Derek Ho <[email protected]> Signed-off-by: sumukhswamy <[email protected]> Signed-off-by: Eric Wei <[email protected]> Signed-off-by: Paul Sebastian <[email protected]> Signed-off-by: Peter Fitzgibbons <[email protected]> Signed-off-by: Shenoy Pratik <[email protected]> Signed-off-by: Simeon Widdis <[email protected]> Co-authored-by: sumukhswamy <[email protected]> Co-authored-by: Eric Wei <[email protected]> Co-authored-by: Paul Sebastian <[email protected]> Co-authored-by: Peter Fitzgibbons <[email protected]> Co-authored-by: Paul Sebastian <[email protected]> Co-authored-by: Shenoy Pratik <[email protected]> Co-authored-by: Simeon Widdis <[email protected]>

* Update mocha (opensearch-project#960) * added code for the change of design of the sidebar, fixed overriding … (opensearch-project#928) * Refactor add_integration_flyout to break out nested methods (opensearch-project#956) * [Log Explorer] Remove top level tabs (opensearch-project#970) * Explorer's timeheader & color change (opensearch-project#959) * Update caching (opensearch-project#965) * Move query_utils to /public (opensearch-project#983) * Remove Zeppelin Parser from Notebook (opensearch-project#985) * Explorer empty bucket (opensearch-project#990) * refactored classnames for sidebar (opensearch-project#933) * Refactor Integrations Backend to abstract IO (opensearch-project#947) * Create React component for Integrations Setup UI (opensearch-project#1009) * Create Data sources plugin with Manage Datasources Flow (opensearch-project#1035) * Manage datasources (opensearch-project#967) * Add acceleration management UI (opensearch-project#989) * Rename data sources to data connections (opensearch-project#1004) * Add fallback to show if user does not have datasource API permissions (opensearch-project#1008) * Add access control tab content (opensearch-project#992) * Delete datasource and Connection Configuration Tab (opensearch-project#1024) * adjust explorer chart color and spacing (opensearch-project#1051) * Configure S3 datasource flow (opensearch-project#1049) * Manage datasources (opensearch-project#967) * Add acceleration management UI (opensearch-project#989) * Rename data sources to data connections (opensearch-project#1004) * Add fallback to show if user does not have datasource API permissions (opensearch-project#1008) * Add access control tab content (opensearch-project#992) * Delete datasource and Connection Configuration Tab (opensearch-project#1024) * modified explorer data grid to follow discover look and feel (opensearch-project#1041) * Create prometheus datasource flow (opensearch-project#1054) * Manage datasources (opensearch-project#967) * Add acceleration management UI (opensearch-project#989) * Rename data sources to data connections (opensearch-project#1004) * Add fallback to show if user does not have datasource API permissions (opensearch-project#1008) * Add access control tab content (opensearch-project#992) * Delete datasource and Connection Configuration Tab (opensearch-project#1024) * added design changes for sidebar (opensearch-project#1061) * updated snapshots (opensearch-project#1062) * Update UI for integrations setup (opensearch-project#1052) * Data sources UI improvements (opensearch-project#1059) * adding redirection to datasources (opensearch-project#1063) * Move convertDateTime, refactor query_utils (opensearch-project#1064) * Setup S3 connection with integrations (opensearch-project#1057) * Use approved svg from UX (opensearch-project#1066) * Fix missing import 'moment' on query_utils. (opensearch-project#1067) --------- Signed-off-by: Derek Ho <[email protected]> Signed-off-by: sumukhswamy <[email protected]> Signed-off-by: Eric Wei <[email protected]> Signed-off-by: Paul Sebastian <[email protected]> Signed-off-by: Peter Fitzgibbons <[email protected]> Signed-off-by: Shenoy Pratik <[email protected]> Signed-off-by: Simeon Widdis <[email protected]> Co-authored-by: sumukhswamy <[email protected]> Co-authored-by: Eric Wei <[email protected]> Co-authored-by: Paul Sebastian <[email protected]> Co-authored-by: Peter Fitzgibbons <[email protected]> Co-authored-by: Paul Sebastian <[email protected]> Co-authored-by: Shenoy Pratik <[email protected]> Co-authored-by: Simeon Widdis <[email protected]> (cherry picked from commit 171acde)

derek-ho and others added 5 commits September 7, 2023 10:36

derek-ho requested review from pjfitzgibbons, ps48, kavithacm, joshuali925, dai-chen, YANG-DB, rupal-bq, mengweieric, vmmusings, Swiddis, penghuo, seankao-az and anirudha as code owners September 20, 2023 21:00

remove acceleration components

ad153ed

Signed-off-by: Shenoy Pratik <[email protected]>

ps48 approved these changes Sep 22, 2023

View reviewed changes

mengweieric approved these changes Sep 25, 2023

View reviewed changes

derek-ho changed the title ~~Merge Feature/flint UI into Main~~ Create Data sources plugin with Manage Datasources Flow Sep 25, 2023

derek-ho merged commit 6efa950 into main Sep 25, 2023
12 of 13 checks passed

derek-ho deleted the feature/flint-UI branch September 25, 2023 17:37

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Create Data sources plugin with Manage Datasources Flow #1035

Create Data sources plugin with Manage Datasources Flow #1035

derek-ho commented Sep 20, 2023

codecov bot commented Sep 20, 2023 •

edited

Loading

ps48 left a comment

derek-ho commented Sep 22, 2023 •

edited

Loading

Create Data sources plugin with Manage Datasources Flow #1035

Create Data sources plugin with Manage Datasources Flow #1035

Conversation

derek-ho commented Sep 20, 2023

Description

Issues Resolved

Check List

codecov bot commented Sep 20, 2023 • edited Loading

Codecov Report

ps48 left a comment

Choose a reason for hiding this comment

derek-ho commented Sep 22, 2023 • edited Loading

codecov bot commented Sep 20, 2023 •

edited

Loading

derek-ho commented Sep 22, 2023 •

edited

Loading