Configure S3 datasource flow #1049

derek-ho · 2023-09-25T17:44:28Z

Description

Adds configure s3 datasource flow
Changes some references of data connections to datasource
Adds directory structure to separate out new and manage files/flows

Issues Resolved

[List any issues this PR will resolve]

Check List

New functionality includes testing.
- All tests pass, including unit test, integration test and doctest
New functionality has been documented.
- New functionality has javadoc added
- New functionality has user manual doc added
Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

* fix name change bug and modify test to test behavior Signed-off-by: Derek Ho <[email protected]> * get rid of lint Signed-off-by: Derek Ho <[email protected]> * test for flyout Signed-off-by: Derek Ho <[email protected]> * flyout to medium size Signed-off-by: Derek Ho <[email protected]> * make accelerate extensible Signed-off-by: Derek Ho <[email protected]> * get datasources and hook up to pplservice Signed-off-by: Derek Ho <[email protected]> * get flint working Signed-off-by: Derek Ho <[email protected]> * add datasource page with steps and buttons on bottom bar Signed-off-by: Derek Ho <[email protected]> * datasources as a new plugin and mostly working Signed-off-by: Derek Ho <[email protected]> * hook up manage to show datasources call Signed-off-by: Derek Ho <[email protected]> * update two tables with descriptions Signed-off-by: Derek Ho <[email protected]> * make some updates to the page Signed-off-by: Derek Ho <[email protected]> * cleanup unused files for data connections Signed-off-by: Derek Ho <[email protected]> * cleanup and add overview panel columns Signed-off-by: Derek Ho <[email protected]> * render tabs Signed-off-by: Derek Ho <[email protected]> * add unit tests Signed-off-by: Derek Ho <[email protected]> * update data test subj and snapshot Signed-off-by: Derek Ho <[email protected]> * Add datasources to management overview Signed-off-by: Derek Ho <[email protected]> * remove spark logo and update snapshot Signed-off-by: Derek Ho <[email protected]> * refactor routes out Signed-off-by: Derek Ho <[email protected]> * separate out the roles Signed-off-by: Derek Ho <[email protected]> * bump version back to 3.0 Signed-off-by: Derek Ho <[email protected]> --------- Signed-off-by: Derek Ho <[email protected]>

* add acceleration management UI skeleton Signed-off-by: Shenoy Pratik <[email protected]> * Create new documentation link for acc Signed-off-by: Shenoy Pratik <[email protected]> * fix typos and minor bugs Signed-off-by: Shenoy Pratik <[email protected]> * update snapshot Signed-off-by: Shenoy Pratik <[email protected]> * update window location to hash Signed-off-by: Shenoy Pratik <[email protected]> * remove unused headers Signed-off-by: Shenoy Pratik <[email protected]> --------- Signed-off-by: Shenoy Pratik <[email protected]>

* rename data sources to data connections Signed-off-by: Derek Ho <[email protected]> * final cleanup Signed-off-by: Derek Ho <[email protected]> * update acceleration breadcrumb Signed-off-by: Derek Ho <[email protected]> * fix API call for data connection page Signed-off-by: Derek Ho <[email protected]> * fix integ test and data test subj and snapshot Signed-off-by: Derek Ho <[email protected]> --------- Signed-off-by: Derek Ho <[email protected]>

…opensearch-project#1008) * add fallback ui for manage and view datasources Signed-off-by: Derek Ho <[email protected]> * always show datasources via pplservice Signed-off-by: Derek Ho <[email protected]> --------- Signed-off-by: Derek Ho <[email protected]>

* basic rendering for the access control tab Signed-off-by: Derek Ho <[email protected]> * hook up basic radio groups and euicombo boxes for query and acceleration permissions Signed-off-by: Derek Ho <[email protected]> * refactor and clean up unuseed inports Signed-off-by: Derek Ho <[email protected]> * remove unused import Signed-off-by: Derek Ho <[email protected]> * fix import and snapshot Signed-off-by: Derek Ho <[email protected]> * fix test Signed-off-by: Derek Ho <[email protected]> * Address PR comments Signed-off-by: Derek Ho <[email protected]> * Address PR comments Signed-off-by: Derek Ho <[email protected]> * Remove unused files and variables Signed-off-by: Derek Ho <[email protected]> --------- Signed-off-by: Derek Ho <[email protected]>

…t#1024) * Address previous PR comments and implement rudimentary delete Signed-off-by: Derek Ho <[email protected]> * Implement modal and instant delete showing up in list Signed-off-by: Derek Ho <[email protected]> * Refactor save or cancel to a shared component, implement hard coded datasource configurations tab Signed-off-by: Derek Ho <[email protected]> * Update test with mock role data Signed-off-by: Derek Ho <[email protected]> * Add most functionality of edit connectiondetails Signed-off-by: Derek Ho <[email protected]> * Address PR comments Signed-off-by: Derek Ho <[email protected]> --------- Signed-off-by: Derek Ho <[email protected]>

Signed-off-by: Derek Ho <[email protected]>

Signed-off-by: Shenoy Pratik <[email protected]>

… tests Signed-off-by: Derek Ho <[email protected]>

…boards-observability into configure Signed-off-by: Derek Ho <[email protected]>

Signed-off-by: Derek Ho <[email protected]>

…vability into configure Signed-off-by: Derek Ho <[email protected]>

codecov · 2023-09-25T18:08:05Z

Codecov Report

Merging #1049 (8251678) into main (6efa950) will increase coverage by 0.00%.
Report is 2 commits behind head on main.
The diff coverage is n/a.

❗ Current head 8251678 differs from pull request most recent head 4d19eb4. Consider uploading reports for the commit 4d19eb4 to get more accurate results

@@           Coverage Diff           @@
##             main    #1049   +/-   ##
=======================================
  Coverage   44.42%   44.43%           
=======================================
  Files         325      325           
  Lines       19055    19072   +17     
  Branches     4631     4634    +3     
=======================================
+ Hits         8465     8474    +9     
- Misses      10046    10054    +8     
  Partials      544      544

Flag	Coverage Δ
dashboards-observability	`44.43% <ø> (+<0.01%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

see 22 files with indirect coverage changes

Signed-off-by: Derek Ho <[email protected]>

public/components/datasources/components/new/configure_datasource.tsx

public/components/datasources/components/manage/data_connection.tsx

Signed-off-by: Derek Ho <[email protected]>

public/components/datasources/components/new/configure_s3_datasource.tsx

Signed-off-by: Derek Ho <[email protected]>

public/components/datasources/components/data_connections_header.tsx

public/components/datasources/components/new/configure_datasource.tsx

public/components/datasources/components/new/configure_s3_datasource.tsx

server/routes/data_connections/data_connections_router.ts

Signed-off-by: Derek Ho <[email protected]>

* Manage datasources (opensearch-project#967) * fix name change bug and modify test to test behavior Signed-off-by: Derek Ho <[email protected]> * get rid of lint Signed-off-by: Derek Ho <[email protected]> * test for flyout Signed-off-by: Derek Ho <[email protected]> * flyout to medium size Signed-off-by: Derek Ho <[email protected]> * make accelerate extensible Signed-off-by: Derek Ho <[email protected]> * get datasources and hook up to pplservice Signed-off-by: Derek Ho <[email protected]> * get flint working Signed-off-by: Derek Ho <[email protected]> * add datasource page with steps and buttons on bottom bar Signed-off-by: Derek Ho <[email protected]> * datasources as a new plugin and mostly working Signed-off-by: Derek Ho <[email protected]> * hook up manage to show datasources call Signed-off-by: Derek Ho <[email protected]> * update two tables with descriptions Signed-off-by: Derek Ho <[email protected]> * make some updates to the page Signed-off-by: Derek Ho <[email protected]> * cleanup unused files for data connections Signed-off-by: Derek Ho <[email protected]> * cleanup and add overview panel columns Signed-off-by: Derek Ho <[email protected]> * render tabs Signed-off-by: Derek Ho <[email protected]> * add unit tests Signed-off-by: Derek Ho <[email protected]> * update data test subj and snapshot Signed-off-by: Derek Ho <[email protected]> * Add datasources to management overview Signed-off-by: Derek Ho <[email protected]> * remove spark logo and update snapshot Signed-off-by: Derek Ho <[email protected]> * refactor routes out Signed-off-by: Derek Ho <[email protected]> * separate out the roles Signed-off-by: Derek Ho <[email protected]> * bump version back to 3.0 Signed-off-by: Derek Ho <[email protected]> --------- Signed-off-by: Derek Ho <[email protected]> * Add acceleration management UI (opensearch-project#989) * add acceleration management UI skeleton Signed-off-by: Shenoy Pratik <[email protected]> * Create new documentation link for acc Signed-off-by: Shenoy Pratik <[email protected]> * fix typos and minor bugs Signed-off-by: Shenoy Pratik <[email protected]> * update snapshot Signed-off-by: Shenoy Pratik <[email protected]> * update window location to hash Signed-off-by: Shenoy Pratik <[email protected]> * remove unused headers Signed-off-by: Shenoy Pratik <[email protected]> --------- Signed-off-by: Shenoy Pratik <[email protected]> * Rename data sources to data connections (opensearch-project#1004) * rename data sources to data connections Signed-off-by: Derek Ho <[email protected]> * final cleanup Signed-off-by: Derek Ho <[email protected]> * update acceleration breadcrumb Signed-off-by: Derek Ho <[email protected]> * fix API call for data connection page Signed-off-by: Derek Ho <[email protected]> * fix integ test and data test subj and snapshot Signed-off-by: Derek Ho <[email protected]> --------- Signed-off-by: Derek Ho <[email protected]> * Add fallback to show if user does not have datasource API permissions (opensearch-project#1008) * add fallback ui for manage and view datasources Signed-off-by: Derek Ho <[email protected]> * always show datasources via pplservice Signed-off-by: Derek Ho <[email protected]> --------- Signed-off-by: Derek Ho <[email protected]> * Add access control tab content (opensearch-project#992) * basic rendering for the access control tab Signed-off-by: Derek Ho <[email protected]> * hook up basic radio groups and euicombo boxes for query and acceleration permissions Signed-off-by: Derek Ho <[email protected]> * refactor and clean up unuseed inports Signed-off-by: Derek Ho <[email protected]> * remove unused import Signed-off-by: Derek Ho <[email protected]> * fix import and snapshot Signed-off-by: Derek Ho <[email protected]> * fix test Signed-off-by: Derek Ho <[email protected]> * Address PR comments Signed-off-by: Derek Ho <[email protected]> * Address PR comments Signed-off-by: Derek Ho <[email protected]> * Remove unused files and variables Signed-off-by: Derek Ho <[email protected]> --------- Signed-off-by: Derek Ho <[email protected]> * Delete datasource and Connection Configuration Tab (opensearch-project#1024) * Address previous PR comments and implement rudimentary delete Signed-off-by: Derek Ho <[email protected]> * Implement modal and instant delete showing up in list Signed-off-by: Derek Ho <[email protected]> * Refactor save or cancel to a shared component, implement hard coded datasource configurations tab Signed-off-by: Derek Ho <[email protected]> * Update test with mock role data Signed-off-by: Derek Ho <[email protected]> * Add most functionality of edit connectiondetails Signed-off-by: Derek Ho <[email protected]> * Address PR comments Signed-off-by: Derek Ho <[email protected]> --------- Signed-off-by: Derek Ho <[email protected]> * Get most of the workflow working for configure datasource Signed-off-by: Derek Ho <[email protected]> * remove acceleration components Signed-off-by: Shenoy Pratik <[email protected]> * Update data connection to datasources in a few more places and update tests Signed-off-by: Derek Ho <[email protected]> * Rename and re-organize in folders Signed-off-by: Derek Ho <[email protected]> * Fix import path in testing file Signed-off-by: Derek Ho <[email protected]> * Get Initial S3 configuration working Signed-off-by: Derek Ho <[email protected]> * Configure S3 Data Source Working Signed-off-by: Derek Ho <[email protected]> * Fix query permissions display Signed-off-by: Derek Ho <[email protected]> * Fix merge conflict Signed-off-by: Derek Ho <[email protected]> * Clean up PR Signed-off-by: Derek Ho <[email protected]> * Add s3 logo in manage table Signed-off-by: Derek Ho <[email protected]> * Fix up PR according to UX feedback Signed-off-by: Derek Ho <[email protected]> * Remove successfully Signed-off-by: Derek Ho <[email protected]> * Update test Signed-off-by: Derek Ho <[email protected]> * Address PR comments Signed-off-by: Derek Ho <[email protected]> * Update test in accordance to useLocation Signed-off-by: Derek Ho <[email protected]> --------- Signed-off-by: Derek Ho <[email protected]> Signed-off-by: Shenoy Pratik <[email protected]> Co-authored-by: Shenoy Pratik <[email protected]>

* Update mocha (#960) * added code for the change of design of the sidebar, fixed overriding … (#928) * Refactor add_integration_flyout to break out nested methods (#956) * [Log Explorer] Remove top level tabs (#970) * Explorer's timeheader & color change (#959) * Update caching (#965) * Move query_utils to /public (#983) * Remove Zeppelin Parser from Notebook (#985) * Explorer empty bucket (#990) * refactored classnames for sidebar (#933) * Refactor Integrations Backend to abstract IO (#947) * Create React component for Integrations Setup UI (#1009) * Create Data sources plugin with Manage Datasources Flow (#1035) * Manage datasources (#967) * Add acceleration management UI (#989) * Rename data sources to data connections (#1004) * Add fallback to show if user does not have datasource API permissions (#1008) * Add access control tab content (#992) * Delete datasource and Connection Configuration Tab (#1024) * adjust explorer chart color and spacing (#1051) * Configure S3 datasource flow (#1049) * Manage datasources (#967) * Add acceleration management UI (#989) * Rename data sources to data connections (#1004) * Add fallback to show if user does not have datasource API permissions (#1008) * Add access control tab content (#992) * Delete datasource and Connection Configuration Tab (#1024) * modified explorer data grid to follow discover look and feel (#1041) * Create prometheus datasource flow (#1054) * Manage datasources (#967) * Add acceleration management UI (#989) * Rename data sources to data connections (#1004) * Add fallback to show if user does not have datasource API permissions (#1008) * Add access control tab content (#992) * Delete datasource and Connection Configuration Tab (#1024) * added design changes for sidebar (#1061) * updated snapshots (#1062) * Update UI for integrations setup (#1052) * Data sources UI improvements (#1059) * adding redirection to datasources (#1063) * Move convertDateTime, refactor query_utils (#1064) * Setup S3 connection with integrations (#1057) * Use approved svg from UX (#1066) * Fix missing import 'moment' on query_utils. (#1067) --------- Signed-off-by: Derek Ho <[email protected]> Signed-off-by: sumukhswamy <[email protected]> Signed-off-by: Eric Wei <[email protected]> Signed-off-by: Paul Sebastian <[email protected]> Signed-off-by: Peter Fitzgibbons <[email protected]> Signed-off-by: Shenoy Pratik <[email protected]> Signed-off-by: Simeon Widdis <[email protected]> Co-authored-by: sumukhswamy <[email protected]> Co-authored-by: Eric Wei <[email protected]> Co-authored-by: Paul Sebastian <[email protected]> Co-authored-by: Peter Fitzgibbons <[email protected]> Co-authored-by: Paul Sebastian <[email protected]> Co-authored-by: Shenoy Pratik <[email protected]> Co-authored-by: Simeon Widdis <[email protected]>

* Update mocha (opensearch-project#960) * added code for the change of design of the sidebar, fixed overriding … (opensearch-project#928) * Refactor add_integration_flyout to break out nested methods (opensearch-project#956) * [Log Explorer] Remove top level tabs (opensearch-project#970) * Explorer's timeheader & color change (opensearch-project#959) * Update caching (opensearch-project#965) * Move query_utils to /public (opensearch-project#983) * Remove Zeppelin Parser from Notebook (opensearch-project#985) * Explorer empty bucket (opensearch-project#990) * refactored classnames for sidebar (opensearch-project#933) * Refactor Integrations Backend to abstract IO (opensearch-project#947) * Create React component for Integrations Setup UI (opensearch-project#1009) * Create Data sources plugin with Manage Datasources Flow (opensearch-project#1035) * Manage datasources (opensearch-project#967) * Add acceleration management UI (opensearch-project#989) * Rename data sources to data connections (opensearch-project#1004) * Add fallback to show if user does not have datasource API permissions (opensearch-project#1008) * Add access control tab content (opensearch-project#992) * Delete datasource and Connection Configuration Tab (opensearch-project#1024) * adjust explorer chart color and spacing (opensearch-project#1051) * Configure S3 datasource flow (opensearch-project#1049) * Manage datasources (opensearch-project#967) * Add acceleration management UI (opensearch-project#989) * Rename data sources to data connections (opensearch-project#1004) * Add fallback to show if user does not have datasource API permissions (opensearch-project#1008) * Add access control tab content (opensearch-project#992) * Delete datasource and Connection Configuration Tab (opensearch-project#1024) * modified explorer data grid to follow discover look and feel (opensearch-project#1041) * Create prometheus datasource flow (opensearch-project#1054) * Manage datasources (opensearch-project#967) * Add acceleration management UI (opensearch-project#989) * Rename data sources to data connections (opensearch-project#1004) * Add fallback to show if user does not have datasource API permissions (opensearch-project#1008) * Add access control tab content (opensearch-project#992) * Delete datasource and Connection Configuration Tab (opensearch-project#1024) * added design changes for sidebar (opensearch-project#1061) * updated snapshots (opensearch-project#1062) * Update UI for integrations setup (opensearch-project#1052) * Data sources UI improvements (opensearch-project#1059) * adding redirection to datasources (opensearch-project#1063) * Move convertDateTime, refactor query_utils (opensearch-project#1064) * Setup S3 connection with integrations (opensearch-project#1057) * Use approved svg from UX (opensearch-project#1066) * Fix missing import 'moment' on query_utils. (opensearch-project#1067) --------- Signed-off-by: Derek Ho <[email protected]> Signed-off-by: sumukhswamy <[email protected]> Signed-off-by: Eric Wei <[email protected]> Signed-off-by: Paul Sebastian <[email protected]> Signed-off-by: Peter Fitzgibbons <[email protected]> Signed-off-by: Shenoy Pratik <[email protected]> Signed-off-by: Simeon Widdis <[email protected]> Co-authored-by: sumukhswamy <[email protected]> Co-authored-by: Eric Wei <[email protected]> Co-authored-by: Paul Sebastian <[email protected]> Co-authored-by: Peter Fitzgibbons <[email protected]> Co-authored-by: Paul Sebastian <[email protected]> Co-authored-by: Shenoy Pratik <[email protected]> Co-authored-by: Simeon Widdis <[email protected]> (cherry picked from commit 171acde)

derek-ho and others added 15 commits September 7, 2023 10:36

Get most of the workflow working for configure datasource

1efd191

Signed-off-by: Derek Ho <[email protected]>

remove acceleration components

ad153ed

Signed-off-by: Shenoy Pratik <[email protected]>

Update data connection to datasources in a few more places and update…

93ad0dd

… tests Signed-off-by: Derek Ho <[email protected]>

Merge branch 'feature/flint-UI' of github.com:opensearch-project/dash…

d98cd8d

…boards-observability into configure Signed-off-by: Derek Ho <[email protected]>

Rename and re-organize in folders

2bd8c27

Signed-off-by: Derek Ho <[email protected]>

Fix import path in testing file

cc7b1a2

Signed-off-by: Derek Ho <[email protected]>

Get Initial S3 configuration working

ed8cfc1

Signed-off-by: Derek Ho <[email protected]>

Configure S3 Data Source Working

0b84fab

Signed-off-by: Derek Ho <[email protected]>

Fix query permissions display

6d8a98f

Signed-off-by: Derek Ho <[email protected]>

derek-ho requested review from pjfitzgibbons, ps48, kavithacm, joshuali925, dai-chen, YANG-DB, rupal-bq, mengweieric, vmmusings, Swiddis, penghuo, seankao-az and anirudha as code owners September 25, 2023 17:44

Merge branch 'main' of github.com:opensearch-project/dashboards-obser…

e8b831f

…vability into configure Signed-off-by: Derek Ho <[email protected]>

derek-ho changed the title ~~Configure~~ Configure S3 datasource flow Sep 25, 2023

Fix merge conflict

7a3f085

Signed-off-by: Derek Ho <[email protected]>

derek-ho commented Sep 25, 2023

View reviewed changes

public/components/datasources/components/new/configure_datasource.tsx Outdated Show resolved Hide resolved

derek-ho commented Sep 25, 2023

View reviewed changes

public/components/datasources/components/manage/data_connection.tsx Show resolved Hide resolved

derek-ho added 4 commits September 25, 2023 14:42

Clean up PR

1ffdfb0

Signed-off-by: Derek Ho <[email protected]>

Add s3 logo in manage table

7a333bf

Signed-off-by: Derek Ho <[email protected]>

Fix up PR according to UX feedback

c5f1eae

Signed-off-by: Derek Ho <[email protected]>

Remove successfully

23699bc

Signed-off-by: Derek Ho <[email protected]>

mengweieric approved these changes Sep 25, 2023

View reviewed changes

public/components/datasources/components/new/configure_s3_datasource.tsx Show resolved Hide resolved

Update test

51bb5d4

Signed-off-by: Derek Ho <[email protected]>

derek-ho force-pushed the configure branch from 8251678 to 51bb5d4 Compare September 26, 2023 19:07

kavithacm approved these changes Sep 26, 2023

View reviewed changes

joshuali925 reviewed Sep 26, 2023

View reviewed changes

public/components/datasources/components/data_connections_header.tsx Outdated Show resolved Hide resolved

joshuali925 reviewed Sep 26, 2023

View reviewed changes

server/routes/data_connections/data_connections_router.ts Outdated Show resolved Hide resolved

derek-ho added 2 commits September 27, 2023 11:35

Address PR comments

680763b

Signed-off-by: Derek Ho <[email protected]>

Update test in accordance to useLocation

4d19eb4

Signed-off-by: Derek Ho <[email protected]>

derek-ho merged commit 79d18df into opensearch-project:main Sep 27, 2023
7 checks passed

derek-ho deleted the configure branch September 27, 2023 15:44

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Configure S3 datasource flow #1049

Configure S3 datasource flow #1049

derek-ho commented Sep 25, 2023 •

edited

Loading

codecov bot commented Sep 25, 2023 •

edited

Loading

Configure S3 datasource flow #1049

Configure S3 datasource flow #1049

Conversation

derek-ho commented Sep 25, 2023 • edited Loading

Description

Issues Resolved

Check List

codecov bot commented Sep 25, 2023 • edited Loading

Codecov Report

derek-ho commented Sep 25, 2023 •

edited

Loading

codecov bot commented Sep 25, 2023 •

edited

Loading