Setup S3 connection with integrations #1057

Swiddis · 2023-09-29T23:12:50Z

Description

Followup to #1052, this PR adds actual S3 integration functionality.

Work in progress -- Main querying pipeline is good, but need to do some work for the UI.

Issues Resolved

N/A

Check List

New functionality includes testing.
- All tests pass, including unit test, integration test and doctest
New functionality has been documented.
- New functionality has javadoc added
- New functionality has user manual doc added
Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Simeon Widdis <[email protected]>

This test is unneeded after 12c4bcf Signed-off-by: Simeon Widdis <[email protected]>

Signed-off-by: Simeon Widdis <[email protected]>

…atalog

Signed-off-by: Simeon Widdis <[email protected]>

Let it be known at on this day, with this commit, I have truly grokked why we don't use `any` in typescript. Signed-off-by: Simeon Widdis <[email protected]>

Signed-off-by: Simeon Widdis <[email protected]>

...c/components/integrations/components/__tests__/__snapshots__/setup_integration.test.tsx.snap

Signed-off-by: Simeon Widdis <[email protected]>

public/components/integrations/components/setup_integration.tsx

kavithacm · 2023-10-04T17:15:51Z

public/components/integrations/components/setup_integration.tsx

+                  if (!result.ok) {
+                    console.error('Query failed', result.error);
+                    setLoading(false);
+                    setToast('Something went wrong.', 'danger');


can we change this toast to more appropriate wording?

What's more appropriate wording? We don't have any error information in the response that we can use

derek-ho · 2023-10-04T17:18:40Z

...c/components/integrations/components/__tests__/__snapshots__/setup_integration.test.tsx.snap

                          "name": "sample",
-                          "type": "logs",
-                          "version": "2.0.0",
+                          "type": "",


Why are you removing type here

Stopped filling a default value when the integration isn't loaded yet -- We only use the type when we go to integration adding so it's safe to keep empty until it's loaded.

public/components/integrations/components/setup_integration.tsx

derek-ho · 2023-10-04T17:20:56Z

public/components/integrations/components/setup_integration.tsx

+        },
+      })
+    ).queryId;
+    while (true) {


Is there better way than while true?

Not sure what kind of polling timeout to put on it, I've heard some queries can take up to 10 mins and don't have any reference for when I should just tell the user that it isn't working

derek-ho

Approving to unblock, left a few comments

derek-ho · 2023-10-04T17:21:27Z

@Swiddis can you fix DCO

public/components/integrations/components/setup_integration.tsx

joshuali925 · 2023-10-04T17:13:54Z

public/components/integrations/components/setup_integration.tsx

+    }
+  } catch (err: any) {
+    console.error(err);
+    return { ok: false, error: err };


i think this works but usually it's error, loading, data. If loading is true then display loading, else if error is not null then display error, otherwise data is useable. for example like https://swr.vercel.app/#overview

this way you don't need to return both ok and error, and loading and sleeping should be distinguished in case UI needs them

public/components/integrations/components/setup_integration.tsx

Signed-off-by: Simeon Widdis <[email protected]>

Swiddis · 2023-10-04T18:10:40Z

Dealt with small-scale PR feedback -- More large-scale changes will need to happen later. Just waiting on tests before merging.

* Update mocha (#960) * added code for the change of design of the sidebar, fixed overriding … (#928) * Refactor add_integration_flyout to break out nested methods (#956) * [Log Explorer] Remove top level tabs (#970) * Explorer's timeheader & color change (#959) * Update caching (#965) * Move query_utils to /public (#983) * Remove Zeppelin Parser from Notebook (#985) * Explorer empty bucket (#990) * refactored classnames for sidebar (#933) * Refactor Integrations Backend to abstract IO (#947) * Create React component for Integrations Setup UI (#1009) * Create Data sources plugin with Manage Datasources Flow (#1035) * Manage datasources (#967) * Add acceleration management UI (#989) * Rename data sources to data connections (#1004) * Add fallback to show if user does not have datasource API permissions (#1008) * Add access control tab content (#992) * Delete datasource and Connection Configuration Tab (#1024) * adjust explorer chart color and spacing (#1051) * Configure S3 datasource flow (#1049) * Manage datasources (#967) * Add acceleration management UI (#989) * Rename data sources to data connections (#1004) * Add fallback to show if user does not have datasource API permissions (#1008) * Add access control tab content (#992) * Delete datasource and Connection Configuration Tab (#1024) * modified explorer data grid to follow discover look and feel (#1041) * Create prometheus datasource flow (#1054) * Manage datasources (#967) * Add acceleration management UI (#989) * Rename data sources to data connections (#1004) * Add fallback to show if user does not have datasource API permissions (#1008) * Add access control tab content (#992) * Delete datasource and Connection Configuration Tab (#1024) * added design changes for sidebar (#1061) * updated snapshots (#1062) * Update UI for integrations setup (#1052) * Data sources UI improvements (#1059) * adding redirection to datasources (#1063) * Move convertDateTime, refactor query_utils (#1064) * Setup S3 connection with integrations (#1057) * Use approved svg from UX (#1066) * Fix missing import 'moment' on query_utils. (#1067) --------- Signed-off-by: Derek Ho <[email protected]> Signed-off-by: sumukhswamy <[email protected]> Signed-off-by: Eric Wei <[email protected]> Signed-off-by: Paul Sebastian <[email protected]> Signed-off-by: Peter Fitzgibbons <[email protected]> Signed-off-by: Shenoy Pratik <[email protected]> Signed-off-by: Simeon Widdis <[email protected]> Co-authored-by: sumukhswamy <[email protected]> Co-authored-by: Eric Wei <[email protected]> Co-authored-by: Paul Sebastian <[email protected]> Co-authored-by: Peter Fitzgibbons <[email protected]> Co-authored-by: Paul Sebastian <[email protected]> Co-authored-by: Shenoy Pratik <[email protected]> Co-authored-by: Simeon Widdis <[email protected]>

* Working on review comments Signed-off-by: ruchika-narang <[email protected]> * Replacing series and dimension with dynamic naming Signed-off-by: ruchika-narang <[email protected]> * Revert code for histogram and treemap after antlr merge because of crashes in app Signed-off-by: ruchika-narang <[email protected]> * Working on review comments, renaming metrics to series Signed-off-by: ruchika-narang <[email protected]> * Removing extra import after rebasing main Signed-off-by: ruchika-narang <[email protected]> * Working on review comments Signed-off-by: ruchika-narang <[email protected]> * Replacing series and dimension with dynamic naming Signed-off-by: ruchika-narang <[email protected]> * Fixed logs view visualizations issues. Signed-off-by: Saisanju Sreevalsakumar <[email protected]> * Resolved review comments for log view fixes 1. Signed-off-by: Saisanju Sreevalsakumar <[email protected]> Signed-off-by: ruchika-narang <[email protected]> Signed-off-by: Saisanju Sreevalsakumar <[email protected]> Co-authored-by: ruchika-narang <[email protected]>

* Update mocha (opensearch-project#960) * added code for the change of design of the sidebar, fixed overriding … (opensearch-project#928) * Refactor add_integration_flyout to break out nested methods (opensearch-project#956) * [Log Explorer] Remove top level tabs (opensearch-project#970) * Explorer's timeheader & color change (opensearch-project#959) * Update caching (opensearch-project#965) * Move query_utils to /public (opensearch-project#983) * Remove Zeppelin Parser from Notebook (opensearch-project#985) * Explorer empty bucket (opensearch-project#990) * refactored classnames for sidebar (opensearch-project#933) * Refactor Integrations Backend to abstract IO (opensearch-project#947) * Create React component for Integrations Setup UI (opensearch-project#1009) * Create Data sources plugin with Manage Datasources Flow (opensearch-project#1035) * Manage datasources (opensearch-project#967) * Add acceleration management UI (opensearch-project#989) * Rename data sources to data connections (opensearch-project#1004) * Add fallback to show if user does not have datasource API permissions (opensearch-project#1008) * Add access control tab content (opensearch-project#992) * Delete datasource and Connection Configuration Tab (opensearch-project#1024) * adjust explorer chart color and spacing (opensearch-project#1051) * Configure S3 datasource flow (opensearch-project#1049) * Manage datasources (opensearch-project#967) * Add acceleration management UI (opensearch-project#989) * Rename data sources to data connections (opensearch-project#1004) * Add fallback to show if user does not have datasource API permissions (opensearch-project#1008) * Add access control tab content (opensearch-project#992) * Delete datasource and Connection Configuration Tab (opensearch-project#1024) * modified explorer data grid to follow discover look and feel (opensearch-project#1041) * Create prometheus datasource flow (opensearch-project#1054) * Manage datasources (opensearch-project#967) * Add acceleration management UI (opensearch-project#989) * Rename data sources to data connections (opensearch-project#1004) * Add fallback to show if user does not have datasource API permissions (opensearch-project#1008) * Add access control tab content (opensearch-project#992) * Delete datasource and Connection Configuration Tab (opensearch-project#1024) * added design changes for sidebar (opensearch-project#1061) * updated snapshots (opensearch-project#1062) * Update UI for integrations setup (opensearch-project#1052) * Data sources UI improvements (opensearch-project#1059) * adding redirection to datasources (opensearch-project#1063) * Move convertDateTime, refactor query_utils (opensearch-project#1064) * Setup S3 connection with integrations (opensearch-project#1057) * Use approved svg from UX (opensearch-project#1066) * Fix missing import 'moment' on query_utils. (opensearch-project#1067) --------- Signed-off-by: Derek Ho <[email protected]> Signed-off-by: sumukhswamy <[email protected]> Signed-off-by: Eric Wei <[email protected]> Signed-off-by: Paul Sebastian <[email protected]> Signed-off-by: Peter Fitzgibbons <[email protected]> Signed-off-by: Shenoy Pratik <[email protected]> Signed-off-by: Simeon Widdis <[email protected]> Co-authored-by: sumukhswamy <[email protected]> Co-authored-by: Eric Wei <[email protected]> Co-authored-by: Paul Sebastian <[email protected]> Co-authored-by: Peter Fitzgibbons <[email protected]> Co-authored-by: Paul Sebastian <[email protected]> Co-authored-by: Shenoy Pratik <[email protected]> Co-authored-by: Simeon Widdis <[email protected]> (cherry picked from commit 171acde)

Swiddis added 30 commits August 17, 2023 15:41

Stub catalog reader interface

c5cea8d

Signed-off-by: Simeon Widdis <[email protected]>

Add basic catalog functionality to new catalog reader

77ad89b

Signed-off-by: Simeon Widdis <[email protected]>

Refactor validation logic with a deeper interface

676591d

Signed-off-by: Simeon Widdis <[email protected]>

Refactor validation logic with a deeper interface

12c4bcf

Signed-off-by: Simeon Widdis <[email protected]>

Remove redundant test.

412480e

This test is unneeded after 12c4bcf Signed-off-by: Simeon Widdis <[email protected]>

Add tests for new validators

6d2bad1

Signed-off-by: Simeon Widdis <[email protected]>

Merge branch 'better-validation' into prototype-network-catalog

c3dd111

Make better failure mode for invalid objects

55d6490

Signed-off-by: Simeon Widdis <[email protected]>

Merge branch 'main' into prototype-network-catalog

c62d28b

Generalize Result type

ba56fb8

Signed-off-by: Simeon Widdis <[email protected]>

Convert backend to use catalog reader (unstable)

cb29208

Signed-off-by: Simeon Widdis <[email protected]>

Repair tests for integrations class (unstable)

c9de36e

Signed-off-by: Simeon Widdis <[email protected]>

Refactor repository for new integration interface

37c7980

Signed-off-by: Simeon Widdis <[email protected]>

Fix outer repository and backend tests

b331090

Signed-off-by: Simeon Widdis <[email protected]>

Add tests for sample data

4b29e17

Signed-off-by: Simeon Widdis <[email protected]>

Merge remote-tracking branch 'upstream/main' into prototype-network-c…

b35dfa8

…atalog

Add CatalogReader JavaDocs

1bef29e

Signed-off-by: Simeon Widdis <[email protected]>

Repair integrations builder

a18cf47

Signed-off-by: Simeon Widdis <[email protected]>

Remove extra commented test

047b14a

Signed-off-by: Simeon Widdis <[email protected]>

Remove unnecessary log statement

e65fe8b

Signed-off-by: Simeon Widdis <[email protected]>

Repair getSchemas behavior to return correct type

00ddb5b

Let it be known at on this day, with this commit, I have truly grokked why we don't use `any` in typescript. Signed-off-by: Simeon Widdis <[email protected]>

Add tests for getSchemas

8af7f4a

Signed-off-by: Simeon Widdis <[email protected]>

Add tests for asset and sample data backend methods

a31963d

Signed-off-by: Simeon Widdis <[email protected]>

Break flyout validation methods out of constructing method

feeaef5

Signed-off-by: Simeon Widdis <[email protected]>

Add tests for extracted flyout methods

b644d11

Signed-off-by: Simeon Widdis <[email protected]>

Switch validation method to use ValidationResult

614613a

Signed-off-by: Simeon Widdis <[email protected]>

Merge branch 'prototype-network-catalog' into feature/integrations-setup

d97b931

Merge branch 'refactor-integ-frontend' into feature/integrations-setup

c8b5eff

Merge branch 'main' into prototype-network-catalog

af78c27

Swap out flyout for hello-world setup page

ab30ed0

Signed-off-by: Simeon Widdis <[email protected]>

Swiddis requested review from rupal-bq, mengweieric, vmmusings, penghuo, seankao-az and anirudha as code owners October 3, 2023 22:05

derek-ho reviewed Oct 3, 2023

View reviewed changes

...c/components/integrations/components/__tests__/__snapshots__/setup_integration.test.tsx.snap Outdated Show resolved Hide resolved

Fix faulty test

d89c004

Signed-off-by: Simeon Widdis <[email protected]>

YANG-DB approved these changes Oct 4, 2023

View reviewed changes

kavithacm reviewed Oct 4, 2023

View reviewed changes

public/components/integrations/components/setup_integration.tsx Outdated Show resolved Hide resolved

kavithacm reviewed Oct 4, 2023

View reviewed changes

derek-ho reviewed Oct 4, 2023

View reviewed changes

kavithacm reviewed Oct 4, 2023

View reviewed changes

public/components/integrations/components/setup_integration.tsx Outdated Show resolved Hide resolved

derek-ho reviewed Oct 4, 2023

View reviewed changes

public/components/integrations/components/setup_integration.tsx Outdated Show resolved Hide resolved

derek-ho reviewed Oct 4, 2023

View reviewed changes

derek-ho approved these changes Oct 4, 2023

View reviewed changes

joshuali925 approved these changes Oct 4, 2023

View reviewed changes

Swiddis added 4 commits October 4, 2023 10:27

Remove debug logging

97c02b2

Signed-off-by: Simeon Widdis <[email protected]>

Remove redundant copy complexity

aaeb16d

Signed-off-by: Simeon Widdis <[email protected]>

Add assets to default state

d1394a6

Signed-off-by: Simeon Widdis <[email protected]>

Fix progress updaters

d70b890

Signed-off-by: Simeon Widdis <[email protected]>

Swiddis merged commit eee56aa into opensearch-project:main Oct 4, 2023
8 of 9 checks passed

Swiddis deleted the setup-s3 branch October 4, 2023 18:28

derek-ho pushed a commit to derek-ho/dashboards-observability that referenced this pull request Oct 4, 2023

Setup S3 connection with integrations (opensearch-project#1057)

4008c02

mengweieric pushed a commit to mengweieric/dashboards-observability that referenced this pull request Oct 15, 2023

Setup S3 connection with integrations (opensearch-project#1057)

e5b07ed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Setup S3 connection with integrations #1057

Setup S3 connection with integrations #1057

Swiddis commented Sep 29, 2023

kavithacm Oct 4, 2023

derek-ho Oct 4, 2023

Swiddis Oct 4, 2023

derek-ho Oct 4, 2023

Swiddis Oct 4, 2023

derek-ho Oct 4, 2023

Swiddis Oct 4, 2023

derek-ho left a comment

derek-ho commented Oct 4, 2023

joshuali925 Oct 4, 2023 •

edited

Loading

Swiddis commented Oct 4, 2023

Setup S3 connection with integrations #1057

Setup S3 connection with integrations #1057

Conversation

Swiddis commented Sep 29, 2023

Description

Issues Resolved

Check List

kavithacm Oct 4, 2023

Choose a reason for hiding this comment

derek-ho Oct 4, 2023

Choose a reason for hiding this comment

Swiddis Oct 4, 2023

Choose a reason for hiding this comment

derek-ho Oct 4, 2023

Choose a reason for hiding this comment

Swiddis Oct 4, 2023

Choose a reason for hiding this comment

derek-ho Oct 4, 2023

Choose a reason for hiding this comment

Swiddis Oct 4, 2023

Choose a reason for hiding this comment

derek-ho left a comment

Choose a reason for hiding this comment

derek-ho commented Oct 4, 2023

joshuali925 Oct 4, 2023 • edited Loading

Choose a reason for hiding this comment

Swiddis commented Oct 4, 2023

joshuali925 Oct 4, 2023 •

edited

Loading