[FEATURE] MDS support in Integrations for observability plugin #2051

sumukhswamy · 2024-08-09T18:20:09Z

Description

Multi data source support in Integrations for observability plugin

Screen.Recording.2024-08-09.at.11.29.25.AM.mov

Issues Resolved

Check List

New functionality includes testing.
- All tests pass, including unit test, integration test and doctest
New functionality has been documented.
- New functionality has javadoc added
- New functionality has user manual doc added
Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Swiddis

Generally LGTM, have a few questions

Swiddis · 2024-08-09T20:54:03Z

public/components/integrations/components/added_integration.tsx

+            componentConfig={{
+              activeOption: [
+                {
+                  id: data?.references?.[0]?.dataSourceMDSId,


How stable is the assumption that the first reference is always the correct dataSource? I could see more reference types being added in the future -- if we keep this is it at least under test so we'll know if we need to update it?

yes it has been updated in the test as well

is this line coming from the data source picker? From the call back function for getting the selected cluster id?

this is coming in from the saved objects created during the set up process of the integrations.

Swiddis · 2024-08-09T21:00:10Z

public/components/integrations/components/setup_integration_inputs.tsx

@@ -95,7 +98,9 @@ const suggestDataSources = async (
        }) ?? []
      );
    } else if (type === 's3' || type === 'securityLake') {
-      const result = (await http.get(DATACONNECTIONS_BASE)) as Array<{
+      const result = (await http.get(
+        `${DATACONNECTIONS_BASE}/dataSourceMDSId=${dataSourceMDSId ?? ''}`


Is this usage correct? This is a URL with an = in it instead of a request param? The http.get method already has a way of using request params with automatic undefined-handling.

wanted to keep it consistent to how other plugins were using and by adding the mds-id here the path is getting differentiated for the other locations where this api is being used

Swiddis · 2024-08-09T21:00:54Z

public/components/integrations/components/setup_integration_inputs.tsx

  lockConnectionType?: boolean;
 }) {
+  const [dataSourceMDSId, setDataSourceMDSId] = useState<string | undefined>('');


What's the difference between undefined and empty string here?

I can make this consistent to be an empty string in all locations

What's the difference between undefined and empty string here?

'' represents the local cluster. I guess the undefined is trying to handle the case that when the MDS is not enabled and the ID is undefined, so at that time we just need to handle the request towards local cluster.

I can make this consistent to be an empty string in all locations

Ideally, we should still handle the undefined and '' separately. But just wanna make sure we check the MDS flag correctly so that it will not go thru the MDS endpoint when MDS is disabled.

server/adaptors/integrations/integrations_manager.ts

Swiddis

Tentative approval based on above

public/components/integrations/components/__tests__/testing_constants.ts

public/components/integrations/components/added_integration_overview_page.tsx

ps48 · 2024-08-12T19:11:12Z

public/components/integrations/components/create_integration_helpers.ts

@@ -204,7 +206,8 @@ export const doExistingDataSourceValidation = async (

 const createComponentMapping = async (
  componentName: string,
-  payload: ComponentMappingPayload
+  payload: ComponentMappingPayload,


Can you please create an issue in https://github.com/opensearch-project/opensearch-js/issues to add missing components in the js client.

public/components/integrations/components/__tests__/testing_constants.ts

server/adaptors/integrations/integrations_builder.ts

server/adaptors/integrations/__test__/builder.test.ts

public/components/integrations/components/setup_integration.tsx

Signed-off-by: sumukhswamy <[email protected]>

* Move the save button to the header control bar Signed-off-by: Ryan Liang <[email protected]> * Update snapshots Signed-off-by: Ryan Liang <[email protected]> * Fix the save button and correct its size + position Signed-off-by: Ryan Liang <[email protected]> * Fix the date picker location Signed-off-by: Ryan Liang <[email protected]> * Rename the navigation in coreRef and switch to use compressed date picker Signed-off-by: Ryan Liang <[email protected]> * Fix the popover Signed-off-by: Ryan Liang <[email protected]> * Rename the button Signed-off-by: Ryan Liang <[email protected]> * Update to latest mockup Signed-off-by: Ryan Liang <[email protected]> * Update snapshots Signed-off-by: Ryan Liang <[email protected]> * Fix the ui issues Signed-off-by: Ryan Liang <[email protected]> * Dummy metrics Signed-off-by: Ryan Liang <[email protected]> * Remove dummy Signed-off-by: Ryan Liang <[email protected]> * update snapshots Signed-off-by: Ryan Liang <[email protected]> * minor changes to match mocks Signed-off-by: Shenoy Pratik <[email protected]> --------- Signed-off-by: Ryan Liang <[email protected]> Signed-off-by: Shenoy Pratik <[email protected]> Co-authored-by: Shenoy Pratik <[email protected]> Signed-off-by: sumukhswamy <[email protected]>

Signed-off-by: sumukhswamy <[email protected]>

Signed-off-by: Sumukh Swamy <[email protected]>

Signed-off-by: sumukhswamy <[email protected]>

opensearch-trigger-bot · 2024-08-26T16:19:09Z

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/dashboards-observability/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/dashboards-observability/backport-2.x
# Create a new branch
git switch --create backport/backport-2051-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 16e2bd556c0e2043e1e29764403ad4c6797e78b1
# Push it to GitHub
git push --set-upstream origin backport/backport-2051-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/dashboards-observability/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-2051-to-2.x.

…earch-project#2051) * Support for MDS in integrations Signed-off-by: sumukhswamy <[email protected]> * added changes for mds support in integrations Signed-off-by: sumukhswamy <[email protected]> * fixed case for Local cluster Signed-off-by: sumukhswamy <[email protected]> * addressed comments for undefined variables Signed-off-by: sumukhswamy <[email protected]> * updated snapshots Signed-off-by: sumukhswamy <[email protected]> * changed the refrences object in integrations instance Signed-off-by: sumukhswamy <[email protected]> * addressed pr comments Signed-off-by: sumukhswamy <[email protected]> * updated snapshots Signed-off-by: sumukhswamy <[email protected]> * [Page Header] New page header for metrics (opensearch-project#2050) * Move the save button to the header control bar Signed-off-by: Ryan Liang <[email protected]> * Update snapshots Signed-off-by: Ryan Liang <[email protected]> * Fix the save button and correct its size + position Signed-off-by: Ryan Liang <[email protected]> * Fix the date picker location Signed-off-by: Ryan Liang <[email protected]> * Rename the navigation in coreRef and switch to use compressed date picker Signed-off-by: Ryan Liang <[email protected]> * Fix the popover Signed-off-by: Ryan Liang <[email protected]> * Rename the button Signed-off-by: Ryan Liang <[email protected]> * Update to latest mockup Signed-off-by: Ryan Liang <[email protected]> * Update snapshots Signed-off-by: Ryan Liang <[email protected]> * Fix the ui issues Signed-off-by: Ryan Liang <[email protected]> * Dummy metrics Signed-off-by: Ryan Liang <[email protected]> * Remove dummy Signed-off-by: Ryan Liang <[email protected]> * update snapshots Signed-off-by: Ryan Liang <[email protected]> * minor changes to match mocks Signed-off-by: Shenoy Pratik <[email protected]> --------- Signed-off-by: Ryan Liang <[email protected]> Signed-off-by: Shenoy Pratik <[email protected]> Co-authored-by: Shenoy Pratik <[email protected]> Signed-off-by: sumukhswamy <[email protected]> * Updated event analytics and integrations components Signed-off-by: sumukhswamy <[email protected]> * updated tests and conditions Signed-off-by: sumukhswamy <[email protected]> * updated snapshots Signed-off-by: sumukhswamy <[email protected]> * updated snapshots Signed-off-by: sumukhswamy <[email protected]> * updated snapshots Signed-off-by: sumukhswamy <[email protected]> * updated the remote and local for snapshots Signed-off-by: sumukhswamy <[email protected]> * updated snapshots Signed-off-by: sumukhswamy <[email protected]> * updated snapshots again Signed-off-by: sumukhswamy <[email protected]> * updated snapshots Signed-off-by: sumukhswamy <[email protected]> * updated snapshots Signed-off-by: sumukhswamy <[email protected]> * updated missing import Signed-off-by: sumukhswamy <[email protected]> --------- Signed-off-by: sumukhswamy <[email protected]> Signed-off-by: Ryan Liang <[email protected]> Signed-off-by: Shenoy Pratik <[email protected]> Signed-off-by: Sumukh Swamy <[email protected]> Co-authored-by: Jialiang Liang <[email protected]> Co-authored-by: Shenoy Pratik <[email protected]>

…2051) (#2096) * [FEATURE] MDS support in Integrations for observability plugin (#2051) * Support for MDS in integrations Signed-off-by: sumukhswamy <[email protected]> * added changes for mds support in integrations Signed-off-by: sumukhswamy <[email protected]> * fixed case for Local cluster Signed-off-by: sumukhswamy <[email protected]> * addressed comments for undefined variables Signed-off-by: sumukhswamy <[email protected]> * updated snapshots Signed-off-by: sumukhswamy <[email protected]> * changed the refrences object in integrations instance Signed-off-by: sumukhswamy <[email protected]> * addressed pr comments Signed-off-by: sumukhswamy <[email protected]> * updated snapshots Signed-off-by: sumukhswamy <[email protected]> * [Page Header] New page header for metrics (#2050) * Move the save button to the header control bar Signed-off-by: Ryan Liang <[email protected]> * Update snapshots Signed-off-by: Ryan Liang <[email protected]> * Fix the save button and correct its size + position Signed-off-by: Ryan Liang <[email protected]> * Fix the date picker location Signed-off-by: Ryan Liang <[email protected]> * Rename the navigation in coreRef and switch to use compressed date picker Signed-off-by: Ryan Liang <[email protected]> * Fix the popover Signed-off-by: Ryan Liang <[email protected]> * Rename the button Signed-off-by: Ryan Liang <[email protected]> * Update to latest mockup Signed-off-by: Ryan Liang <[email protected]> * Update snapshots Signed-off-by: Ryan Liang <[email protected]> * Fix the ui issues Signed-off-by: Ryan Liang <[email protected]> * Dummy metrics Signed-off-by: Ryan Liang <[email protected]> * Remove dummy Signed-off-by: Ryan Liang <[email protected]> * update snapshots Signed-off-by: Ryan Liang <[email protected]> * minor changes to match mocks Signed-off-by: Shenoy Pratik <[email protected]> --------- Signed-off-by: Ryan Liang <[email protected]> Signed-off-by: Shenoy Pratik <[email protected]> Co-authored-by: Shenoy Pratik <[email protected]> Signed-off-by: sumukhswamy <[email protected]> * Updated event analytics and integrations components Signed-off-by: sumukhswamy <[email protected]> * updated tests and conditions Signed-off-by: sumukhswamy <[email protected]> * updated snapshots Signed-off-by: sumukhswamy <[email protected]> * updated snapshots Signed-off-by: sumukhswamy <[email protected]> * updated snapshots Signed-off-by: sumukhswamy <[email protected]> * updated the remote and local for snapshots Signed-off-by: sumukhswamy <[email protected]> * updated snapshots Signed-off-by: sumukhswamy <[email protected]> * updated snapshots again Signed-off-by: sumukhswamy <[email protected]> * updated snapshots Signed-off-by: sumukhswamy <[email protected]> * updated snapshots Signed-off-by: sumukhswamy <[email protected]> * updated missing import Signed-off-by: sumukhswamy <[email protected]> --------- Signed-off-by: sumukhswamy <[email protected]> Signed-off-by: Ryan Liang <[email protected]> Signed-off-by: Shenoy Pratik <[email protected]> Signed-off-by: Sumukh Swamy <[email protected]> Co-authored-by: Jialiang Liang <[email protected]> Co-authored-by: Shenoy Pratik <[email protected]> * linter fix Signed-off-by: sumukhswamy <[email protected]> --------- Signed-off-by: sumukhswamy <[email protected]> Signed-off-by: Ryan Liang <[email protected]> Signed-off-by: Shenoy Pratik <[email protected]> Signed-off-by: Sumukh Swamy <[email protected]> Co-authored-by: Jialiang Liang <[email protected]> Co-authored-by: Shenoy Pratik <[email protected]>

sumukhswamy requested review from ps48, kavithacm, derek-ho, joshuali925, dai-chen, YANG-DB, rupal-bq, mengweieric, vamsimanohar, Swiddis, penghuo, seankao-az, anirudha and paulstn as code owners August 9, 2024 18:20

sumukhswamy added enhancement New feature or request backport 2.x labels Aug 9, 2024

sumukhswamy force-pushed the integrations-mds branch from 029c0ca to 95487ed Compare August 9, 2024 19:36

Swiddis reviewed Aug 9, 2024

View reviewed changes

Swiddis approved these changes Aug 9, 2024

View reviewed changes

ps48 reviewed Aug 12, 2024

View reviewed changes

public/components/integrations/components/__tests__/testing_constants.ts Outdated Show resolved Hide resolved

ps48 reviewed Aug 12, 2024

View reviewed changes

public/components/integrations/components/added_integration_overview_page.tsx Outdated Show resolved Hide resolved

ps48 reviewed Aug 12, 2024

View reviewed changes

sumukhswamy force-pushed the integrations-mds branch from 39545a4 to 935ff53 Compare August 12, 2024 20:58

RyanL1997 approved these changes Aug 14, 2024

View reviewed changes

ps48 reviewed Aug 16, 2024

View reviewed changes

public/components/integrations/components/__tests__/testing_constants.ts Outdated Show resolved Hide resolved

ps48 reviewed Aug 16, 2024

View reviewed changes

server/adaptors/integrations/integrations_builder.ts Outdated Show resolved Hide resolved

ps48 reviewed Aug 16, 2024

View reviewed changes

server/adaptors/integrations/__test__/builder.test.ts Outdated Show resolved Hide resolved

ps48 reviewed Aug 16, 2024

View reviewed changes

public/components/integrations/components/setup_integration.tsx Outdated Show resolved Hide resolved

sumukhswamy force-pushed the integrations-mds branch 2 times, most recently from 483bdf6 to 964bbe6 Compare August 20, 2024 00:29

sumukhswamy and others added 18 commits August 23, 2024 10:02

Support for MDS in integrations

09409dc

Signed-off-by: sumukhswamy <[email protected]>

added changes for mds support in integrations

2553b6f

Signed-off-by: sumukhswamy <[email protected]>

fixed case for Local cluster

5bf68f4

Signed-off-by: sumukhswamy <[email protected]>

addressed comments for undefined variables

77c697d

Signed-off-by: sumukhswamy <[email protected]>

updated snapshots

f12d658

Signed-off-by: sumukhswamy <[email protected]>

changed the refrences object in integrations instance

ec78b7b

Signed-off-by: sumukhswamy <[email protected]>

addressed pr comments

9b55ab0

Signed-off-by: sumukhswamy <[email protected]>

updated snapshots

24cb4d9

Signed-off-by: sumukhswamy <[email protected]>

Updated event analytics and integrations components

618955f

Signed-off-by: sumukhswamy <[email protected]>

updated tests and conditions

2801fd0

Signed-off-by: sumukhswamy <[email protected]>

updated snapshots

8611d53

Signed-off-by: sumukhswamy <[email protected]>

updated snapshots

97e7c94

Signed-off-by: sumukhswamy <[email protected]>

updated snapshots

98290d7

Signed-off-by: sumukhswamy <[email protected]>

updated the remote and local for snapshots

072e857

Signed-off-by: sumukhswamy <[email protected]>

updated snapshots

99d7521

Signed-off-by: sumukhswamy <[email protected]>

updated snapshots again

21f593b

Signed-off-by: sumukhswamy <[email protected]>

updated snapshots

b373625

Signed-off-by: sumukhswamy <[email protected]>

sumukhswamy force-pushed the integrations-mds branch from 9649ef8 to b373625 Compare August 23, 2024 17:22

sumukhswamy added 3 commits August 25, 2024 13:46

Merge branch 'main' into integrations-mds

58cca98

Signed-off-by: Sumukh Swamy <[email protected]>

updated snapshots

32ec0b5

Signed-off-by: sumukhswamy <[email protected]>

updated missing import

b260b89

Signed-off-by: sumukhswamy <[email protected]>

ps48 merged commit 16e2bd5 into opensearch-project:main Aug 26, 2024
12 of 18 checks passed

opensearch-trigger-bot bot added the backport-failed label Aug 26, 2024

sumukhswamy mentioned this pull request Aug 26, 2024

[Backport 2.X] MDS support in Integrations for observability plugin (#2051) #2096

Merged

6 tasks

RyanL1997 mentioned this pull request Sep 9, 2024

[MDS] Add MDS support for Integration opensearch-project/OpenSearch-Dashboards#8008

Merged

7 tasks

joshuali925 mentioned this pull request Sep 11, 2024

[BUG] Integrations doesn't create objects using selected data source #2159

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[FEATURE] MDS support in Integrations for observability plugin #2051

[FEATURE] MDS support in Integrations for observability plugin #2051

sumukhswamy commented Aug 9, 2024 •

edited

Loading

Swiddis left a comment

Swiddis Aug 9, 2024

sumukhswamy Aug 9, 2024

RyanL1997 Aug 9, 2024

sumukhswamy Aug 12, 2024

Swiddis Aug 9, 2024

sumukhswamy Aug 9, 2024

Swiddis Aug 9, 2024

sumukhswamy Aug 9, 2024

RyanL1997 Aug 9, 2024 •

edited

Loading

Swiddis left a comment

ps48 Aug 12, 2024

opensearch-trigger-bot bot commented Aug 26, 2024

[FEATURE] MDS support in Integrations for observability plugin #2051

[FEATURE] MDS support in Integrations for observability plugin #2051

Conversation

sumukhswamy commented Aug 9, 2024 • edited Loading

Description

Issues Resolved

Check List

Swiddis left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

RyanL1997 Aug 9, 2024 • edited Loading

Choose a reason for hiding this comment

Swiddis left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

opensearch-trigger-bot bot commented Aug 26, 2024

sumukhswamy commented Aug 9, 2024 •

edited

Loading

RyanL1997 Aug 9, 2024 •

edited

Loading