Update Observability Data Sources with Type and Redirection #8492

paulstn · 2024-10-04T18:11:38Z

Description

Added data source type to the data sources page

For S3:

And Prometheus:

Updating redirection to Discover now that Explorer cannot be redirected to

For the flows in associated objects and accelerations that require redirection, all instances can be separated out into two kinds:

for tables and skipping indices: redirection to a 'flint' datasource with the datasource, database, and table name.
for covering indices and materialized views: redirection to the associated index name.
For both kinds of redirection, now the MDS id has to be used to query correctly. All of these flows were updated accordingly.

There is one last kind of redirection with this card:

That previously would redirect to Log Explorer with only the datasource and nothing else. In the current version of Discover, doing this would be impossible, and even alternatives such as opening the DataSet picker modal in discover with the datasource selected isn't feasible. Currently it is set to redirect to Discover without the datasource.

Issues Resolved

Resolves: #8256

Screenshot

Testing the changes

Testing with MDS off:

Screen.Recording.2024-10-15.at.8.06.04.PM.mov

(video cut for length but query eventually completes)

When query:enhancements:enabled is off, every redirection point is disabled as such:

This is done for

assc. obj. table
assc. obj. flyout
acc. table
acc. flyout

Changelog

fix: Updated DataSource Management to redirect to Discover as well as display the type of the DataSource

Check List

All tests pass
- yarn test:jest
- yarn test:jest_integration
New functionality includes testing.
New functionality has been documented.
Update CHANGELOG.md
Commits are signed per the DCO using --signoff

Signed-off-by: Paul Sebastian <[email protected]>

github-actions · 2024-10-04T18:12:10Z

❌ Empty Changelog Section

The Changelog section in your PR description is empty. Please add a valid changelog entry or entries. If you did add a changelog entry, check to make sure that it was not accidentally included inside the comment block in the Changelog section.

codecov · 2024-10-04T18:27:43Z

Codecov Report

Attention: Patch coverage is 77.41935% with 7 lines in your changes missing coverage. Please review.

Project coverage is 60.87%. Comparing base (650166d) to head (d2197e8).
Report is 22 commits behind head on main.

Files with missing lines	Patch %	Lines
..._management/utils/associated_objects_tab_utils.tsx	70.00%	1 Missing and 2 partials ⚠️
...ted_object_management/associated_objects_table.tsx	60.00%	2 Missing ⚠️
...nts/acceleration_management/acceleration_table.tsx	75.00%	1 Missing ⚠️
...nnection_detail/direct_query_connection_detail.tsx	0.00%	1 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #8492   +/-   ##
=======================================
  Coverage   60.87%   60.87%           
=======================================
  Files        3790     3790           
  Lines       90215    90228   +13     
  Branches    14142    14146    +4     
=======================================
+ Hits        54915    54924    +9     
- Misses      31836    31840    +4     
  Partials     3464     3464

Flag	Coverage Δ
Linux_1	`29.15% <14.28%> (?)`
Linux_2	`56.39% <ø> (ø)`
Linux_3	`37.72% <77.41%> (?)`
Linux_4	`29.90% <14.28%> (-0.01%)`	⬇️
Windows_1	`29.17% <14.28%> (-0.01%)`	⬇️
Windows_2	`56.34% <ø> (ø)`
Windows_3	`37.72% <77.41%> (+<0.01%)`	⬆️
Windows_4	`29.90% <14.28%> (-0.01%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

github-actions · 2024-10-15T18:26:49Z

❌ Empty Changelog Section

The Changelog section in your PR description is empty. Please add a valid changelog entry or entries. If you did add a changelog entry, check to make sure that it was not accidentally included inside the comment block in the Changelog section.

…ch-Dashboards into update-obs-datasources-redir-page

Signed-off-by: Paul Sebastian <[email protected]>

github-actions · 2024-10-15T23:11:10Z

❌ Empty Changelog Section

The Changelog section in your PR description is empty. Please add a valid changelog entry or entries. If you did add a changelog entry, check to make sure that it was not accidentally included inside the comment block in the Changelog section.

github-actions · 2024-10-15T23:14:52Z

❌ Entry Too Long

Entry is 129 characters long, which is 29 characters longer than the maximum allowed length of 100 characters. Please revise your entry to be within the maximum length.

github-actions · 2024-10-15T23:33:16Z

❌ Entry Too Long

Entry is 119 characters long, which is 19 characters longer than the maximum allowed length of 100 characters. Please revise your entry to be within the maximum length.

Signed-off-by: Paul Sebastian <[email protected]>

RyanL1997

Hi @paulstn , thanks for taking this on. I just left some comments. In addition, could you also add

Resolve [BUG] Redirection issue for direct query datasource in dashboards management #8256

In the PR description, so that the above issue can be auto resolved by merging this PR.

...mponents/direct_query_data_sources_components/acceleration_management/acceleration_table.tsx

..._data_sources_components/associated_object_management/utils/associated_objects_tab_utils.tsx

...ts/direct_query_data_sources_components/connection_detail/direct_query_connection_detail.tsx

RyanL1997 · 2024-10-16T02:10:26Z

Also, can we record some testing videos, for both MDS enabled and disabled?

Signed-off-by: Paul Sebastian <[email protected]>

sejli

Looks good so far, just some general questions and nitpicks.

sejli · 2024-10-16T20:04:17Z

...direct_query_data_sources_components/acceleration_management/acceleration_details_flyout.tsx

@@ -231,8 +232,9 @@ export const AccelerationDetailsFlyout = (props: AccelerationDetailsFlyoutProps)
  const DiscoverIcon = () => {
    return (
      <EuiButtonEmpty
+        isDisabled={!getUiSettings().get('query:enhancements:enabled')}


nit: data_source_management doesn't take a dependency on the data plugin, but could maybe add it here since we also have DEFAULT_DATA_SOURCE_UI_SETTINGS_ID.

Would there be any benefit to a dependency on the data plugin if we can just check the ui settings? Or were you pointing me to adding this UI setting as a constant to that folder?

Was just pointing you to add it as a constant and export it in the data source management plugin.

sejli · 2024-10-16T20:06:31Z

changelogs/fragments/8492.yml

@@ -0,0 +1,2 @@
+fix:
+- Updated DataSource Management to redirect to Discover as well as display the type of the DataSource ([#8492](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/8492))


Just FYI, the release notes generator requires the CHANGELOG description to be under 100 characters. Not sure if it counts the PR number, but the description itself is at 99 characters.

Oh yeah my original description was over 100 characters and when I shortened it the changelog was accepted so I'd assume that the changebot checking the character count would be consistent with the release notes generator. That's just an assumption though.

...a_sources_components/associated_object_management/associated_objects_details_flyout.test.tsx

sejli · 2024-10-16T20:13:15Z

..._data_sources_components/associated_object_management/utils/associated_objects_tab_utils.tsx

-      datasourceType: DATA_SOURCE_TYPES.S3Glue,
-    },
+  application.navigateToApp('data-explorer', {
+    path: `discover#?_a=(discover:(columns:!(_source),isDirty:!f,sort:!()),metadata:(view:discover))&_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-15m,to:now))&_q=(filters:!(),query:(dataset:(dataSource:(id:'${


nit: Feel like these two functions have a lot of duplicate code, could be worth a try to either combine them or wrap the actual path in another function.

Originally when it was just the 'state' being redirected to explorer through navigateToApp, that made a lot of sense. However, since Discover's url redirection has different forms between redirecting to a flint index vs an index pattern, the differences i would need to put in between both strings would be too much for it to be worth it, as I think it would greatly reduce readability.

..._data_sources_components/associated_object_management/utils/associated_objects_tab_utils.tsx

Signed-off-by: Paul Sebastian <[email protected]>

ashwin-pc

Proxy approval for Ryan since he isnt a maintainer yet but is more familiar with this section of the codebase than me.

* add type to datasources description area Signed-off-by: Paul Sebastian <[email protected]> * complete merge Signed-off-by: Paul Sebastian <[email protected]> * keep imports Signed-off-by: Paul Sebastian <[email protected]> * update redirection to discover Signed-off-by: Paul Sebastian <[email protected]> * let only datasource redirection to explorer use plain discover Signed-off-by: Paul Sebastian <[email protected]> * Changeset file for PR #8492 created/updated * update for case where flint is within default cluster Signed-off-by: Paul Sebastian <[email protected]> * rename helper functions Signed-off-by: Paul Sebastian <[email protected]> * fix tests Signed-off-by: Paul Sebastian <[email protected]> * disabled redir on query enhancements disabled Signed-off-by: Paul Sebastian <[email protected]> * shelter against rison encoding error Signed-off-by: Paul Sebastian <[email protected]> * try catch getting ui settings Signed-off-by: Paul Sebastian <[email protected]> * update tests Signed-off-by: Paul Sebastian <[email protected]> --------- Signed-off-by: Paul Sebastian <[email protected]> Co-authored-by: opensearch-changeset-bot[bot] <154024398+opensearch-changeset-bot[bot]@users.noreply.github.com> (cherry picked from commit 5d74306) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…8673) * add type to datasources description area * complete merge * keep imports * update redirection to discover * let only datasource redirection to explorer use plain discover * Changeset file for PR #8492 created/updated * update for case where flint is within default cluster * rename helper functions * fix tests * disabled redir on query enhancements disabled * shelter against rison encoding error * try catch getting ui settings * update tests --------- (cherry picked from commit 5d74306) Signed-off-by: Paul Sebastian <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: opensearch-changeset-bot[bot] <154024398+opensearch-changeset-bot[bot]@users.noreply.github.com>

…ch-project#8492) * add type to datasources description area Signed-off-by: Paul Sebastian <[email protected]> * complete merge Signed-off-by: Paul Sebastian <[email protected]> * keep imports Signed-off-by: Paul Sebastian <[email protected]> * update redirection to discover Signed-off-by: Paul Sebastian <[email protected]> * let only datasource redirection to explorer use plain discover Signed-off-by: Paul Sebastian <[email protected]> * Changeset file for PR opensearch-project#8492 created/updated * update for case where flint is within default cluster Signed-off-by: Paul Sebastian <[email protected]> * rename helper functions Signed-off-by: Paul Sebastian <[email protected]> * fix tests Signed-off-by: Paul Sebastian <[email protected]> * disabled redir on query enhancements disabled Signed-off-by: Paul Sebastian <[email protected]> * shelter against rison encoding error Signed-off-by: Paul Sebastian <[email protected]> * try catch getting ui settings Signed-off-by: Paul Sebastian <[email protected]> * update tests Signed-off-by: Paul Sebastian <[email protected]> --------- Signed-off-by: Paul Sebastian <[email protected]> Co-authored-by: opensearch-changeset-bot[bot] <154024398+opensearch-changeset-bot[bot]@users.noreply.github.com>

add type to datasources description area

5d242f9

Signed-off-by: Paul Sebastian <[email protected]>

paulstn force-pushed the update-obs-datasources-redir-page branch from c5ea920 to 5d242f9 Compare October 4, 2024 18:11

github-actions bot added the valued-contributor label Oct 4, 2024

github-actions bot added the failed changeset label Oct 4, 2024

paulstn added 5 commits October 15, 2024 18:50

Merge branch 'main' of https://github.com/opensearch-project/OpenSear…

f5a7109

…ch-Dashboards into update-obs-datasources-redir-page

complete merge

d439950

Signed-off-by: Paul Sebastian <[email protected]>

keep imports

e94032f

Signed-off-by: Paul Sebastian <[email protected]>

update redirection to discover

b5b90af

Signed-off-by: Paul Sebastian <[email protected]>

let only datasource redirection to explorer use plain discover

833ebb1

Signed-off-by: Paul Sebastian <[email protected]>

paulstn marked this pull request as ready for review October 15, 2024 23:32

paulstn requested review from ananzh, kavilla, AMoo-Miki, ashwin-pc, joshuarrrr, abbyhu2000, zengyan-amazon, zhongnansu, manasvinibs, ZilongX, Flyingliuhub, curq, bandinib-amzn, SuZhou-Joe and ruanyl as code owners October 15, 2024 23:32

paulstn requested review from virajsanghvi, sejli, joshuali925 and huyaboo as code owners October 15, 2024 23:32

Changeset file for PR opensearch-project#8492 created/updated

76b02cd

github-actions bot removed the failed changeset label Oct 15, 2024

paulstn added 3 commits October 16, 2024 01:12

update for case where flint is within default cluster

f81c9dd

Signed-off-by: Paul Sebastian <[email protected]>

rename helper functions

e616392

Signed-off-by: Paul Sebastian <[email protected]>

fix tests

4af3fdf

Signed-off-by: Paul Sebastian <[email protected]>

RyanL1997 reviewed Oct 16, 2024

View reviewed changes

paulstn added 3 commits October 16, 2024 17:56

disabled redir on query enhancements disabled

8be2f8e

Signed-off-by: Paul Sebastian <[email protected]>

shelter against rison encoding error

4bb86fd

Signed-off-by: Paul Sebastian <[email protected]>

Merge branch 'main' into update-obs-datasources-redir-page

1d2e635

sejli reviewed Oct 16, 2024

View reviewed changes

paulstn added 3 commits October 16, 2024 23:59

try catch getting ui settings

f9fb6a9

Signed-off-by: Paul Sebastian <[email protected]>

update tests

911f22d

Signed-off-by: Paul Sebastian <[email protected]>

Merge branch 'main' into update-obs-datasources-redir-page

d2197e8

RyanL1997 approved these changes Oct 18, 2024

View reviewed changes

sejli approved these changes Oct 21, 2024

View reviewed changes

ashwin-pc approved these changes Oct 21, 2024

View reviewed changes

ashwin-pc added backport 2.x v2.18.0 labels Oct 21, 2024

ashwin-pc merged commit 5d74306 into opensearch-project:main Oct 21, 2024
71 of 72 checks passed

paulstn deleted the update-obs-datasources-redir-page branch October 21, 2024 23:19

opensearch-trigger-bot bot mentioned this pull request Oct 21, 2024

[Backport 2.x] Update Observability Data Sources with Type and Redirection #8673

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update Observability Data Sources with Type and Redirection #8492

Update Observability Data Sources with Type and Redirection #8492

paulstn commented Oct 4, 2024 •

edited

Loading

github-actions bot commented Oct 4, 2024

codecov bot commented Oct 4, 2024 •

edited

Loading

github-actions bot commented Oct 15, 2024

github-actions bot commented Oct 15, 2024

github-actions bot commented Oct 15, 2024

github-actions bot commented Oct 15, 2024

RyanL1997 left a comment

RyanL1997 commented Oct 16, 2024 •

edited

Loading

sejli left a comment

sejli Oct 16, 2024

paulstn Oct 16, 2024

sejli Oct 21, 2024

sejli Oct 16, 2024

paulstn Oct 16, 2024

sejli Oct 16, 2024

paulstn Oct 16, 2024

ashwin-pc left a comment

		@@ -0,0 +1,2 @@
		fix:
		- Updated DataSource Management to redirect to Discover as well as display the type of the DataSource ([#8492](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/8492))

Update Observability Data Sources with Type and Redirection #8492

Update Observability Data Sources with Type and Redirection #8492

Conversation

paulstn commented Oct 4, 2024 • edited Loading

Description

Added data source type to the data sources page

Updating redirection to Discover now that Explorer cannot be redirected to

Issues Resolved

Screenshot

Testing the changes

Changelog

Check List

github-actions bot commented Oct 4, 2024

❌ Empty Changelog Section

codecov bot commented Oct 4, 2024 • edited Loading

Codecov Report

github-actions bot commented Oct 15, 2024

❌ Empty Changelog Section

github-actions bot commented Oct 15, 2024

❌ Empty Changelog Section

github-actions bot commented Oct 15, 2024

❌ Entry Too Long

github-actions bot commented Oct 15, 2024

❌ Entry Too Long

RyanL1997 left a comment

Choose a reason for hiding this comment

RyanL1997 commented Oct 16, 2024 • edited Loading

sejli left a comment

Choose a reason for hiding this comment

sejli Oct 16, 2024

Choose a reason for hiding this comment

paulstn Oct 16, 2024

Choose a reason for hiding this comment

sejli Oct 21, 2024

Choose a reason for hiding this comment

sejli Oct 16, 2024

Choose a reason for hiding this comment

paulstn Oct 16, 2024

Choose a reason for hiding this comment

sejli Oct 16, 2024

Choose a reason for hiding this comment

paulstn Oct 16, 2024

Choose a reason for hiding this comment

ashwin-pc left a comment

Choose a reason for hiding this comment

paulstn commented Oct 4, 2024 •

edited

Loading

codecov bot commented Oct 4, 2024 •

edited

Loading

RyanL1997 commented Oct 16, 2024 •

edited

Loading