[MDS] Support for Timeline #6385

huyaboo · 2024-04-09T21:25:40Z

Description

Previously, MDS did not support Timeline. This PR adds a new field data_source_name similar to #5927 that enables users to specify a unique data source in which to query data from. This allows users to query from multiple indices and multiple data sources via the .opensearch() function.

Issues Resolved

Closes #6009 and is related to #6410

Screenshot

Screen.Recording.2024-04-10.at.3.27.37.PM.mov

Testing the changes

If MDS is disabled:

Adding a data_source_name will throw an error server-side and a toast message will be shown indicating that MDS is disabled
Local Cluster queries function as usual

If MDS is enabled:

Create a data source(s): In the recording, Source A and Source B were connected
Navigate to Timeline visualization
Add data_source_name="<data source name>" at the end of the .opensearch() expression. Order matters with Timeline expressions as arguments can be named or unnamed.
Assuming the data source has the index and data, visualization will update

Check List

All tests pass
- yarn test:jest
- yarn test:jest_integration
New functionality includes testing.
New functionality has been documented.
Update CHANGELOG.md
Commits are signed per the DCO using --signoff

bandinib-amzn · 2024-04-11T20:40:54Z

Before rebase:
All checks have passed
64 successful, 1 skipped, and 1 neutral checks

CHANGELOG.md

src/plugins/vis_type_timeline/server/lib/fetch_data_source_id.test.ts

bandinib-amzn · 2024-04-11T20:48:20Z

src/plugins/vis_type_timeline/server/lib/fetch_data_source_id.ts

+import { getDataSourceEnabled } from './services';
+import { OpenSearchFunctionConfig } from '../types';
+
+export const fetchDataSourceIdByName = async (


Didn't we define similar function for vega? In vega, we are fetching data source id by name, right? Correct me if I'm missing something.

Yes but Vega fetches data source id client-side while Timeline fetches data source id information server-side. I agree that this is something that needs to be refactored into one utility function in data source plugin and I cut a new issue #6417

Is there an ETA to refactor this function? Just asking because if we don't actively track and then we will keep using these two separate function when they are performing same functionality.

I'm not sure if this is a huge priority at the moment. Most plugins are not blocked by this refactor, but once the UI components are finalized (these are blocking issues), I will pick up this task. This should be started within this sprint or at the most, next sprint.

bandinib-amzn · 2024-04-11T20:49:01Z

src/plugins/vis_type_timeline/server/lib/fetch_data_source_id.ts

+      throw new Error(
+        `Expected exactly 1 result for data_source_name "${config.data_source_name}" but got ${possibleDataSourceIds.length} results`


Is this user facing error?

Yes. Timeline parses the expression server-side and throws an Error if the expression is incorrectly written. This will behave the same way.

src/plugins/vis_type_timeline/server/series_functions/opensearch/index.js

bandinib-amzn · 2024-04-11T20:51:44Z

src/plugins/vis_type_timeline/server/types.ts

+  /**
+   * @deprecated The interval picker should be used instead
+   */
+  interval: string | null;


what is meaning of comment? It is deprecated and we are still using it?

In the Timeline expression, there is an option for the user to specify an interval. However, as noted in the opensearch function args, there is a picker react component that users should use instead. I assume interval is kept as to not break any existing visualization so I marked it as deprecated.

OpenSearch-Dashboards/src/plugins/vis_type_timeline/server/series_functions/opensearch/index.js

Lines 109 to 115 in 458501f

{

name: 'interval', // You really shouldn't use this, use the interval picker instead

types: ['string', 'null'],

help: i18n.translate('timeline.help.functions.opensearch.args.intervalHelpText', {

defaultMessage: `**DO NOT USE THIS**. It's fun for debugging fit functions, but you really should use the interval picker`,

}),

},

zhongnansu · 2024-04-11T22:29:48Z

src/plugins/vis_type_timeline/server/types.ts

+  /**
+   * @deprecated The interval picker should be used instead
+   */
+  interval: string | null;


bandinib-amzn · 2024-04-15T21:15:28Z

src/plugins/vis_type_timeline/server/lib/fetch_data_source_id.ts

+import { getDataSourceEnabled } from './services';
+import { OpenSearchFunctionConfig } from '../types';
+
+export const fetchDataSourceIdByName = async (


Is there an ETA to refactor this function? Just asking because if we don't actively track and then we will keep using these two separate function when they are performing same functionality.

bandinib-amzn · 2024-04-15T21:18:11Z

src/plugins/vis_type_timeline/server/lib/fetch_data_source_id.ts

+) => {
+  if (config.data_source_name) {
+    if (!getDataSourceEnabled().enabled) {
+      throw new Error('To query from multiple data sources, first enable the data sources feature');


nit: shouldn't be enable the data source feature?

Signed-off-by: Huy Nguyen <[email protected]>

bandinib-amzn · 2024-04-15T23:20:57Z

src/plugins/vis_type_timeline/server/lib/fetch_data_source_id.test.ts

+    await expect(
+      fetchDataSourceIdByName({ ...config, data_source_name: 'Some Data Source' }, client)
+    ).rejects.toThrowError(
+      'To query from multiple data sources, first enable the data sources feature'


this also needs to fix as well, right? as per this commit a274600

Signed-off-by: Huy Nguyen <[email protected]>

BionIT · 2024-04-16T04:41:15Z

src/plugins/vis_type_timeline/server/handlers/chain_runner.js

@@ -47,7 +47,9 @@ export default function chainRunner(tlConfig) {
  let sheet;

  function throwWithCell(cell, exception) {
-    throw new Error(' in cell #' + (cell + 1) + ': ' + exception.message);
+    const e = new Error(exception.message);
+    e.name = `Expression parse error in cell #${cell + 1}`;


does the name field matter?

Yes: server-side error messages are propagated to the client and the toast error message will show (this is existing behavior and UX is aligned). This is needed to make the toast more readable (stack information is kept hidden from client).

BionIT · 2024-04-16T04:46:33Z

src/plugins/vis_type_timeline/server/lib/fetch_data_source_id.ts

+  config: OpenSearchFunctionConfig,
+  client: SavedObjectsClientContract
+) => {
+  if (config.data_source_name) {


can we use early return https://github.com/opensearch-project/OpenSearch-Dashboards/blob/main/DEVELOPER_GUIDE.md#returnthrow-early-from-functions

Signed-off-by: Huy Nguyen <[email protected]>

* Add MDS support to Timeline Signed-off-by: Huy Nguyen <[email protected]> * Refactor to function and add unit tests Signed-off-by: Huy Nguyen <[email protected]> * Fix typo in args Signed-off-by: Huy Nguyen <[email protected]> * Refactor build request to pass unit tests Signed-off-by: Huy Nguyen <[email protected]> * Add to CHANGELOG Signed-off-by: Huy Nguyen <[email protected]> * Refactor error messages + address comments Signed-off-by: Huy Nguyen <[email protected]> * Fix ut Signed-off-by: Huy Nguyen <[email protected]> * Change to data source feature Signed-off-by: Huy Nguyen <[email protected]> * Fix UT Signed-off-by: Huy Nguyen <[email protected]> * Address comments Signed-off-by: Huy Nguyen <[email protected]> --------- Signed-off-by: Huy Nguyen <[email protected]> (cherry picked from commit 73e5d78) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> # Conflicts: # CHANGELOG.md

* Add MDS support to Timeline Signed-off-by: Huy Nguyen <[email protected]> * Refactor to function and add unit tests Signed-off-by: Huy Nguyen <[email protected]> * Fix typo in args Signed-off-by: Huy Nguyen <[email protected]> * Refactor build request to pass unit tests Signed-off-by: Huy Nguyen <[email protected]> * Add to CHANGELOG Signed-off-by: Huy Nguyen <[email protected]> * Refactor error messages + address comments Signed-off-by: Huy Nguyen <[email protected]> * Fix ut Signed-off-by: Huy Nguyen <[email protected]> * Change to data source feature Signed-off-by: Huy Nguyen <[email protected]> * Fix UT Signed-off-by: Huy Nguyen <[email protected]> * Address comments Signed-off-by: Huy Nguyen <[email protected]> --------- Signed-off-by: Huy Nguyen <[email protected]> (cherry picked from commit 73e5d78) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> # Conflicts: # CHANGELOG.md Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: ZilongX <[email protected]>

github-actions bot added the repeat-contributor label Apr 9, 2024

huyaboo force-pushed the timeline branch from eecfd89 to 54febd4 Compare April 10, 2024 16:56

huyaboo marked this pull request as ready for review April 10, 2024 17:24

huyaboo requested review from ananzh, kavilla, AMoo-Miki, ashwin-pc, joshuarrrr, abbyhu2000, zengyan-amazon, zhongnansu, manasvinibs, ZilongX, Flyingliuhub, curq, bandinib-amzn, SuZhou-Joe, ruanyl, BionIT and xinruiba as code owners April 10, 2024 17:24

bandinib-amzn force-pushed the timeline branch from 317a786 to 458501f Compare April 11, 2024 20:41

bandinib-amzn reviewed Apr 11, 2024

View reviewed changes

zhongnansu previously approved these changes Apr 11, 2024

View reviewed changes

huyaboo dismissed zhongnansu’s stale review via eb50cd4 April 12, 2024 00:45

huyaboo force-pushed the timeline branch from 458501f to eb50cd4 Compare April 12, 2024 00:45

zhongnansu force-pushed the timeline branch from a82cb3c to 7594e96 Compare April 12, 2024 19:15

seraphjiang mentioned this pull request Apr 13, 2024

[Feature][MDS] Add data source support to Timeline visualization #6009

Closed

bandinib-amzn reviewed Apr 15, 2024

View reviewed changes

huyaboo force-pushed the timeline branch from 7594e96 to a693dbb Compare April 15, 2024 22:15

huyaboo added 5 commits April 15, 2024 22:55

Refactor build request to pass unit tests

140a1c6

Signed-off-by: Huy Nguyen <[email protected]>

Add to CHANGELOG

4e13978

Signed-off-by: Huy Nguyen <[email protected]>

Refactor error messages + address comments

28edd7f

Signed-off-by: Huy Nguyen <[email protected]>

Fix ut

b4411b3

Signed-off-by: Huy Nguyen <[email protected]>

Change to data source feature

a274600

Signed-off-by: Huy Nguyen <[email protected]>

huyaboo force-pushed the timeline branch from d9ea50c to a274600 Compare April 15, 2024 22:55

bandinib-amzn added backport 2.x v2.15.0 labels Apr 15, 2024

bandinib-amzn reviewed Apr 15, 2024

View reviewed changes

Fix UT

e8a41b4

Signed-off-by: Huy Nguyen <[email protected]>

bandinib-amzn previously approved these changes Apr 16, 2024

View reviewed changes

huyaboo added 2 commits April 15, 2024 17:21

Merge branch 'main' into timeline

6146194

Merge branch 'main' into timeline

b3ee293

BionIT reviewed Apr 16, 2024

View reviewed changes

huyaboo added 2 commits April 16, 2024 11:06

Merge branch 'main' into timeline

cff410a

Address comments

541cbdd

Signed-off-by: Huy Nguyen <[email protected]>

huyaboo dismissed bandinib-amzn’s stale review via 541cbdd April 16, 2024 18:14

BionIT approved these changes Apr 16, 2024

View reviewed changes

Merge branch 'main' into timeline

503e77b

bandinib-amzn approved these changes Apr 16, 2024

View reviewed changes

bandinib-amzn merged commit 73e5d78 into opensearch-project:main Apr 16, 2024
130 checks passed

opensearch-trigger-bot bot mentioned this pull request Apr 16, 2024

[Backport 2.x] [MDS] Support for Timeline #6493

Merged

huyaboo deleted the timeline branch April 18, 2024 00:15

BionIT mentioned this pull request Jun 3, 2024

[Meta] [2.15] Security, bug fix and enhancemanet for dashboards anywhere #6600

Closed

45 tasks

zhongnansu added the multiple datasource multiple datasource project label Jun 6, 2024

This was referenced Jun 10, 2024

Add missing change log for 2.15 MDS PRs #6986

Merged

Add Release note for 2.15.0 #7006

Merged

zhongnansu added the enhancement New feature or request label Jun 12, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[MDS] Support for Timeline #6385

[MDS] Support for Timeline #6385

huyaboo commented Apr 9, 2024 •

edited

Loading

bandinib-amzn commented Apr 11, 2024

bandinib-amzn Apr 11, 2024

huyaboo Apr 11, 2024

bandinib-amzn Apr 15, 2024

huyaboo Apr 15, 2024

bandinib-amzn Apr 11, 2024

huyaboo Apr 11, 2024

bandinib-amzn Apr 11, 2024

zhongnansu Apr 11, 2024

huyaboo Apr 11, 2024 •

edited

Loading

zhongnansu Apr 11, 2024

bandinib-amzn Apr 15, 2024

bandinib-amzn Apr 15, 2024

huyaboo Apr 15, 2024

bandinib-amzn Apr 15, 2024

BionIT Apr 16, 2024

huyaboo Apr 16, 2024 •

edited

Loading

BionIT Apr 16, 2024

huyaboo Apr 16, 2024

		throw new Error(
		`Expected exactly 1 result for data_source_name "${config.data_source_name}" but got ${possibleDataSourceIds.length} results`

	{
	name: 'interval', // You really shouldn't use this, use the interval picker instead
	types: ['string', 'null'],
	help: i18n.translate('timeline.help.functions.opensearch.args.intervalHelpText', {
	defaultMessage: `DO NOT USE THIS. It's fun for debugging fit functions, but you really should use the interval picker`,
	}),
	},

[MDS] Support for Timeline #6385

[MDS] Support for Timeline #6385

Conversation

huyaboo commented Apr 9, 2024 • edited Loading

Description

Issues Resolved

Screenshot

Testing the changes

Check List

bandinib-amzn commented Apr 11, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

huyaboo Apr 11, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

huyaboo Apr 16, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

huyaboo commented Apr 9, 2024 •

edited

Loading

huyaboo Apr 11, 2024 •

edited

Loading

huyaboo Apr 16, 2024 •

edited

Loading