Add otel metrics #1314

kavithacm · 2023-12-20T18:13:21Z

Description

Adding Otel metrics as a data source and also providing visualizations for it

Issues Resolved

[List any issues this PR will resolve]

Check List

New functionality includes testing.
- All tests pass, including unit test, integration test and doctest
New functionality has been documented.
- New functionality has javadoc added
- New functionality has user manual doc added
Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

public/components/metrics/redux/slices/metrics_slice.ts

public/components/metrics/sidebar/data_source_picker.tsx

public/components/metrics/sidebar/index_picker.tsx

pjfitzgibbons · 2023-12-20T22:14:12Z

public/components/metrics/sidebar/metrics_accordion.tsx

@@ -34,7 +34,7 @@ export const MetricsAccordion = (props: IMetricNameProps) => {
      {metricsList.length > 100 && <p>Use search bar to focus listed Metrics.</p>}
      <ul className="metricsList">
        {metricsList.slice(0, 100).map((metric: any) => (
-          <li key={metric.id} className="metricsListContainer" data-test-subj={dataTestSubj}>
+          <li key={metric?.id} className="metricsListContainer" data-test-subj={dataTestSubj}>


This is technically a dangerous safe-operator. We need to understand why the list was iterated with null/undefineds?

pjfitzgibbons · 2023-12-20T22:15:12Z

public/components/metrics/sidebar/sidebar.tsx

    dispatch(deSelectMetric(metric));
  };

  return (
    <I18nProvider>
      <section className="sidebarHeight">
+        <DataSourcePicker


You'll need to look at new Sidebar. Largely, this component, with no attributes, is what will go here. None of the change above will be necessary -- all the functionality is in metricSlice!

public/components/metrics/top_menu/metrics_export_panel.tsx

public/components/metrics/top_menu/top_menu.tsx

public/services/saved_objects/event_analytics/saved_objects.ts

ps48 · 2024-01-04T21:52:28Z

@kavithacm It would be great if you could add some more context about what is being added as a feature in this PR. Maybe a link to github issue, a design RFC or can just attach a demo video.

common/constants/shared.ts

common/types/metrics.ts

public/components/common/query_utils/index.ts

public/components/custom_panels/helpers/utils.tsx

public/components/custom_panels/panel_modules/panel_grid/panel_grid_so.tsx

Signed-off-by: Kavitha Conjeevaram Mohan <[email protected]>

ps48 · 2024-01-11T18:39:42Z

common/types/metrics.ts

@@ -13,10 +14,15 @@ export interface MetricType extends VisualizationType {
  w: number;
  h: number;
  query: {
-    type: 'savedCustomMetric' | 'prometheusMetric';
+    type: 'savedCustomMetric' | 'prometheusMetric' | typeof OTEL_METRIC_SUBTYPE;


nit. Can we move all the three in a single type

moved it, will update now

ps48 · 2024-01-11T18:40:27Z

public/components/common/query_utils/index.ts

-import { isEmpty } from 'lodash';
-import { SearchMetaData } from '../../event_analytics/redux/slices/search_meta_data_slice';
+import _, { isEmpty } from 'lodash';
+import { SearchMetaData } from 'public/components/event_analytics/redux/slices/search_meta_data_slice';


should this be a relative path for import?

not sure how this changed, reverted it back

ps48 · 2024-01-11T18:42:19Z

public/components/custom_panels/helpers/__tests__/utils.test.tsx

@@ -21,13 +21,21 @@ import {
 } from '../../../../../test/panels_constants';
 import {
  displayVisualization,
+  fetchAggregatedBinCount,
+  fetchSampleOTDocument,
+  // fetchSampleOTDocument,


can remove the commented line

Signed-off-by: Kavitha Conjeevaram Mohan <[email protected]>

ps48 · 2024-01-11T20:51:43Z

public/components/custom_panels/helpers/__tests__/utils.test.tsx

@@ -29,6 +29,8 @@ import {
 } from '../utils';
 import { convertDateTime } from '../../../common/query_utils';

+jest.setTimeout(60000);


let's remove this

ps48 · 2024-01-11T20:52:22Z

common/types/metrics.ts

@@ -4,6 +4,8 @@
 */

 import { VisualizationType } from './custom_panels';
+import { OTEL_METRIC_SUBTYPE } from '../constants/shared';
+type AllMetricTypes = 'savedCustomMetric' | 'prometheusMetric' | typeof OTEL_METRIC_SUBTYPE;


may be MetricTypes

Signed-off-by: Kavitha Conjeevaram Mohan <[email protected]>

ps48 · 2024-01-12T00:50:24Z

public/components/visualizations/saved_object_visualization.tsx

@@ -2,6 +2,7 @@
 * Copyright OpenSearch Contributors
 * SPDX-License-Identifier: Apache-2.0
 */
+/* eslint-disable @typescript-eslint/no-unused-vars */


I think this was skipped for lint check.

ps48

Can you please update the snapshots or rebase from main: #1352 before merging this PR? We want to see the unit tests green at the least. The snapshots are changed due to an upstream OSD change.

opensearch-trigger-bot · 2024-01-16T18:01:16Z

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/dashboards-observability/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/dashboards-observability/backport-2.x
# Create a new branch
git switch --create backport/backport-1314-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 f2a66a9a1c2b9dbe50b8218864e34ca570188906
# Push it to GitHub
git push --set-upstream origin backport/backport-1314-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/dashboards-observability/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-1314-to-2.x.

* List all available otel docs Signed-off-by: Kavitha Conjeevaram Mohan <[email protected]> * Upgraded Signed-off-by: Kavitha Conjeevaram Mohan <[email protected]> * fetching bin count Signed-off-by: Kavitha Conjeevaram Mohan <[email protected]> * Histogram working Signed-off-by: Kavitha Conjeevaram Mohan <[email protected]> * Cleaned up code Signed-off-by: Kavitha Conjeevaram Mohan <[email protected]> * Viz working with real time data Signed-off-by: Kavitha Conjeevaram Mohan <[email protected]> * Successfully saving to dashboards Signed-off-by: Kavitha Conjeevaram Mohan <[email protected]> * Fixed sidebar jest test Signed-off-by: Kavitha Conjeevaram Mohan <[email protected]> * View otel metric in dashboards Signed-off-by: Kavitha Conjeevaram Mohan <[email protected]> * update snapshot and remove commented lines Signed-off-by: Kavitha Conjeevaram Mohan <[email protected]> * Otel metrics working end to end Signed-off-by: Kavitha Conjeevaram Mohan <[email protected]> * Added support for OSD dashboards Signed-off-by: Kavitha Conjeevaram Mohan <[email protected]> * Remove commented code Signed-off-by: Kavitha Conjeevaram Mohan <[email protected]> * Fixed jest test and removed console comments Signed-off-by: Kavitha Conjeevaram Mohan <[email protected]> * Added test and addressed comments Signed-off-by: Kavitha Conjeevaram Mohan <[email protected]> * minor Signed-off-by: Kavitha Conjeevaram Mohan <[email protected]> * fix es lint Signed-off-by: Kavitha Conjeevaram Mohan <[email protected]> * comments Signed-off-by: Kavitha Conjeevaram Mohan <[email protected]> * change to get request Signed-off-by: Kavitha Conjeevaram Mohan <[email protected]> * remove missed commented line Signed-off-by: Kavitha Conjeevaram Mohan <[email protected]> --------- Signed-off-by: Kavitha Conjeevaram Mohan <[email protected]>

…oject#1314 Signed-off-by: sumukhswamy <[email protected]>

* resolved conflicts and added the manual cherry pick for #1314 Signed-off-by: sumukhswamy <[email protected]> * removed duplicates Signed-off-by: sumukhswamy <[email protected]> * updated snapshots Signed-off-by: sumukhswamy <[email protected]> --------- Signed-off-by: sumukhswamy <[email protected]> Co-authored-by: Kavitha Conjeevaram Mohan <[email protected]>

pjfitzgibbons reviewed Dec 20, 2023

View reviewed changes

kavithacm force-pushed the data-prepper-metrics branch 2 times, most recently from ad0b9ce to 203b6e9 Compare December 31, 2023 09:31

kavithacm added the backport 2.x label Dec 31, 2023

kavithacm marked this pull request as ready for review December 31, 2023 09:45

kavithacm requested review from ps48, derek-ho, joshuali925, dai-chen, YANG-DB, rupal-bq, mengweieric, vamsimanohar, Swiddis, penghuo, seankao-az, anirudha, paulstn and sumukhswamy as code owners December 31, 2023 09:45