Skip to content

Commit

Permalink
[8.x] [Investigation app] add entities route and investigation Contex…
Browse files Browse the repository at this point in the history
…tual Insight (#194432) (#195158)

# Backport

This will backport the following commits from `main` to `8.x`:
- [[Investigation app] add entities route and investigation Contextual
Insight (#194432)](#194432)

<!--- Backport version: 8.9.8 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Dominique
Clarke","email":"[email protected]"},"sourceCommit":{"committedDate":"2024-10-04T17:58:28Z","message":"[Investigation
app] add entities route and investigation Contextual Insight
(#194432)\n\n## Summary\r\n\r\nAdds a route that can be used to fetch
entities related to an\r\ninvestigation.\r\n\r\nThe route fetches
associated entities by service name, host name, or\r\ncontainer id. It
then identifies the associated indices and datastreams.\r\n\r\nThe
discovered entities are passed to the contextual insight to
inform\r\nthe
LLM.\r\n\r\n\r\n![image](https://github.com/user-attachments/assets/855a8d68-b039-4557-ba23-5661cd961021)\r\n\r\nThis
PR represents the first step in developing an AI-informed\r\nhypothesis
at the beginning of the investigation. Over time, further\r\ninsights
will be provided to the LLM to deepen it's investigative\r\nanalysis and
propose a more helpful root cause hypothesis.\r\n\r\n###
Testing\r\n\r\n1. Create some APM data. I'm using the otel demo and
triggering a\r\nfailure via the flagd service. Since this is in flux,
you can reach out\r\nto me about this workflow. However, you can also
create APM data via\r\n`synth-trace`.\r\n2. Create an custom threshold
rule that you expect to trigger an alert.\r\nI created mine to using
`http.response.status_code: 500 /\r\nhttp.response.status_code : *` and
set a low threshold base on the\r\namount of failures in my current test
data. Be sure to also group the\r\nalert by `service.name`\r\n3. Wait
for the alert to fire, then visit the alert details page and\r\nstart an
investigation\r\n4. notice the contextual insight. Expand it to see more
information\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine
<[email protected]>","sha":"e4bb435b48560852b37e4de54fb9c05cf5a7f3b1","branchLabelMapping":{"^v9.0.0$":"main","^v8.16.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","v9.0.0","backport:prev-minor","ci:project-deploy-observability","Team:obs-ux-management","v8.16.0"],"number":194432,"url":"https://github.com/elastic/kibana/pull/194432","mergeCommit":{"message":"[Investigation
app] add entities route and investigation Contextual Insight
(#194432)\n\n## Summary\r\n\r\nAdds a route that can be used to fetch
entities related to an\r\ninvestigation.\r\n\r\nThe route fetches
associated entities by service name, host name, or\r\ncontainer id. It
then identifies the associated indices and datastreams.\r\n\r\nThe
discovered entities are passed to the contextual insight to
inform\r\nthe
LLM.\r\n\r\n\r\n![image](https://github.com/user-attachments/assets/855a8d68-b039-4557-ba23-5661cd961021)\r\n\r\nThis
PR represents the first step in developing an AI-informed\r\nhypothesis
at the beginning of the investigation. Over time, further\r\ninsights
will be provided to the LLM to deepen it's investigative\r\nanalysis and
propose a more helpful root cause hypothesis.\r\n\r\n###
Testing\r\n\r\n1. Create some APM data. I'm using the otel demo and
triggering a\r\nfailure via the flagd service. Since this is in flux,
you can reach out\r\nto me about this workflow. However, you can also
create APM data via\r\n`synth-trace`.\r\n2. Create an custom threshold
rule that you expect to trigger an alert.\r\nI created mine to using
`http.response.status_code: 500 /\r\nhttp.response.status_code : *` and
set a low threshold base on the\r\namount of failures in my current test
data. Be sure to also group the\r\nalert by `service.name`\r\n3. Wait
for the alert to fire, then visit the alert details page and\r\nstart an
investigation\r\n4. notice the contextual insight. Expand it to see more
information\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine
<[email protected]>","sha":"e4bb435b48560852b37e4de54fb9c05cf5a7f3b1"}},"sourceBranch":"main","suggestedTargetBranches":["8.x"],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","labelRegex":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/194432","number":194432,"mergeCommit":{"message":"[Investigation
app] add entities route and investigation Contextual Insight
(#194432)\n\n## Summary\r\n\r\nAdds a route that can be used to fetch
entities related to an\r\ninvestigation.\r\n\r\nThe route fetches
associated entities by service name, host name, or\r\ncontainer id. It
then identifies the associated indices and datastreams.\r\n\r\nThe
discovered entities are passed to the contextual insight to
inform\r\nthe
LLM.\r\n\r\n\r\n![image](https://github.com/user-attachments/assets/855a8d68-b039-4557-ba23-5661cd961021)\r\n\r\nThis
PR represents the first step in developing an AI-informed\r\nhypothesis
at the beginning of the investigation. Over time, further\r\ninsights
will be provided to the LLM to deepen it's investigative\r\nanalysis and
propose a more helpful root cause hypothesis.\r\n\r\n###
Testing\r\n\r\n1. Create some APM data. I'm using the otel demo and
triggering a\r\nfailure via the flagd service. Since this is in flux,
you can reach out\r\nto me about this workflow. However, you can also
create APM data via\r\n`synth-trace`.\r\n2. Create an custom threshold
rule that you expect to trigger an alert.\r\nI created mine to using
`http.response.status_code: 500 /\r\nhttp.response.status_code : *` and
set a low threshold base on the\r\namount of failures in my current test
data. Be sure to also group the\r\nalert by `service.name`\r\n3. Wait
for the alert to fire, then visit the alert details page and\r\nstart an
investigation\r\n4. notice the contextual insight. Expand it to see more
information\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine
<[email protected]>","sha":"e4bb435b48560852b37e4de54fb9c05cf5a7f3b1"}},{"branch":"8.x","label":"v8.16.0","labelRegex":"^v8.16.0$","isSourceBranch":false,"state":"NOT_CREATED"}]}]
BACKPORT-->

Co-authored-by: Rickyanto Ang <[email protected]>
  • Loading branch information
dominiqueclarke and animehart authored Oct 7, 2024
1 parent abc351f commit c099f33
Show file tree
Hide file tree
Showing 25 changed files with 1,485 additions and 53 deletions.
48 changes: 48 additions & 0 deletions packages/kbn-investigation-shared/src/rest_specs/entity.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the "Elastic License
* 2.0", the "GNU Affero General Public License v3.0 only", and the "Server Side
* Public License v 1"; you may not use this file except in compliance with, at
* your election, the "Elastic License 2.0", the "GNU Affero General Public
* License v3.0 only", or the "Server Side Public License, v 1".
*/

import { z } from '@kbn/zod';

const metricsSchema = z.object({
failedTransactionRate: z.number().optional(),
latency: z.number().optional(),
throughput: z.number().optional(),
logErrorRate: z.number().optional(),
logRate: z.number().optional(),
});

const entitySchema = z.object({
id: z.string(),
definitionId: z.string(),
definitionVersion: z.string(),
displayName: z.string(),
firstSeenTimestamp: z.string(),
lastSeenTimestamp: z.string(),
identityFields: z.array(z.string()),
schemaVersion: z.string(),
type: z.string(),
metrics: metricsSchema,
});

const entitySourceSchema = z.object({
dataStream: z.string().optional(),
});

const entityWithSourceSchema = z.intersection(
entitySchema,
z.object({
sources: z.array(entitySourceSchema),
})
);

type EntityWithSource = z.output<typeof entityWithSourceSchema>;
type EntitySource = z.output<typeof entitySourceSchema>;

export { entitySchema, entityWithSourceSchema };
export type { EntityWithSource, EntitySource };
34 changes: 34 additions & 0 deletions packages/kbn-investigation-shared/src/rest_specs/get_entities.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the "Elastic License
* 2.0", the "GNU Affero General Public License v3.0 only", and the "Server Side
* Public License v 1"; you may not use this file except in compliance with, at
* your election, the "Elastic License 2.0", the "GNU Affero General Public
* License v3.0 only", or the "Server Side Public License, v 1".
*/

import { z } from '@kbn/zod';
import { entityWithSourceSchema } from './entity';

const getEntitiesParamsSchema = z
.object({
query: z
.object({
'service.name': z.string(),
'service.environment': z.string(),
'host.name': z.string(),
'container.id': z.string(),
})
.partial(),
})
.partial();

const getEntitiesResponseSchema = z.object({
entities: z.array(entityWithSourceSchema),
});

type GetEntitiesParams = z.infer<typeof getEntitiesParamsSchema.shape.query>;
type GetEntitiesResponse = z.output<typeof getEntitiesResponseSchema>;

export { getEntitiesParamsSchema, getEntitiesResponseSchema };
export type { GetEntitiesParams, GetEntitiesResponse };
4 changes: 4 additions & 0 deletions packages/kbn-investigation-shared/src/rest_specs/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,8 @@ export type * from './update_item';
export type * from './update_note';
export type * from './event';
export type * from './get_events';
export type * from './entity';
export type * from './get_entities';

export * from './create';
export * from './create_item';
Expand All @@ -48,3 +50,5 @@ export * from './update_item';
export * from './update_note';
export * from './event';
export * from './get_events';
export * from './entity';
export * from './get_entities';
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,6 @@
"configPath": ["xpack", "investigateApp"],
"requiredPlugins": [
"investigate",
"observabilityAIAssistant",
"observabilityShared",
"lens",
"dataViews",
Expand All @@ -28,7 +27,7 @@
"kibanaReact",
"kibanaUtils",
],
"optionalPlugins": [],
"optionalPlugins": ["observabilityAIAssistant"],
"extraPublicDirs": []
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,16 @@ export const investigationKeys = {
[...investigationKeys.detail(investigationId), 'notes'] as const,
detailItems: (investigationId: string) =>
[...investigationKeys.detail(investigationId), 'items'] as const,
entities: ({
investigationId,
...params
}: {
investigationId: string;
serviceName?: string;
serviceEnvironment?: string;
hostName?: string;
containerId?: string;
}) => [...investigationKeys.detail(investigationId), 'entities', params] as const,
};

export type InvestigationKeys = typeof investigationKeys;
Original file line number Diff line number Diff line change
Expand Up @@ -7,10 +7,11 @@

import { useQuery } from '@tanstack/react-query';
import { BASE_RAC_ALERTS_API_PATH, EcsFieldsResponse } from '@kbn/rule-registry-plugin/common';
import { useKibana } from '../../../hooks/use_kibana';
import { type GetInvestigationResponse, alertOriginSchema } from '@kbn/investigation-shared';
import { useKibana } from './use_kibana';

export interface AlertParams {
id?: string;
export interface UseFetchAlertParams {
investigation?: GetInvestigationResponse;
}

export interface UseFetchAlertResponse {
Expand All @@ -22,20 +23,22 @@ export interface UseFetchAlertResponse {
data: EcsFieldsResponse | undefined | null;
}

export function useFetchAlert({ id }: AlertParams): UseFetchAlertResponse {
export function useFetchAlert({ investigation }: UseFetchAlertParams): UseFetchAlertResponse {
const {
core: {
http,
notifications: { toasts },
},
} = useKibana();
const alertOriginInvestigation = alertOriginSchema.safeParse(investigation?.origin);
const alertId = alertOriginInvestigation.success ? alertOriginInvestigation.data.id : undefined;

const { isInitialLoading, isLoading, isError, isSuccess, isRefetching, data } = useQuery({
queryKey: ['fetchAlert', id],
queryKey: ['fetchAlert', investigation?.id],
queryFn: async ({ signal }) => {
return await http.get<EcsFieldsResponse>(BASE_RAC_ALERTS_API_PATH, {
query: {
id,
id: alertId,
},
signal,
});
Expand All @@ -47,7 +50,7 @@ export function useFetchAlert({ id }: AlertParams): UseFetchAlertResponse {
title: 'Something went wrong while fetching alert',
});
},
enabled: Boolean(id),
enabled: Boolean(alertId),
});

return {
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,67 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/

import { useQuery } from '@tanstack/react-query';
import { GetEntitiesResponse } from '@kbn/investigation-shared';
import { useKibana } from './use_kibana';
import { investigationKeys } from './query_key_factory';

export interface EntityParams {
investigationId: string;
serviceName?: string;
serviceEnvironment?: string;
hostName?: string;
containerId?: string;
}

export function useFetchEntities({
investigationId,
serviceName,
serviceEnvironment,
hostName,
containerId,
}: EntityParams) {
const {
core: { http },
} = useKibana();

const { isInitialLoading, isLoading, isError, isSuccess, isRefetching, data } = useQuery({
queryKey: investigationKeys.entities({
investigationId,
serviceName,
serviceEnvironment,
hostName,
containerId,
}),
queryFn: async ({ signal }) => {
return await http.get<GetEntitiesResponse>('/api/observability/investigation/entities', {
query: {
'service.name': serviceName,
'service.environment': serviceEnvironment,
'host.name': hostName,
'container.id': containerId,
},
version: '2023-10-31',
signal,
});
},
refetchOnWindowFocus: false,
onError: (error: Error) => {
// ignore error
},
enabled: Boolean(investigationId && (serviceName || hostName || containerId)),
});

return {
data,
isInitialLoading,
isLoading,
isRefetching,
isSuccess,
isError,
};
}
Original file line number Diff line number Diff line change
Expand Up @@ -5,13 +5,12 @@
* 2.0.
*/

import { alertOriginSchema } from '@kbn/investigation-shared';
import { ALERT_REASON, ALERT_START, ALERT_STATUS } from '@kbn/rule-data-utils';
import type { EcsFieldsResponse } from '@kbn/rule-registry-plugin/common';
import dedent from 'dedent';
import { useEffect } from 'react';
import { useKibana } from '../../../hooks/use_kibana';
import { useInvestigation } from '../contexts/investigation_context';
import { useKibana } from './use_kibana';
import { useInvestigation } from '../pages/details/contexts/investigation_context';
import { useFetchAlert } from './use_fetch_alert';

export function useScreenContext() {
Expand All @@ -22,9 +21,7 @@ export function useScreenContext() {
} = useKibana();

const { investigation } = useInvestigation();
const alertOriginInvestigation = alertOriginSchema.safeParse(investigation?.origin);
const alertId = alertOriginInvestigation.success ? alertOriginInvestigation.data.id : undefined;
const { data: alertDetails, isLoading: isAlertDetailsLoading } = useFetchAlert({ id: alertId });
const { data: alertDetails, isLoading: isAlertDetailsLoading } = useFetchAlert({ investigation });

useEffect(() => {
if (!investigation || isAlertDetailsLoading) {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ import { EuiLoadingSpinner, EuiFlexItem } from '@elastic/eui';
import { css } from '@emotion/css';
import { ReactEmbeddableRenderer } from '@kbn/embeddable-plugin/public';
import type { GlobalWidgetParameters } from '@kbn/investigate-plugin/public';
import { useAbortableAsync } from '@kbn/observability-ai-assistant-plugin/public';
import { useAbortableAsync } from '@kbn/observability-utils/hooks/use_abortable_async';
import React, { useEffect, useMemo, useRef, useState } from 'react';
import { v4 } from 'uuid';
import { ErrorMessage } from '../../components/error_message';
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ import type { ESQLSearchResponse } from '@kbn/es-types';
import { i18n } from '@kbn/i18n';
import { type GlobalWidgetParameters } from '@kbn/investigate-plugin/public';
import type { Suggestion } from '@kbn/lens-plugin/public';
import { useAbortableAsync } from '@kbn/observability-ai-assistant-plugin/public';
import { useAbortableAsync } from '@kbn/observability-utils/hooks/use_abortable_async';
import React, { useMemo } from 'react';
import { ErrorMessage } from '../../components/error_message';
import { useKibana } from '../../hooks/use_kibana';
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ import type { ESQLColumn, ESQLRow } from '@kbn/es-types';
import { GlobalWidgetParameters } from '@kbn/investigate-plugin/public';
import { Item } from '@kbn/investigation-shared';
import type { Suggestion } from '@kbn/lens-plugin/public';
import { useAbortableAsync } from '@kbn/observability-ai-assistant-plugin/public';
import { useAbortableAsync } from '@kbn/observability-utils/hooks/use_abortable_async';
import React, { useEffect, useMemo, useState } from 'react';
import { ErrorMessage } from '../../../../components/error_message';
import { SuggestVisualizationList } from '../../../../components/suggest_visualization_list';
Expand Down
Loading

0 comments on commit c099f33

Please sign in to comment.