Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Security Solution] Elastic Security Assistant #156933

Merged
merged 53 commits into from
Jun 2, 2023

Conversation

spong
Copy link
Member

@spong spong commented May 5, 2023

[Security Solution] Elastic Security Assistant

The Elastic Security Assistant has entered the chat, integrating generative AI and large language models (LLMs) into the workflows of Elastic Security users.

Bring your alerts, events, rules, and data quality checks into the conversation.

Kapture.2023-06-02.at.14.56.09.mp4

This PR merges a feature branch developed by @spong and @andrew-goldstein , seeded by @jamesspi 's prototype of the assistant. Connectivity to LLMs is provided the Generative AI Connector , developed by @stephmilovic . This PR includes:

  • A new reusable Kibana package containing the assistant: x-pack/packages/kbn-elastic-assistant
    • See the How to embed the Assistant in other parts of Kibana for details
  • Assistant integration into Elastic Security Solution workflows (e.g. alerts, cases, Timeline, rules, data quality)

An assistant trained on the Elastic stack and Elastic Security

The Generative AI Connector connects the assistant to OpenAI and Azure OpenAI models trained with knowledge of the Elastic stack and the Elastic Security solution, including:

This training enables the assistant to offer fully interactive chat experiences that include:

  • alert summarization
  • interactive query generation
  • workflow suggestions
  • generating ingestion configurations that conform to the Elastic Common Schema
  • your imagination

using context from Elastic Security.

Take action from your conversations

The Actions (from assistant response):

  • Send KQL to Timeline
  • Send EQL to Timeline
  • Send Elasticsearch DSL to Timeline
  • Send Note to timeline
  • Create new case
  • Add to existing case
  • Copy to clipboard

Feature Flag

This functionality has been introduced behind the assistantEnabled feature flag. Please add this configuration to your kibana.yml or Kibana Cloud User Settings configuration to enable:

xpack.securitySolution.enableExperimental: ['assistantEnabled']

Components architecture diagram

Untitled Diagram drawio (1)

How to embed the Assistant in other parts of Kibana

Follow the general instructions in x-pack/packages/kbn-elastic-assistant/index.ts to integrate the assistant into a Kibana app.

Step 1 - Wrap your Kibana app in the AssistantProvider component

// Step 1: Wrap your Kibana app in the `AssistantProvider` component. This typically
// happens in the root of your app. Optionally provide a custom title for the assistant:

/** provides context (from the app) to the assistant, and injects Kibana services, like `http` */
export { AssistantProvider } from './impl/assistant_context';

Step 2: Add the AssistantOverlay component to your app

// Step 2: Add the `AssistantOverlay` component to your app. This component displays the assistant
// overlay in a modal, bound to a shortcut key:

/** modal overlay for Elastic Assistant conversations */
export { AssistantOverlay } from './impl/assistant/assistant_overlay';

// In addition to the `AssistantOverlay`, or as an alternative, you may use the `Assistant` component
// to display the assistant without the modal overlay:

/** this component renders the Assistant without the modal overlay to, for example, render it in a Timeline tab */
export { Assistant } from './impl/assistant';

Step 3: Wherever you want to bring context into the assistant, use the any combination of the following

// Step 3: Wherever you want to bring context into the assistant, use the any combination of the following
// components and hooks:
// - `NewChat` component
// - `NewChatById` component
// - `useAssistantOverlay` hook

/**
 * `NewChat` displays a _New chat_ icon button, providing all the context
 * necessary to start a new chat. You may optionally style the button icon,
 * or override the default _New chat_ text with custom content, like `🪄✨`
 *
 * USE THIS WHEN: All the data necessary to start a new chat is available
 * in the same part of the React tree as the _New chat_ button.
 */
export { NewChat } from './impl/new_chat';

/**
 * `NewChatByID` displays a _New chat_ icon button by providing only the `promptContextId`
 * of a context that was (already) registered by the `useAssistantOverlay` hook. You may
 * optionally style the button icon, or override the default _New chat_ text with custom
 * content, like {'🪄✨'}
 *
 * USE THIS WHEN: all the data necessary to start a new chat is NOT available
 * in the same part of the React tree as the _New chat_ button. When paired
 * with the `useAssistantOverlay` hook, this option enables context to be be
 * registered where the data is available, and then the _New chat_ button can be displayed
 * in another part of the tree.
 */
export { NewChatById } from './impl/new_chat_by_id';

/**
 * `useAssistantOverlay` is a hook that registers context with the assistant overlay, and
 * returns an optional `showAssistantOverlay` function to display the assistant overlay.
 * As an alterative to using the `showAssistantOverlay` returned from this hook, you may
 * use the `NewChatById` component and pass it the `promptContextId` returned by this hook.
 *
 * USE THIS WHEN: You want to register context in one part of the tree, and then show
 * a _New chat_ button in another part of the tree without passing around the data, or when
 * you want to build a custom `New chat` button with features not not provided by the
 * `NewChat` component.
 */
export { useAssistantOverlay } from './impl/assistant/use_assistant_overlay';

Co-authored-by: Garrett Spong [email protected]
Co-authored-by: Andrew Macri [email protected]

@spong spong added skip-ci ci:no-auto-commit Disable auto-committing changes on CI labels May 5, 2023
@spong spong force-pushed the security-assistant-playground branch from e477152 to 562e5d9 Compare May 8, 2023 19:46
andrew-goldstein and others added 20 commits May 8, 2023 21:58
…k and New chat button #5

- adds the useSecurityAssistantQuery hook
- adds a `New chat` button that renders a query in a popover
- adds hooks that provide context to the Security Assistant
…g overlay with conversationId and promptContextId
…Prompts / Sends Prompt Context #7

- adds the query editor
- passes context and system prompt to the api
… quick prompts and wired them back up to the prompt textarea
- Add note to Timeline
- Common New Chat button for alert flyout and Timeline
- Updated context descriptions
- fixes i18n errors (expressions are not allowed in template literals)
…ssistant package #11

- bootstraps the `@kbn/elastic-assistant` package in `x-pack/packages/kbn-elastic-assistant`
andrew-goldstein and others added 5 commits May 23, 2023 08:54
…bn/elastic-assistant package #12

## [Security Solution] Security Assistant: Moves the assistant to the `@kbn/elastic-assistant` package

- moves most assistant code to `x-pack/packages/kbn-elastic-assistant`
- fixes linter issues remaining from ON week
@elasticmachine
Copy link
Contributor

Pinging @elastic/security-solution (Team: SecuritySolution)

@spong spong added the release_note:feature Makes this part of the condensed release notes label Jun 2, 2023
import type { TimelineEventsDetailsItem } from '../../../../common/search_strategy';

import type { BrowserFields } from '../../../../common/search_strategy/index_fields';
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This changes is not related to the PR context.

@@ -53,6 +53,7 @@ const enrichedHostIpData: AlertSummaryRow['description'] = {
};

const mockCount = 90019001;

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could be excluded from PR as well


_setValue(value);
(value: T | ((prev: T) => T)) => {
if (typeof value === 'function') {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Where this is used and how this changes is related to the current PR?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's used in the useConversationStore wrapper to store state in localstorage. _setValue needed to be extend to support providing the prev value as a function param when updating:

export const useConversationStore = (): UseConversationStore => {
const [conversations, setConversations] = useLocalStorage<Record<string, Conversation>>({
defaultValue: BASE_SECURITY_CONVERSATIONS,
key: LOCAL_STORAGE_KEY,
isInvalidDefault: (valueFromStorage) => {
return !valueFromStorage;
},
});

Copy link
Contributor

@YulNaumenko YulNaumenko left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested locally and all is good. Have some minor comment related to the code and some to the user workflows. It could be done as a follow up work.
Amazing work!!!

@spong spong removed request for pzl and parkiino June 2, 2023 20:28
@kibana-ci
Copy link
Collaborator

💛 Build succeeded, but was flaky

Failed CI Steps

Test Failures

  • [job] [logs] FTR Configs #21 / Alerting alerts_as_data alerts as data flapping should set flapping and flapping_history for flapping alerts that settle on active

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id before after diff
securitySolution 3912 4103 +191

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id before after diff
@kbn/elastic-assistant - 55 +55

Any counts in public APIs

Total count of every any typed public API. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats any for more detailed information.

id before after diff
@kbn/elastic-assistant - 4 +4

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id before after diff
securitySolution 9.4MB 10.7MB ⚠️ +1.4MB

Public APIs missing exports

Total count of every type that is part of your API that should be exported but is not. This will cause broken links in the API documentation system. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats exports for more detailed information.

id before after diff
@kbn/elastic-assistant - 3 +3
stackConnectors 0 1 +1
total +4

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id before after diff
securitySolution 51.0KB 51.4KB +359.0B
Unknown metric groups

API count

id before after diff
@kbn/elastic-assistant - 75 +75

ESLint disabled line counts

id before after diff
@kbn/elastic-assistant - 15 +15
enterpriseSearch 19 21 +2
securitySolution 414 418 +4
total +21

Total ESLint disabled count

id before after diff
@kbn/elastic-assistant - 15 +15
enterpriseSearch 20 22 +2
securitySolution 498 502 +4
total +21

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @spong @andrew-goldstein

@andrew-goldstein andrew-goldstein merged commit 4e38817 into elastic:main Jun 2, 2023
@andrew-goldstein andrew-goldstein deleted the security-assistant-playground branch June 2, 2023 21:19
@kibanamachine
Copy link
Contributor

💔 All backports failed

Status Branch Result
8.8 Backport failed because of merge conflicts

Manual backport

To create the backport manually run:

node scripts/backport --pr 156933

Questions ?

Please refer to the Backport tool documentation

spong added a commit to spong/kibana that referenced this pull request Jun 2, 2023
## [Security Solution] Elastic Security Assistant

The _Elastic Security Assistant_ has entered the chat, integrating generative AI and large language models (LLMs) into the workflows of Elastic Security users.

Bring your alerts, events, rules, and data quality checks into the conversation.

<https://github.com/elastic/kibana/assets/2946766/31d65c78-5692-4817-b726-820c5df0801c>

This PR merges a feature branch developed by @spong and @andrew-goldstein , seeded by @jamesspi 's prototype of the assistant. Connectivity to LLMs is provided the [Generative AI Connector](<elastic#157228>) , developed by @stephmilovic . This PR includes:

- A new reusable Kibana package containing the assistant: `x-pack/packages/kbn-elastic-assistant`
  - See the `How to embed the Assistant in other parts of Kibana` for details
- Assistant integration into Elastic Security Solution workflows (e.g. alerts, cases, Timeline, rules, data quality)

### An assistant trained on the Elastic stack and Elastic Security

The [Generative AI Connector](<elastic#157228>) connects the assistant to OpenAI and Azure OpenAI models trained with knowledge of the Elastic stack and the Elastic Security solution, including:

- The Elastic open [Detection Rules](https://github.com/elastic/detection-rules)
- The [Elastic Common Schema (ECS)](https://www.elastic.co/guide/en/ecs/current/index.html)
- Elastic query languages, including [KQL](https://www.elastic.co/guide/en/kibana/current/kuery-query.html), [EQL](https://www.elastic.co/guide/en/elasticsearch/reference/current/eql-syntax.html), and the [Elastic Query DSL](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl.html)
- [Elasticsearch API documentation](https://www.elastic.co/guide/en/elasticsearch/reference/8.8/rest-apis.html)

This training enables the assistant to offer fully interactive chat experiences that include:

- alert summarization
- interactive query generation
- workflow suggestions
- generating ingestion configurations that conform to the Elastic Common Schema
- your imagination

using context from Elastic Security.

### Take action from your conversations

The Actions (from assistant response):

- Send KQL to Timeline
- Send EQL to Timeline
- Send Elasticsearch DSL to Timeline
- Send Note to timeline
- Create new case
- Add to existing case
- Copy to clipboard

### Components architecture diagram

![Untitled Diagram drawio (1)](https://github.com/elastic/kibana/assets/55110838/8f446313-629f-4646-b44d-0cb0ca74aeaf)

### How to embed the Assistant in other parts of Kibana

Follow the general instructions in `x-pack/packages/kbn-elastic-assistant/index.ts` to integrate the assistant into a Kibana app.

#### Step 1 - Wrap your Kibana app in the `AssistantProvider` component

```ts
// Step 1: Wrap your Kibana app in the `AssistantProvider` component. This typically
// happens in the root of your app. Optionally provide a custom title for the assistant:

/** provides context (from the app) to the assistant, and injects Kibana services, like `http` */
export { AssistantProvider } from './impl/assistant_context';
```

#### Step 2: Add the `AssistantOverlay` component to your app

```ts
// Step 2: Add the `AssistantOverlay` component to your app. This component displays the assistant
// overlay in a modal, bound to a shortcut key:

/** modal overlay for Elastic Assistant conversations */
export { AssistantOverlay } from './impl/assistant/assistant_overlay';

// In addition to the `AssistantOverlay`, or as an alternative, you may use the `Assistant` component
// to display the assistant without the modal overlay:

/** this component renders the Assistant without the modal overlay to, for example, render it in a Timeline tab */
export { Assistant } from './impl/assistant';
```

#### Step 3: Wherever you want to bring context into the assistant, use the any combination of the following

```ts
// Step 3: Wherever you want to bring context into the assistant, use the any combination of the following
// components and hooks:
// - `NewChat` component
// - `NewChatById` component
// - `useAssistantOverlay` hook

/**
 * `NewChat` displays a _New chat_ icon button, providing all the context
 * necessary to start a new chat. You may optionally style the button icon,
 * or override the default _New chat_ text with custom content, like `🪄✨`
 *
 * USE THIS WHEN: All the data necessary to start a new chat is available
 * in the same part of the React tree as the _New chat_ button.
 */
export { NewChat } from './impl/new_chat';

/**
 * `NewChatByID` displays a _New chat_ icon button by providing only the `promptContextId`
 * of a context that was (already) registered by the `useAssistantOverlay` hook. You may
 * optionally style the button icon, or override the default _New chat_ text with custom
 * content, like {'🪄✨'}
 *
 * USE THIS WHEN: all the data necessary to start a new chat is NOT available
 * in the same part of the React tree as the _New chat_ button. When paired
 * with the `useAssistantOverlay` hook, this option enables context to be be
 * registered where the data is available, and then the _New chat_ button can be displayed
 * in another part of the tree.
 */
export { NewChatById } from './impl/new_chat_by_id';

/**
 * `useAssistantOverlay` is a hook that registers context with the assistant overlay, and
 * returns an optional `showAssistantOverlay` function to display the assistant overlay.
 * As an alterative to using the `showAssistantOverlay` returned from this hook, you may
 * use the `NewChatById` component and pass it the `promptContextId` returned by this hook.
 *
 * USE THIS WHEN: You want to register context in one part of the tree, and then show
 * a _New chat_ button in another part of the tree without passing around the data, or when
 * you want to build a custom `New chat` button with features not not provided by the
 * `NewChat` component.
 */
export { useAssistantOverlay } from './impl/assistant/use_assistant_overlay';
```

Co-authored-by: Garrett Spong <[email protected]>
Co-authored-by: Andrew Macri <[email protected]>
(cherry picked from commit 4e38817)

# Conflicts:
#	.github/CODEOWNERS
#	x-pack/plugins/security_solution/public/common/components/event_details/translations.ts
#	x-pack/plugins/security_solution/public/timelines/components/timeline/tabs_content/index.tsx
@spong
Copy link
Member Author

spong commented Jun 2, 2023

💚 All backports created successfully

Status Branch Result
8.8

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

spong added a commit that referenced this pull request Jun 2, 2023
# Backport

This will backport the following commits from `main` to `8.8`:
- [[Security Solution] Elastic Security Assistant
(#156933)](#156933)

<!--- Backport version: 8.9.7 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Garrett
Spong","email":"[email protected]"},"sourceCommit":{"committedDate":"2023-06-02T21:19:10Z","message":"[Security
Solution] Elastic Security Assistant (#156933)\n\n## [Security Solution]
Elastic Security Assistant\r\n\r\nThe _Elastic Security Assistant_ has
entered the chat, integrating generative AI and large language models
(LLMs) into the workflows of Elastic Security users.\r\n\r\nBring your
alerts, events, rules, and data quality checks into the
conversation.\r\n\r\n<https://github.com/elastic/kibana/assets/2946766/31d65c78-5692-4817-b726-820c5df0801c>\r\n\r\nThis
PR merges a feature branch developed by @spong and @andrew-goldstein ,
seeded by @jamesspi 's prototype of the assistant. Connectivity to LLMs
is provided the [Generative AI
Connector](<#157228>) , developed
by @stephmilovic . This PR includes:\r\n\r\n- A new reusable Kibana
package containing the assistant:
`x-pack/packages/kbn-elastic-assistant`\r\n - See the `How to embed the
Assistant in other parts of Kibana` for details\r\n- Assistant
integration into Elastic Security Solution workflows (e.g. alerts,
cases, Timeline, rules, data quality)\r\n\r\n### An assistant trained on
the Elastic stack and Elastic Security\r\n\r\nThe [Generative AI
Connector](<#157228>) connects the
assistant to OpenAI and Azure OpenAI models trained with knowledge of
the Elastic stack and the Elastic Security solution, including:\r\n\r\n-
The Elastic open [Detection
Rules](https://github.com/elastic/detection-rules)\r\n- The [Elastic
Common Schema
(ECS)](https://www.elastic.co/guide/en/ecs/current/index.html)\r\n-
Elastic query languages, including
[KQL](https://www.elastic.co/guide/en/kibana/current/kuery-query.html),
[EQL](https://www.elastic.co/guide/en/elasticsearch/reference/current/eql-syntax.html),
and the [Elastic Query
DSL](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl.html)\r\n-
[Elasticsearch API
documentation](https://www.elastic.co/guide/en/elasticsearch/reference/8.8/rest-apis.html)\r\n\r\nThis
training enables the assistant to offer fully interactive chat
experiences that include:\r\n\r\n- alert summarization\r\n- interactive
query generation\r\n- workflow suggestions\r\n- generating ingestion
configurations that conform to the Elastic Common Schema\r\n- your
imagination\r\n\r\nusing context from Elastic Security.\r\n\r\n### Take
action from your conversations\r\n\r\nThe Actions (from assistant
response):\r\n\r\n- Send KQL to Timeline\r\n- Send EQL to Timeline\r\n-
Send Elasticsearch DSL to Timeline\r\n- Send Note to timeline\r\n-
Create new case\r\n- Add to existing case\r\n- Copy to
clipboard\r\n\r\n### Components architecture diagram\r\n\r\n![Untitled
Diagram drawio
(1)](https://github.com/elastic/kibana/assets/55110838/8f446313-629f-4646-b44d-0cb0ca74aeaf)\r\n\r\n###
How to embed the Assistant in other parts of Kibana\r\n\r\nFollow the
general instructions in `x-pack/packages/kbn-elastic-assistant/index.ts`
to integrate the assistant into a Kibana app.\r\n\r\n#### Step 1 - Wrap
your Kibana app in the `AssistantProvider` component\r\n\r\n```ts\r\n//
Step 1: Wrap your Kibana app in the `AssistantProvider` component. This
typically\r\n// happens in the root of your app. Optionally provide a
custom title for the assistant:\r\n\r\n/** provides context (from the
app) to the assistant, and injects Kibana services, like `http`
*/\r\nexport { AssistantProvider } from
'./impl/assistant_context';\r\n```\r\n\r\n#### Step 2: Add the
`AssistantOverlay` component to your app\r\n\r\n```ts\r\n// Step 2: Add
the `AssistantOverlay` component to your app. This component displays
the assistant\r\n// overlay in a modal, bound to a shortcut
key:\r\n\r\n/** modal overlay for Elastic Assistant conversations
*/\r\nexport { AssistantOverlay } from
'./impl/assistant/assistant_overlay';\r\n\r\n// In addition to the
`AssistantOverlay`, or as an alternative, you may use the `Assistant`
component\r\n// to display the assistant without the modal
overlay:\r\n\r\n/** this component renders the Assistant without the
modal overlay to, for example, render it in a Timeline tab */\r\nexport
{ Assistant } from './impl/assistant';\r\n```\r\n\r\n#### Step 3:
Wherever you want to bring context into the assistant, use the any
combination of the following\r\n\r\n```ts\r\n// Step 3: Wherever you
want to bring context into the assistant, use the any combination of the
following\r\n// components and hooks:\r\n// - `NewChat` component\r\n//
- `NewChatById` component\r\n// - `useAssistantOverlay`
hook\r\n\r\n/**\r\n * `NewChat` displays a _New chat_ icon button,
providing all the context\r\n * necessary to start a new chat. You may
optionally style the button icon,\r\n * or override the default _New
chat_ text with custom content, like `🪄✨`\r\n *\r\n * USE THIS WHEN: All
the data necessary to start a new chat is available\r\n * in the same
part of the React tree as the _New chat_ button.\r\n */\r\nexport {
NewChat } from './impl/new_chat';\r\n\r\n/**\r\n * `NewChatByID`
displays a _New chat_ icon button by providing only the
`promptContextId`\r\n * of a context that was (already) registered by
the `useAssistantOverlay` hook. You may\r\n * optionally style the
button icon, or override the default _New chat_ text with custom\r\n *
content, like {'🪄✨'}\r\n *\r\n * USE THIS WHEN: all the data necessary
to start a new chat is NOT available\r\n * in the same part of the React
tree as the _New chat_ button. When paired\r\n * with the
`useAssistantOverlay` hook, this option enables context to be be\r\n *
registered where the data is available, and then the _New chat_ button
can be displayed\r\n * in another part of the tree.\r\n */\r\nexport {
NewChatById } from './impl/new_chat_by_id';\r\n\r\n/**\r\n *
`useAssistantOverlay` is a hook that registers context with the
assistant overlay, and\r\n * returns an optional `showAssistantOverlay`
function to display the assistant overlay.\r\n * As an alterative to
using the `showAssistantOverlay` returned from this hook, you may\r\n *
use the `NewChatById` component and pass it the `promptContextId`
returned by this hook.\r\n *\r\n * USE THIS WHEN: You want to register
context in one part of the tree, and then show\r\n * a _New chat_ button
in another part of the tree without passing around the data, or when\r\n
* you want to build a custom `New chat` button with features not not
provided by the\r\n * `NewChat` component.\r\n */\r\nexport {
useAssistantOverlay } from
'./impl/assistant/use_assistant_overlay';\r\n```\r\n\r\nCo-authored-by:
Garrett Spong <[email protected]>\r\nCo-authored-by: Andrew Macri
<[email protected]>","sha":"4e38817a4ddfea8aff336c22788bc3a30ece5908","branchLabelMapping":{"^v8.9.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["Team:
SecuritySolution","release_note:feature","ci:no-auto-commit","v8.9.0","v8.8.1"],"number":156933,"url":"https://github.com/elastic/kibana/pull/156933","mergeCommit":{"message":"[Security
Solution] Elastic Security Assistant (#156933)\n\n## [Security Solution]
Elastic Security Assistant\r\n\r\nThe _Elastic Security Assistant_ has
entered the chat, integrating generative AI and large language models
(LLMs) into the workflows of Elastic Security users.\r\n\r\nBring your
alerts, events, rules, and data quality checks into the
conversation.\r\n\r\n<https://github.com/elastic/kibana/assets/2946766/31d65c78-5692-4817-b726-820c5df0801c>\r\n\r\nThis
PR merges a feature branch developed by @spong and @andrew-goldstein ,
seeded by @jamesspi 's prototype of the assistant. Connectivity to LLMs
is provided the [Generative AI
Connector](<#157228>) , developed
by @stephmilovic . This PR includes:\r\n\r\n- A new reusable Kibana
package containing the assistant:
`x-pack/packages/kbn-elastic-assistant`\r\n - See the `How to embed the
Assistant in other parts of Kibana` for details\r\n- Assistant
integration into Elastic Security Solution workflows (e.g. alerts,
cases, Timeline, rules, data quality)\r\n\r\n### An assistant trained on
the Elastic stack and Elastic Security\r\n\r\nThe [Generative AI
Connector](<#157228>) connects the
assistant to OpenAI and Azure OpenAI models trained with knowledge of
the Elastic stack and the Elastic Security solution, including:\r\n\r\n-
The Elastic open [Detection
Rules](https://github.com/elastic/detection-rules)\r\n- The [Elastic
Common Schema
(ECS)](https://www.elastic.co/guide/en/ecs/current/index.html)\r\n-
Elastic query languages, including
[KQL](https://www.elastic.co/guide/en/kibana/current/kuery-query.html),
[EQL](https://www.elastic.co/guide/en/elasticsearch/reference/current/eql-syntax.html),
and the [Elastic Query
DSL](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl.html)\r\n-
[Elasticsearch API
documentation](https://www.elastic.co/guide/en/elasticsearch/reference/8.8/rest-apis.html)\r\n\r\nThis
training enables the assistant to offer fully interactive chat
experiences that include:\r\n\r\n- alert summarization\r\n- interactive
query generation\r\n- workflow suggestions\r\n- generating ingestion
configurations that conform to the Elastic Common Schema\r\n- your
imagination\r\n\r\nusing context from Elastic Security.\r\n\r\n### Take
action from your conversations\r\n\r\nThe Actions (from assistant
response):\r\n\r\n- Send KQL to Timeline\r\n- Send EQL to Timeline\r\n-
Send Elasticsearch DSL to Timeline\r\n- Send Note to timeline\r\n-
Create new case\r\n- Add to existing case\r\n- Copy to
clipboard\r\n\r\n### Components architecture diagram\r\n\r\n![Untitled
Diagram drawio
(1)](https://github.com/elastic/kibana/assets/55110838/8f446313-629f-4646-b44d-0cb0ca74aeaf)\r\n\r\n###
How to embed the Assistant in other parts of Kibana\r\n\r\nFollow the
general instructions in `x-pack/packages/kbn-elastic-assistant/index.ts`
to integrate the assistant into a Kibana app.\r\n\r\n#### Step 1 - Wrap
your Kibana app in the `AssistantProvider` component\r\n\r\n```ts\r\n//
Step 1: Wrap your Kibana app in the `AssistantProvider` component. This
typically\r\n// happens in the root of your app. Optionally provide a
custom title for the assistant:\r\n\r\n/** provides context (from the
app) to the assistant, and injects Kibana services, like `http`
*/\r\nexport { AssistantProvider } from
'./impl/assistant_context';\r\n```\r\n\r\n#### Step 2: Add the
`AssistantOverlay` component to your app\r\n\r\n```ts\r\n// Step 2: Add
the `AssistantOverlay` component to your app. This component displays
the assistant\r\n// overlay in a modal, bound to a shortcut
key:\r\n\r\n/** modal overlay for Elastic Assistant conversations
*/\r\nexport { AssistantOverlay } from
'./impl/assistant/assistant_overlay';\r\n\r\n// In addition to the
`AssistantOverlay`, or as an alternative, you may use the `Assistant`
component\r\n// to display the assistant without the modal
overlay:\r\n\r\n/** this component renders the Assistant without the
modal overlay to, for example, render it in a Timeline tab */\r\nexport
{ Assistant } from './impl/assistant';\r\n```\r\n\r\n#### Step 3:
Wherever you want to bring context into the assistant, use the any
combination of the following\r\n\r\n```ts\r\n// Step 3: Wherever you
want to bring context into the assistant, use the any combination of the
following\r\n// components and hooks:\r\n// - `NewChat` component\r\n//
- `NewChatById` component\r\n// - `useAssistantOverlay`
hook\r\n\r\n/**\r\n * `NewChat` displays a _New chat_ icon button,
providing all the context\r\n * necessary to start a new chat. You may
optionally style the button icon,\r\n * or override the default _New
chat_ text with custom content, like `🪄✨`\r\n *\r\n * USE THIS WHEN: All
the data necessary to start a new chat is available\r\n * in the same
part of the React tree as the _New chat_ button.\r\n */\r\nexport {
NewChat } from './impl/new_chat';\r\n\r\n/**\r\n * `NewChatByID`
displays a _New chat_ icon button by providing only the
`promptContextId`\r\n * of a context that was (already) registered by
the `useAssistantOverlay` hook. You may\r\n * optionally style the
button icon, or override the default _New chat_ text with custom\r\n *
content, like {'🪄✨'}\r\n *\r\n * USE THIS WHEN: all the data necessary
to start a new chat is NOT available\r\n * in the same part of the React
tree as the _New chat_ button. When paired\r\n * with the
`useAssistantOverlay` hook, this option enables context to be be\r\n *
registered where the data is available, and then the _New chat_ button
can be displayed\r\n * in another part of the tree.\r\n */\r\nexport {
NewChatById } from './impl/new_chat_by_id';\r\n\r\n/**\r\n *
`useAssistantOverlay` is a hook that registers context with the
assistant overlay, and\r\n * returns an optional `showAssistantOverlay`
function to display the assistant overlay.\r\n * As an alterative to
using the `showAssistantOverlay` returned from this hook, you may\r\n *
use the `NewChatById` component and pass it the `promptContextId`
returned by this hook.\r\n *\r\n * USE THIS WHEN: You want to register
context in one part of the tree, and then show\r\n * a _New chat_ button
in another part of the tree without passing around the data, or when\r\n
* you want to build a custom `New chat` button with features not not
provided by the\r\n * `NewChat` component.\r\n */\r\nexport {
useAssistantOverlay } from
'./impl/assistant/use_assistant_overlay';\r\n```\r\n\r\nCo-authored-by:
Garrett Spong <[email protected]>\r\nCo-authored-by: Andrew Macri
<[email protected]>","sha":"4e38817a4ddfea8aff336c22788bc3a30ece5908"}},"sourceBranch":"main","suggestedTargetBranches":["8.8"],"targetPullRequestStates":[{"branch":"main","label":"v8.9.0","labelRegex":"^v8.9.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/156933","number":156933,"mergeCommit":{"message":"[Security
Solution] Elastic Security Assistant (#156933)\n\n## [Security Solution]
Elastic Security Assistant\r\n\r\nThe _Elastic Security Assistant_ has
entered the chat, integrating generative AI and large language models
(LLMs) into the workflows of Elastic Security users.\r\n\r\nBring your
alerts, events, rules, and data quality checks into the
conversation.\r\n\r\n<https://github.com/elastic/kibana/assets/2946766/31d65c78-5692-4817-b726-820c5df0801c>\r\n\r\nThis
PR merges a feature branch developed by @spong and @andrew-goldstein ,
seeded by @jamesspi 's prototype of the assistant. Connectivity to LLMs
is provided the [Generative AI
Connector](<#157228>) , developed
by @stephmilovic . This PR includes:\r\n\r\n- A new reusable Kibana
package containing the assistant:
`x-pack/packages/kbn-elastic-assistant`\r\n - See the `How to embed the
Assistant in other parts of Kibana` for details\r\n- Assistant
integration into Elastic Security Solution workflows (e.g. alerts,
cases, Timeline, rules, data quality)\r\n\r\n### An assistant trained on
the Elastic stack and Elastic Security\r\n\r\nThe [Generative AI
Connector](<#157228>) connects the
assistant to OpenAI and Azure OpenAI models trained with knowledge of
the Elastic stack and the Elastic Security solution, including:\r\n\r\n-
The Elastic open [Detection
Rules](https://github.com/elastic/detection-rules)\r\n- The [Elastic
Common Schema
(ECS)](https://www.elastic.co/guide/en/ecs/current/index.html)\r\n-
Elastic query languages, including
[KQL](https://www.elastic.co/guide/en/kibana/current/kuery-query.html),
[EQL](https://www.elastic.co/guide/en/elasticsearch/reference/current/eql-syntax.html),
and the [Elastic Query
DSL](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl.html)\r\n-
[Elasticsearch API
documentation](https://www.elastic.co/guide/en/elasticsearch/reference/8.8/rest-apis.html)\r\n\r\nThis
training enables the assistant to offer fully interactive chat
experiences that include:\r\n\r\n- alert summarization\r\n- interactive
query generation\r\n- workflow suggestions\r\n- generating ingestion
configurations that conform to the Elastic Common Schema\r\n- your
imagination\r\n\r\nusing context from Elastic Security.\r\n\r\n### Take
action from your conversations\r\n\r\nThe Actions (from assistant
response):\r\n\r\n- Send KQL to Timeline\r\n- Send EQL to Timeline\r\n-
Send Elasticsearch DSL to Timeline\r\n- Send Note to timeline\r\n-
Create new case\r\n- Add to existing case\r\n- Copy to
clipboard\r\n\r\n### Components architecture diagram\r\n\r\n![Untitled
Diagram drawio
(1)](https://github.com/elastic/kibana/assets/55110838/8f446313-629f-4646-b44d-0cb0ca74aeaf)\r\n\r\n###
How to embed the Assistant in other parts of Kibana\r\n\r\nFollow the
general instructions in `x-pack/packages/kbn-elastic-assistant/index.ts`
to integrate the assistant into a Kibana app.\r\n\r\n#### Step 1 - Wrap
your Kibana app in the `AssistantProvider` component\r\n\r\n```ts\r\n//
Step 1: Wrap your Kibana app in the `AssistantProvider` component. This
typically\r\n// happens in the root of your app. Optionally provide a
custom title for the assistant:\r\n\r\n/** provides context (from the
app) to the assistant, and injects Kibana services, like `http`
*/\r\nexport { AssistantProvider } from
'./impl/assistant_context';\r\n```\r\n\r\n#### Step 2: Add the
`AssistantOverlay` component to your app\r\n\r\n```ts\r\n// Step 2: Add
the `AssistantOverlay` component to your app. This component displays
the assistant\r\n// overlay in a modal, bound to a shortcut
key:\r\n\r\n/** modal overlay for Elastic Assistant conversations
*/\r\nexport { AssistantOverlay } from
'./impl/assistant/assistant_overlay';\r\n\r\n// In addition to the
`AssistantOverlay`, or as an alternative, you may use the `Assistant`
component\r\n// to display the assistant without the modal
overlay:\r\n\r\n/** this component renders the Assistant without the
modal overlay to, for example, render it in a Timeline tab */\r\nexport
{ Assistant } from './impl/assistant';\r\n```\r\n\r\n#### Step 3:
Wherever you want to bring context into the assistant, use the any
combination of the following\r\n\r\n```ts\r\n// Step 3: Wherever you
want to bring context into the assistant, use the any combination of the
following\r\n// components and hooks:\r\n// - `NewChat` component\r\n//
- `NewChatById` component\r\n// - `useAssistantOverlay`
hook\r\n\r\n/**\r\n * `NewChat` displays a _New chat_ icon button,
providing all the context\r\n * necessary to start a new chat. You may
optionally style the button icon,\r\n * or override the default _New
chat_ text with custom content, like `🪄✨`\r\n *\r\n * USE THIS WHEN: All
the data necessary to start a new chat is available\r\n * in the same
part of the React tree as the _New chat_ button.\r\n */\r\nexport {
NewChat } from './impl/new_chat';\r\n\r\n/**\r\n * `NewChatByID`
displays a _New chat_ icon button by providing only the
`promptContextId`\r\n * of a context that was (already) registered by
the `useAssistantOverlay` hook. You may\r\n * optionally style the
button icon, or override the default _New chat_ text with custom\r\n *
content, like {'🪄✨'}\r\n *\r\n * USE THIS WHEN: all the data necessary
to start a new chat is NOT available\r\n * in the same part of the React
tree as the _New chat_ button. When paired\r\n * with the
`useAssistantOverlay` hook, this option enables context to be be\r\n *
registered where the data is available, and then the _New chat_ button
can be displayed\r\n * in another part of the tree.\r\n */\r\nexport {
NewChatById } from './impl/new_chat_by_id';\r\n\r\n/**\r\n *
`useAssistantOverlay` is a hook that registers context with the
assistant overlay, and\r\n * returns an optional `showAssistantOverlay`
function to display the assistant overlay.\r\n * As an alterative to
using the `showAssistantOverlay` returned from this hook, you may\r\n *
use the `NewChatById` component and pass it the `promptContextId`
returned by this hook.\r\n *\r\n * USE THIS WHEN: You want to register
context in one part of the tree, and then show\r\n * a _New chat_ button
in another part of the tree without passing around the data, or when\r\n
* you want to build a custom `New chat` button with features not not
provided by the\r\n * `NewChat` component.\r\n */\r\nexport {
useAssistantOverlay } from
'./impl/assistant/use_assistant_overlay';\r\n```\r\n\r\nCo-authored-by:
Garrett Spong <[email protected]>\r\nCo-authored-by: Andrew Macri
<[email protected]>","sha":"4e38817a4ddfea8aff336c22788bc3a30ece5908"}},{"branch":"8.8","label":"v8.8.1","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"}]}]
BACKPORT-->
@nicpenning
Copy link

This is looking rad. Is this something that can be tested via feature flag as mentioned above in 8.8.1 or come later in 8.9.0?

Also, will this require OpenAI and/or Azure OpenAI subscriptions to operate? Looking to understand some pre-requisites so when this drops we can immediately test out.

This looks like a force multiplier for us analysts out there.

@jamesspi
Copy link

jamesspi commented Jun 6, 2023

@nicpenning , Answered you on the community slack :)

ashokaditya added a commit that referenced this pull request Jul 11, 2023
…cypress test (#161533)

## Summary

Unskip test that was timing out at indexing test hosts. The timeout has
since been increased in /pull/159518

It was added in /pull/157777 and skipped in
/pull/156933 thus should be backported to `8.8.1` and
`8.9.0`
ashokaditya added a commit that referenced this pull request Jul 12, 2023
…cypress test (#161533)

## Summary

Unskip test that was timing out at indexing test hosts. The timeout has
since been increased in /pull/159518

It was added in /pull/157777 and skipped in
/pull/156933 thus should be backported to `8.8.1` and
`8.9.0`

(cherry picked from commit f6f5986)

# Conflicts:
#	x-pack/plugins/security_solution/public/management/cypress/e2e/mocked_data/reponse_actions_history.cy.ts
ashokaditya added a commit to ashokaditya/kibana that referenced this pull request Jul 12, 2023
…cypress test (elastic#161533)

## Summary

Unskip test that was timing out at indexing test hosts. The timeout has
since been increased in elastic/pull/159518

It was added in elastic/pull/157777 and skipped in
elastic/pull/156933 thus should be backported to `8.8.1` and
`8.9.0`

(cherry picked from commit f6f5986)

# Conflicts:
#	x-pack/plugins/security_solution/public/management/cypress/e2e/mocked_data/reponse_actions_history.cy.ts
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ci:no-auto-commit Disable auto-committing changes on CI release_note:feature Makes this part of the condensed release notes Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. v8.8.1 v8.9.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.