-
Notifications
You must be signed in to change notification settings - Fork 8.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Security Assistant] Adds new Knowledge Base Management Settings UI #192665
Conversation
…ore helpers when working between v1/v2 schemas
…create global issue, and cleanup AddEntry button and flyout UI
…ifying modelId on kbSetup, and adding an option to not add ESQL docs on setup
// ssl: false as ML vocab API is broken with SSL enabled | ||
.filter( | ||
(a: string) => | ||
!( | ||
a.startsWith('--elasticsearch.hosts=') || | ||
a.startsWith('--elasticsearch.ssl.certificateAuthorities=') | ||
) | ||
), |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
As noted in the description, we must run with ssl:false
for certain ml API's to function. See #193477 for details.
Allowing (only)
What is the use case for an |
x-pack/packages/kbn-elastic-assistant-common/env/http-client.env.json
Outdated
Show resolved
Hide resolved
…age if FF disabled, fixed modelId override only if FF is enabled, and renamed assistant_all user
I was mistaken in my original message, you are correct here @andrew-goldstein -- thank you for questioning this behavior! 🙏 I have confirmed with @jamesspi that RBAC should behave as following:
As discussed in our pair session with @patrykkopycinski, I will re-work the I will update and un-skip this API integration test to ensure this behavior is correct so we have this confidence going forward. edit: this is done and test un-skipped and new test added in 4d151a5 |
...c-assistant/impl/knowledge_base/knowledge_base_settings_management/document_entry_editor.tsx
Show resolved
Hide resolved
...c-assistant/impl/knowledge_base/knowledge_base_settings_management/document_entry_editor.tsx
Show resolved
Hide resolved
[setEntry] | ||
); | ||
// TODO: KB-RBAC Disable global option if no RBAC | ||
const sharingOptions = [ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: Consider memoizing this non-primitive value, because it's passed as a prop to a component below
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Will do! I will be extracting the whole sharing field into its own component in the next UI PR as it's used in both entry types.
return onIndexClicked || onDocumentClicked ? ( | ||
<EuiPopover | ||
button={ | ||
<EuiButton iconType="arrowDown" iconSide="right" onClick={onButtonClick}> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Consider disabling the New
button when the KB hasn't been setup yet to avoid the error toaster in the screenshot below:
I'm also wondering if the Setup Knowledge Base
CTA positioned at the end of the sentence in the screenshot above should be the same size text as the Learn more
link, or perhaps be more prominent like it is in the assistant:
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Will disable the New
button if not available 👍
And there's still some pending UI changes around the setup button. I chatted with @bojanasan and we decided to hide the table and show the same button as in the assistant as you recommend here.
💛 Build succeeded, but was flaky
Failed CI StepsMetrics [docs]Module Count
Public APIs missing comments
Async chunks
Unknown metric groupsAPI count
ESLint disabled line counts
References to deprecated APIs
Total ESLint disabled count
History
To update your PR or re-run it, just comment with: cc @spong |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks @spong for the new settings UI! 🙏
✅ Desk tested locally
LGTM 🚀
💔 All backports failed
Manual backportTo create the backport manually run:
Questions ?Please refer to the Backport tool documentation |
💚 All backports created successfully
Note: Successful backport PRs will be merged automatically after passing CI. Questions ?Please refer to the Backport tool documentation |
…lastic#192665) ## Summary This PR updates the Knowledge Base Management Settings page to use the new `entries` API introduced in elastic#186566. Many thanks to @angorayc for her work on the Assistant Management Settings overhaul, and initial implementation of this new KB Management UI over in elastic#186847. <p align="center"> <img width="600" src="https://github.com/user-attachments/assets/0a82587e-f33c-45f1-9165-1a676d6db5fa" /> </p> ### Feature Flag & Setup The changes in this PR, as with the other [recent V2 KB enhancements](elastic#186566), are behind the following feature flag: ``` xpack.securitySolution.enableExperimental: - 'assistantKnowledgeBaseByDefault' ``` ~They also require a code change in the `AIAssistantService` to enable the new mapping (since setup happens on plugin start before FF registration), so be sure to update `fieldMap` to `knowledgeBaseFieldMapV2` below before testing:~ This is no longer the case as of [cdec104](elastic@cdec104). Just changing the above feature flag is now sufficient, just note that if upgrading and the KB was previously setup, you'll need to manually delete the data stream (`DELETE /_data_stream/.kibana-elastic-ai-assistant-knowledge-base-default`) or the management table will be littered with the old ESQL docs instead of being a single aggregate entry. Once configured, the new Knowledge Base Management Settings will become available in Stack Management. The old settings UI is currently still available via the Settings Modal, but will soon be removed and replaced with links to the new interface via the Assistant Settings Context Menu (replacing the existing `cog`). Please see the designs ([Security GenAI](https://www.figma.com/design/BMvpY9EhcPIaoOS7LSrkL0/%5B8.15%2C-%5D-GenAI-Security-Settings?node-id=51-25207&node-type=canvas&t=t3vZSPhMxQhScJVt-0) / [Unified AI Assistant](https://www.figma.com/design/xN20zMRNtMlirWB6n9n1xJ/Unified-AI-Assistant-Settings?node-id=0-1&node-type=canvas&t=3RDYE7h2DjLlFlcN-0)) for all changes. > [!IMPORTANT] > There are no migrations in place between the legacy and v2 KB mappings, so be sure to start with a clean ES data directory. ### Testing To aid with developing the UI, I took the opportunity to start fleshing out the KB Entries API integration tests. These live in [x-pack/test/security_solution_api_integration/test_suites/genai/knowledge_base/entries](https://github.com/spong/kibana/tree/7ae6be136ad992b2163df13b55118556b01b6cb9/x-pack/test/security_solution_api_integration/test_suites/genai/knowledge_base/entries), and are currently configured to only run on `@ess`, as running `tiny_elser` in serverless and MKI environments can be tricky (more on that later). To start the server and run the tests, from the `x-pack/test/security_solution_api_integration/` directory run `yarn genai_kb_entries:server:ess`, and once started, `yarn genai_kb_entries:runner:ess`. ##### Changes in support of testing In order to setup the API integration tests for use with the Knowledge Base, some functional changes needed to be made to the assistant/config: 1. Since ELSER is a heavy model to run in CI, the ML folks have created `pt_tiny_elser` for use in testing. Unfortunately, the `getELSER()` helper off the `ml` client that we use to get the `modelld` for installing ELSER, ingest pipelines, etc, cannot be overridden ([elastic#193633](elastic#193633)), so we must have some other means of doing that. So to get things working in the test env, I've plumbed through an optional `modelId` override to the POST knowledge base route (`/ internal/ elastic_assistant/ knowledge_base/{resource?}?modelId=pt_tiny_elser`). This then overrides the aiAssistantService `getELSER()` function [when fetching](https://github.com/elastic/kibana/blob/645b3b863be16d70b8a7130a84b248c19729c340/x-pack/plugins/elastic_assistant/server/ai_assistant_service/index.ts#L334-L354) a `kbDataClient` using the request, which appears to be the only way to also trigger a reinitialization of the ingest pipeline (which required the `modelId`), since that usually only occurs on plugin start. If there is a cleaner way to perform this reinitialization, please let me know! 2. Turns out [`getService('ml').importTrainedModel()`](https://github.com/elastic/kibana/blob/f18224c6869ae52228da3764ca9a427106b872fb/x-pack/test/functional/services/ml/api.ts#L1575-L1587) can't be run in test env's with `ssl:true`, which is the default security config. You can read more about that issue in [elastic#193477](elastic#193477), but the current workaround is to turn off `ssl` for this specific test configuration, so that's why [`ess.config.ts`](https://github.com/spong/kibana/blob/cf73d4c7fcd69207a9625046456a94212da833c7/x-pack/test/security_solution_api_integration/test_suites/genai/knowledge_base/entries/trial_license_complete_tier/configs/ess.config.ts#L22) looks a little different. If there's a better way to manage this config, also please let me know! ##### Additional notes We don't currently have a `securityAssistant` API client/service to use in integration tests, so I've just been creating one-off functions using `supertest` for now. I don't have the bandwidth to work this now, but perhaps @MadameSheema / @muskangulati-qasource could lend a hand here? I did need to test multi-user and multi-space scenarios, so I ported over the same [auth helpers](https://github.com/elastic/kibana/tree/dc26f1012f35c2445028a87dcc8cb3f063e058b0/x-pack/test/security_solution_api_integration/test_suites/genai/knowledge_base/entries/utils/auth) I saw used in other suites. Would be nice if these were bundled into the client as well ala how the o11y folks have done it [here](https://github.com/elastic/kibana/blob/e9f23aa98e3abadd491be61b17e7daa3cc110cdb/x-pack/test/observability_ai_assistant_api_integration/tests/knowledge_base/knowledge_base.spec.ts#L27-L34). Perhaps this is also on the list of things for @maximpn to generate from OAS's.... 🙃 ### RBAC In plumbing the UI, I've tried to place `// TODO: KB-RBAC` tags in all the places I came across that will require an RBAC check/change. This includes some of the API integration tests, which I currently have skipped as they would fail without RBAC. ### Other notable changes * There are now dedicated `legacy` and `v2` helper functions when managing persistence/retrieval of knowledge base entries. This should help with tearing out the old KB later, and better readability now. * I've tried to remove dependency on the `ElasticsearchStore` as much as possible. The store's only use should now be within tools as a retriever [here](https://github.com/elastic/kibana/blob/de89153368848397df823c062e907a607d347dff/x-pack/plugins/elastic_assistant/server/routes/helpers.ts#L397-L405), and in post_evaluate [here](https://github.com/elastic/kibana/blob/de89153368848397df823c062e907a607d347dff/x-pack/plugins/elastic_assistant/server/routes/evaluate/post_evaluate.ts#L170-L179). If we adopt the new [`naturalLanguageToESQL`](elastic#192042) tool in `8.16` (or update our existing ESQL tool to use the `kbDataClient` for retrieval), we should be able to get rid of this entirely. * Added a [`spaces_roles_users_data.http`](https://github.com/elastic/kibana/blob/7447394fe39d5e2e098c266c14875d3aa17d3067/x-pack/packages/kbn-elastic-assistant-common/impl/utils/spaces_roles_users_data.http#L1) file for adding spaces, roles, users, and a sample `slackbot` index for use with [sample `IndexEntries` here](https://github.com/elastic/kibana/blob/7447394fe39d5e2e098c266c14875d3aa17d3067/x-pack/packages/kbn-elastic-assistant-common/impl/schemas/knowledge_base/entries/crud_knowledge_base_entries_route.http#L18-L56). ### // TODO In effort to make incremental progress and facilitate early knowledge share with @patrykkopycinski, I'm capping this PR where it's at, and so here are the remaining items to complete full integration of the new Knowledge Base Management Settings interface: - [ ] Support `Update` action - [ ] Move from `EuiInMemoryTable` - [ ] Finalize `Setup` UI - [ ] Cleanup `Save` loaders - [ ] Plumb through `{{knowledge_history}}` prompt template and include use's `required` entries All this work is behind the aforementioned feature flag and required code change, and this changeset has also been manually upgrade tested to ensure there are no issues that would impact the regularly scheduled serverless releases. This is more of a note to reviewers when testing that full functionality is not present. ### Checklist - [X] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md) - [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials * Feature currently behind feature flag. Documentation to be added before flag is removed. Tracked in elastic/security-docs#5337 - [X] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios --------- Co-authored-by: kibanamachine <[email protected]> Co-authored-by: Elastic Machine <[email protected]> (cherry picked from commit 63730ea)
…s UI (#192665) (#194074) # Backport This will backport the following commits from `main` to `8.x`: - [[Security Assistant] Adds new Knowledge Base Management Settings UI (#192665)](#192665) <!--- Backport version: 8.9.8 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Garrett Spong","email":"[email protected]"},"sourceCommit":{"committedDate":"2024-09-25T20:38:18Z","message":"[Security Assistant] Adds new Knowledge Base Management Settings UI (#192665)\n\n## Summary\r\n\r\nThis PR updates the Knowledge Base Management Settings page to use the\r\nnew `entries` API introduced in\r\nhttps://github.com//pull/186566. Many thanks to @angorayc\r\nfor her work on the Assistant Management Settings overhaul, and initial\r\nimplementation of this new KB Management UI over in\r\nhttps://github.com//pull/186847.\r\n\r\n<p align=\"center\">\r\n<img width=\"600\"\r\nsrc=\"https://github.com/user-attachments/assets/0a82587e-f33c-45f1-9165-1a676d6db5fa\"\r\n/>\r\n</p> \r\n\r\n\r\n\r\n### Feature Flag & Setup\r\nThe changes in this PR, as with the other [recent V2 KB\r\nenhancements](#186566), are behind\r\nthe following feature flag:\r\n```\r\nxpack.securitySolution.enableExperimental:\r\n - 'assistantKnowledgeBaseByDefault'\r\n```\r\n\r\n~They also require a code change in the `AIAssistantService` to enable\r\nthe new mapping (since setup happens on plugin start before FF\r\nregistration), so be sure to update `fieldMap` to\r\n`knowledgeBaseFieldMapV2` below before testing:~\r\n\r\nThis is no longer the case as of\r\n[cdec104](https://github.com/elastic/kibana/pull/192665/commits/cdec10402f2e9b889598693f9f415c98ccd9855c).\r\nJust changing the above feature flag is now sufficient, just note that\r\nif upgrading and the KB was previously setup, you'll need to manually\r\ndelete the data stream (`DELETE\r\n/_data_stream/.kibana-elastic-ai-assistant-knowledge-base-default`) or\r\nthe management table will be littered with the old ESQL docs instead of\r\nbeing a single aggregate entry.\r\n\r\nOnce configured, the new Knowledge Base Management Settings will become\r\navailable in Stack Management. The old settings UI is currently still\r\navailable via the Settings Modal, but will soon be removed and replaced\r\nwith links to the new interface via the Assistant Settings Context Menu\r\n(replacing the existing `cog`). Please see the designs ([Security\r\nGenAI](https://www.figma.com/design/BMvpY9EhcPIaoOS7LSrkL0/%5B8.15%2C-%5D-GenAI-Security-Settings?node-id=51-25207&node-type=canvas&t=t3vZSPhMxQhScJVt-0)\r\n/ [Unified AI\r\nAssistant](https://www.figma.com/design/xN20zMRNtMlirWB6n9n1xJ/Unified-AI-Assistant-Settings?node-id=0-1&node-type=canvas&t=3RDYE7h2DjLlFlcN-0))\r\nfor all changes.\r\n\r\n> [!IMPORTANT]\r\n> There are no migrations in place between the legacy and v2 KB\r\nmappings, so be sure to start with a clean ES data directory.\r\n\r\n### Testing\r\n\r\nTo aid with developing the UI, I took the opportunity to start fleshing\r\nout the KB Entries API integration tests. These live in\r\n[x-pack/test/security_solution_api_integration/test_suites/genai/knowledge_base/entries](https://github.com/spong/kibana/tree/7ae6be136ad992b2163df13b55118556b01b6cb9/x-pack/test/security_solution_api_integration/test_suites/genai/knowledge_base/entries),\r\nand are currently configured to only run on `@ess`, as running\r\n`tiny_elser` in serverless and MKI environments can be tricky (more on\r\nthat later).\r\n\r\nTo start the server and run the tests, from the\r\n`x-pack/test/security_solution_api_integration/` directory run `yarn\r\ngenai_kb_entries:server:ess`, and once started, `yarn\r\ngenai_kb_entries:runner:ess`.\r\n\r\n##### Changes in support of testing\r\n\r\nIn order to setup the API integration tests for use with the Knowledge\r\nBase, some functional changes needed to be made to the assistant/config:\r\n\r\n1. Since ELSER is a heavy model to run in CI, the ML folks have created\r\n`pt_tiny_elser` for use in testing. Unfortunately, the `getELSER()`\r\nhelper off the `ml` client that we use to get the `modelld` for\r\ninstalling ELSER, ingest pipelines, etc, cannot be overridden\r\n([#193633](#193633)), so we must\r\nhave some other means of doing that. So to get things working in the\r\ntest env, I've plumbed through an optional `modelId` override to the\r\nPOST knowledge base route (`/ internal/ elastic_assistant/\r\nknowledge_base/{resource?}?modelId=pt_tiny_elser`). This then overrides\r\nthe aiAssistantService `getELSER()` function [when\r\nfetching](https://github.com/elastic/kibana/blob/645b3b863be16d70b8a7130a84b248c19729c340/x-pack/plugins/elastic_assistant/server/ai_assistant_service/index.ts#L334-L354)\r\na `kbDataClient` using the request, which appears to be the only way to\r\nalso trigger a reinitialization of the ingest pipeline (which required\r\nthe `modelId`), since that usually only occurs on plugin start. If there\r\nis a cleaner way to perform this reinitialization, please let me know!\r\n\r\n2. Turns out\r\n[`getService('ml').importTrainedModel()`](https://github.com/elastic/kibana/blob/f18224c6869ae52228da3764ca9a427106b872fb/x-pack/test/functional/services/ml/api.ts#L1575-L1587)\r\ncan't be run in test env's with `ssl:true`, which is the default\r\nsecurity config. You can read more about that issue in\r\n[#193477](#193477), but the\r\ncurrent workaround is to turn off `ssl` for this specific test\r\nconfiguration, so that's why\r\n[`ess.config.ts`](https://github.com/spong/kibana/blob/cf73d4c7fcd69207a9625046456a94212da833c7/x-pack/test/security_solution_api_integration/test_suites/genai/knowledge_base/entries/trial_license_complete_tier/configs/ess.config.ts#L22)\r\nlooks a little different. If there's a better way to manage this config,\r\nalso please let me know!\r\n\r\n##### Additional notes\r\n\r\nWe don't currently have a `securityAssistant` API client/service to use\r\nin integration tests, so I've just been creating one-off functions using\r\n`supertest` for now. I don't have the bandwidth to work this now, but\r\nperhaps @MadameSheema / @muskangulati-qasource could lend a hand here? I\r\ndid need to test multi-user and multi-space scenarios, so I ported over\r\nthe same [auth\r\nhelpers](https://github.com/elastic/kibana/tree/dc26f1012f35c2445028a87dcc8cb3f063e058b0/x-pack/test/security_solution_api_integration/test_suites/genai/knowledge_base/entries/utils/auth)\r\nI saw used in other suites. Would be nice if these were bundled into the\r\nclient as well ala how the o11y folks have done it\r\n[here](https://github.com/elastic/kibana/blob/e9f23aa98e3abadd491be61b17e7daa3cc110cdb/x-pack/test/observability_ai_assistant_api_integration/tests/knowledge_base/knowledge_base.spec.ts#L27-L34).\r\nPerhaps this is also on the list of things for @maximpn to generate from\r\nOAS's.... 🙃\r\n\r\n### RBAC\r\nIn plumbing the UI, I've tried to place `// TODO: KB-RBAC` tags in all\r\nthe places I came across that will require an RBAC check/change. This\r\nincludes some of the API integration tests, which I currently have\r\nskipped as they would fail without RBAC.\r\n\r\n### Other notable changes\r\n\r\n* There are now dedicated `legacy` and `v2` helper functions when\r\nmanaging persistence/retrieval of knowledge base entries. This should\r\nhelp with tearing out the old KB later, and better readability now.\r\n* I've tried to remove dependency on the `ElasticsearchStore` as much as\r\npossible. The store's only use should now be within tools as a retriever\r\n[here](https://github.com/elastic/kibana/blob/de89153368848397df823c062e907a607d347dff/x-pack/plugins/elastic_assistant/server/routes/helpers.ts#L397-L405),\r\nand in post_evaluate\r\n[here](https://github.com/elastic/kibana/blob/de89153368848397df823c062e907a607d347dff/x-pack/plugins/elastic_assistant/server/routes/evaluate/post_evaluate.ts#L170-L179).\r\nIf we adopt the new\r\n[`naturalLanguageToESQL`](https://github.com/elastic/kibana/pull/192042)\r\ntool in `8.16` (or update our existing ESQL tool to use the\r\n`kbDataClient` for retrieval), we should be able to get rid of this\r\nentirely.\r\n* Added a\r\n[`spaces_roles_users_data.http`](https://github.com/elastic/kibana/blob/7447394fe39d5e2e098c266c14875d3aa17d3067/x-pack/packages/kbn-elastic-assistant-common/impl/utils/spaces_roles_users_data.http#L1)\r\nfile for adding spaces, roles, users, and a sample `slackbot` index for\r\nuse with [sample `IndexEntries`\r\nhere](https://github.com/elastic/kibana/blob/7447394fe39d5e2e098c266c14875d3aa17d3067/x-pack/packages/kbn-elastic-assistant-common/impl/schemas/knowledge_base/entries/crud_knowledge_base_entries_route.http#L18-L56).\r\n\r\n### // TODO\r\nIn effort to make incremental progress and facilitate early knowledge\r\nshare with @patrykkopycinski, I'm capping this PR where it's at, and so\r\nhere are the remaining items to complete full integration of the new\r\nKnowledge Base Management Settings interface:\r\n\r\n- [ ] Support `Update` action\r\n- [ ] Move from `EuiInMemoryTable` \r\n- [ ] Finalize `Setup` UI\r\n- [ ] Cleanup `Save` loaders\r\n- [ ] Plumb through `{{knowledge_history}}` prompt template and include\r\nuse's `required` entries\r\n\r\nAll this work is behind the aforementioned feature flag and required\r\ncode change, and this changeset has also been manually upgrade tested to\r\nensure there are no issues that would impact the regularly scheduled\r\nserverless releases. This is more of a note to reviewers when testing\r\nthat full functionality is not present.\r\n\r\n\r\n\r\n\r\n### Checklist\r\n\r\n- [X] Any text added follows [EUI's writing\r\nguidelines](https://elastic.github.io/eui/#/guidelines/writing), uses\r\nsentence case text and includes [i18n\r\nsupport](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)\r\n- [ ]\r\n[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)\r\nwas added for features that require explanation or tutorials\r\n* Feature currently behind feature flag. Documentation to be added\r\nbefore flag is removed. Tracked in\r\nhttps://github.com/elastic/security-docs/issues/5337\r\n- [X] [Unit or functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere updated or added to match the most common scenarios\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine <[email protected]>\r\nCo-authored-by: Elastic Machine <[email protected]>","sha":"63730ea0c9d9b036a05cb919b25b6d19c2ea8f03","branchLabelMapping":{"^v9.0.0$":"main","^v8.16.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","v9.0.0","Feature:Security Assistant","Team:Security Generative AI","v8.16.0","backport:version"],"number":192665,"url":"https://github.com/elastic/kibana/pull/192665","mergeCommit":{"message":"[Security Assistant] Adds new Knowledge Base Management Settings UI (#192665)\n\n## Summary\r\n\r\nThis PR updates the Knowledge Base Management Settings page to use the\r\nnew `entries` API introduced in\r\nhttps://github.com//pull/186566. Many thanks to @angorayc\r\nfor her work on the Assistant Management Settings overhaul, and initial\r\nimplementation of this new KB Management UI over in\r\nhttps://github.com//pull/186847.\r\n\r\n<p align=\"center\">\r\n<img width=\"600\"\r\nsrc=\"https://github.com/user-attachments/assets/0a82587e-f33c-45f1-9165-1a676d6db5fa\"\r\n/>\r\n</p> \r\n\r\n\r\n\r\n### Feature Flag & Setup\r\nThe changes in this PR, as with the other [recent V2 KB\r\nenhancements](#186566), are behind\r\nthe following feature flag:\r\n```\r\nxpack.securitySolution.enableExperimental:\r\n - 'assistantKnowledgeBaseByDefault'\r\n```\r\n\r\n~They also require a code change in the `AIAssistantService` to enable\r\nthe new mapping (since setup happens on plugin start before FF\r\nregistration), so be sure to update `fieldMap` to\r\n`knowledgeBaseFieldMapV2` below before testing:~\r\n\r\nThis is no longer the case as of\r\n[cdec104](https://github.com/elastic/kibana/pull/192665/commits/cdec10402f2e9b889598693f9f415c98ccd9855c).\r\nJust changing the above feature flag is now sufficient, just note that\r\nif upgrading and the KB was previously setup, you'll need to manually\r\ndelete the data stream (`DELETE\r\n/_data_stream/.kibana-elastic-ai-assistant-knowledge-base-default`) or\r\nthe management table will be littered with the old ESQL docs instead of\r\nbeing a single aggregate entry.\r\n\r\nOnce configured, the new Knowledge Base Management Settings will become\r\navailable in Stack Management. The old settings UI is currently still\r\navailable via the Settings Modal, but will soon be removed and replaced\r\nwith links to the new interface via the Assistant Settings Context Menu\r\n(replacing the existing `cog`). Please see the designs ([Security\r\nGenAI](https://www.figma.com/design/BMvpY9EhcPIaoOS7LSrkL0/%5B8.15%2C-%5D-GenAI-Security-Settings?node-id=51-25207&node-type=canvas&t=t3vZSPhMxQhScJVt-0)\r\n/ [Unified AI\r\nAssistant](https://www.figma.com/design/xN20zMRNtMlirWB6n9n1xJ/Unified-AI-Assistant-Settings?node-id=0-1&node-type=canvas&t=3RDYE7h2DjLlFlcN-0))\r\nfor all changes.\r\n\r\n> [!IMPORTANT]\r\n> There are no migrations in place between the legacy and v2 KB\r\nmappings, so be sure to start with a clean ES data directory.\r\n\r\n### Testing\r\n\r\nTo aid with developing the UI, I took the opportunity to start fleshing\r\nout the KB Entries API integration tests. These live in\r\n[x-pack/test/security_solution_api_integration/test_suites/genai/knowledge_base/entries](https://github.com/spong/kibana/tree/7ae6be136ad992b2163df13b55118556b01b6cb9/x-pack/test/security_solution_api_integration/test_suites/genai/knowledge_base/entries),\r\nand are currently configured to only run on `@ess`, as running\r\n`tiny_elser` in serverless and MKI environments can be tricky (more on\r\nthat later).\r\n\r\nTo start the server and run the tests, from the\r\n`x-pack/test/security_solution_api_integration/` directory run `yarn\r\ngenai_kb_entries:server:ess`, and once started, `yarn\r\ngenai_kb_entries:runner:ess`.\r\n\r\n##### Changes in support of testing\r\n\r\nIn order to setup the API integration tests for use with the Knowledge\r\nBase, some functional changes needed to be made to the assistant/config:\r\n\r\n1. Since ELSER is a heavy model to run in CI, the ML folks have created\r\n`pt_tiny_elser` for use in testing. Unfortunately, the `getELSER()`\r\nhelper off the `ml` client that we use to get the `modelld` for\r\ninstalling ELSER, ingest pipelines, etc, cannot be overridden\r\n([#193633](#193633)), so we must\r\nhave some other means of doing that. So to get things working in the\r\ntest env, I've plumbed through an optional `modelId` override to the\r\nPOST knowledge base route (`/ internal/ elastic_assistant/\r\nknowledge_base/{resource?}?modelId=pt_tiny_elser`). This then overrides\r\nthe aiAssistantService `getELSER()` function [when\r\nfetching](https://github.com/elastic/kibana/blob/645b3b863be16d70b8a7130a84b248c19729c340/x-pack/plugins/elastic_assistant/server/ai_assistant_service/index.ts#L334-L354)\r\na `kbDataClient` using the request, which appears to be the only way to\r\nalso trigger a reinitialization of the ingest pipeline (which required\r\nthe `modelId`), since that usually only occurs on plugin start. If there\r\nis a cleaner way to perform this reinitialization, please let me know!\r\n\r\n2. Turns out\r\n[`getService('ml').importTrainedModel()`](https://github.com/elastic/kibana/blob/f18224c6869ae52228da3764ca9a427106b872fb/x-pack/test/functional/services/ml/api.ts#L1575-L1587)\r\ncan't be run in test env's with `ssl:true`, which is the default\r\nsecurity config. You can read more about that issue in\r\n[#193477](#193477), but the\r\ncurrent workaround is to turn off `ssl` for this specific test\r\nconfiguration, so that's why\r\n[`ess.config.ts`](https://github.com/spong/kibana/blob/cf73d4c7fcd69207a9625046456a94212da833c7/x-pack/test/security_solution_api_integration/test_suites/genai/knowledge_base/entries/trial_license_complete_tier/configs/ess.config.ts#L22)\r\nlooks a little different. If there's a better way to manage this config,\r\nalso please let me know!\r\n\r\n##### Additional notes\r\n\r\nWe don't currently have a `securityAssistant` API client/service to use\r\nin integration tests, so I've just been creating one-off functions using\r\n`supertest` for now. I don't have the bandwidth to work this now, but\r\nperhaps @MadameSheema / @muskangulati-qasource could lend a hand here? I\r\ndid need to test multi-user and multi-space scenarios, so I ported over\r\nthe same [auth\r\nhelpers](https://github.com/elastic/kibana/tree/dc26f1012f35c2445028a87dcc8cb3f063e058b0/x-pack/test/security_solution_api_integration/test_suites/genai/knowledge_base/entries/utils/auth)\r\nI saw used in other suites. Would be nice if these were bundled into the\r\nclient as well ala how the o11y folks have done it\r\n[here](https://github.com/elastic/kibana/blob/e9f23aa98e3abadd491be61b17e7daa3cc110cdb/x-pack/test/observability_ai_assistant_api_integration/tests/knowledge_base/knowledge_base.spec.ts#L27-L34).\r\nPerhaps this is also on the list of things for @maximpn to generate from\r\nOAS's.... 🙃\r\n\r\n### RBAC\r\nIn plumbing the UI, I've tried to place `// TODO: KB-RBAC` tags in all\r\nthe places I came across that will require an RBAC check/change. This\r\nincludes some of the API integration tests, which I currently have\r\nskipped as they would fail without RBAC.\r\n\r\n### Other notable changes\r\n\r\n* There are now dedicated `legacy` and `v2` helper functions when\r\nmanaging persistence/retrieval of knowledge base entries. This should\r\nhelp with tearing out the old KB later, and better readability now.\r\n* I've tried to remove dependency on the `ElasticsearchStore` as much as\r\npossible. The store's only use should now be within tools as a retriever\r\n[here](https://github.com/elastic/kibana/blob/de89153368848397df823c062e907a607d347dff/x-pack/plugins/elastic_assistant/server/routes/helpers.ts#L397-L405),\r\nand in post_evaluate\r\n[here](https://github.com/elastic/kibana/blob/de89153368848397df823c062e907a607d347dff/x-pack/plugins/elastic_assistant/server/routes/evaluate/post_evaluate.ts#L170-L179).\r\nIf we adopt the new\r\n[`naturalLanguageToESQL`](https://github.com/elastic/kibana/pull/192042)\r\ntool in `8.16` (or update our existing ESQL tool to use the\r\n`kbDataClient` for retrieval), we should be able to get rid of this\r\nentirely.\r\n* Added a\r\n[`spaces_roles_users_data.http`](https://github.com/elastic/kibana/blob/7447394fe39d5e2e098c266c14875d3aa17d3067/x-pack/packages/kbn-elastic-assistant-common/impl/utils/spaces_roles_users_data.http#L1)\r\nfile for adding spaces, roles, users, and a sample `slackbot` index for\r\nuse with [sample `IndexEntries`\r\nhere](https://github.com/elastic/kibana/blob/7447394fe39d5e2e098c266c14875d3aa17d3067/x-pack/packages/kbn-elastic-assistant-common/impl/schemas/knowledge_base/entries/crud_knowledge_base_entries_route.http#L18-L56).\r\n\r\n### // TODO\r\nIn effort to make incremental progress and facilitate early knowledge\r\nshare with @patrykkopycinski, I'm capping this PR where it's at, and so\r\nhere are the remaining items to complete full integration of the new\r\nKnowledge Base Management Settings interface:\r\n\r\n- [ ] Support `Update` action\r\n- [ ] Move from `EuiInMemoryTable` \r\n- [ ] Finalize `Setup` UI\r\n- [ ] Cleanup `Save` loaders\r\n- [ ] Plumb through `{{knowledge_history}}` prompt template and include\r\nuse's `required` entries\r\n\r\nAll this work is behind the aforementioned feature flag and required\r\ncode change, and this changeset has also been manually upgrade tested to\r\nensure there are no issues that would impact the regularly scheduled\r\nserverless releases. This is more of a note to reviewers when testing\r\nthat full functionality is not present.\r\n\r\n\r\n\r\n\r\n### Checklist\r\n\r\n- [X] Any text added follows [EUI's writing\r\nguidelines](https://elastic.github.io/eui/#/guidelines/writing), uses\r\nsentence case text and includes [i18n\r\nsupport](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)\r\n- [ ]\r\n[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)\r\nwas added for features that require explanation or tutorials\r\n* Feature currently behind feature flag. Documentation to be added\r\nbefore flag is removed. Tracked in\r\nhttps://github.com/elastic/security-docs/issues/5337\r\n- [X] [Unit or functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere updated or added to match the most common scenarios\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine <[email protected]>\r\nCo-authored-by: Elastic Machine <[email protected]>","sha":"63730ea0c9d9b036a05cb919b25b6d19c2ea8f03"}},"sourceBranch":"main","suggestedTargetBranches":["8.x"],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","labelRegex":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/192665","number":192665,"mergeCommit":{"message":"[Security Assistant] Adds new Knowledge Base Management Settings UI (#192665)\n\n## Summary\r\n\r\nThis PR updates the Knowledge Base Management Settings page to use the\r\nnew `entries` API introduced in\r\nhttps://github.com//pull/186566. Many thanks to @angorayc\r\nfor her work on the Assistant Management Settings overhaul, and initial\r\nimplementation of this new KB Management UI over in\r\nhttps://github.com//pull/186847.\r\n\r\n<p align=\"center\">\r\n<img width=\"600\"\r\nsrc=\"https://github.com/user-attachments/assets/0a82587e-f33c-45f1-9165-1a676d6db5fa\"\r\n/>\r\n</p> \r\n\r\n\r\n\r\n### Feature Flag & Setup\r\nThe changes in this PR, as with the other [recent V2 KB\r\nenhancements](#186566), are behind\r\nthe following feature flag:\r\n```\r\nxpack.securitySolution.enableExperimental:\r\n - 'assistantKnowledgeBaseByDefault'\r\n```\r\n\r\n~They also require a code change in the `AIAssistantService` to enable\r\nthe new mapping (since setup happens on plugin start before FF\r\nregistration), so be sure to update `fieldMap` to\r\n`knowledgeBaseFieldMapV2` below before testing:~\r\n\r\nThis is no longer the case as of\r\n[cdec104](https://github.com/elastic/kibana/pull/192665/commits/cdec10402f2e9b889598693f9f415c98ccd9855c).\r\nJust changing the above feature flag is now sufficient, just note that\r\nif upgrading and the KB was previously setup, you'll need to manually\r\ndelete the data stream (`DELETE\r\n/_data_stream/.kibana-elastic-ai-assistant-knowledge-base-default`) or\r\nthe management table will be littered with the old ESQL docs instead of\r\nbeing a single aggregate entry.\r\n\r\nOnce configured, the new Knowledge Base Management Settings will become\r\navailable in Stack Management. The old settings UI is currently still\r\navailable via the Settings Modal, but will soon be removed and replaced\r\nwith links to the new interface via the Assistant Settings Context Menu\r\n(replacing the existing `cog`). Please see the designs ([Security\r\nGenAI](https://www.figma.com/design/BMvpY9EhcPIaoOS7LSrkL0/%5B8.15%2C-%5D-GenAI-Security-Settings?node-id=51-25207&node-type=canvas&t=t3vZSPhMxQhScJVt-0)\r\n/ [Unified AI\r\nAssistant](https://www.figma.com/design/xN20zMRNtMlirWB6n9n1xJ/Unified-AI-Assistant-Settings?node-id=0-1&node-type=canvas&t=3RDYE7h2DjLlFlcN-0))\r\nfor all changes.\r\n\r\n> [!IMPORTANT]\r\n> There are no migrations in place between the legacy and v2 KB\r\nmappings, so be sure to start with a clean ES data directory.\r\n\r\n### Testing\r\n\r\nTo aid with developing the UI, I took the opportunity to start fleshing\r\nout the KB Entries API integration tests. These live in\r\n[x-pack/test/security_solution_api_integration/test_suites/genai/knowledge_base/entries](https://github.com/spong/kibana/tree/7ae6be136ad992b2163df13b55118556b01b6cb9/x-pack/test/security_solution_api_integration/test_suites/genai/knowledge_base/entries),\r\nand are currently configured to only run on `@ess`, as running\r\n`tiny_elser` in serverless and MKI environments can be tricky (more on\r\nthat later).\r\n\r\nTo start the server and run the tests, from the\r\n`x-pack/test/security_solution_api_integration/` directory run `yarn\r\ngenai_kb_entries:server:ess`, and once started, `yarn\r\ngenai_kb_entries:runner:ess`.\r\n\r\n##### Changes in support of testing\r\n\r\nIn order to setup the API integration tests for use with the Knowledge\r\nBase, some functional changes needed to be made to the assistant/config:\r\n\r\n1. Since ELSER is a heavy model to run in CI, the ML folks have created\r\n`pt_tiny_elser` for use in testing. Unfortunately, the `getELSER()`\r\nhelper off the `ml` client that we use to get the `modelld` for\r\ninstalling ELSER, ingest pipelines, etc, cannot be overridden\r\n([#193633](#193633)), so we must\r\nhave some other means of doing that. So to get things working in the\r\ntest env, I've plumbed through an optional `modelId` override to the\r\nPOST knowledge base route (`/ internal/ elastic_assistant/\r\nknowledge_base/{resource?}?modelId=pt_tiny_elser`). This then overrides\r\nthe aiAssistantService `getELSER()` function [when\r\nfetching](https://github.com/elastic/kibana/blob/645b3b863be16d70b8a7130a84b248c19729c340/x-pack/plugins/elastic_assistant/server/ai_assistant_service/index.ts#L334-L354)\r\na `kbDataClient` using the request, which appears to be the only way to\r\nalso trigger a reinitialization of the ingest pipeline (which required\r\nthe `modelId`), since that usually only occurs on plugin start. If there\r\nis a cleaner way to perform this reinitialization, please let me know!\r\n\r\n2. Turns out\r\n[`getService('ml').importTrainedModel()`](https://github.com/elastic/kibana/blob/f18224c6869ae52228da3764ca9a427106b872fb/x-pack/test/functional/services/ml/api.ts#L1575-L1587)\r\ncan't be run in test env's with `ssl:true`, which is the default\r\nsecurity config. You can read more about that issue in\r\n[#193477](#193477), but the\r\ncurrent workaround is to turn off `ssl` for this specific test\r\nconfiguration, so that's why\r\n[`ess.config.ts`](https://github.com/spong/kibana/blob/cf73d4c7fcd69207a9625046456a94212da833c7/x-pack/test/security_solution_api_integration/test_suites/genai/knowledge_base/entries/trial_license_complete_tier/configs/ess.config.ts#L22)\r\nlooks a little different. If there's a better way to manage this config,\r\nalso please let me know!\r\n\r\n##### Additional notes\r\n\r\nWe don't currently have a `securityAssistant` API client/service to use\r\nin integration tests, so I've just been creating one-off functions using\r\n`supertest` for now. I don't have the bandwidth to work this now, but\r\nperhaps @MadameSheema / @muskangulati-qasource could lend a hand here? I\r\ndid need to test multi-user and multi-space scenarios, so I ported over\r\nthe same [auth\r\nhelpers](https://github.com/elastic/kibana/tree/dc26f1012f35c2445028a87dcc8cb3f063e058b0/x-pack/test/security_solution_api_integration/test_suites/genai/knowledge_base/entries/utils/auth)\r\nI saw used in other suites. Would be nice if these were bundled into the\r\nclient as well ala how the o11y folks have done it\r\n[here](https://github.com/elastic/kibana/blob/e9f23aa98e3abadd491be61b17e7daa3cc110cdb/x-pack/test/observability_ai_assistant_api_integration/tests/knowledge_base/knowledge_base.spec.ts#L27-L34).\r\nPerhaps this is also on the list of things for @maximpn to generate from\r\nOAS's.... 🙃\r\n\r\n### RBAC\r\nIn plumbing the UI, I've tried to place `// TODO: KB-RBAC` tags in all\r\nthe places I came across that will require an RBAC check/change. This\r\nincludes some of the API integration tests, which I currently have\r\nskipped as they would fail without RBAC.\r\n\r\n### Other notable changes\r\n\r\n* There are now dedicated `legacy` and `v2` helper functions when\r\nmanaging persistence/retrieval of knowledge base entries. This should\r\nhelp with tearing out the old KB later, and better readability now.\r\n* I've tried to remove dependency on the `ElasticsearchStore` as much as\r\npossible. The store's only use should now be within tools as a retriever\r\n[here](https://github.com/elastic/kibana/blob/de89153368848397df823c062e907a607d347dff/x-pack/plugins/elastic_assistant/server/routes/helpers.ts#L397-L405),\r\nand in post_evaluate\r\n[here](https://github.com/elastic/kibana/blob/de89153368848397df823c062e907a607d347dff/x-pack/plugins/elastic_assistant/server/routes/evaluate/post_evaluate.ts#L170-L179).\r\nIf we adopt the new\r\n[`naturalLanguageToESQL`](https://github.com/elastic/kibana/pull/192042)\r\ntool in `8.16` (or update our existing ESQL tool to use the\r\n`kbDataClient` for retrieval), we should be able to get rid of this\r\nentirely.\r\n* Added a\r\n[`spaces_roles_users_data.http`](https://github.com/elastic/kibana/blob/7447394fe39d5e2e098c266c14875d3aa17d3067/x-pack/packages/kbn-elastic-assistant-common/impl/utils/spaces_roles_users_data.http#L1)\r\nfile for adding spaces, roles, users, and a sample `slackbot` index for\r\nuse with [sample `IndexEntries`\r\nhere](https://github.com/elastic/kibana/blob/7447394fe39d5e2e098c266c14875d3aa17d3067/x-pack/packages/kbn-elastic-assistant-common/impl/schemas/knowledge_base/entries/crud_knowledge_base_entries_route.http#L18-L56).\r\n\r\n### // TODO\r\nIn effort to make incremental progress and facilitate early knowledge\r\nshare with @patrykkopycinski, I'm capping this PR where it's at, and so\r\nhere are the remaining items to complete full integration of the new\r\nKnowledge Base Management Settings interface:\r\n\r\n- [ ] Support `Update` action\r\n- [ ] Move from `EuiInMemoryTable` \r\n- [ ] Finalize `Setup` UI\r\n- [ ] Cleanup `Save` loaders\r\n- [ ] Plumb through `{{knowledge_history}}` prompt template and include\r\nuse's `required` entries\r\n\r\nAll this work is behind the aforementioned feature flag and required\r\ncode change, and this changeset has also been manually upgrade tested to\r\nensure there are no issues that would impact the regularly scheduled\r\nserverless releases. This is more of a note to reviewers when testing\r\nthat full functionality is not present.\r\n\r\n\r\n\r\n\r\n### Checklist\r\n\r\n- [X] Any text added follows [EUI's writing\r\nguidelines](https://elastic.github.io/eui/#/guidelines/writing), uses\r\nsentence case text and includes [i18n\r\nsupport](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)\r\n- [ ]\r\n[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)\r\nwas added for features that require explanation or tutorials\r\n* Feature currently behind feature flag. Documentation to be added\r\nbefore flag is removed. Tracked in\r\nhttps://github.com/elastic/security-docs/issues/5337\r\n- [X] [Unit or functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere updated or added to match the most common scenarios\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine <[email protected]>\r\nCo-authored-by: Elastic Machine <[email protected]>","sha":"63730ea0c9d9b036a05cb919b25b6d19c2ea8f03"}},{"branch":"8.x","label":"v8.16.0","labelRegex":"^v8.16.0$","isSourceBranch":false,"state":"NOT_CREATED"}]}] BACKPORT-->
…194354) ## Summary This PR is a follow up to #192665 and addresses a bunch of feedback and fixes including: - [X] Adds support for updating/editing entries - [X] Fixes initial loading experience of the KB Settings Setup/Table - [X] Fixes two bugs where `semantic_text` and `text` must be declared for `IndexEntries` to work - [X] Add new Settings Context Menu items for KB and Alerts - [X] Add support for `required` entries in initial prompt * See [this trace](https://smith.langchain.com/public/84a17a31-8ce8-4bd9-911e-38a854484dd8/r) for included knowledge. Note that the KnowledgeBaseRetrievalTool was not selected. * Note: All prompts were updated to include the `{knowledge_history}` placeholder, and _not behind the feature flag_, as this will just be the empty case until the feature flag is enabled. TODO (in this or follow-up PR): - [ ] Add suggestions to `index` and `fields` inputs - [ ] Adds URL deeplinking to securityAssistantManagement - [ ] Fix bug where updating entry does not re-create embeddings (see [comment](#194354 (comment))) - [ ] Fix loading indicators when adding/editing entries - [ ] API integration tests for update API (@e40pud) ### Checklist Delete any items that are not applicable to this PR. - [X] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md) - [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials * Docs being tracked in elastic/security-docs#5337 for when feature flag is enabled - [ ] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios --------- Co-authored-by: kibanamachine <[email protected]> Co-authored-by: Patryk Kopycinski <[email protected]>
…lastic#194354) ## Summary This PR is a follow up to elastic#192665 and addresses a bunch of feedback and fixes including: - [X] Adds support for updating/editing entries - [X] Fixes initial loading experience of the KB Settings Setup/Table - [X] Fixes two bugs where `semantic_text` and `text` must be declared for `IndexEntries` to work - [X] Add new Settings Context Menu items for KB and Alerts - [X] Add support for `required` entries in initial prompt * See [this trace](https://smith.langchain.com/public/84a17a31-8ce8-4bd9-911e-38a854484dd8/r) for included knowledge. Note that the KnowledgeBaseRetrievalTool was not selected. * Note: All prompts were updated to include the `{knowledge_history}` placeholder, and _not behind the feature flag_, as this will just be the empty case until the feature flag is enabled. TODO (in this or follow-up PR): - [ ] Add suggestions to `index` and `fields` inputs - [ ] Adds URL deeplinking to securityAssistantManagement - [ ] Fix bug where updating entry does not re-create embeddings (see [comment](elastic#194354 (comment))) - [ ] Fix loading indicators when adding/editing entries - [ ] API integration tests for update API (@e40pud) ### Checklist Delete any items that are not applicable to this PR. - [X] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md) - [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials * Docs being tracked in elastic/security-docs#5337 for when feature flag is enabled - [ ] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios --------- Co-authored-by: kibanamachine <[email protected]> Co-authored-by: Patryk Kopycinski <[email protected]> (cherry picked from commit 7df3672)
…xes (#194354) (#195644) # Backport This will backport the following commits from `main` to `8.x`: - [[Security Assistant] V2 Knowledge Base Settings feedback and fixes (#194354)](#194354) <!--- Backport version: 9.4.3 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Garrett Spong","email":"[email protected]"},"sourceCommit":{"committedDate":"2024-10-09T16:17:47Z","message":"[Security Assistant] V2 Knowledge Base Settings feedback and fixes (#194354)\n\n## Summary\r\n\r\nThis PR is a follow up to #192665 and addresses a bunch of feedback and\r\nfixes including:\r\n\r\n- [X] Adds support for updating/editing entries\r\n- [X] Fixes initial loading experience of the KB Settings Setup/Table\r\n- [X] Fixes two bugs where `semantic_text` and `text` must be declared\r\nfor `IndexEntries` to work\r\n- [X] Add new Settings Context Menu items for KB and Alerts\r\n - [X] Add support for `required` entries in initial prompt\r\n* See [this\r\ntrace](https://smith.langchain.com/public/84a17a31-8ce8-4bd9-911e-38a854484dd8/r)\r\nfor included knowledge. Note that the KnowledgeBaseRetrievalTool was not\r\nselected.\r\n* Note: All prompts were updated to include the `{knowledge_history}`\r\nplaceholder, and _not behind the feature flag_, as this will just be the\r\nempty case until the feature flag is enabled.\r\n\r\nTODO (in this or follow-up PR):\r\n - [ ] Add suggestions to `index` and `fields` inputs\r\n - [ ] Adds URL deeplinking to securityAssistantManagement\r\n- [ ] Fix bug where updating entry does not re-create embeddings (see\r\n[comment](https://github.com/elastic/kibana/pull/194354#discussion_r1786475496))\r\n - [ ] Fix loading indicators when adding/editing entries\r\n - [ ] API integration tests for update API (@e40pud)\r\n\r\n\r\n### Checklist\r\n\r\nDelete any items that are not applicable to this PR.\r\n\r\n- [X] Any text added follows [EUI's writing\r\nguidelines](https://elastic.github.io/eui/#/guidelines/writing), uses\r\nsentence case text and includes [i18n\r\nsupport](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)\r\n- [ ]\r\n[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)\r\nwas added for features that require explanation or tutorials\r\n* Docs being tracked in\r\nhttps://github.com/elastic/security-docs/issues/5337 for when feature\r\nflag is enabled\r\n- [ ] [Unit or functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere updated or added to match the most common scenarios\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine <[email protected]>\r\nCo-authored-by: Patryk Kopycinski <[email protected]>","sha":"7df36721923159f45bc4fdbd26f76b20ad84249a","branchLabelMapping":{"^v9.0.0$":"main","^v8.16.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","v9.0.0","Feature:Security Assistant","Team:Security Generative AI","v8.16.0","backport:version"],"title":"[Security Assistant] V2 Knowledge Base Settings feedback and fixes","number":194354,"url":"https://github.com/elastic/kibana/pull/194354","mergeCommit":{"message":"[Security Assistant] V2 Knowledge Base Settings feedback and fixes (#194354)\n\n## Summary\r\n\r\nThis PR is a follow up to #192665 and addresses a bunch of feedback and\r\nfixes including:\r\n\r\n- [X] Adds support for updating/editing entries\r\n- [X] Fixes initial loading experience of the KB Settings Setup/Table\r\n- [X] Fixes two bugs where `semantic_text` and `text` must be declared\r\nfor `IndexEntries` to work\r\n- [X] Add new Settings Context Menu items for KB and Alerts\r\n - [X] Add support for `required` entries in initial prompt\r\n* See [this\r\ntrace](https://smith.langchain.com/public/84a17a31-8ce8-4bd9-911e-38a854484dd8/r)\r\nfor included knowledge. Note that the KnowledgeBaseRetrievalTool was not\r\nselected.\r\n* Note: All prompts were updated to include the `{knowledge_history}`\r\nplaceholder, and _not behind the feature flag_, as this will just be the\r\nempty case until the feature flag is enabled.\r\n\r\nTODO (in this or follow-up PR):\r\n - [ ] Add suggestions to `index` and `fields` inputs\r\n - [ ] Adds URL deeplinking to securityAssistantManagement\r\n- [ ] Fix bug where updating entry does not re-create embeddings (see\r\n[comment](https://github.com/elastic/kibana/pull/194354#discussion_r1786475496))\r\n - [ ] Fix loading indicators when adding/editing entries\r\n - [ ] API integration tests for update API (@e40pud)\r\n\r\n\r\n### Checklist\r\n\r\nDelete any items that are not applicable to this PR.\r\n\r\n- [X] Any text added follows [EUI's writing\r\nguidelines](https://elastic.github.io/eui/#/guidelines/writing), uses\r\nsentence case text and includes [i18n\r\nsupport](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)\r\n- [ ]\r\n[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)\r\nwas added for features that require explanation or tutorials\r\n* Docs being tracked in\r\nhttps://github.com/elastic/security-docs/issues/5337 for when feature\r\nflag is enabled\r\n- [ ] [Unit or functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere updated or added to match the most common scenarios\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine <[email protected]>\r\nCo-authored-by: Patryk Kopycinski <[email protected]>","sha":"7df36721923159f45bc4fdbd26f76b20ad84249a"}},"sourceBranch":"main","suggestedTargetBranches":["8.x"],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","branchLabelMappingKey":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/194354","number":194354,"mergeCommit":{"message":"[Security Assistant] V2 Knowledge Base Settings feedback and fixes (#194354)\n\n## Summary\r\n\r\nThis PR is a follow up to #192665 and addresses a bunch of feedback and\r\nfixes including:\r\n\r\n- [X] Adds support for updating/editing entries\r\n- [X] Fixes initial loading experience of the KB Settings Setup/Table\r\n- [X] Fixes two bugs where `semantic_text` and `text` must be declared\r\nfor `IndexEntries` to work\r\n- [X] Add new Settings Context Menu items for KB and Alerts\r\n - [X] Add support for `required` entries in initial prompt\r\n* See [this\r\ntrace](https://smith.langchain.com/public/84a17a31-8ce8-4bd9-911e-38a854484dd8/r)\r\nfor included knowledge. Note that the KnowledgeBaseRetrievalTool was not\r\nselected.\r\n* Note: All prompts were updated to include the `{knowledge_history}`\r\nplaceholder, and _not behind the feature flag_, as this will just be the\r\nempty case until the feature flag is enabled.\r\n\r\nTODO (in this or follow-up PR):\r\n - [ ] Add suggestions to `index` and `fields` inputs\r\n - [ ] Adds URL deeplinking to securityAssistantManagement\r\n- [ ] Fix bug where updating entry does not re-create embeddings (see\r\n[comment](https://github.com/elastic/kibana/pull/194354#discussion_r1786475496))\r\n - [ ] Fix loading indicators when adding/editing entries\r\n - [ ] API integration tests for update API (@e40pud)\r\n\r\n\r\n### Checklist\r\n\r\nDelete any items that are not applicable to this PR.\r\n\r\n- [X] Any text added follows [EUI's writing\r\nguidelines](https://elastic.github.io/eui/#/guidelines/writing), uses\r\nsentence case text and includes [i18n\r\nsupport](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)\r\n- [ ]\r\n[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)\r\nwas added for features that require explanation or tutorials\r\n* Docs being tracked in\r\nhttps://github.com/elastic/security-docs/issues/5337 for when feature\r\nflag is enabled\r\n- [ ] [Unit or functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere updated or added to match the most common scenarios\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine <[email protected]>\r\nCo-authored-by: Patryk Kopycinski <[email protected]>","sha":"7df36721923159f45bc4fdbd26f76b20ad84249a"}},{"branch":"8.x","label":"v8.16.0","branchLabelMappingKey":"^v8.16.0$","isSourceBranch":false,"state":"NOT_CREATED"}]}] BACKPORT--> Co-authored-by: Garrett Spong <[email protected]>
Summary
This PR updates the Knowledge Base Management Settings page to use the new
entries
API introduced in #186566. Many thanks to @angorayc for her work on the Assistant Management Settings overhaul, and initial implementation of this new KB Management UI over in #186847.Feature Flag & Setup
The changes in this PR, as with the other recent V2 KB enhancements, are behind the following feature flag:
They also require a code change in theAIAssistantService
to enable the new mapping (since setup happens on plugin start before FF registration), so be sure to updatefieldMap
toknowledgeBaseFieldMapV2
below before testing:This is no longer the case as of cdec104. Just changing the above feature flag is now sufficient, just note that if upgrading and the KB was previously setup, you'll need to manually delete the data stream (
DELETE /_data_stream/.kibana-elastic-ai-assistant-knowledge-base-default
) or the management table will be littered with the old ESQL docs instead of being a single aggregate entry.Once configured, the new Knowledge Base Management Settings will become available in Stack Management. The old settings UI is currently still available via the Settings Modal, but will soon be removed and replaced with links to the new interface via the Assistant Settings Context Menu (replacing the existing
cog
). Please see the designs (Security GenAI / Unified AI Assistant) for all changes.Important
There are no migrations in place between the legacy and v2 KB mappings, so be sure to start with a clean ES data directory.
Testing
To aid with developing the UI, I took the opportunity to start fleshing out the KB Entries API integration tests. These live in x-pack/test/security_solution_api_integration/test_suites/genai/knowledge_base/entries, and are currently configured to only run on
@ess
, as runningtiny_elser
in serverless and MKI environments can be tricky (more on that later).To start the server and run the tests, from the
x-pack/test/security_solution_api_integration/
directory runyarn genai_kb_entries:server:ess
, and once started,yarn genai_kb_entries:runner:ess
.Changes in support of testing
In order to setup the API integration tests for use with the Knowledge Base, some functional changes needed to be made to the assistant/config:
Since ELSER is a heavy model to run in CI, the ML folks have created
pt_tiny_elser
for use in testing. Unfortunately, thegetELSER()
helper off theml
client that we use to get themodelld
for installing ELSER, ingest pipelines, etc, cannot be overridden (#193633), so we must have some other means of doing that. So to get things working in the test env, I've plumbed through an optionalmodelId
override to the POST knowledge base route (/ internal/ elastic_assistant/ knowledge_base/{resource?}?modelId=pt_tiny_elser
). This then overrides the aiAssistantServicegetELSER()
function when fetching akbDataClient
using the request, which appears to be the only way to also trigger a reinitialization of the ingest pipeline (which required themodelId
), since that usually only occurs on plugin start. If there is a cleaner way to perform this reinitialization, please let me know!Turns out
getService('ml').importTrainedModel()
can't be run in test env's withssl:true
, which is the default security config. You can read more about that issue in #193477, but the current workaround is to turn offssl
for this specific test configuration, so that's whyess.config.ts
looks a little different. If there's a better way to manage this config, also please let me know!Additional notes
We don't currently have a
securityAssistant
API client/service to use in integration tests, so I've just been creating one-off functions usingsupertest
for now. I don't have the bandwidth to work this now, but perhaps @MadameSheema / @muskangulati-qasource could lend a hand here? I did need to test multi-user and multi-space scenarios, so I ported over the same auth helpers I saw used in other suites. Would be nice if these were bundled into the client as well ala how the o11y folks have done it here. Perhaps this is also on the list of things for @maximpn to generate from OAS's.... 🙃RBAC
In plumbing the UI, I've tried to place
// TODO: KB-RBAC
tags in all the places I came across that will require an RBAC check/change. This includes some of the API integration tests, which I currently have skipped as they would fail without RBAC.Other notable changes
legacy
andv2
helper functions when managing persistence/retrieval of knowledge base entries. This should help with tearing out the old KB later, and better readability now.ElasticsearchStore
as much as possible. The store's only use should now be within tools as a retriever here, and in post_evaluate here. If we adopt the newnaturalLanguageToESQL
tool in8.16
(or update our existing ESQL tool to use thekbDataClient
for retrieval), we should be able to get rid of this entirely.spaces_roles_users_data.http
file for adding spaces, roles, users, and a sampleslackbot
index for use with sampleIndexEntries
here.// TODO
In effort to make incremental progress and facilitate early knowledge share with @patrykkopycinski, I'm capping this PR where it's at, and so here are the remaining items to complete full integration of the new Knowledge Base Management Settings interface:
Update
actionEuiInMemoryTable
Setup
UISave
loaders{{knowledge_history}}
prompt template and include use'srequired
entriesAll this work is behind the aforementioned feature flag and required code change, and this changeset has also been manually upgrade tested to ensure there are no issues that would impact the regularly scheduled serverless releases. This is more of a note to reviewers when testing that full functionality is not present.
Checklist