-
Notifications
You must be signed in to change notification settings - Fork 8.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Security Solution][Detections] Implement endpoint for fetching installed Fleet integrations #132667
[Security Solution][Detections] Implement endpoint for fetching installed Fleet integrations #132667
Conversation
22d8531
to
9c7a7a8
Compare
74648a5
to
2166628
Compare
…ntegrations, Required Fields, and Setup (#131475) ## Summary Adds UI for new Rule Fields `Related Integrations`, `Required Fields`, and `Setup` to both the Rules Table and Rule Details pages. On the Rules Table a new column is added that shows the number of related integrations, and upon clicking will show you details about those integrations and links off to the integration page in fleet. On the Rule Details page `Setup` is added as a tab pill within the About section (if provided), and `Related Integrations` and `Required Fields` are displayed in the Definition section. Once package/integration install data is added in #132667, the UI will show the installed status of an integration, and whether or not the installed version satisfies the related integration dependency. NOTE: Until then, please follow the test instructions below for how to add a custom rule and return mock data to test the `installed/uninstalled` UI. ##### Related Issues * elastic/security-team#2083 (internal) * elastic/security-team#558 (internal) * elastic/security-team#2856 (internal) * elastic/security-team#1801 (internal) * elastic/security-team#3624 (internal) ##### Related Links * elastic/security-docs#2015 * [Figma mocks](https://www.figma.com/file/zZs8TOrYsp13T6Z2HoMMFN/%5B8.2%5D-Associate-prebuilt-rules-with-Related-Integrations?node-id=0%3A1) ##### Steps to test In this initial implementation these new fields are only visible with Prebuilt Rules, and so there is limited API support and currently no UI for editing them. If a Prebuilt Rule is duplicated, these fields are emptied (set to `''` or `[]`). When a Rule is exported these fields are included (as empty values), and it is possible to edit the `ndjson` and re-import and then see these fields for the Custom Rule (but still not editable in the UI). This is expected behavior, and is actually a nice and easy way to test. Here is a sample export you can paste into an `test.ndjson` file and import to test this feature. You can modify the `package`/`version` fields to test corner cases like if a package is installed but it's the wrong version. ``` {"id":"6cc39c80-da3a-11ec-9fce-65c1a0bee904","updated_at":"2022-05-23T01:48:23.422Z","updated_by":"elastic","created_at":"2022-05-23T01:48:20.940Z","created_by":"elastic","name":"Testing #131475, don't mind me...","tags":["Elastic","Endpoint Security"],"interval":"5m","enabled":false,"description":"Generates a detection alert each time an Elastic Endpoint Security alert is received. Enabling this rule allows you to immediately begin investigating your Endpoint alerts.","risk_score":47,"severity":"medium","license":"Elastic License v2","output_index":".siem-signals-default","meta":{"from":"5m"},"rule_name_override":"message","timestamp_override":"event.ingested","author":["Elastic"],"false_positives":[],"from":"now-600s","rule_id":"2c66bf23-6ae9-4eb2-859e-446bea181ae9","max_signals":10000,"risk_score_mapping":[{"field":"event.risk_score","operator":"equals","value":""}],"severity_mapping":[{"field":"event.severity","operator":"equals","severity":"low","value":"21"},{"field":"event.severity","operator":"equals","severity":"medium","value":"47"},{"field":"event.severity","operator":"equals","severity":"high","value":"73"},{"field":"event.severity","operator":"equals","severity":"critical","value":"99"}],"threat":[],"to":"now","references":[],"version":7,"exceptions_list":[{"id":"endpoint_list","list_id":"endpoint_list","namespace_type":"agnostic","type":"endpoint"}],"immutable":false,"related_integrations":[{"package":"system","version":"1.6.4"},{"package":"aws","integration":"cloudtrail","version":"1.11.0"}],"required_fields":[{"ecs":true,"name":"event.code","type":"keyword"},{"ecs":true,"name":"message","type":"match_only_text"},{"ecs":false,"name":"winlog.event_data.AttributeLDAPDisplayName","type":"keyword"},{"ecs":false,"name":"winlog.event_data.AttributeValue","type":"keyword"},{"ecs":false,"name":"winlog.event_data.ShareName","type":"keyword"},{"ecs":false,"name":"winlog.event_data.RelativeTargetName","type":"keyword"},{"ecs":false,"name":"winlog.event_data.AccessList","type":"keyword"}],"setup":"## Config\\n\\nThe 'Audit Detailed File Share' audit policy must be configured (Success Failure).\\nSteps to implement the logging policy with with Advanced Audit Configuration:\\n\\n```\\nComputer Configuration > \\nPolicies > \\nWindows Settings > \\nSecurity Settings > \\nAdvanced Audit Policies Configuration > \\nAudit Policies > \\nObject Access > \\nAudit Detailed File Share (Success,Failure)\\n```\\n\\nThe 'Audit Directory Service Changes' audit policy must be configured (Success Failure).\\nSteps to implement the logging policy with with Advanced Audit Configuration:\\n\\n```\\nComputer Configuration > \\nPolicies > \\nWindows Settings > \\nSecurity Settings > \\nAdvanced Audit Policies Configuration > \\nAudit Policies > \\nDS Access > \\nAudit Directory Service Changes (Success,Failure)\\n```\\n","type":"query","language":"kuery","index":["logs-endpoint.alerts-*"],"query":"event.kind:alert and event.module:(endpoint and not endgame)\\n","filters":[],"throttle":"no_actions","actions":[]} {"exported_count":1,"exported_rules_count":1,"missing_rules":[],"missing_rules_count":0,"exported_exception_list_count":0,"exported_exception_list_item_count":0,"missing_exception_list_item_count":0,"missing_exception_list_items":[],"missing_exception_lists":[],"missing_exception_lists_count":0} ``` ##### Existing plumbing for showing integration install state This PR includes a `useInstalledIntegrations` hook wired up to the `DETECTION_ENGINE_INSTALLED_INTEGRATIONS_URL` route to be added in #132667. I plumbed the initial logic as if that API returned an array of integrations in the same format stored by the rule (i.e. the `RelatedIntegrationArray` type), so this will need to be adapted when integrating this feature. There's also a `packages[]` that can be provided to `useInstalledIntegrations` to constrain the search against all installed integrations, but this may not be used in the initial API. To test the Installed Integrations UI, just uncomment the mock data return in `use_installed_integrations.tsx`. ##### Additional Notes/Todo: * Probably want to revisit the copy on the Rules Table integrations popover -- it reads a little off if we don't have have integration install data. Will work with docs folks on this. * No additional overflow logic was added for `Related Integrations`/`Required Fields`. We don't really have an overflow pattern for these description list items, so instead of just adding support for these two fields (as like another description list item that's a popover), would like to solve this for generically for all items. * TODO: Increase test coverage * ##### Rule Details <p align="center"> <img width="650" src="https://user-images.githubusercontent.com/2946766/169636465-fa9ac477-2175-40ea-8064-bc194e7c3cbc.png" /> </p> <p align="center"> <img width="650" src="https://user-images.githubusercontent.com/2946766/169745657-a35cf8b8-fe9b-4580-b637-1c98e4e6f90a.png" /> </p> <p align="center"> <img width="650" src="https://user-images.githubusercontent.com/2946766/169745451-a02612b3-dd5b-46dc-a168-8823f9b1753d.png" /> </p> ##### Rule Details without `Installed Integrations` API changes <p align="center"> <img width="650" src="https://user-images.githubusercontent.com/2946766/169745986-6bef1d55-c305-4931-9845-96a0da76d030.png" /> </p> ##### Rules Table <p align="center"> <img width="650" src="https://user-images.githubusercontent.com/2946766/169636693-0d10bf67-9981-4815-a069-1439a7c5e1ce.png" /> </p> <p align="center"> <img width="650" src="https://user-images.githubusercontent.com/2946766/169636541-ac279369-c753-4184-b25b-c4352586f347.png" /> </p> ##### Rules Table without `Installed Integrations` API changes <p align="center"> <img width="650" src="https://user-images.githubusercontent.com/2946766/169636630-983277aa-6211-487c-a50f-8cf000446436.png" /> </p> <p align="center"> <img width="650" src="https://user-images.githubusercontent.com/2946766/169636649-fb03f44f-e28e-4a3f-8944-d3e300ed94c8.png" /> </p> ##### Version mismatch In cases where the related package/integration is installed, but the version is not satisfied, a warning icon/tooltip will display next to the integration link letting the user know the installed vs targeted version. I just wanted to make sure this case was handled so copy/UI isn't final -- any feedback welcome here! 🙂 <p align="center"> <img width="650" src="https://user-images.githubusercontent.com/2946766/169730187-6269ce6c-6833-4455-83b8-e98c8dcfa387.png" /> </p> <p align="center"> <img width="650" src="https://user-images.githubusercontent.com/2946766/169745221-c0ac5e90-a89b-420a-8ade-deedfc1d1194.png" /> </p> ### Checklist Delete any items that are not applicable to this PR. - [X] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md) - [X] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials - Collaborating with docs folks on this dedicated docs issue: elastic/security-docs#2015 - [X] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - Some added, but need to follow-up with additional tests around versioning logic once we finalize installed integrations API - [X] Any UI touched in this PR is usable by keyboard only (learn more about [keyboard accessibility](https://webaim.org/techniques/keyboard/)) (https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US)) - [X] This renders correctly on smaller devices using a responsive layout. (You can test this [in your browser](https://www.browserstack.com/guide/responsive-testing-on-local-server))
2166628
to
d3864e1
Compare
getInternalFleetServices: jest.fn(() => { | ||
// TODO: Mock EndpointInternalFleetServicesInterface and return the mocked object. | ||
throw new Error('Not implemented'); | ||
}), | ||
getScopedFleetServices: jest.fn((req: KibanaRequest) => { | ||
// TODO: Mock EndpointScopedFleetServicesInterface and return the mocked object. | ||
throw new Error('Not implemented'); | ||
}), |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Will be done in a follow-up PR with test coverage.
Pinging @elastic/security-detections-response (Team:Detections and Resp) |
Pinging @elastic/security-solution (Team: SecuritySolution) |
/** | ||
* Returns an array of installed Fleet integrations and their packages. | ||
*/ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
May be worth adding a quick summary on the current method for collecting this information -- that we're first querying all packagePolicies
to determine all installed integrations, then querying for each applicable package to augment with additional data.
💚 Build SucceededMetrics [docs]Module Count
Public APIs missing comments
Public APIs missing exports
History
To update your PR or re-run it, just comment with: cc @banderror |
/** | ||
* Basic information about an installed Fleet package. | ||
*/ | ||
export interface InstalledPackageBasicInfo { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We may want to add iconPath
in the future so we can easily reference the package/integrations icon in the UI. There's no requirement for this now, and would be easy to add later with how you've structured things over in createInstalledIntegrationSet
, but just wanted to mention this additional data that may be useful.
logger, | ||
core, | ||
plugins, | ||
endpointAppContextService: this.endpointAppContextService, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
These services are already exposed on the endpointAppContextService
a few lines below:
kibana/x-pack/plugins/security_solution/server/plugin.ts
Lines 164 to 169 in d3864e1
const endpointContext: EndpointAppContext = { | |
logFactory: pluginContext.logger, | |
service: this.endpointAppContextService, | |
config: (): Promise<ConfigType> => Promise.resolve(config), | |
experimentalFeatures, | |
}; |
So another option would be just wiring up the endpointContext
through initRoutes
and passing the existing context to the route ala:
readRulesRoute(router, logger, endpointAppContext);
It's been a bit since I've been in our plugin setup code, so not sure of all the trade-offs here, but wanted to mention since this was the route (har har 😅) I took over in my initial spike branch.
integrations: Map<string, InstalledIntegrationBasicInfo>; | ||
} | ||
|
||
export const createInstalledIntegrationSet = (): IInstalledIntegrationSet => { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
As with the route, may be good to have a brief summary following the logic/construction of this data.
set.getPackages().map((packageInfo) => { | ||
return fleet.packages.getRegistryPackage( | ||
packageInfo.package_name, | ||
packageInfo.package_version | ||
); | ||
}) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We should reach out to the fleet folks to see why there's an HTTP API for getting all packages but not a corresponding client API. Feels weird that we need to make a request for each individual package. Will be interesting to see how this scales on deployments with >100+ installed packages 😅
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Checked out, verified locally, code reviewed, aaaand pair code reviewed with @banderror as well -- LGTM! 🙂 🚀:tada:
Thanks for the quick turnaround here @banderror, and for taking the time earlier today to review and discuss both the new interfaces and our options for constructing this data from the existing fleet services, appreciate it!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Alerts area changes LGTM
…o interface (#132847) ## Summary Wires up the new Installed Integrations API added in #132667 to the new Related Integrations UI added in #131475. #### Additional changes include (though not all necessary for this specific PR): - [X] Updates integrations badge icon to `package` on Rules Table - [ ] Add Kibana Advanced Setting for disabling integrations badge on Rules Table - [ ] Add loaders where necessary since there can now be API delay - [ ] Separate description step components to specific files Please see #131475 for screenshots and additional details. #### Steps to test In this initial implementation these new fields are only visible with Prebuilt Rules, and so there is limited API support and currently no UI for editing them. If a Prebuilt Rule is duplicated, these fields are emptied (set to `''` or `[]`). When a Rule is exported these fields are included (as empty values), and it is possible to edit the `ndjson` and re-import and then see these fields for the Custom Rule (but still not editable in the UI). This is expected behavior, and is actually a nice and easy way to test. Here is a sample export you can paste into a `test.ndjson` file and import to test this feature. You can modify the `package`/`version` fields to test corner cases like if a package is installed but it's the wrong version. ``` {"id":"6cc39c80-da3a-11ec-9fce-65c1a0bee904","updated_at":"2022-05-23T01:48:23.422Z","updated_by":"elastic","created_at":"2022-05-23T01:48:20.940Z","created_by":"elastic","name":"Testing #131475, don't mind me...","tags":["Elastic","Endpoint Security"],"interval":"5m","enabled":false,"description":"Generates a detection alert each time an Elastic Endpoint Security alert is received. Enabling this rule allows you to immediately begin investigating your Endpoint alerts.","risk_score":47,"severity":"medium","license":"Elastic License v2","output_index":".siem-signals-default","meta":{"from":"5m"},"rule_name_override":"message","timestamp_override":"event.ingested","author":["Elastic"],"false_positives":[],"from":"now-600s","rule_id":"2c66bf23-6ae9-4eb2-859e-446bea181ae9","max_signals":10000,"risk_score_mapping":[{"field":"event.risk_score","operator":"equals","value":""}],"severity_mapping":[{"field":"event.severity","operator":"equals","severity":"low","value":"21"},{"field":"event.severity","operator":"equals","severity":"medium","value":"47"},{"field":"event.severity","operator":"equals","severity":"high","value":"73"},{"field":"event.severity","operator":"equals","severity":"critical","value":"99"}],"threat":[],"to":"now","references":[],"version":7,"exceptions_list":[{"id":"endpoint_list","list_id":"endpoint_list","namespace_type":"agnostic","type":"endpoint"}],"immutable":false,"related_integrations":[{"package":"system","version":"1.6.4"},{"package":"aws","integration":"cloudtrail","version":"1.11.0"}],"required_fields":[{"ecs":true,"name":"event.code","type":"keyword"},{"ecs":true,"name":"message","type":"match_only_text"},{"ecs":false,"name":"winlog.event_data.AttributeLDAPDisplayName","type":"keyword"},{"ecs":false,"name":"winlog.event_data.AttributeValue","type":"keyword"},{"ecs":false,"name":"winlog.event_data.ShareName","type":"keyword"},{"ecs":false,"name":"winlog.event_data.RelativeTargetName","type":"keyword"},{"ecs":false,"name":"winlog.event_data.AccessList","type":"keyword"}],"setup":"## Config\\n\\nThe 'Audit Detailed File Share' audit policy must be configured (Success Failure).\\nSteps to implement the logging policy with with Advanced Audit Configuration:\\n\\n```\\nComputer Configuration > \\nPolicies > \\nWindows Settings > \\nSecurity Settings > \\nAdvanced Audit Policies Configuration > \\nAudit Policies > \\nObject Access > \\nAudit Detailed File Share (Success,Failure)\\n```\\n\\nThe 'Audit Directory Service Changes' audit policy must be configured (Success Failure).\\nSteps to implement the logging policy with with Advanced Audit Configuration:\\n\\n```\\nComputer Configuration > \\nPolicies > \\nWindows Settings > \\nSecurity Settings > \\nAdvanced Audit Policies Configuration > \\nAudit Policies > \\nDS Access > \\nAudit Directory Service Changes (Success,Failure)\\n```\\n","type":"query","language":"kuery","index":["logs-endpoint.alerts-*"],"query":"event.kind:alert and event.module:(endpoint and not endgame)\\n","filters":[],"throttle":"no_actions","actions":[]} {"exported_count":1,"exported_rules_count":1,"missing_rules":[],"missing_rules_count":0,"exported_exception_list_count":0,"exported_exception_list_item_count":0,"missing_exception_list_item_count":0,"missing_exception_list_items":[],"missing_exception_lists":[],"missing_exception_lists_count":0} ```
Addresses partially: https://github.com/elastic/security-team/issues/2856, https://github.com/elastic/security-team/issues/3624 (internal tickets)
Summary
Adds a new detections endpoint that returns installed Fleet integrations. It is to be used on the Rule Management and Rule Details pages (see #131475 for context and screenshots). This endpoint is
internal
- no need to document it.Next steps
package_name
andis_enabled
(will be done in a separate PR).Checklist
Delete any items that are not applicable to this PR.