[Security Solution] - Security solution ES|QL configurable via advanced setting #181616
Conversation
michaelolo24
changed the title
michaelolo24
added
Team:Threat Hunting:Investigations
There was a problem hiding this comment.
Neat! tested locally, the Timeline all works as expected.
We will only need to add the setting to our Security project settings list for serverless (code).
Nevertheless, we will have to be careful and do all the testing the moment we add this setting in serverless, since it is defaulted to true, just by adding it to this list we'll be enabling all ESQL features by default.
|
Thanks for improving ES|QL configuration in Kibana! We register ES|QL rule type in plugin https://github.com/elastic/kibana/blob/8.14/x-pack/plugins/security_solution/server/plugin.ts#L322-L324 if (config.settings.ESQLEnabled && !experimentalFeatures.esqlRulesDisabled) {
plugins.alerting.registerType(securityRuleTypeWrapper(createEsqlAlertType(ruleOptions)));
}I think this new advanced setting should be accounted here too. |
|
@michaelolo24 there are a couple of places on the server side that will need to be updated: kibana/x-pack/plugins/security_solution/server/plugin.ts Lines 323 to 325 in 60c6cdb |
| @@ -22,7 +18,6 @@ export interface ConfigSettings { | |||
| */ | |||
| export const defaultSettings: ConfigSettings = Object.freeze({ | |||
| ILMEnabled: true, | |||
| ESQLEnabled: true, | |||
There was a problem hiding this comment.
Remember to remove it from the serverless config
kibana/config/serverless.security.yml
Line 26 in 60c6cdb
|
Thanks @semd and @vitaliidm for the comments and letting me know about some spots left to update! I'll make the changes that you asked. I would like QA to take a look at this branch before it's merged, but will follow up with product about timelines for getting this in. Thanks again! |
@vitaliidm afaik we don't have access to the |
|
@stratoula do you have any problems with security making use of |
In that case new option should not affect rule type at all. |
michaelolo24
force-pushed
the
link-security-to-esql-advanced-setting
branch
from
2e45cfe to
2ef799e
Compare
|
@michaelolo24 I am so sorry i missed the ping. I am def fine of using it. We decided to not rename it (as changing this will mean that the customers that have already set it will need to re-set it). But we changed the description so it doesnt mention only Discover now. 
I think it would better to not create your own setting, I don't think we need so many settings for ESQL and will confuse the users even more. This setting doesnt provide any security against running ESQL queries, it just makes it harder from the UI. With that being said let me bring again with my team the discussion on renaming and moving to a general section. |
Great, thanks so much @stratoula . Makes sense. Yep, will wait on the final confirmation 😄 |
@vitaliidm and @semd after speaking with @paulewing we made the decision to align with how the rest of the stack currently works. When the advanced setting for ESQL is disabled, users are still able to edit old saved searches as well as any existing esql stack rules. They just are unable to create any new esql rules or new esql saved searches when the advanced setting is disabled. I had to make a minor change from the last update to get that logic to work for timeline, but the PR should match the expected scenario above. |
|
@michaelolo24 we decided to move to General section and rename to I will try to ask for fast review and also backport to 8.14 when CI is green |
@michaelolo24, @paulewing users still would be able to create ES|QL rule through API. Is this fine? |
michaelolo24
force-pushed
the
link-security-to-esql-advanced-setting
branch
from
2ef799e to
a4f8e9e
Compare
💚 All backports created successfully
Note: Successful backport PRs will be merged automatically after passing CI. Questions ?Please refer to the Backport tool documentation |
…ed setting (elastic#181616) ## Summary This PR links the ESQL functionality in security solution to the `discover:enableESQL` advanced setting. The advanced setting will only be present in ESS, but not serverless The way this should work to maintain parity with the rest of Kibana such as discover and stack rules: - By default ES|QL will be enabled across all Kibana - When the ES|QL advanced setting is disabled: - Timeline - ES|QL tab should not be accessible on any newly created timelines - Existing Timelines with an ES|QL query should still have the tab accessible - Rules - New ES|QL rule should not be available to be created in the *Rule Creation* workflow - Existing ES|QL rules should still run and be able to be edited **Timeline Demo Video:** https://github.com/elastic/kibana/assets/17211684/d5429be9-de37-43e2-882d-687b3371beb4 **Rules Demo Video:** https://github.com/elastic/kibana/assets/17211684/7df2fd11-bd2b-4e50-ad97-b6e1d0f7867a --------- Co-authored-by: Vitalii Dmyterko <[email protected]> Co-authored-by: kibanamachine <[email protected]> (cherry picked from commit 963391e)
💚 All backports created successfully
Note: Successful backport PRs will be merged automatically after passing CI. Questions ?Please refer to the Backport tool documentation |
💚 All backports created successfully
Note: Successful backport PRs will be merged automatically after passing CI. Questions ?Please refer to the Backport tool documentation |
…ed setting (elastic#181616) ## Summary This PR links the ESQL functionality in security solution to the `discover:enableESQL` advanced setting. The advanced setting will only be present in ESS, but not serverless The way this should work to maintain parity with the rest of Kibana such as discover and stack rules: - By default ES|QL will be enabled across all Kibana - When the ES|QL advanced setting is disabled: - Timeline - ES|QL tab should not be accessible on any newly created timelines - Existing Timelines with an ES|QL query should still have the tab accessible - Rules - New ES|QL rule should not be available to be created in the *Rule Creation* workflow - Existing ES|QL rules should still run and be able to be edited **Timeline Demo Video:** https://github.com/elastic/kibana/assets/17211684/d5429be9-de37-43e2-882d-687b3371beb4 **Rules Demo Video:** https://github.com/elastic/kibana/assets/17211684/7df2fd11-bd2b-4e50-ad97-b6e1d0f7867a --------- Co-authored-by: Vitalii Dmyterko <[email protected]> Co-authored-by: kibanamachine <[email protected]> (cherry picked from commit 963391e)
… advanced setting (#181616) (#182517) # Backport This will backport the following commits from `main` to `8.14`: - [[Security Solution] - Security solution ES|QL configurable via advanced setting (#181616)](#181616) <!--- Backport version: 8.9.8 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Michael Olorunnisola","email":"[email protected]"},"sourceCommit":{"committedDate":"2024-05-02T17:02:46Z","message":"[Security Solution] - Security solution ES|QL configurable via advanced setting (#181616)\n\n## Summary\r\n\r\nThis PR links the ESQL functionality in security solution to the\r\n`discover:enableESQL` advanced setting. The advanced setting will only\r\nbe present in ESS, but not serverless\r\n\r\nThe way this should work to maintain parity with the rest of Kibana such\r\nas discover and stack rules:\r\n\r\n- By default ES|QL will be enabled across all Kibana\r\n- When the ES|QL advanced setting is disabled:\r\n - Timeline\r\n - ES|QL tab should not be accessible on any newly created timelines\r\n- Existing Timelines with an ES|QL query should still have the tab\r\naccessible\r\n - Rules\r\n- New ES|QL rule should not be available to be created in the *Rule\r\nCreation* workflow\r\n - Existing ES|QL rules should still run and be able to be edited\r\n\r\n\r\n\r\n**Timeline Demo Video:**\r\n\r\n\r\nhttps://github.com/elastic/kibana/assets/17211684/d5429be9-de37-43e2-882d-687b3371beb4\r\n\r\n**Rules Demo Video:**\r\n\r\n\r\n\r\nhttps://github.com/elastic/kibana/assets/17211684/7df2fd11-bd2b-4e50-ad97-b6e1d0f7867a\r\n\r\n---------\r\n\r\nCo-authored-by: Vitalii Dmyterko <[email protected]>\r\nCo-authored-by: kibanamachine <[email protected]>","sha":"963391ed0f7513c24c4827444feb971e460c53c0","branchLabelMapping":{"^v8.15.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:enhancement","Team:Threat Hunting:Investigations","ci:build-serverless-image","v8.14.0","v8.15.0"],"number":181616,"url":"https://github.com/elastic/kibana/pull/181616","mergeCommit":{"message":"[Security Solution] - Security solution ES|QL configurable via advanced setting (#181616)\n\n## Summary\r\n\r\nThis PR links the ESQL functionality in security solution to the\r\n`discover:enableESQL` advanced setting. The advanced setting will only\r\nbe present in ESS, but not serverless\r\n\r\nThe way this should work to maintain parity with the rest of Kibana such\r\nas discover and stack rules:\r\n\r\n- By default ES|QL will be enabled across all Kibana\r\n- When the ES|QL advanced setting is disabled:\r\n - Timeline\r\n - ES|QL tab should not be accessible on any newly created timelines\r\n- Existing Timelines with an ES|QL query should still have the tab\r\naccessible\r\n - Rules\r\n- New ES|QL rule should not be available to be created in the *Rule\r\nCreation* workflow\r\n - Existing ES|QL rules should still run and be able to be edited\r\n\r\n\r\n\r\n**Timeline Demo Video:**\r\n\r\n\r\nhttps://github.com/elastic/kibana/assets/17211684/d5429be9-de37-43e2-882d-687b3371beb4\r\n\r\n**Rules Demo Video:**\r\n\r\n\r\n\r\nhttps://github.com/elastic/kibana/assets/17211684/7df2fd11-bd2b-4e50-ad97-b6e1d0f7867a\r\n\r\n---------\r\n\r\nCo-authored-by: Vitalii Dmyterko <[email protected]>\r\nCo-authored-by: kibanamachine <[email protected]>","sha":"963391ed0f7513c24c4827444feb971e460c53c0"}},"sourceBranch":"main","suggestedTargetBranches":["8.14"],"targetPullRequestStates":[{"branch":"8.14","label":"v8.14.0","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v8.15.0","labelRegex":"^v8.15.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/181616","number":181616,"mergeCommit":{"message":"[Security Solution] - Security solution ES|QL configurable via advanced setting (#181616)\n\n## Summary\r\n\r\nThis PR links the ESQL functionality in security solution to the\r\n`discover:enableESQL` advanced setting. The advanced setting will only\r\nbe present in ESS, but not serverless\r\n\r\nThe way this should work to maintain parity with the rest of Kibana such\r\nas discover and stack rules:\r\n\r\n- By default ES|QL will be enabled across all Kibana\r\n- When the ES|QL advanced setting is disabled:\r\n - Timeline\r\n - ES|QL tab should not be accessible on any newly created timelines\r\n- Existing Timelines with an ES|QL query should still have the tab\r\naccessible\r\n - Rules\r\n- New ES|QL rule should not be available to be created in the *Rule\r\nCreation* workflow\r\n - Existing ES|QL rules should still run and be able to be edited\r\n\r\n\r\n\r\n**Timeline Demo Video:**\r\n\r\n\r\nhttps://github.com/elastic/kibana/assets/17211684/d5429be9-de37-43e2-882d-687b3371beb4\r\n\r\n**Rules Demo Video:**\r\n\r\n\r\n\r\nhttps://github.com/elastic/kibana/assets/17211684/7df2fd11-bd2b-4e50-ad97-b6e1d0f7867a\r\n\r\n---------\r\n\r\nCo-authored-by: Vitalii Dmyterko <[email protected]>\r\nCo-authored-by: kibanamachine <[email protected]>","sha":"963391ed0f7513c24c4827444feb971e460c53c0"}}]}] BACKPORT--> --------- Co-authored-by: Kibana Machine <[email protected]>
…hould disable ESQL Tab. (#182816) ## Summary handles #182798 Recently there was PR : #181616 which does not disable ES|QL tab in timeline even if feature flag is disabled when : - User has already created a ESQL Query in timeline and saved the timeline. This PR makes sure when below feature flag exists, then `ES|QL` tab will be definitely disabled even when user has a saved timeline with ES|QL Query in it. ```yaml xpack.securitySolution.enableExperimental: - timelineEsqlTabDisabled ``` ## Desk Testing Guidelines 1. Remove above Feature Flag 2. Go to Timeline and Create a Timeline with ESQL Query 3. Save the timeline. 4. Go To advanced Settings and search for `esql` and disable the `enableESQL` setting. 5. Go back to the timeline saved in step 3. 6. ✅ Assert that the `ES|QL` tab is still there. 7. ✅ Assert that the `ES|QL` tab is NOT there in a new timeline. 8. Add above feature flag - which disables the esql Tab. 9. Go back to the timeline saved in step 3. 10. ✅ Assert that the `ES|QL` tab is no longer there. 11. ✅ Assert that the `ES|QL` tab is NOT there in a new timeline. ### Checklist - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
…hould disable ESQL Tab. (elastic#182816) ## Summary handles elastic#182798 Recently there was PR : elastic#181616 which does not disable ES|QL tab in timeline even if feature flag is disabled when : - User has already created a ESQL Query in timeline and saved the timeline. This PR makes sure when below feature flag exists, then `ES|QL` tab will be definitely disabled even when user has a saved timeline with ES|QL Query in it. ```yaml xpack.securitySolution.enableExperimental: - timelineEsqlTabDisabled ``` ## Desk Testing Guidelines 1. Remove above Feature Flag 2. Go to Timeline and Create a Timeline with ESQL Query 3. Save the timeline. 4. Go To advanced Settings and search for `esql` and disable the `enableESQL` setting. 5. Go back to the timeline saved in step 3. 6. ✅ Assert that the `ES|QL` tab is still there. 7. ✅ Assert that the `ES|QL` tab is NOT there in a new timeline. 8. Add above feature flag - which disables the esql Tab. 9. Go back to the timeline saved in step 3. 10. ✅ Assert that the `ES|QL` tab is no longer there. 11. ✅ Assert that the `ES|QL` tab is NOT there in a new timeline. ### Checklist - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios (cherry picked from commit c43da3e)
… flag should disable ESQL Tab. (#182816) (#182905) # Backport This will backport the following commits from `main` to `8.14`: - [[Security Solution] Timeline : Disabling Timeline ESQL feature flag should disable ESQL Tab. (#182816)](#182816) <!--- Backport version: 9.4.3 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Jatin Kathuria","email":"[email protected]"},"sourceCommit":{"committedDate":"2024-05-08T06:21:30Z","message":"[Security Solution] Timeline : Disabling Timeline ESQL feature flag should disable ESQL Tab. (#182816)\n\n## Summary\r\n\r\nhandles https://github.com/elastic/kibana/issues/182798\r\n\r\nRecently there was PR : https://github.com/elastic/kibana/pull/181616\r\nwhich does not disable ES|QL tab in timeline even if feature flag is\r\ndisabled when :\r\n- User has already created a ESQL Query in timeline and saved the\r\ntimeline.\r\n\r\nThis PR makes sure when below feature flag exists, then `ES|QL` tab will\r\nbe definitely disabled even when user has a saved timeline with ES|QL\r\nQuery in it.\r\n\r\n```yaml\r\nxpack.securitySolution.enableExperimental:\r\n - timelineEsqlTabDisabled\r\n\r\n```\r\n\r\n## Desk Testing Guidelines\r\n\r\n1. Remove above Feature Flag\r\n2. Go to Timeline and Create a Timeline with ESQL Query\r\n3. Save the timeline.\r\n4. Go To advanced Settings and search for `esql` and disable the\r\n`enableESQL` setting.\r\n5. Go back to the timeline saved in step 3. \r\n6. ✅ Assert that the `ES|QL` tab is still there.\r\n7. ✅ Assert that the `ES|QL` tab is NOT there in a new timeline.\r\n8. Add above feature flag - which disables the esql Tab.\r\n9. Go back to the timeline saved in step 3.\r\n10. ✅ Assert that the `ES|QL` tab is no longer there.\r\n11. ✅ Assert that the `ES|QL` tab is NOT there in a new timeline.\r\n\r\n\r\n### Checklist\r\n\r\n- [x] [Unit or functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere updated or added to match the most common scenarios","sha":"c43da3e4349a6bb13cdab507530efd926147c30b","branchLabelMapping":{"^v8.15.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:fix","Team:Threat Hunting:Investigations","backport:prev-minor","v8.14.0","v8.15.0"],"title":"[Security Solution] Timeline : Disabling Timeline ESQL feature flag should disable ESQL Tab. ","number":182816,"url":"https://github.com/elastic/kibana/pull/182816","mergeCommit":{"message":"[Security Solution] Timeline : Disabling Timeline ESQL feature flag should disable ESQL Tab. (#182816)\n\n## Summary\r\n\r\nhandles https://github.com/elastic/kibana/issues/182798\r\n\r\nRecently there was PR : https://github.com/elastic/kibana/pull/181616\r\nwhich does not disable ES|QL tab in timeline even if feature flag is\r\ndisabled when :\r\n- User has already created a ESQL Query in timeline and saved the\r\ntimeline.\r\n\r\nThis PR makes sure when below feature flag exists, then `ES|QL` tab will\r\nbe definitely disabled even when user has a saved timeline with ES|QL\r\nQuery in it.\r\n\r\n```yaml\r\nxpack.securitySolution.enableExperimental:\r\n - timelineEsqlTabDisabled\r\n\r\n```\r\n\r\n## Desk Testing Guidelines\r\n\r\n1. Remove above Feature Flag\r\n2. Go to Timeline and Create a Timeline with ESQL Query\r\n3. Save the timeline.\r\n4. Go To advanced Settings and search for `esql` and disable the\r\n`enableESQL` setting.\r\n5. Go back to the timeline saved in step 3. \r\n6. ✅ Assert that the `ES|QL` tab is still there.\r\n7. ✅ Assert that the `ES|QL` tab is NOT there in a new timeline.\r\n8. Add above feature flag - which disables the esql Tab.\r\n9. Go back to the timeline saved in step 3.\r\n10. ✅ Assert that the `ES|QL` tab is no longer there.\r\n11. ✅ Assert that the `ES|QL` tab is NOT there in a new timeline.\r\n\r\n\r\n### Checklist\r\n\r\n- [x] [Unit or functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere updated or added to match the most common scenarios","sha":"c43da3e4349a6bb13cdab507530efd926147c30b"}},"sourceBranch":"main","suggestedTargetBranches":["8.14"],"targetPullRequestStates":[{"branch":"8.14","label":"v8.14.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v8.15.0","branchLabelMappingKey":"^v8.15.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/182816","number":182816,"mergeCommit":{"message":"[Security Solution] Timeline : Disabling Timeline ESQL feature flag should disable ESQL Tab. (#182816)\n\n## Summary\r\n\r\nhandles https://github.com/elastic/kibana/issues/182798\r\n\r\nRecently there was PR : https://github.com/elastic/kibana/pull/181616\r\nwhich does not disable ES|QL tab in timeline even if feature flag is\r\ndisabled when :\r\n- User has already created a ESQL Query in timeline and saved the\r\ntimeline.\r\n\r\nThis PR makes sure when below feature flag exists, then `ES|QL` tab will\r\nbe definitely disabled even when user has a saved timeline with ES|QL\r\nQuery in it.\r\n\r\n```yaml\r\nxpack.securitySolution.enableExperimental:\r\n - timelineEsqlTabDisabled\r\n\r\n```\r\n\r\n## Desk Testing Guidelines\r\n\r\n1. Remove above Feature Flag\r\n2. Go to Timeline and Create a Timeline with ESQL Query\r\n3. Save the timeline.\r\n4. Go To advanced Settings and search for `esql` and disable the\r\n`enableESQL` setting.\r\n5. Go back to the timeline saved in step 3. \r\n6. ✅ Assert that the `ES|QL` tab is still there.\r\n7. ✅ Assert that the `ES|QL` tab is NOT there in a new timeline.\r\n8. Add above feature flag - which disables the esql Tab.\r\n9. Go back to the timeline saved in step 3.\r\n10. ✅ Assert that the `ES|QL` tab is no longer there.\r\n11. ✅ Assert that the `ES|QL` tab is NOT there in a new timeline.\r\n\r\n\r\n### Checklist\r\n\r\n- [x] [Unit or functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere updated or added to match the most common scenarios","sha":"c43da3e4349a6bb13cdab507530efd926147c30b"}}]}] BACKPORT--> Co-authored-by: Jatin Kathuria <[email protected]>
Summary
This PR links the ESQL functionality in security solution to the
discover:enableESQLadvanced setting. The advanced setting will only be present in ESS, but not serverlessThe way this should work to maintain parity with the rest of Kibana such as discover and stack rules:
Timeline Demo Video:
Screen.Recording.2024-04-29.at.3.52.51.PM.mov
Rules Demo Video:
Screen.Recording.2024-04-29.at.3.54.24.PM.mov