[8.x] [Security Solution] Extend upgrade perform endpoint logic (#191439

) (#196471)

# Backport

This will backport the following commits from `main` to `8.x`:
- [[Security Solution] Extend upgrade perform endpoint logic
(#191439)](#191439)

<!--- Backport version: 9.4.3 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Juan Pablo
Djeredjian","email":"[email protected]"},"sourceCommit":{"committedDate":"2024-10-16T01:51:25Z","message":"[Security
Solution] Extend upgrade perform endpoint logic (#191439)\n\nFixes:
#166376 (main ticket)\r\nFixes:
#186544 (handling of\r\nspecific
fields)\r\nFixes: #180195
(replace PATCH\r\nwith PUT logic on rule upgrade)\r\n\r\n##
Summary\r\n\r\n- Enhances the `/upgrade/_perform` endpoint to upgrade
rules in a way\r\nthat works with prebuilt rules customized by users and
resolve conflicts\r\nbetween user customizations and updates from
Elastic.\r\n- Handles special fields under the hood (see below)\r\n-
Replaces the update prebuilt rule logic to work with PUT instead
of\r\nPATCH.\r\n\r\n### Rough implementation plan\r\n- For each
`upgradeableRule`, we attempt to build the payload necessary\r\nto pass
to `upgradePrebuiltRules()`, which is of type\r\n`PrebuiltRuleAsset`. So
we retrieve the field names from\r\n`FIELDS_PAYLOAD_BY_RULE_TYPE` and
loop through them.\r\n- If any of those `field`s are non-upgreadable,
(i.e. its value needs to\r\nbe handled under the hood) we do so in
`determineFieldUpgradeStatus`.\r\n- Otherwise, we continue to build a
`FieldUpgradeSpecifier` for each\r\nfield, which will help us determine
if that field needs to be set to the\r\nbase, current, target version,
OR if it needs to be calculated as a\r\nMERGED value, or it is passed in
the request payload as a RESOLVED\r\nvalue.\r\n- Notice that we are
iterating over \"flat\" (non-grouped) fields which\r\nare part of the
`PrebuiltRuleAsset` schema. This means that mapping is\r\nnecessary
between these flat fields and the diffable (grouped) fields\r\nthat are
used in the API contract, part of `DiffableRule`. For example,\r\nif we
try to determine the value for the `query` field, we will need
to\r\nlook up for its value in the `eql_query` field if the target rule
is\r\n`eql` or in `esql_query` if the target rule is `esql`. All
these\r\nmappings can be found in
`diffable_rule_fields_mappings.ts`.\r\n- Once a `FieldUpgradeSpecifier`
has been retrieved for each field of\r\nthe payload we are building,
retrieve its actual value: either fetching\r\nit from the base, current
or target versions of the rule, from the three\r\nway diff calculation,
or retrieving it from the request payload if it\r\nresolved.\r\n- Do
this for all upgreadable rules, and the pass the payload array
into\r\n`upgradePrebuiltRules()`.\r\n- **IMPORTANT:** The upgrade
prebuilt rules logic has been changed from\r\nPATCH to PUT. That means
that if the next version of a rule removes a\r\nfield, and the user
updates to that target version, those fields will be\r\nundefined in the
resulting rule. **Additional example:** a installs a\r\nrule, and
creates a `timeline_id` for it rule by modifying it. If\r\nneither the
next version (target version) still does not have a\r\n`timeline_id`
field for it, and the user updates to that target version\r\nfully
(without resolving the conflict), that field will not exist\r\nanymore
in the resulting rule.\r\n\r\n## Acceptance criteria\r\n\r\n- [x] Extend
the contract of the API endpoint according to
the\r\n[POC](https://github.com/elastic/kibana/pull/144060):\r\n- [x]
Add the ability to pick the `MERGED` version for rule upgrades.
If\r\nthe `MERGED` version is selected, the diffs are recalculated and
the\r\nrule fields are updated to the result of the diff calculation.
This is\r\nonly possible if all field diffs return a `conflict` value of
either\r\n`NO`. If any fields returns a value of `NON_SOLVABLE` or
`SOLVABLE`,\r\nreject the request with an error specifying that there
are conflicts,\r\nand that they must be resolved on a per-field
basis.\r\n- [x] Calculate diffs inside this endpoint, when the value
of\r\n`pick_version` is `MERGED`.\r\n- [x] Add the ability to specify
rule field versions, to update specific\r\nfields to different
`pick_versions`: `BASE' | 'CURRENT' | 'TARGET' |\r\n'MERGED' |
'RESOLVED'` (See `FieldUpgradeRequest`
in\r\n[PoC](#144060) for
details)\r\n\r\n## Handling of special fields\r\n\r\nSpecific fields are
handled under the hood based
on\r\nhttps://github.com//issues/186544\r\n\r\nSee
implementation
in\r\n`x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/api/perform_rule_upgrade/determine_field_upgrade_status.ts`,\r\nwhich
imports fields to handle under the hood:\r\n-
`DiffableFieldsToOmit`\r\n- `FieldsToUpdateToCurrentVersion`\r\n\r\n##
Edge cases\r\n\r\n- [x] If target version of rule has a **rule type
change**, check that\r\nall `pick_version`, at all levels, match
`TARGET`. Otherwise, create new\r\nerror and add to ruleErrors
array.\r\n- [x] if a rule has a specific `targetVersion.type` (for
example, EQL)\r\nand the user includes in its `fields` object of the
request payload any\r\nfields which do not match that rule type (in this
case, for example,\r\nsending in `machine_learning_job_id` as part of
`fields`), throw an\r\nerror for that rule.\r\n- [x] Calculation of
field diffs: what happens if some fields have a\r\nconflict value of
`NON_SOLVABLE`:\r\n- [x] If the whole rule is being updated to `MERGED`,
and **ANY** fields\r\nreturn with a `NON_SOLVABLE` conflict, reject the
whole update for that\r\nrule: create new error and add to ruleErrors
array.\r\n- [x] **EXCEPTION** for case above: the whole rule is being
updated to\r\n`MERGED`, and one or more of the fields return with a
`NON_SOLVABLE`\r\nconflict, BUT those same fields have a specific
`pick_version` for them\r\nin the `fields` object which **ARE NOT**
`MERGED`. No error should be\r\nreported in this case.\r\n- [x] The
whole rule is being updated to any `pick_version` other than\r\nMERGED,
but any specific field in the `fields` object is set to upgrade\r\nto
`MERGED`, and the diff for that fields returns a
`NON_SOLVABLE`\r\nconflict. In that case, create new error and add to
ruleErrors array.\r\n\r\n### TODO\r\n\r\n- [[Security Solution] Add
InvestigationFields and AlertSuppression\r\nfields to the upgrade
workflow\r\n[#190597]](https://github.com/elastic/kibana/issues/190597):\r\nInvestigationFields
is already working, but AlertSuppression is still\r\ncurrently handled
under the hood to update to current version.\r\n\r\n\r\n### For
maintainers\r\n\r\n- [ ] This was checked for breaking API changes and
was
[labeled\r\nappropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)\r\n\r\n---------\r\n\r\nCo-authored-by:
Maxim Palenov
<[email protected]>","sha":"7c3887309cec54cc21e1abf8a2522afa49147712","branchLabelMapping":{"^v9.0.0$":"main","^v8.16.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:enhancement","Team:Fleet","v9.0.0","Team:Detections
and Resp","Team: SecuritySolution","Team:Detection Rule
Management","Feature:Prebuilt Detection
Rules","backport:prev-minor","v8.16.0"],"title":"[Security Solution]
Extend upgrade perform endpoint
logic","number":191439,"url":"https://github.com/elastic/kibana/pull/191439","mergeCommit":{"message":"[Security
Solution] Extend upgrade perform endpoint logic (#191439)\n\nFixes:
#166376 (main ticket)\r\nFixes:
#186544 (handling of\r\nspecific
fields)\r\nFixes: #180195
(replace PATCH\r\nwith PUT logic on rule upgrade)\r\n\r\n##
Summary\r\n\r\n- Enhances the `/upgrade/_perform` endpoint to upgrade
rules in a way\r\nthat works with prebuilt rules customized by users and
resolve conflicts\r\nbetween user customizations and updates from
Elastic.\r\n- Handles special fields under the hood (see below)\r\n-
Replaces the update prebuilt rule logic to work with PUT instead
of\r\nPATCH.\r\n\r\n### Rough implementation plan\r\n- For each
`upgradeableRule`, we attempt to build the payload necessary\r\nto pass
to `upgradePrebuiltRules()`, which is of type\r\n`PrebuiltRuleAsset`. So
we retrieve the field names from\r\n`FIELDS_PAYLOAD_BY_RULE_TYPE` and
loop through them.\r\n- If any of those `field`s are non-upgreadable,
(i.e. its value needs to\r\nbe handled under the hood) we do so in
`determineFieldUpgradeStatus`.\r\n- Otherwise, we continue to build a
`FieldUpgradeSpecifier` for each\r\nfield, which will help us determine
if that field needs to be set to the\r\nbase, current, target version,
OR if it needs to be calculated as a\r\nMERGED value, or it is passed in
the request payload as a RESOLVED\r\nvalue.\r\n- Notice that we are
iterating over \"flat\" (non-grouped) fields which\r\nare part of the
`PrebuiltRuleAsset` schema. This means that mapping is\r\nnecessary
between these flat fields and the diffable (grouped) fields\r\nthat are
used in the API contract, part of `DiffableRule`. For example,\r\nif we
try to determine the value for the `query` field, we will need
to\r\nlook up for its value in the `eql_query` field if the target rule
is\r\n`eql` or in `esql_query` if the target rule is `esql`. All
these\r\nmappings can be found in
`diffable_rule_fields_mappings.ts`.\r\n- Once a `FieldUpgradeSpecifier`
has been retrieved for each field of\r\nthe payload we are building,
retrieve its actual value: either fetching\r\nit from the base, current
or target versions of the rule, from the three\r\nway diff calculation,
or retrieving it from the request payload if it\r\nresolved.\r\n- Do
this for all upgreadable rules, and the pass the payload array
into\r\n`upgradePrebuiltRules()`.\r\n- **IMPORTANT:** The upgrade
prebuilt rules logic has been changed from\r\nPATCH to PUT. That means
that if the next version of a rule removes a\r\nfield, and the user
updates to that target version, those fields will be\r\nundefined in the
resulting rule. **Additional example:** a installs a\r\nrule, and
creates a `timeline_id` for it rule by modifying it. If\r\nneither the
next version (target version) still does not have a\r\n`timeline_id`
field for it, and the user updates to that target version\r\nfully
(without resolving the conflict), that field will not exist\r\nanymore
in the resulting rule.\r\n\r\n## Acceptance criteria\r\n\r\n- [x] Extend
the contract of the API endpoint according to
the\r\n[POC](https://github.com/elastic/kibana/pull/144060):\r\n- [x]
Add the ability to pick the `MERGED` version for rule upgrades.
If\r\nthe `MERGED` version is selected, the diffs are recalculated and
the\r\nrule fields are updated to the result of the diff calculation.
This is\r\nonly possible if all field diffs return a `conflict` value of
either\r\n`NO`. If any fields returns a value of `NON_SOLVABLE` or
`SOLVABLE`,\r\nreject the request with an error specifying that there
are conflicts,\r\nand that they must be resolved on a per-field
basis.\r\n- [x] Calculate diffs inside this endpoint, when the value
of\r\n`pick_version` is `MERGED`.\r\n- [x] Add the ability to specify
rule field versions, to update specific\r\nfields to different
`pick_versions`: `BASE' | 'CURRENT' | 'TARGET' |\r\n'MERGED' |
'RESOLVED'` (See `FieldUpgradeRequest`
in\r\n[PoC](#144060) for
details)\r\n\r\n## Handling of special fields\r\n\r\nSpecific fields are
handled under the hood based
on\r\nhttps://github.com//issues/186544\r\n\r\nSee
implementation
in\r\n`x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/api/perform_rule_upgrade/determine_field_upgrade_status.ts`,\r\nwhich
imports fields to handle under the hood:\r\n-
`DiffableFieldsToOmit`\r\n- `FieldsToUpdateToCurrentVersion`\r\n\r\n##
Edge cases\r\n\r\n- [x] If target version of rule has a **rule type
change**, check that\r\nall `pick_version`, at all levels, match
`TARGET`. Otherwise, create new\r\nerror and add to ruleErrors
array.\r\n- [x] if a rule has a specific `targetVersion.type` (for
example, EQL)\r\nand the user includes in its `fields` object of the
request payload any\r\nfields which do not match that rule type (in this
case, for example,\r\nsending in `machine_learning_job_id` as part of
`fields`), throw an\r\nerror for that rule.\r\n- [x] Calculation of
field diffs: what happens if some fields have a\r\nconflict value of
`NON_SOLVABLE`:\r\n- [x] If the whole rule is being updated to `MERGED`,
and **ANY** fields\r\nreturn with a `NON_SOLVABLE` conflict, reject the
whole update for that\r\nrule: create new error and add to ruleErrors
array.\r\n- [x] **EXCEPTION** for case above: the whole rule is being
updated to\r\n`MERGED`, and one or more of the fields return with a
`NON_SOLVABLE`\r\nconflict, BUT those same fields have a specific
`pick_version` for them\r\nin the `fields` object which **ARE NOT**
`MERGED`. No error should be\r\nreported in this case.\r\n- [x] The
whole rule is being updated to any `pick_version` other than\r\nMERGED,
but any specific field in the `fields` object is set to upgrade\r\nto
`MERGED`, and the diff for that fields returns a
`NON_SOLVABLE`\r\nconflict. In that case, create new error and add to
ruleErrors array.\r\n\r\n### TODO\r\n\r\n- [[Security Solution] Add
InvestigationFields and AlertSuppression\r\nfields to the upgrade
workflow\r\n[#190597]](https://github.com/elastic/kibana/issues/190597):\r\nInvestigationFields
is already working, but AlertSuppression is still\r\ncurrently handled
under the hood to update to current version.\r\n\r\n\r\n### For
maintainers\r\n\r\n- [ ] This was checked for breaking API changes and
was
[labeled\r\nappropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)\r\n\r\n---------\r\n\r\nCo-authored-by:
Maxim Palenov
<[email protected]>","sha":"7c3887309cec54cc21e1abf8a2522afa49147712"}},"sourceBranch":"main","suggestedTargetBranches":["8.x"],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","branchLabelMappingKey":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/191439","number":191439,"mergeCommit":{"message":"[Security
Solution] Extend upgrade perform endpoint logic (#191439)\n\nFixes:
#166376 (main ticket)\r\nFixes:
#186544 (handling of\r\nspecific
fields)\r\nFixes: #180195
(replace PATCH\r\nwith PUT logic on rule upgrade)\r\n\r\n##
Summary\r\n\r\n- Enhances the `/upgrade/_perform` endpoint to upgrade
rules in a way\r\nthat works with prebuilt rules customized by users and
resolve conflicts\r\nbetween user customizations and updates from
Elastic.\r\n- Handles special fields under the hood (see below)\r\n-
Replaces the update prebuilt rule logic to work with PUT instead
of\r\nPATCH.\r\n\r\n### Rough implementation plan\r\n- For each
`upgradeableRule`, we attempt to build the payload necessary\r\nto pass
to `upgradePrebuiltRules()`, which is of type\r\n`PrebuiltRuleAsset`. So
we retrieve the field names from\r\n`FIELDS_PAYLOAD_BY_RULE_TYPE` and
loop through them.\r\n- If any of those `field`s are non-upgreadable,
(i.e. its value needs to\r\nbe handled under the hood) we do so in
`determineFieldUpgradeStatus`.\r\n- Otherwise, we continue to build a
`FieldUpgradeSpecifier` for each\r\nfield, which will help us determine
if that field needs to be set to the\r\nbase, current, target version,
OR if it needs to be calculated as a\r\nMERGED value, or it is passed in
the request payload as a RESOLVED\r\nvalue.\r\n- Notice that we are
iterating over \"flat\" (non-grouped) fields which\r\nare part of the
`PrebuiltRuleAsset` schema. This means that mapping is\r\nnecessary
between these flat fields and the diffable (grouped) fields\r\nthat are
used in the API contract, part of `DiffableRule`. For example,\r\nif we
try to determine the value for the `query` field, we will need
to\r\nlook up for its value in the `eql_query` field if the target rule
is\r\n`eql` or in `esql_query` if the target rule is `esql`. All
these\r\nmappings can be found in
`diffable_rule_fields_mappings.ts`.\r\n- Once a `FieldUpgradeSpecifier`
has been retrieved for each field of\r\nthe payload we are building,
retrieve its actual value: either fetching\r\nit from the base, current
or target versions of the rule, from the three\r\nway diff calculation,
or retrieving it from the request payload if it\r\nresolved.\r\n- Do
this for all upgreadable rules, and the pass the payload array
into\r\n`upgradePrebuiltRules()`.\r\n- **IMPORTANT:** The upgrade
prebuilt rules logic has been changed from\r\nPATCH to PUT. That means
that if the next version of a rule removes a\r\nfield, and the user
updates to that target version, those fields will be\r\nundefined in the
resulting rule. **Additional example:** a installs a\r\nrule, and
creates a `timeline_id` for it rule by modifying it. If\r\nneither the
next version (target version) still does not have a\r\n`timeline_id`
field for it, and the user updates to that target version\r\nfully
(without resolving the conflict), that field will not exist\r\nanymore
in the resulting rule.\r\n\r\n## Acceptance criteria\r\n\r\n- [x] Extend
the contract of the API endpoint according to
the\r\n[POC](https://github.com/elastic/kibana/pull/144060):\r\n- [x]
Add the ability to pick the `MERGED` version for rule upgrades.
If\r\nthe `MERGED` version is selected, the diffs are recalculated and
the\r\nrule fields are updated to the result of the diff calculation.
This is\r\nonly possible if all field diffs return a `conflict` value of
either\r\n`NO`. If any fields returns a value of `NON_SOLVABLE` or
`SOLVABLE`,\r\nreject the request with an error specifying that there
are conflicts,\r\nand that they must be resolved on a per-field
basis.\r\n- [x] Calculate diffs inside this endpoint, when the value
of\r\n`pick_version` is `MERGED`.\r\n- [x] Add the ability to specify
rule field versions, to update specific\r\nfields to different
`pick_versions`: `BASE' | 'CURRENT' | 'TARGET' |\r\n'MERGED' |
'RESOLVED'` (See `FieldUpgradeRequest`
in\r\n[PoC](#144060) for
details)\r\n\r\n## Handling of special fields\r\n\r\nSpecific fields are
handled under the hood based
on\r\nhttps://github.com//issues/186544\r\n\r\nSee
implementation
in\r\n`x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/api/perform_rule_upgrade/determine_field_upgrade_status.ts`,\r\nwhich
imports fields to handle under the hood:\r\n-
`DiffableFieldsToOmit`\r\n- `FieldsToUpdateToCurrentVersion`\r\n\r\n##
Edge cases\r\n\r\n- [x] If target version of rule has a **rule type
change**, check that\r\nall `pick_version`, at all levels, match
`TARGET`. Otherwise, create new\r\nerror and add to ruleErrors
array.\r\n- [x] if a rule has a specific `targetVersion.type` (for
example, EQL)\r\nand the user includes in its `fields` object of the
request payload any\r\nfields which do not match that rule type (in this
case, for example,\r\nsending in `machine_learning_job_id` as part of
`fields`), throw an\r\nerror for that rule.\r\n- [x] Calculation of
field diffs: what happens if some fields have a\r\nconflict value of
`NON_SOLVABLE`:\r\n- [x] If the whole rule is being updated to `MERGED`,
and **ANY** fields\r\nreturn with a `NON_SOLVABLE` conflict, reject the
whole update for that\r\nrule: create new error and add to ruleErrors
array.\r\n- [x] **EXCEPTION** for case above: the whole rule is being
updated to\r\n`MERGED`, and one or more of the fields return with a
`NON_SOLVABLE`\r\nconflict, BUT those same fields have a specific
`pick_version` for them\r\nin the `fields` object which **ARE NOT**
`MERGED`. No error should be\r\nreported in this case.\r\n- [x] The
whole rule is being updated to any `pick_version` other than\r\nMERGED,
but any specific field in the `fields` object is set to upgrade\r\nto
`MERGED`, and the diff for that fields returns a
`NON_SOLVABLE`\r\nconflict. In that case, create new error and add to
ruleErrors array.\r\n\r\n### TODO\r\n\r\n- [[Security Solution] Add
InvestigationFields and AlertSuppression\r\nfields to the upgrade
workflow\r\n[#190597]](https://github.com/elastic/kibana/issues/190597):\r\nInvestigationFields
is already working, but AlertSuppression is still\r\ncurrently handled
under the hood to update to current version.\r\n\r\n\r\n### For
maintainers\r\n\r\n- [ ] This was checked for breaking API changes and
was
[labeled\r\nappropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)\r\n\r\n---------\r\n\r\nCo-authored-by:
Maxim Palenov
<[email protected]>","sha":"7c3887309cec54cc21e1abf8a2522afa49147712"}},{"branch":"8.x","label":"v8.16.0","branchLabelMappingKey":"^v8.16.0$","isSourceBranch":false,"state":"NOT_CREATED"}]}]
BACKPORT-->

Co-authored-by: Juan Pablo Djeredjian <[email protected]>

x-pack/plugins/security_solution/common/api/detection_engine/prebuilt_rules/model/diff/diffable_rule/diffable_rule.ts

@@ -174,6 +174,17 @@ export const DiffableNewTermsFields = z.object({
   alert_suppression: AlertSuppression.optional(),
 });
 
+export const DiffableFieldsByTypeUnion = z.discriminatedUnion('type', [
+  DiffableCustomQueryFields,
+  DiffableSavedQueryFields,
+  DiffableEqlFields,
+  DiffableEsqlFields,
+  DiffableThreatMatchFields,
+  DiffableThresholdFields,
+  DiffableMachineLearningFields,
+  DiffableNewTermsFields,
+]);
+
 /**
  * Represents a normalized rule object that is suitable for passing to the diff algorithm.
  * Every top-level field of a diffable rule can be compared separately on its own.
@@ -200,18 +211,6 @@ export const DiffableNewTermsFields = z.object({
  * NOTE: Every top-level field in a DiffableRule MUST BE LOGICALLY INDEPENDENT from other
  * top-level fields.
  */
-
-export const DiffableFieldsByTypeUnion = z.discriminatedUnion('type', [
-  DiffableCustomQueryFields,
-  DiffableSavedQueryFields,
-  DiffableEqlFields,
-  DiffableEsqlFields,
-  DiffableThreatMatchFields,
-  DiffableThresholdFields,
-  DiffableMachineLearningFields,
-  DiffableNewTermsFields,
-]);
-
 export type DiffableRule = z.infer<typeof DiffableRule>;
 export const DiffableRule = z.intersection(DiffableCommonFields, DiffableFieldsByTypeUnion);
 
@@ -246,3 +245,22 @@ export const DiffableAllFields = DiffableCommonFields.merge(
   .merge(DiffableMachineLearningFields.omit({ type: true }))
   .merge(DiffableNewTermsFields.omit({ type: true }))
   .merge(z.object({ type: DiffableRuleTypes }));
+
+const getRuleTypeFields = (schema: z.ZodObject<z.ZodRawShape>): string[] =>
+  Object.keys(schema.shape);
+
+const createDiffableFieldsPerRuleType = (specificFields: z.ZodObject<z.ZodRawShape>): string[] => [
+  ...getRuleTypeFields(DiffableCommonFields),
+  ...getRuleTypeFields(specificFields),
+];
+
+export const DIFFABLE_RULE_TYPE_FIELDS_MAP = new Map<DiffableRuleTypes, string[]>([
+  ['query', createDiffableFieldsPerRuleType(DiffableCustomQueryFields)],
+  ['saved_query', createDiffableFieldsPerRuleType(DiffableSavedQueryFields)],
+  ['eql', createDiffableFieldsPerRuleType(DiffableEqlFields)],
+  ['esql', createDiffableFieldsPerRuleType(DiffableEsqlFields)],
+  ['threat_match', createDiffableFieldsPerRuleType(DiffableThreatMatchFields)],
+  ['threshold', createDiffableFieldsPerRuleType(DiffableThresholdFields)],
+  ['machine_learning', createDiffableFieldsPerRuleType(DiffableMachineLearningFields)],
+  ['new_terms', createDiffableFieldsPerRuleType(DiffableNewTermsFields)],
+]);

x-pack/plugins/security_solution/common/api/detection_engine/prebuilt_rules/perform_rule_upgrade/perform_rule_upgrade_route.ts

@@ -11,30 +11,65 @@ import { RuleResponse } from '../../model/rule_schema/rule_schemas.gen';
 import { AggregatedPrebuiltRuleError, DiffableAllFields } from '../model';
 import { RuleSignatureId, RuleVersion } from '../../model';
 
+export type Mode = z.infer<typeof Mode>;
+export const Mode = z.enum(['ALL_RULES', 'SPECIFIC_RULES']);
+export type ModeEnum = typeof Mode.enum;
+export const ModeEnum = Mode.enum;
+
 export type PickVersionValues = z.infer<typeof PickVersionValues>;
 export const PickVersionValues = z.enum(['BASE', 'CURRENT', 'TARGET', 'MERGED']);
 export type PickVersionValuesEnum = typeof PickVersionValues.enum;
 export const PickVersionValuesEnum = PickVersionValues.enum;
 
+// Specific handling of special fields according to:
+// https://github.com/elastic/kibana/issues/186544
+export const FIELDS_TO_UPGRADE_TO_CURRENT_VERSION = [
+  'enabled',
+  'exceptions_list',
+  'alert_suppression',
+  'actions',
+  'throttle',
+  'response_actions',
+  'meta',
+  'output_index',
+  'namespace',
+  'alias_purpose',
+  'alias_target_id',
+  'outcome',
+  'concurrent_searches',
+  'items_per_search',
+] as const;
+
+export const NON_UPGRADEABLE_DIFFABLE_FIELDS = [
+  'type',
+  'rule_id',
+  'version',
+  'author',
+  'license',
+] as const;
+
+type NON_UPGRADEABLE_DIFFABLE_FIELDS_TO_OMIT_TYPE = {
+  readonly [key in (typeof NON_UPGRADEABLE_DIFFABLE_FIELDS)[number]]: true;
+};
+
+// This transformation is needed to have Zod's `omit` accept the rule fields that need to be omitted
+export const DiffableFieldsToOmit = NON_UPGRADEABLE_DIFFABLE_FIELDS.reduce((acc, field) => {
+  return { ...acc, [field]: true };
+}, {} as NON_UPGRADEABLE_DIFFABLE_FIELDS_TO_OMIT_TYPE);
+
 /**
  * Fields upgradable by the /upgrade/_perform endpoint.
  * Specific fields are omitted because they are not upgradeable, and
  * handled under the hood by endpoint logic.
  * See: https://github.com/elastic/kibana/issues/186544
  */
 export type DiffableUpgradableFields = z.infer<typeof DiffableUpgradableFields>;
-export const DiffableUpgradableFields = DiffableAllFields.omit({
-  type: true,
-  rule_id: true,
-  version: true,
-  author: true,
-  license: true,
-});
+export const DiffableUpgradableFields = DiffableAllFields.omit(DiffableFieldsToOmit);
 
 export type FieldUpgradeSpecifier<T> = z.infer<
   ReturnType<typeof fieldUpgradeSpecifier<z.ZodType<T>>>
 >;
-const fieldUpgradeSpecifier = <T extends z.ZodTypeAny>(fieldSchema: T) =>
+export const fieldUpgradeSpecifier = <T extends z.ZodTypeAny>(fieldSchema: T) =>
   z.discriminatedUnion('pick_version', [
     z
       .object({

x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/api/get_prebuilt_rules_status/get_prebuilt_rules_status_route.ts

@@ -13,7 +13,7 @@ import { buildSiemResponse } from '../../../routes/utils';
 import { createPrebuiltRuleAssetsClient } from '../../logic/rule_assets/prebuilt_rule_assets_client';
 import { createPrebuiltRuleObjectsClient } from '../../logic/rule_objects/prebuilt_rule_objects_client';
 import { fetchRuleVersionsTriad } from '../../logic/rule_versions/fetch_rule_versions_triad';
-import { getVersionBuckets } from '../../model/rule_versions/get_version_buckets';
+import { getRuleGroups } from '../../model/rule_groups/get_rule_groups';
 
 export const getPrebuiltRulesStatusRoute = (router: SecuritySolutionPluginRouter) => {
   router.versioned
@@ -44,7 +44,7 @@ export const getPrebuiltRulesStatusRoute = (router: SecuritySolutionPluginRouter
             ruleObjectsClient,
           });
           const { currentRules, installableRules, upgradeableRules, totalAvailableRules } =
-            getVersionBuckets(ruleVersionsMap);
+            getRuleGroups(ruleVersionsMap);
 
           const body: GetPrebuiltRulesStatusResponseBody = {
             stats: {

x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/api/perform_rule_installation/perform_rule_installation_route.ts

@@ -25,9 +25,9 @@ import { createPrebuiltRuleAssetsClient } from '../../logic/rule_assets/prebuilt
 import { createPrebuiltRules } from '../../logic/rule_objects/create_prebuilt_rules';
 import { createPrebuiltRuleObjectsClient } from '../../logic/rule_objects/prebuilt_rule_objects_client';
 import { fetchRuleVersionsTriad } from '../../logic/rule_versions/fetch_rule_versions_triad';
-import { getVersionBuckets } from '../../model/rule_versions/get_version_buckets';
 import { performTimelinesInstallation } from '../../logic/perform_timelines_installation';
 import { PREBUILT_RULES_OPERATION_SOCKET_TIMEOUT_MS } from '../../constants';
+import { getRuleGroups } from '../../model/rule_groups/get_rule_groups';
 
 export const performRuleInstallationRoute = (router: SecuritySolutionPluginRouter) => {
   router.versioned
@@ -80,7 +80,7 @@ export const performRuleInstallationRoute = (router: SecuritySolutionPluginRoute
             ruleObjectsClient,
             versionSpecifiers: mode === 'ALL_RULES' ? undefined : request.body.rules,
           });
-          const { currentRules, installableRules } = getVersionBuckets(ruleVersionsMap);
+          const { currentRules, installableRules } = getRuleGroups(ruleVersionsMap);
 
           // Perform all the checks we can before we start the upgrade process
           if (mode === 'SPECIFIC_RULES') {

x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/api/perform_rule_upgrade/assert_diffable_fields_match_rule_type.test.ts

@@ -0,0 +1,60 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License
+ * 2.0; you may not use this file except in compliance with the Elastic License
+ * 2.0.
+ */
+
+import { assertDiffableFieldsMatchRuleType } from './assert_diffable_fields_match_rule_type';
+import { DIFFABLE_RULE_TYPE_FIELDS_MAP } from '../../../../../../common/api/detection_engine';
+
+describe('assertDiffableFieldsMatchRuleType', () => {
+  describe('valid scenarios -', () => {
+    it('should validate all fields in DIFFABLE_RULE_TYPE_FIELDS_MAP', () => {
+      DIFFABLE_RULE_TYPE_FIELDS_MAP.forEach((fields, ruleType) => {
+        expect(() => {
+          assertDiffableFieldsMatchRuleType(fields, ruleType);
+        }).not.toThrow();
+      });
+    });
+
+    it('should not throw an error for valid upgradeable fields', () => {
+      expect(() => {
+        assertDiffableFieldsMatchRuleType(['name', 'description', 'severity'], 'query');
+      }).not.toThrow();
+    });
+
+    it('should handle valid rule type correctly', () => {
+      expect(() => {
+        assertDiffableFieldsMatchRuleType(['eql_query'], 'eql');
+      }).not.toThrow();
+    });
+
+    it('should handle empty upgradeable fields array', () => {
+      expect(() => {
+        assertDiffableFieldsMatchRuleType([], 'query');
+      }).not.toThrow();
+    });
+  });
+
+  describe('invalid scenarios -', () => {
+    it('should throw an error for invalid upgradeable fields', () => {
+      expect(() => {
+        assertDiffableFieldsMatchRuleType(['invalid_field'], 'query');
+      }).toThrow("invalid_field is not a valid upgradeable field for type 'query'");
+    });
+
+    it('should throw for incompatible rule types', () => {
+      expect(() => {
+        assertDiffableFieldsMatchRuleType(['eql_query'], 'query');
+      }).toThrow("eql_query is not a valid upgradeable field for type 'query'");
+    });
+
+    it('should throw an error for an unknown rule type', () => {
+      expect(() => {
+        // @ts-expect-error - unknown rule
+        assertDiffableFieldsMatchRuleType(['name'], 'unknown_type');
+      }).toThrow();
+    });
+  });
+});

x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/api/perform_rule_upgrade/assert_diffable_fields_match_rule_type.ts

@@ -0,0 +1,40 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License
+ * 2.0; you may not use this file except in compliance with the Elastic License
+ * 2.0.
+ */
+
+import type { DiffableRuleTypes } from '../../../../../../common/api/detection_engine';
+import { DIFFABLE_RULE_TYPE_FIELDS_MAP } from '../../../../../../common/api/detection_engine';
+
+/**
+ * Validates that the upgradeable (diffable) fields match the target rule type's diffable fields.
+ *
+ * This function is used in the rule upgrade process to ensure that the fields
+ * specified for upgrade in the request body are valid for the target rule type.
+ * It checks each upgradeable field provided in body.rule[].fields against the
+ * set of diffable fields for the target rule type.
+ *
+ * @param {string[]} diffableFields - An array of field names to be upgraded.
+ * @param {string} ruleType - A rule type (e.g., 'query', 'eql', 'machine_learning').
+ * @throws {Error} If an upgradeable field is not valid for the target rule type.
+ *
+ * @examples
+ * assertDiffableFieldsMatchRuleType(['kql_query', 'severity'], 'query');
+ * assertDiffableFieldsMatchRuleType(['esql_query', 'description'], 'esql');
+ * assertDiffableFieldsMatchRuleType(['machine_learning_job_id'], 'eql'); // throws error
+ *
+ * @see {@link DIFFABLE_RULE_TYPE_FIELDS_MAP} in diffable_rule.ts for the mapping of rule types to their diffable fields.
+ */
+export const assertDiffableFieldsMatchRuleType = (
+  diffableFields: string[],
+  ruleType: DiffableRuleTypes
+) => {
+  const diffableFieldsForType = new Set(DIFFABLE_RULE_TYPE_FIELDS_MAP.get(ruleType));
+  for (const diffableField of diffableFields) {
+    if (!diffableFieldsForType.has(diffableField)) {
+      throw new Error(`${diffableField} is not a valid upgradeable field for type '${ruleType}'`);
+    }
+  }
+};