Skip to content

Commit

Permalink
[Rules migration] Add rules migrations update route (#11209) (#200815)
Browse files Browse the repository at this point in the history
## Summary

Changes in this PR:
* Added `update` route to handle bulk rule migrations docs updates
* Exposed `id` field in `RuleMigration` object needed for ES bulk update
operation
* Updated SIEM migrations schemas to use `NonEmptyString` when it is
needed

## Testing locally

Enable the flag
```
xpack.securitySolution.enableExperimental: ['siemMigrationsEnabled']
```

Create and start a rule migration. Then use `update` API to updated
corresponding docs.

cURL request examples:

<details>
  <summary>Rules migration `create` POST request</summary>

```
curl --location --request POST 'http://elastic:changeme@localhost:5601/internal/siem_migrations/rules' \
--header 'kbn-xsrf;' \
--header 'x-elastic-internal-origin: security-solution' \
--header 'elastic-api-version: 1' \
--header 'Content-Type: application/json' \
--data '[
    {
        "id": "f8c325ea-506e-4105-8ccf-da1492e90115",
        "vendor": "splunk",
        "title": "Linux Auditd Add User Account Type",
        "description": "The following analytic detects the suspicious add user account type. This behavior is critical for a SOC to monitor because it may indicate attempts to gain unauthorized access or maintain control over a system. Such actions could be signs of malicious activity. If confirmed, this could lead to serious consequences, including a compromised system, unauthorized access to sensitive data, or even a wider breach affecting the entire network. Detecting and responding to these signs early is essential to prevent potential security incidents.",
        "query": "sourcetype=\"linux:audit\" type=ADD_USER \n| rename hostname as dest \n| stats count min(_time) as firstTime max(_time) as lastTime by exe pid dest res UID type \n| `security_content_ctime(firstTime)` \n| `security_content_ctime(lastTime)`\n| search *",
        "query_language":"spl",
        "mitre_attack_ids": [
            "T1136"
        ]
    },
    {
        "id": "7b87c556-0ca4-47e0-b84c-6cd62a0a3e90",
        "vendor": "splunk",
        "title": "Linux Auditd Change File Owner To Root",
        "description": "The following analytic detects the use of the '\''chown'\'' command to change a file owner to '\''root'\'' on a Linux system. It leverages Linux Auditd telemetry, specifically monitoring command-line executions and process details. This activity is significant as it may indicate an attempt to escalate privileges by adversaries, malware, or red teamers. If confirmed malicious, this action could allow an attacker to gain root-level access, leading to full control over the compromised host and potential persistence within the environment.",
        "query": "`linux_auditd` `linux_auditd_normalized_proctitle_process`\r\n| rename host as dest \r\n| where LIKE (process_exec, \"%chown %root%\") \r\n| stats count min(_time) as firstTime max(_time) as lastTime by process_exec proctitle normalized_proctitle_delimiter dest \r\n| `security_content_ctime(firstTime)` \r\n| `security_content_ctime(lastTime)`\r\n| `linux_auditd_change_file_owner_to_root_filter`",
        "query_language": "spl",
        "mitre_attack_ids": [
            "T1222"
        ]
    }
]'
```
</details>

<details>
  <summary>Rules migration `start` task request</summary>

- Assuming the connector `azureOpenAiGPT4o` is already created in the
local environment.
- Using the {{`migration_id`}} from the first POST request response

```
curl --location --request PUT 'http://elastic:changeme@localhost:5601/internal/siem_migrations/rules/{{migration_id}}/start' \
--header 'kbn-xsrf;' \
--header 'x-elastic-internal-origin: security-solution' \
--header 'elastic-api-version: 1' \
--header 'Content-Type: application/json' \
--data '{
    "connectorId": "azureOpenAiGPT4o"
}'
```
</details>

<details>
  <summary>Rules migration rules documents request</summary>

- Using the {{`migration_id`}} from the first POST request response.

```
curl --location --request GET 'http://elastic:changeme@localhost:5601/internal/siem_migrations/rules/{{migration_id}}' \
--header 'kbn-xsrf;' \
--header 'x-elastic-internal-origin: security-solution' \
--header 'elastic-api-version: 1'
```
</details>

<details>
  <summary>Rules migration `update` PUT request</summary>

- Using the {{`rule_migration_id_1`}} and {{`rule_migration_id_2`}} from
previous GET request response

```
curl --location --request PUT 'http://elastic:changeme@localhost:5601/internal/siem_migrations/rules' \
--header 'kbn-xsrf;' \
--header 'x-elastic-internal-origin: security-solution' \
--header 'elastic-api-version: 1'
--data '[
    {
        "comments": [
            "## Migration Summary\n- The `FROM` command is used to select the `logs-*` index pattern.\n- The `RENAME` command is used to rename the `host` field to `dest`.\n- The `WHERE` command filters the rows where `process_exec` contains the pattern `*chown *root*`.\n- The `STATS` command is used to aggregate the data, counting the number of occurrences and finding the minimum and maximum timestamps, grouped by `process_exec`, `proctitle`, `normalized_proctitle_delimiter`, and `dest`.\n- The macros `security_content_ctime` and `linux_auditd_change_file_owner_to_root_filter` are placeholders for the corresponding Splunk macros.",
            "Additional comment 2.0"
        ],
        "translation_result": "full",
        "id": "{{rule_migration_id_1}}"
    },
    {
        "created_by": "elastic2.0",
        "elastic_rule": {
            "severity": "high",
            "title": "Linux Auditd Change File Owner To Root (UPDATED)"
        },
        "id": "{{rule_migration_id_2}}"
    }
]'
```
</details>

---------

Co-authored-by: kibanamachine <[email protected]>
(cherry picked from commit f6ac2cf)
  • Loading branch information
e40pud committed Nov 20, 2024
1 parent 4bbe24c commit 12e94d3
Show file tree
Hide file tree
Showing 17 changed files with 408 additions and 85 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -378,6 +378,8 @@ import type {
StartRuleMigrationResponse,
StopRuleMigrationRequestParamsInput,
StopRuleMigrationResponse,
UpdateRuleMigrationRequestBodyInput,
UpdateRuleMigrationResponse,
UpsertRuleMigrationResourcesRequestParamsInput,
UpsertRuleMigrationResourcesRequestBodyInput,
UpsertRuleMigrationResourcesResponse,
Expand Down Expand Up @@ -2086,6 +2088,22 @@ detection engine rules.
})
.catch(catchAxiosErrorFormatAndThrow);
}
/**
* Updates rules migrations attributes
*/
async updateRuleMigration(props: UpdateRuleMigrationProps) {
this.log.info(`${new Date().toISOString()} Calling API UpdateRuleMigration`);
return this.kbnClient
.request<UpdateRuleMigrationResponse>({
path: '/internal/siem_migrations/rules',
headers: {
[ELASTIC_HTTP_VERSION_HEADER]: '1',
},
method: 'PUT',
body: props.body,
})
.catch(catchAxiosErrorFormatAndThrow);
}
async uploadAssetCriticalityRecords(props: UploadAssetCriticalityRecordsProps) {
this.log.info(`${new Date().toISOString()} Calling API UploadAssetCriticalityRecords`);
return this.kbnClient
Expand Down Expand Up @@ -2394,6 +2412,9 @@ export interface TriggerRiskScoreCalculationProps {
export interface UpdateRuleProps {
body: UpdateRuleRequestBodyInput;
}
export interface UpdateRuleMigrationProps {
body: UpdateRuleMigrationRequestBodyInput;
}
export interface UploadAssetCriticalityRecordsProps {
attachment: FormData;
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -19,14 +19,17 @@ import { ArrayFromString } from '@kbn/zod-helpers';

import {
OriginalRule,
ElasticRulePartial,
RuleMigrationTranslationResult,
RuleMigrationComments,
RuleMigrationAllTaskStats,
RuleMigration,
RuleMigrationTaskStats,
RuleMigrationResourceData,
RuleMigrationResourceType,
RuleMigrationResource,
} from '../../rule_migration.gen';
import { ConnectorId, LangSmithOptions } from '../common.gen';
import { NonEmptyString, ConnectorId, LangSmithOptions } from '../../common.gen';

export type CreateRuleMigrationRequestBody = z.infer<typeof CreateRuleMigrationRequestBody>;
export const CreateRuleMigrationRequestBody = z.array(OriginalRule);
Expand All @@ -37,15 +40,15 @@ export const CreateRuleMigrationResponse = z.object({
/**
* The migration id created.
*/
migration_id: z.string(),
migration_id: NonEmptyString,
});

export type GetAllStatsRuleMigrationResponse = z.infer<typeof GetAllStatsRuleMigrationResponse>;
export const GetAllStatsRuleMigrationResponse = RuleMigrationAllTaskStats;

export type GetRuleMigrationRequestParams = z.infer<typeof GetRuleMigrationRequestParams>;
export const GetRuleMigrationRequestParams = z.object({
migration_id: z.string(),
migration_id: NonEmptyString,
});
export type GetRuleMigrationRequestParamsInput = z.input<typeof GetRuleMigrationRequestParams>;

Expand All @@ -66,7 +69,7 @@ export type GetRuleMigrationResourcesRequestParams = z.infer<
typeof GetRuleMigrationResourcesRequestParams
>;
export const GetRuleMigrationResourcesRequestParams = z.object({
migration_id: z.string(),
migration_id: NonEmptyString,
});
export type GetRuleMigrationResourcesRequestParamsInput = z.input<
typeof GetRuleMigrationResourcesRequestParams
Expand All @@ -77,7 +80,7 @@ export const GetRuleMigrationResourcesResponse = z.array(RuleMigrationResource);

export type GetRuleMigrationStatsRequestParams = z.infer<typeof GetRuleMigrationStatsRequestParams>;
export const GetRuleMigrationStatsRequestParams = z.object({
migration_id: z.string(),
migration_id: NonEmptyString,
});
export type GetRuleMigrationStatsRequestParamsInput = z.input<
typeof GetRuleMigrationStatsRequestParams
Expand All @@ -88,7 +91,7 @@ export const GetRuleMigrationStatsResponse = RuleMigrationTaskStats;

export type StartRuleMigrationRequestParams = z.infer<typeof StartRuleMigrationRequestParams>;
export const StartRuleMigrationRequestParams = z.object({
migration_id: z.string(),
migration_id: NonEmptyString,
});
export type StartRuleMigrationRequestParamsInput = z.input<typeof StartRuleMigrationRequestParams>;

Expand All @@ -109,7 +112,7 @@ export const StartRuleMigrationResponse = z.object({

export type StopRuleMigrationRequestParams = z.infer<typeof StopRuleMigrationRequestParams>;
export const StopRuleMigrationRequestParams = z.object({
migration_id: z.string(),
migration_id: NonEmptyString,
});
export type StopRuleMigrationRequestParamsInput = z.input<typeof StopRuleMigrationRequestParams>;

Expand All @@ -121,11 +124,42 @@ export const StopRuleMigrationResponse = z.object({
stopped: z.boolean(),
});

export type UpdateRuleMigrationRequestBody = z.infer<typeof UpdateRuleMigrationRequestBody>;
export const UpdateRuleMigrationRequestBody = z.array(
z.object({
/**
* The rule migration id
*/
id: NonEmptyString,
/**
* The migrated elastic rule attributes to update.
*/
elastic_rule: ElasticRulePartial.optional(),
/**
* The rule translation result.
*/
translation_result: RuleMigrationTranslationResult.optional(),
/**
* The comments for the migration including a summary from the LLM in markdown.
*/
comments: RuleMigrationComments.optional(),
})
);
export type UpdateRuleMigrationRequestBodyInput = z.input<typeof UpdateRuleMigrationRequestBody>;

export type UpdateRuleMigrationResponse = z.infer<typeof UpdateRuleMigrationResponse>;
export const UpdateRuleMigrationResponse = z.object({
/**
* Indicates rules migrations have been updated.
*/
updated: z.boolean(),
});

export type UpsertRuleMigrationResourcesRequestParams = z.infer<
typeof UpsertRuleMigrationResourcesRequestParams
>;
export const UpsertRuleMigrationResourcesRequestParams = z.object({
migration_id: z.string(),
migration_id: NonEmptyString,
});
export type UpsertRuleMigrationResourcesRequestParamsInput = z.input<
typeof UpsertRuleMigrationResourcesRequestParams
Expand All @@ -134,7 +168,16 @@ export type UpsertRuleMigrationResourcesRequestParamsInput = z.input<
export type UpsertRuleMigrationResourcesRequestBody = z.infer<
typeof UpsertRuleMigrationResourcesRequestBody
>;
export const UpsertRuleMigrationResourcesRequestBody = z.array(RuleMigrationResourceData);
export const UpsertRuleMigrationResourcesRequestBody = z.array(
RuleMigrationResourceData.merge(
z.object({
/**
* The rule resource migration id
*/
id: NonEmptyString,
})
)
);
export type UpsertRuleMigrationResourcesRequestBodyInput = z.input<
typeof UpsertRuleMigrationResourcesRequestBody
>;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,6 @@ info:
title: SIEM Rules Migration API
version: '1'
paths:

# Rule migrations APIs

/internal/siem_migrations/rules:
Expand Down Expand Up @@ -33,8 +32,52 @@ paths:
- migration_id
properties:
migration_id:
type: string
description: The migration id created.
$ref: '../../common.schema.yaml#/components/schemas/NonEmptyString'

put:
summary: Updates rules migrations
operationId: UpdateRuleMigration
x-codegen-enabled: true
description: Updates rules migrations attributes
tags:
- SIEM Rule Migrations
requestBody:
required: true
content:
application/json:
schema:
type: array
items:
type: object
required:
- id
properties:
id:
description: The rule migration id
$ref: '../../common.schema.yaml#/components/schemas/NonEmptyString'
elastic_rule:
description: The migrated elastic rule attributes to update.
$ref: '../../rule_migration.schema.yaml#/components/schemas/ElasticRulePartial'
translation_result:
description: The rule translation result.
$ref: '../../rule_migration.schema.yaml#/components/schemas/RuleMigrationTranslationResult'
comments:
description: The comments for the migration including a summary from the LLM in markdown.
$ref: '../../rule_migration.schema.yaml#/components/schemas/RuleMigrationComments'
responses:
200:
description: Indicates rules migrations have been updated correctly.
content:
application/json:
schema:
type: object
required:
- updated
properties:
updated:
type: boolean
description: Indicates rules migrations have been updated.

/internal/siem_migrations/rules/stats:
get:
Expand Down Expand Up @@ -67,8 +110,8 @@ paths:
in: path
required: true
schema:
type: string
description: The migration id to start
$ref: '../../common.schema.yaml#/components/schemas/NonEmptyString'
responses:
200:
description: Indicates rule migration have been retrieved correctly.
Expand All @@ -94,8 +137,8 @@ paths:
in: path
required: true
schema:
type: string
description: The migration id to start
$ref: '../../common.schema.yaml#/components/schemas/NonEmptyString'
requestBody:
required: true
content:
Expand All @@ -106,9 +149,9 @@ paths:
- connector_id
properties:
connector_id:
$ref: '../common.schema.yaml#/components/schemas/ConnectorId'
$ref: '../../common.schema.yaml#/components/schemas/ConnectorId'
langsmith_options:
$ref: '../common.schema.yaml#/components/schemas/LangSmithOptions'
$ref: '../../common.schema.yaml#/components/schemas/LangSmithOptions'
responses:
200:
description: Indicates the migration start request has been processed successfully.
Expand Down Expand Up @@ -138,8 +181,8 @@ paths:
in: path
required: true
schema:
type: string
description: The migration id to start
$ref: '../../common.schema.yaml#/components/schemas/NonEmptyString'
responses:
200:
description: Indicates the migration stats has been retrieved correctly.
Expand All @@ -163,8 +206,8 @@ paths:
in: path
required: true
schema:
type: string
description: The migration id to stop
$ref: '../../common.schema.yaml#/components/schemas/NonEmptyString'
responses:
200:
description: Indicates migration task stop has been processed successfully.
Expand Down Expand Up @@ -197,16 +240,24 @@ paths:
in: path
required: true
schema:
type: string
description: The migration id to attach the resources
$ref: '../../common.schema.yaml#/components/schemas/NonEmptyString'
requestBody:
required: true
content:
application/json:
schema:
type: array
items:
$ref: '../../rule_migration.schema.yaml#/components/schemas/RuleMigrationResourceData'
allOf:
- $ref: '../../rule_migration.schema.yaml#/components/schemas/RuleMigrationResourceData'
- type: object
required:
- id
properties:
id:
description: The rule resource migration id
$ref: '../../common.schema.yaml#/components/schemas/NonEmptyString'
responses:
200:
description: Indicates migration resources have been created or updated correctly.
Expand Down Expand Up @@ -234,8 +285,8 @@ paths:
in: path
required: true
schema:
type: string
description: The migration id to attach the resources
$ref: '../../common.schema.yaml#/components/schemas/NonEmptyString'
- name: type
in: query
required: false
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -10,12 +10,21 @@
* This file is automatically generated by the OpenAPI Generator, @kbn/openapi-generator.
*
* info:
* title: SIEM Rule Migrations API common components
* title: SIEM Rule Migration common components
* version: not applicable
*/

import { z } from '@kbn/zod';

/**
* A string that is not empty and does not contain only whitespace
*/
export type NonEmptyString = z.infer<typeof NonEmptyString>;
export const NonEmptyString = z
.string()
.min(1)
.regex(/^(?! *$).+$/);

/**
* The GenAI connector id to use.
*/
Expand Down
Original file line number Diff line number Diff line change
@@ -1,11 +1,16 @@
openapi: 3.0.3
info:
title: SIEM Rule Migrations API common components
title: SIEM Rule Migration common components
version: 'not applicable'
paths: {}
components:
x-codegen-enabled: true
schemas:
NonEmptyString:
type: string
pattern: ^(?! *$).+$
minLength: 1
description: A string that is not empty and does not contain only whitespace
ConnectorId:
type: string
description: The GenAI connector id to use.
Expand Down
Loading

0 comments on commit 12e94d3

Please sign in to comment.