Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Security Solution] Test plan for rule type field diff algorithm #193372

Merged
merged 3 commits into from
Sep 30, 2024
+109 −8
Merged

[Security Solution] Test plan for rule type field diff algorithm #193372

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
20f3b69
adds cases to test plan
dplumlee Sep 18, 2024
81313bb
Merge remote-tracking branch 'upstream/main' into rule-type-diff-algo…
dplumlee Sep 30, 2024
4b87bc7
adds comments
dplumlee Sep 30, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
117 changes: 109 additions & 8 deletions ...sting/test_plans/detection_response/prebuilt_rules/upgrade_review_algorithms.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,12 +17,16 @@ Status: `in progress`.
- [Rule field doesn't have an update and has no custom value - `AAA`](#rule-field-doesnt-have-an-update-and-has-no-custom-value---aaa)
- [**Scenario: `AAA` - Rule field is any type**](#scenario-aaa---rule-field-is-any-type)
- [Rule field doesn't have an update but has a custom value - `ABA`](#rule-field-doesnt-have-an-update-but-has-a-custom-value---aba)
- [**Scenario: `ABA` - Rule field is any type**](#scenario-aba---rule-field-is-any-type)
- [**Scenario: `ABA` - Rule field is any type except rule `type`**](#scenario-aba---rule-field-is-any-type-except-rule-type)
- [**Scenario: `ABA` - Rule field is rule `type`**](#scenario-aba---rule-field-is-rule-type)
- [Rule field has an update and doesn't have a custom value - `AAB`](#rule-field-has-an-update-and-doesnt-have-a-custom-value---aab)
- [**Scenario: `AAB` - Rule field is any type**](#scenario-aab---rule-field-is-any-type)
- [**Scenario: `AAB` - Rule field is any type except rule `type`**](#scenario-aab---rule-field-is-any-type-except-rule-type)
- [**Scenario: `AAB` - Rule field is rule `type`**](#scenario-aab---rule-field-is-rule-type)
- [Rule field has an update and a custom value that are the same - `ABB`](#rule-field-has-an-update-and-a-custom-value-that-are-the-same---abb)
- [**Scenario: `ABB` - Rule field is any type**](#scenario-abb---rule-field-is-any-type)
- [**Scenario: `ABB` - Rule field is any type except rule `type`**](#scenario-abb---rule-field-is-any-type-except-rule-type)
- [**Scenario: `ABB` - Rule field is rule `type`**](#scenario-abb---rule-field-is-rule-type)
- [Rule field has an update and a custom value that are NOT the same - `ABC`](#rule-field-has-an-update-and-a-custom-value-that-are-not-the-same---abc)
- [**Scenario: `ABC` - Rule field is rule `type`**](#scenario-abc---rule-field-is-rule-type)
- [**Scenario: `ABC` - Rule field is a number or single line string**](#scenario-abc---rule-field-is-a-number-or-single-line-string)
- [**Scenario: `ABC` - Rule field is a mergeable multi line string**](#scenario-abc---rule-field-is-a-mergeable-multi-line-string)
- [**Scenario: `ABC` - Rule field is a non-mergeable multi line string**](#scenario-abc---rule-field-is-a-non-mergeable-multi-line-string)
Expand All @@ -37,6 +41,7 @@ Status: `in progress`.
- [**Scenario: `-AB` - Rule field is an array of scalar values**](#scenario--ab---rule-field-is-an-array-of-scalar-values)
- [**Scenario: `-AB` - Rule field is a solvable `data_source` object**](#scenario--ab---rule-field-is-a-solvable-data_source-object)
- [**Scenario: `-AB` - Rule field is a non-solvable `data_source` object**](#scenario--ab---rule-field-is-a-non-solvable-data_source-object)
- [**Scenario: `-AB` - Rule field is rule `type`**](#scenario--ab---rule-field-is-rule-type)

## Useful information

Expand Down Expand Up @@ -74,7 +79,7 @@ Status: `in progress`.

#### **Scenario: `AAA` - Rule field is any type**

**Automation**: 10 integration tests with mock rules + a set of unit tests for each algorithm
**Automation**: 11 integration tests with mock rules + a set of unit tests for each algorithm

```Gherkin
Given <field_name> field is not customized by the user (current version == base version)
Expand All @@ -85,6 +90,7 @@ And <field_name> field should not be shown in the upgrade preview UI

Examples:
| algorithm | field_name | base_version | current_version | target_version | merged_version |
| rule type | type | "query" | "query" | "query" | "query" |
| single line string | name | "A" | "A" | "A" | "A" |
| multi line string | description | "My description.\nThis is a second line." | "My description.\nThis is a second line." | "My description.\nThis is a second line." | "My description.\nThis is a second line." |
| number | risk_score | 1 | 1 | 1 | 1 |
Expand All @@ -99,7 +105,7 @@ Examples:

### Rule field doesn't have an update but has a custom value - `ABA`

#### **Scenario: `ABA` - Rule field is any type**
#### **Scenario: `ABA` - Rule field is any type except rule `type`**

**Automation**: 10 integration tests with mock rules + a set of unit tests for each algorithm

Expand All @@ -124,9 +130,27 @@ Examples:
| esql_query | esql_query | {query: "FROM query WHERE true", language: "esql"} | {query: "FROM query WHERE false", language: "esql"} | {query: "FROM query WHERE true", language: "esql"} | {query: "FROM query WHERE false", language: "esql"} |
```

#### **Scenario: `ABA` - Rule field is rule `type`**

**Automation**: 1 integration test with mock rules + a set of unit tests for each algorithm

```Gherkin
Given <field_name> field is customized by the user (current version != base version)
And <field_name> field is not updated by Elastic in this upgrade (target version == base version)
Then for <field_name> field the diff algorithm should output the target version as the merged one with a non-solvable conflict
And <field_name> field should be returned from the `upgrade/_review` API endpoint
And <field_name> field should be shown in the upgrade preview UI

Examples:
| algorithm | field_name | base_version | current_version | target_version | merged_version |
| rule type | type | "query" | "saved_query" | "query" | "query" |
```

Notes: `type` field can only be changed between `query` and `saved_query` rule types in the UI and API via normal conventions, but the logic for others is still covered

### Rule field has an update and doesn't have a custom value - `AAB`

#### **Scenario: `AAB` - Rule field is any type**
#### **Scenario: `AAB` - Rule field is any type except rule `type`**

**Automation**: 10 integration tests with mock rules + a set of unit tests for each algorithm

Expand All @@ -151,9 +175,27 @@ Examples:
| esql_query | esql_query | {query: "FROM query WHERE true", language: "esql"} | {query: "FROM query WHERE true", language: "esql"} | {query: "FROM query WHERE false", language: "esql"} | {query: "FROM query WHERE false", language: "esql"} |
```

#### **Scenario: `AAB` - Rule field is rule `type`**

**Automation**: 1 integration test with mock rules + a set of unit tests for each algorithm

```Gherkin
Given <field_name> field is not customized by the user (current version == base version)
And <field_name> field is updated by Elastic in this upgrade (target version != base version)
Then for <field_name> field the diff algorithm should output the target version as the merged one with a non-solvable conflict
And <field_name> field should be returned from the `upgrade/_review` API endpoint
And <field_name> field should be shown in the upgrade preview UI

Examples:
| algorithm | field_name | base_version | current_version | target_version | merged_version |
| rule type | type | "query" | "query" | "saved_query" | "saved_query" |
```

Notes: `type` field can only be changed between `query` and `saved_query` rule types in the UI and API via normal conventions, but the logic for others is still covered

### Rule field has an update and a custom value that are the same - `ABB`

#### **Scenario: `ABB` - Rule field is any type**
#### **Scenario: `ABB` - Rule field is any type except rule `type`**

**Automation**: 10 integration tests with mock rules + a set of unit tests for each algorithm

Expand All @@ -179,8 +221,46 @@ Examples:
| esql_query | esql_query | {query: "FROM query WHERE true", language: "esql"} | {query: "FROM query WHERE false", language: "esql"} | {query: "FROM query WHERE false", language: "esql"} | {query: "FROM query WHERE false", language: "esql"} |
```

#### **Scenario: `ABB` - Rule field is rule `type`**
Copy link
Contributor

@jpdjere jpdjere Sep 26, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@dplumlee The same in this PR for possible and not possible cases:

#193369 (comment)

Maybe @pborgonovi has an opinion on this: some of the scenarios are not possible in our system (see comment above), but technically we can create an outcome for them. What should be the way forward from a testing perspective? Should we write in the test plan that the scenarios are not possible? Should they have test cases anyways (even though the mock data that needs to be generated in order to create those scenarios is technically not possible to achieve in the normal flow of the application)?

Copy link
Contributor

@pborgonovi pborgonovi Sep 27, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @dplumlee and @jpdjere

For edge scenarios that are not possible within the current flow of our application but can be tested using mock data, I recommend we still include them in our test plan.

We should include an explanation as comment of why these scenarios are not achievable in the normal flow, outlining any relevant technical limitations. This will help manage expectations and ensure everyone is aware of the system's current capabilities. Including these scenarios can also help future-proof the system and ensure it doesn’t break if they arise due to future changes. In the corresponding test cases, we should indicate that the tests depend on mock data.

Let me know if that makes sense.


**Automation**: 1 integration test with mock rules + a set of unit tests for each algorithm

```Gherkin
Given <field_name> field is customized by the user (current version != base version)
And <field_name> field is updated by Elastic in this upgrade (target version != base version)
And customized <field_name> field is the same as the Elastic update in this upgrade (current version == target version)
Then for <field_name> field the diff algorithm should output the target version as the merged one with a non-solvable conflict
And <field_name> field should be returned from the `upgrade/_review` API endpoint
And <field_name> field should be shown in the upgrade preview UI

Examples:
| algorithm | field_name | base_version | current_version | target_version | merged_version |
| rule type | type | "query" | "saved_query" | "saved_query" | "saved_query" |
```

Notes: `type` field can only be changed between `query` and `saved_query` rule types in the UI and API via normal conventions, but the logic for others is still covered

### Rule field has an update and a custom value that are NOT the same - `ABC`

#### **Scenario: `ABC` - Rule field is rule `type`**

**Automation**: 1 integration test with mock rules + a set of unit tests for the algorithms

```Gherkin
Given <field_name> field is customized by the user (current version != base version)
And <field_name> field is updated by Elastic in this upgrade (target version != base version)
And customized <field_name> field is different than the Elastic update in this upgrade (current version != target version)
Then for <field_name> field the diff algorithm should output the target version as the merged one with a non-solvable conflict
And <field_name> field should be returned from the `upgrade/_review` API endpoint
And <field_name> field should be shown in the upgrade preview UI

Examples:
| algorithm | field_name | base_version | current_version | target_version | merged_version |
| rule type | type | "query" | "saved_query" | "threshold" | "threshold" |
```

Notes: `type` field can only be changed between `query` and `saved_query` rule types in the UI and API via normal conventions, but the logic for others is still covered. This test case scenario cannot currently be reached.

#### **Scenario: `ABC` - Rule field is a number or single line string**

**Automation**: 2 integration tests with mock rules + a set of unit tests for the algorithms
Expand Down Expand Up @@ -328,7 +408,7 @@ Examples:

#### **Scenario: `-AA` - Rule field is any type**

**Automation**: 9 integration tests with mock rules + a set of unit tests for each algorithm
**Automation**: 11 integration tests with mock rules + a set of unit tests for each algorithm

```Gherkin
Given at least 1 installed prebuilt rule has a new version available
Expand All @@ -340,6 +420,7 @@ And <field_name> field should not be shown in the upgrade preview UI

Examples:
| algorithm | field_name | base_version | current_version | target_version | merged_version |
| rule type | type | N/A | "query" | "query" | "query" |
| single line string | name | N/A | "A" | "A" | "A" |
| multi line string | description | N/A | "My description.\nThis is a second line." | "My description.\nThis is a second line." | "My description.\nThis is a second line." |
| number | risk_score | N/A | 1 | 1 | 1 |
Expand Down Expand Up @@ -438,3 +519,23 @@ Examples:
| algorithm | base_version | current_version | target_version | merged_version |
| data_source | N/A | {type: "index_patterns", "index_patterns": ["one", "two", "three"]} | {type: "data_view", "data_view_id": "A"} | {type: "data_view", "data_view_id": "A"} |
```

#### **Scenario: `-AB` - Rule field is rule `type`**

**Automation**: 1 integration test with mock rules + a set of unit tests for the algorithm

```Gherkin
Given at least 1 installed prebuilt rule has a new version available
And the base version of the rule cannot be determined
And customized data_source field is different than the Elastic update in this upgrade (current version != target version)
And current version and target version are not both array fields in data_source
Then for data_source field the diff algorithm should output the target version as the merged version with a non-solvable conflict
And data_source field should be returned from the `upgrade/_review` API endpoint
And data_source field should be shown in the upgrade preview UI

Examples:
| algorithm | base_version | current_version | target_version | merged_version |
| rule type | N/A | "query" | "saved_query" | "saved_query" |
```

Notes: `type` field can only be changed between `query` and `saved_query` rule types in the UI and API via normal conventions, but the logic for others is still covered