Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Security Solution] Create test plan for prebuilt rule installation and upgrade workflows #160685

Merged
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
banderror marked this conversation as resolved.
Show resolved Hide resolved
Original file line number Diff line number Diff line change
@@ -0,0 +1,190 @@
# Rule Immutability / Customization
banderror marked this conversation as resolved.
Show resolved Hide resolved

## Test Plan for 2nd Milestone of Customizing prebuilt detection rules
banderror marked this conversation as resolved.
Show resolved Hide resolved
banderror marked this conversation as resolved.
Show resolved Hide resolved

### Useful information

Ticket: [https://github.com/elastic/security-team/issues/1974](https://github.com/elastic/security-team/issues/1974)

**Assumptions**
- The current test plan is only for Milestone 2 of the Rule Immutability/Customization feature to be released in 8.9. It does not pretend to cover any scenario for past or future milestones. Scenarios and flows are sensitive to change in future Milestones.
- Below scenarios only apply to prebuilt rules.
- Most of our users are on the 7.17.x version, that’s why the 8.x version is specified on scenarios, because this TestPlan is considering a minimum version of 8.x.
banderror marked this conversation as resolved.
Show resolved Hide resolved
- The rule Customization feature should be available to users on the Basic license and higher.


### Scenarios

### Notifications
banderror marked this conversation as resolved.
Show resolved Hide resolved

#### **Scenario: No callout messages are displayed when user does not have prebuilt rules installed**
banderror marked this conversation as resolved.
Show resolved Hide resolved

**GIVEN** user doesn't have any 8.x prebuilt rules installed
**AND** user is running a fresh instance
**WHEN** user navigates to the Rules Management Page
**THEN** no callouts message should be displayed

#### **Scenario: No callout messages are displayed when there are no pending installs/updates**

**GIVEN** user has the latest version of prebuilt rules `<prebuilt_rules_status>`
**WHEN** user navigates to the Rules Management Page
**THEN** no callout message is displayed for `<prebuilt_rules_status>` rules

*CASE 1: `<prebuilt_rules_status>` = installed*
*CASE 2: `<prebuilt_rules_status>` = updated*

banderror marked this conversation as resolved.
Show resolved Hide resolved
#### **Scenario: Callout message is displayed when there are new prebuilt rules available to install**

**GIVEN** user already has 8.x prebuilt rules installed
**AND** there are new prebuilt rules available to install
**WHEN** user navigates to the Rules Management Page
**THEN** user should see a callout message to install new prebuilt rules
**AND** the number of new rules available to install should be displayed on the +Add Elastic Rules link

#### **Scenario: Callout message is displayed when there are new updates on already installed prebuilt rules**

**GIVEN** user already has 8.x prebuilt rules installed
**AND** there are new updates available for those prebuilt rules
**WHEN** users navigate to the Rules Management Page
**THEN** users should see an update callout message
**AND** the number of outdated rules should be displayed on the Rules Updates tab

#### **Scenario: User is notified of available prebuilt rules to install when a rule is deleted**

**GIVEN** user has the latest version of prebuilt rules installed
**WHEN** user navigates to Rules Management Page
**AND** user deletes some prebuilt rules
**THEN** user should see a callout message with the same amount of prebuilt rules ready to install




### Prebuilt Rules Installation

#### **Scenario: User without any installed prebuilt rule can install `<amount>` prebuilt rules**

**GIVEN** a user that doesn’t have prebuilt rules installed
**WHEN** user navigates to Add Elastic Rules Page
**THEN** available prebuilt rules are displayed on Elastic Rules table
**AND** user can install `<amount>` prebuilt Rules
**AND** successfully installed message is displayed after installation
**AND** installed rules are removed from Elastic Rules table
**AND** rules to install counter is decreased accordingly

*CASE 1: `<amount>` = All*
*CASE 2: `<amount>` = Selected*
banderror marked this conversation as resolved.
Show resolved Hide resolved

#### **Scenario: User performing a clean install for prebuilt rules sees a loading skeleton until installation is completed**

**GIVEN** a user that is on Rules Management Page
**WHEN** user installs all prebuilt rules through Add Elastic Rules button/link
**THEN** a loading skeleton is displayed until the installation is completed

banderror marked this conversation as resolved.
Show resolved Hide resolved




### Prebuilt Rules Update

#### **Scenario: Users can update prebuilt rules**

**GIVEN** user already has 8.x prebuilt rules installed in Kibana
**AND** there are new updates available for those prebuilt rules
**AND** user is on Rules Management Page
**WHEN** user navigates to the Rules Update tab
**THEN** user should see all the prebuilt rules
banderror marked this conversation as resolved.
Show resolved Hide resolved
**AND** user can update outdated prebuilt rules
**AND** successfully updated message is displayed
**AND** Rules Upgrade tab counter is decreased according to the number of updated rules




### Installation / Update Failure

#### **Scenario: Error message is displayed when any prebuilt rules operation fails**

**GIVEN** user is `<action>` prebuilt rules
**WHEN** the installation or update process fails
**THEN** user should see an error message
**AND** prebuilt rules are not installed/updated
**AND** the callout message for pending installs/updates is still displayed on Rules Management Page
banderror marked this conversation as resolved.
Show resolved Hide resolved

*CASE 1: `<action>` = installing all*
*CASE 2: `<action>` = installing selected*
*CASE 3: `<action>` = Updating selected*
banderror marked this conversation as resolved.
Show resolved Hide resolved



### Add Elastic Rules Page

#### **Scenario: New workflow elements are displayed on Rules Management Page**

**GIVEN** a user that doesn’t have `security_detection_engine` package installed
**WHEN** user is on Rules Management Page
**THEN** “+Add Elastic rules” menu with available Rules counter is displayed
**AND** Rule Updates tab is displayed
**AND** “+Add Elastic rules” button is displayed on empty Rules Table

#### **Scenario: Rules settings persist on Add Elastic Rules table**

**GIVEN** a user has Rules listed on Add Elastic Rules page
**WHEN** user reloads the page
**THEN** the rule state should persist for all the rules
*CASE 1: after refreshing the table*
banderror marked this conversation as resolved.
Show resolved Hide resolved
*CASE 2: after switching table pagination*
*CASE 3: After filtering and clear filters*

#### **Scenario: User can navigate back to Rules Management page**

**GIVEN** a user is on Add Rules Page
**WHEN** user navigates back to Rules Management page
**THEN** Rules Management Page is properly displayed

#### **Scenario: User can filter prebuilt rules by query or by tag**

**GIVEN** a user is on Add Rules Page
**WHEN** user filters by `<filter>`
**THEN** Add Rules Table is properly updated

*CASE 1: `<filter>` = Query filter on search bar*
banderror marked this conversation as resolved.
Show resolved Hide resolved
*CASE 2: `<filter>` = Tag filter*



banderror marked this conversation as resolved.
Show resolved Hide resolved


### Authorization / RBAC

#### **Scenario: User with read privileges on security solution cannot install prebuilt rules**

**GIVEN** a user with Security: read privileges on Security solution
**WHEN** user navigates to Add Elastic Rules Page
**THEN** user can see available prebuilt rules to install
**AND** user cannot Install those prebuilt rules

#### **Scenario: User with read privileges on security solution cannot update prebuilt rules**

**GIVEN** a user with Security: read privileges on Security solution
**WHEN** user navigates to Rule Updates Tab on Rules Management Page
**THEN** user can see new updates for installed prebuilt rules
**AND** user cannot Update those prebuilt rules




### Kibana upgrade

#### **Scenario: User can operate with prebuilt rules when user upgrades from version `<version>` to 8.9 version**

**GIVEN** a user that is upgrading from version `<version>` to version 8.9
**AND** the `<version>` instance contains already installed prebuilt rules
**WHEN** the upgrade is complete
**THEN** user can install new prebuilt rules
**AND** remove installed prebuilt rules
**AND** update prebuilt rules from `<version>` to 8.9

| version |
|----------|
| 8.7 |
| 7.17.x |