Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Security Solution] Specific Cypress executions for Rule Management team #171868

Merged
Merged
Show file tree
Hide file tree
Changes from 16 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -95,6 +95,32 @@ steps:
- exit_status: '*'
limit: 1

- command: .buildkite/scripts/steps/functional/security_serverless_rule_management.sh
MadameSheema marked this conversation as resolved.
Show resolved Hide resolved
label: 'Serverless Rule Management - Security Solution Cypress Tests'
if: "build.env('SKIP_CYPRESS') != '1' && build.env('SKIP_CYPRESS') != 'true'"
agents:
queue: n2-4-spot
depends_on: build
timeout_in_minutes: 60
parallelism: 8
retry:
automatic:
- exit_status: '*'
limit: 1

- command: .buildkite/scripts/steps/functional/security_serverless_rule_management_prebuilt_rules.sh
label: 'Serverless Rule Management - Prebuilt Rules - Security Solution Cypress Tests'
if: "build.env('SKIP_CYPRESS') != '1' && build.env('SKIP_CYPRESS') != 'true'"
agents:
queue: n2-4-spot
depends_on: build
timeout_in_minutes: 60
parallelism: 6
retry:
automatic:
- exit_status: '*'
limit: 1

- command: .buildkite/scripts/steps/functional/defend_workflows_serverless.sh
label: 'Defend Workflows Cypress Tests on Serverless'
if: "build.env('SKIP_CYPRESS') != '1' && build.env('SKIP_CYPRESS') != 'true'"
Expand Down
48 changes: 48 additions & 0 deletions .buildkite/pipelines/on_merge.yml
Original file line number Diff line number Diff line change
Expand Up @@ -115,6 +115,54 @@ steps:
- exit_status: '*'
limit: 1

- command: .buildkite/scripts/steps/functional/security_serverless_rule_management.sh
label: 'Serverless Rule Management - Security Solution Cypress Tests'
agents:
queue: n2-4-spot
depends_on: build
timeout_in_minutes: 60
parallelism: 8
retry:
automatic:
- exit_status: '*'
limit: 1

- command: .buildkite/scripts/steps/functional/security_serverless_rule_management_prebuilt_rules.sh
label: 'Serverless Rule Management - Prebuilt Rules - Security Solution Cypress Tests'
agents:
queue: n2-4-spot
depends_on: build
timeout_in_minutes: 60
parallelism: 6
retry:
automatic:
- exit_status: '*'
limit: 1

- command: .buildkite/scripts/steps/functional/security_solution_rule_management.sh
label: 'Rule Management - Security Solution Cypress Tests'
agents:
queue: n2-4-spot
depends_on: build
timeout_in_minutes: 60
parallelism: 8
retry:
automatic:
- exit_status: '*'
limit: 1

- command: .buildkite/scripts/steps/functional/security_solution_rule_management_prebuilt_rules.sh
delanni marked this conversation as resolved.
Show resolved Hide resolved
label: 'Rule Management - Prebuilt Rules - Security Solution Cypress Tests'
agents:
queue: n2-4-spot
depends_on: build
timeout_in_minutes: 60
parallelism: 6
retry:
automatic:
- exit_status: '*'
limit: 1

- command: .buildkite/scripts/steps/functional/security_solution.sh
label: 'Security Solution Cypress Tests'
agents:
Expand Down
48 changes: 48 additions & 0 deletions .buildkite/pipelines/pull_request/base.yml
Original file line number Diff line number Diff line change
Expand Up @@ -93,6 +93,30 @@ steps:
- exit_status: '*'
limit: 1

- command: .buildkite/scripts/steps/functional/security_serverless_rule_management.sh
label: 'Serverless Rule Management - Security Solution Cypress Tests'
agents:
queue: n2-4-spot
depends_on: build
timeout_in_minutes: 60
parallelism: 8
retry:
automatic:
- exit_status: '*'
limit: 1

- command: .buildkite/scripts/steps/functional/security_serverless_rule_management_prebuilt_rules.sh
label: 'Serverless Rule Management - Prebuilt Rules - Security Solution Cypress Tests'
agents:
queue: n2-4-spot
depends_on: build
timeout_in_minutes: 60
parallelism: 6
retry:
automatic:
- exit_status: '*'
limit: 1

- command: .buildkite/scripts/steps/functional/security_solution.sh
label: 'Security Solution Cypress Tests'
agents:
Expand All @@ -117,6 +141,30 @@ steps:
- exit_status: '*'
limit: 1

- command: .buildkite/scripts/steps/functional/security_solution_rule_management.sh
label: 'Rule Management - Security Solution Cypress Tests'
agents:
queue: n2-4-spot
depends_on: build
timeout_in_minutes: 60
parallelism: 8
retry:
automatic:
- exit_status: '*'
limit: 1

- command: .buildkite/scripts/steps/functional/security_solution_rule_management_prebuilt_rules.sh
label: 'Rule Management - Prebuilt Rules - Security Solution Cypress Tests'
agents:
queue: n2-4-spot
depends_on: build
timeout_in_minutes: 60
parallelism: 6
retry:
automatic:
- exit_status: '*'
limit: 1

- command: .buildkite/scripts/steps/functional/security_solution_investigations.sh
label: 'Investigations - Security Solution Cypress Tests'
agents:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,30 @@ steps:
# TODO : Revise the timeout when the pipeline will be officially integrated with the quality gate.
timeout_in_minutes: 300
parallelism: 8
retry:
automatic:
- exit_status: '*'
limit: 1

- command: .buildkite/scripts/pipelines/security_solution_quality_gate/security_solution_cypress/mki_security_solution_cypress.sh cypress:run:qa:serverless:rule_management
label: 'Serverless MKI QA Rule Management - Security Solution Cypress Tests'
agents:
queue: n2-4-spot
# TODO : Revise the timeout when the pipeline will be officially integrated with the quality gate.
timeout_in_minutes: 300
parallelism: 8
banderror marked this conversation as resolved.
Show resolved Hide resolved
retry:
automatic:
- exit_status: '*'
limit: 1

- command: .buildkite/scripts/pipelines/security_solution_quality_gate/security_solution_cypress/mki_security_solution_cypress.sh cypress:run:qa:serverless:rule_management:prebuilt_rules
label: 'Serverless MKI QA Rule Management - Prebuilt Rules - Security Solution Cypress Tests'
agents:
queue: n2-4-spot
# TODO : Revise the timeout when the pipeline will be officially integrated with the quality gate.
timeout_in_minutes: 300
parallelism: 6
retry:
automatic:
- exit_status: '*'
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
#!/usr/bin/env bash

set -euo pipefail

source .buildkite/scripts/steps/functional/common.sh
source .buildkite/scripts/steps/functional/common_cypress.sh

export JOB=kibana-security-solution-chrome
export KIBANA_INSTALL_DIR=${KIBANA_BUILD_LOCATION}

echo "--- Rule Management Cypress Tests on Serverless"

cd x-pack/test/security_solution_cypress

set +e
yarn cypress:rule_management:run:serverless; status=$?; yarn junit:merge || :; exit $status
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
#!/usr/bin/env bash

set -euo pipefail

source .buildkite/scripts/steps/functional/common.sh
source .buildkite/scripts/steps/functional/common_cypress.sh

export JOB=kibana-security-solution-chrome
export KIBANA_INSTALL_DIR=${KIBANA_BUILD_LOCATION}

echo "--- Rule Management - Prebuilt Rules - Cypress Tests on Serverless"

cd x-pack/test/security_solution_cypress

set +e
yarn cypress:rule_management:prebuilt_rules:run:serverless; status=$?; yarn junit:merge || :; exit $status
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
#!/usr/bin/env bash

set -euo pipefail

source .buildkite/scripts/steps/functional/common.sh
source .buildkite/scripts/steps/functional/common_cypress.sh

export JOB=kibana-security-solution-chrome
export KIBANA_INSTALL_DIR=${KIBANA_BUILD_LOCATION}

echo "--- Rule Management - Security Solution Cypress Tests"

cd x-pack/test/security_solution_cypress

set +e
yarn cypress:rule_management:run:ess; status=$?; yarn junit:merge || :; exit $status
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
#!/usr/bin/env bash

set -euo pipefail

source .buildkite/scripts/steps/functional/common.sh
source .buildkite/scripts/steps/functional/common_cypress.sh

export JOB=kibana-security-solution-chrome
export KIBANA_INSTALL_DIR=${KIBANA_BUILD_LOCATION}

echo "--- Rule Management - Prebuilt Rules - Security Solution Cypress Tests"

cd x-pack/test/security_solution_cypress

set +e
yarn cypress:rule_management:prebuilt_rules:run:ess; status=$?; yarn junit:merge || :; exit $status
2 changes: 0 additions & 2 deletions .github/CODEOWNERS
Original file line number Diff line number Diff line change
Expand Up @@ -1318,9 +1318,7 @@ x-pack/test/security_solution_cypress/cypress/tasks/expandable_flyout @elastic/
/x-pack/plugins/security_solution/common/api/detection_engine/rule_monitoring @elastic/security-detection-rule-management
/x-pack/plugins/security_solution/common/detection_engine/rule_management @elastic/security-detection-rule-management

/x-pack/test/security_solution_cypress/cypress/e2e/detection_response/prebuilt_rules @elastic/security-detection-rule-management
/x-pack/test/security_solution_cypress/cypress/e2e/detection_response/rule_management @elastic/security-detection-rule-management
/x-pack/test/security_solution_cypress/cypress/e2e/detection_response/rule_details @elastic/security-detection-rule-management
/x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules @elastic/security-detection-rule-management
MadameSheema marked this conversation as resolved.
Show resolved Hide resolved
/x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/rule_management @elastic/security-detection-rule-management
/x-pack/test/security_solution_api_integration/test_suites/detections_response/default_license/prebuilt_rules @elastic/security-detection-rule-management
Expand Down
23 changes: 17 additions & 6 deletions x-pack/test/security_solution_cypress/cypress/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -62,19 +62,25 @@ Run the tests with the following yarn scripts from `x-pack/test/security_solutio
| cypress | Runs the default Cypress command |
| cypress:open:ess | Opens the Cypress UI with all tests in the `e2e` directory. This also runs a local kibana and ES instance. The kibana instance will reload when you make code changes. This is the recommended way to debug and develop tests. |
| cypress:open:serverless | Opens the Cypress UI with all tests in the `e2e` directory. This also runs a mocked serverless environment. The kibana instance will reload when you make code changes. This is the recommended way to debug and develop tests. |
| cypress:run:ess | Runs all tests tagged as ESS placed in the `e2e` directory excluding `investigations` and `explore` directories in headless mode |
| cypress:run:ess | Runs all tests tagged as ESS placed in the `e2e` directory excluding `investigations`,`explore` and `detection_response/rule_management` directories in headless mode |
| cypress:run:cases:ess | Runs all tests under `explore/cases` in the `e2e` directory related to the Cases area team in headless mode |
| cypress:ess | Runs all ESS tests with the specified configuration in headless mode and produces a report using `cypress-multi-reporters` |
| cypress:rule_management:run:ess | Runs all tests tagged as ESS in the `e2e/detection_response/rule_management` excluding `e2e/detection_response/rule_management/prebuilt_rules` directory in headless mode |
| cypress:rule_management:prebuilt_rules:run:ess | Runs all tests tagged as ESS in the `e2e/detection_response/rule_management/prebuilt_rules` directory in headless mode |
| cypress:run:respops:ess | Runs all tests related to the Response Ops area team, specifically tests in `detection_alerts`, `detection_rules`, and `exceptions` directories in headless mode |
| cypress:run:serverless | Runs all tests tagged as SERVERLESS in the `e2e` directory excluding `investigations` and `explore` directories in headless mode |
| cypress:investigations:run:ess | Runs all tests tagged as ESS in the `e2e/investigations` directory in headless mode |
| cypress:run:serverless | Runs all tests tagged as SERVERLESS in the `e2e` directory excluding `investigations`, `explore` and `rule_management` directories in headless mode |
| cypress:rule_management:run:serverless | Runs all tests tagged as SERVERLESS in the `e2e/detection_response/rule_management` excluding `e2e/detection_response/rule_management/prebuilt_rules` directory in headless mode |
| cypress:rule_management:prebuilt_rules:run:serverless | Runs all tests tagged as ESS in the `e2e/detection_response/rule_management/prebuilt_rules` directory in headless mode |
| cypress:investigations:run:ess | Runs all tests tagged as SERVERLESS in the `e2e/investigations` directory in headless mode |
| cypress:explore:run:ess | Runs all tests tagged as ESS in the `e2e/explore` directory in headless mode |
| cypress:investigations:run:serverless | Runs all tests tagged as SERVERLESS in the `e2e/investigations` directory in headless mode |
| cypress:explore:run:serverless | Runs all tests tagged as SERVERLESS in the `e2e/explore` directory in headless mode |
| cypress:open:qa:serverless | Opens the Cypress UI with all tests in the `e2e` directory tagged as SERVERLESS. This also creates an MKI project in console.qa enviornment. The kibana instance will reload when you make code changes. This is the recommended way to debug tests in QA. Follow the readme in order to learn about the known limitations. |
| cypress:run:qa:serverless | Runs all tests tagged as SERVERLESS placed in the `e2e` directory excluding `investigations` and `explore` directories in headless mode using the QA environment and real MKI projects.|
| cypress:run:qa:serverless | Runs all tests tagged as SERVERLESS placed in the `e2e` directory excluding `investigations`, `explore` and `rule_management` directories in headless mode using the QA environment and real MKI projects.|
| cypress:run:qa:serverless:explore | Runs all tests tagged as SERVERLESS in the `e2e/explore` directory in headless mode using the QA environment and real MKI prorjects. |
| cypress:run:qa:serverless:investigations | Runs all tests tagged as SERVERLESS in the `e2e/investigations` directory in headless mode using the QA environment and reak MKI projects. |
| cypress:run:qa:serverless:rule_management | Runs all tests tagged as SERVERLESS in the `e2e/detection_response/rule_management` directory, excluding `e2e/detection_response/rule_management/prebuilt_rules` in headless mode using the QA environment and reak MKI projects. |
| cypress:run:qa:serverless:rule_management:prebuilt_rules | Runs all tests tagged as SERVERLESS in the `e2e/detection_response/rule_management/prebuilt_rules` directory in headless mode using the QA environment and reak MKI projects. |
| junit:merge | Merges individual test reports into a single report and moves the report to the `junit` directory |

Please note that all the headless mode commands do not open the Cypress UI and are typically used in CI/CD environments. The scripts that open the Cypress UI are useful for development and debugging.
Expand All @@ -94,7 +100,7 @@ Below you can find the folder structure used on our Cypress tests.

Cypress convention starting version 10 (previously known as integration). Contains the specs that are going to be executed.

### e2e/explore and e2e/investigations
### Area teams folders

These directories contain tests which are run in their own Buildkite pipeline.

Expand All @@ -103,7 +109,8 @@ If you belong to one of the teams listed in the table, please add new e2e specs
| Directory | Area team |
| -- | -- |
| `e2e/explore` | Threat Hunting Explore |
| `e2e/investigations | Threat Hunting Investigations |
| `e2e/investigations` | Threat Hunting Investigations |
| `e2e/detection_response/rule_management` | Detection Rule Management |

### fixtures/

Expand Down Expand Up @@ -203,6 +210,8 @@ Run the tests with the following yarn scripts from `x-pack/test/security_solutio
| cypress:run:serverless | Runs all tests tagged as SERVERLESS in the `e2e` directory excluding `investigations` and `explore` directories in headless mode |
| cypress:investigations:run:serverless | Runs all tests tagged as SERVERLESS in the `e2e/investigations` directory in headless mode |
| cypress:explore:run:serverless | Runs all tests tagged as SERVERLESS in the `e2e/explore` directory in headless mode |
| cypress:rule_management:run:serverless | Runs all tests tagged as SERVERLESS in the `e2e/detection_response/rule_management` excluding `e2e/detection_response/rule_management/prebuilt_rules` directory in headless mode |
| cypress:rule_management:prebuilt_rules:run:serverless | Runs all tests tagged as ESS in the `e2e/detection_response/rule_management/prebuilt_rules` directory in headless mode |

Please note that all the headless mode commands do not open the Cypress UI and are typically used in CI/CD environments. The scripts that open the Cypress UI are useful for development and debugging.

Expand Down Expand Up @@ -248,6 +257,8 @@ Run the tests with the following yarn scripts from `x-pack/test/security_solutio
| cypress:run:qa:serverless | Runs all tests tagged as SERVERLESS placed in the `e2e` directory excluding `investigations` and `explore` directories in headless mode using the QA environment and real MKI projects.|
| cypress:run:qa:serverless:explore | Runs all tests tagged as SERVERLESS in the `e2e/explore` directory in headless mode using the QA environment and real MKI prorjects. |
| cypress:run:qa:serverless:investigations | Runs all tests tagged as SERVERLESS in the `e2e/investigations` directory in headless mode using the QA environment and reak MKI projects. |
| cypress:run:qa:serverless:rule_management | Runs all tests tagged as SERVERLESS in the `e2e/detection_response/rule_management` directory, excluding `e2e/detection_response/rule_management/prebuilt_rules` in headless mode using the QA environment and reak MKI projects. |
| cypress:run:qa:serverless:rule_management:prebuilt_rules | Runs all tests tagged as SERVERLESS in the `e2e/detection_response/rule_management/prebuilt_rules` directory in headless mode using the QA environment and reak MKI projects. |

Please note that all the headless mode commands do not open the Cypress UI and are typically used in CI/CD environments. The scripts that open the Cypress UI are useful for development and debugging.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -12,14 +12,14 @@ import {
} from '@kbn/security-solution-plugin/common/constants';
import { ROLES } from '@kbn/security-solution-plugin/common/test';

import { createRuleAssetSavedObject } from '../../../helpers/rules';
import { createRuleAssetSavedObject } from '../../../../helpers/rules';
import {
createAndInstallMockedPrebuiltRules,
installPrebuiltRuleAssets,
preventPrebuiltRulesPackageInstallation,
} from '../../../tasks/api_calls/prebuilt_rules';
import { visit } from '../../../tasks/navigation';
import { RULES_MANAGEMENT_URL } from '../../../urls/rules_management';
} from '../../../../tasks/api_calls/prebuilt_rules';
import { visit } from '../../../../tasks/navigation';
import { RULES_MANAGEMENT_URL } from '../../../../urls/rules_management';
import {
ADD_ELASTIC_RULES_BTN,
getInstallSingleRuleButtonByRuleId,
Expand All @@ -31,8 +31,8 @@ import {
RULES_UPDATES_TAB,
RULE_CHECKBOX,
UPGRADE_ALL_RULES_BUTTON,
} from '../../../screens/alerts_detection_rules';
import { login } from '../../../tasks/login';
} from '../../../../screens/alerts_detection_rules';
import { login } from '../../../../tasks/login';

// Rule to test update
const RULE_1_ID = 'rule_1';
Expand Down
Loading