Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Security Solution] Specific Cypress executions for Rule Management team #171868

Merged
merged 19 commits into from
Nov 28, 2023
Merged

[Security Solution] Specific Cypress executions for Rule Management team #171868

merged 19 commits into from
Nov 28, 2023

Conversation

MadameSheema
Copy link
Member

@MadameSheema MadameSheema commented Nov 23, 2023 •
edited by kibanamachine
Loading

Addresses: #153661

Summary

This PR is part of the effort we are making to have our cypres serverless tests ready for the second quality gate (QA environment - real serverless project and also part of the initial effort started by #153664.

With the introduced changes, we are creating specific Rule Management Cypress executions for both ESS and serverless with the aim of:

  • To help to identify quickly the ownership of the tests in case of failure.
  • To help to raise specific flakiness inside the tests of the team.

In this PR:

  • We are creating a specific folder for the rule_management team
  • We are creating 2 different executions (just prebuilt rules and all the tests excluding prebuilt rules) for ESS, serverless and the quality gate
  • We are integrating everything with buildkite
  • We are updating the readme

@MadameSheema MadameSheema changed the title new rule management folder [Security Solution] Specific Cypress executions for Rule Management team Nov 23, 2023
@MadameSheema MadameSheema self-assigned this Nov 23, 2023
@MadameSheema MadameSheema added Team:Detections and Resp Security Detection Response Team Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. Team:Detection Rule Management Security Detection Rule Management Team v8.12.0 release_note:skip Skip the PR/issue when compiling release notes labels Nov 23, 2023
@MadameSheema MadameSheema marked this pull request as ready for review November 23, 2023 15:09
@MadameSheema MadameSheema requested review from a team as code owners November 23, 2023 15:09
@elasticmachine
Copy link
Contributor

Pinging @elastic/security-detections-response (Team:Detections and Resp)

@elasticmachine
Copy link
Contributor

Pinging @elastic/security-solution (Team: SecuritySolution)

@banderror banderror requested review from banderror and removed request for dplumlee November 23, 2023 17:12
delanni
delanni approved these changes Nov 24, 2023
View reviewed changes
Copy link
Contributor

@delanni delanni left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

new test suite additions from the ops perspective 👍

Keep in mind, the current test slice runs on these suites are quite fast (8-12 minutes). Since several other tests are running for ~30 minutes, it could be OK to reduce parallelism, and have fewer, but longer running slices (this would reduce the sum overhead from agent provisioning/bootstrap)

@banderror
Copy link
Contributor

@MadameSheema Looking into it

banderror
banderror requested changes Nov 27, 2023
View reviewed changes
Copy link
Contributor

@banderror banderror left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Everything LGTM, it's just some paths need to be fixed in x-pack/test/security_solution_cypress/package.json.

I also posted a nit comment about folder structure, but it shouldn't block the PR.

...ecurity_solution_cypress/cypress/e2e/rule_management/authorization/all_rules_read_only.cy.ts Outdated
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: Hey @MadameSheema, was it necessary to move rule management tests outside of the x-pack/test/security_solution_cypress/cypress/e2e/detection_response folder to x-pack/test/security_solution_cypress/cypress/e2e/rule_management? Can we move them back?

I'd prefer having the following structure:

  • x-pack/test/security_solution_cypress/cypress/e2e/detection_response/detection_engine
  • x-pack/test/security_solution_cypress/cypress/e2e/detection_response/rule_management

cc @yctercero

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

++ on suggested change. I'm somewhat hesitant to base folder structure on teams (as we've updated teams several times), but I think it's needed in order to group the configs, right?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@yctercero is around making it easier to maintain the scripts we use as well as checking as knowing by a quick look who is the owner. Another thing that I noticed when we didn't have this structure is that several tests didn't have ownership at all making hard to follow some automated flows on operations side.

x-pack/test/security_solution_cypress/package.json Outdated Show resolved Hide resolved
.buildkite/pipelines/security_solution/security_solution_cypress.yml Show resolved Hide resolved
@kibana-ci
Copy link
Collaborator

💚 Build Succeeded

Metrics [docs]

✅ unchanged

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @MadameSheema

banderror
banderror approved these changes Nov 28, 2023
View reviewed changes
Copy link
Contributor

@banderror banderror left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks @MadameSheema 👍

@banderror
Copy link
Contributor

@MadameSheema I added the v8.11.2 label because I think it's always worth backporting changes in tests to minimize merge conflicts in the future. Often when we fix a bug and backport the fix, we backport some changes in tests too. Another example would be fixing flakiness - over the last few months, we've been backporting such PRs to reduce flakiness in release branches as well.

@MadameSheema MadameSheema merged commit 242cb6f into elastic:main Nov 28, 2023
2 checks passed
@MadameSheema MadameSheema deleted the rule-management-cypress-execution branch November 28, 2023 13:36
@kibanamachine
Copy link
Contributor

💔 All backports failed

Status Branch Result
❌ 8.11 Backport failed because of merge conflicts

You might need to backport the following PRs to 8.11:
- [Security Solution] fix cypress config to run all tests (#169942)
- [Security Solution] Adds more cypress tests on serverless for investigations team (#169379)
- [Security Solution] Adding serverlessQA tag (#167494)

Manual backport

To create the backport manually run:

node scripts/backport --pr 171868

Questions ?

Please refer to the Backport tool documentation

@jbudz jbudz added the reverted label Nov 28, 2023
@jbudz
Copy link
Member

jbudz commented Nov 28, 2023

This was reverted with 177dbd1 due to a parse error in on_merge.yml. See https://buildkite.com/elastic/kibana-on-merge/builds/38581#018c1627-73c5-4f15-9c17-f45b82626364.

@delanni delanni mentioned this pull request Nov 28, 2023
Closed
@MadameSheema MadameSheema restored the rule-management-cypress-execution branch November 28, 2023 14:12
vitaliidm pushed a commit to vitaliidm/kibana that referenced this pull request Nov 28, 2023
@jbudz
Revert "[Security Solution] Specific Cypress executions for `Rule Man…
177dbd1 
…agement` team (elastic#171868)"

This reverts commit 242cb6f.
This was referenced Nov 28, 2023
Merged
Merged
@MadameSheema MadameSheema deleted the rule-management-cypress-execution branch June 28, 2024 10:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@yctercero yctercero yctercero left review comments

@delanni delanni delanni approved these changes

@dkirchan dkirchan dkirchan approved these changes

@banderror banderror banderror approved these changes

@jpdjere jpdjere jpdjere approved these changes

Assignees

@MadameSheema MadameSheema

Labels
release_note:skip Skip the PR/issue when compiling release notes reverted Team:Detection Rule Management Security Detection Rule Management Team Team:Detections and Resp Security Detection Response Team Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. v8.11.2 v8.12.0
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
10 participants
@MadameSheema @elasticmachine @jpdjere @banderror @kibana-ci @kibanamachine @jbudz @delanni @yctercero @dkirchan