Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Security Solution] Adding serverlessQA tag #167494

Merged
merged 7 commits into from
Oct 13, 2023
Merged

[Security Solution] Adding serverlessQA tag #167494

merged 7 commits into from
Oct 13, 2023

Conversation

MadameSheema
Copy link
Member

@MadameSheema MadameSheema commented Sep 28, 2023 •
edited by kibanamachine
Loading

Summary

Currently, we have our cypress tests properly integrated with the first quality gate. 


In this quality gate, our tests are executed as part of the PR process using the serverless FTR environment. This environment uses the latest elasticsearch snapshot and the kibana source code, but it is not a real serverless environment because is not a deployed project on the cloud.

In order to continue assuring that we don’t introduce new issues, we want to execute automated tests in the second quality gate as well. The second quality gate is a deployed serverless project in the QA environment.

We want to start moving slowly in that direction and at the same time make sure we don't introduce flakiness, and the tests perform well.

In this PR we are creating the @serverlessQA tag. Tests including that label will be executed on the second quality gate. 
We are also adding the label to one of the tests we know that is stable in serverless to test the full integration.

Note that currently, we have some known limitations in this gate pending to be solved and we are working on it:

  • Execution of the tests is sequential
  • No differentiation between teams during the execution is done over the AET umbrella
  • The environment is set in complete so tests that exercise the essential behaviour should not be added.
  • Please ping me in case you want to add more tests into that gate to asses the risk before we have the gate tested and all the serverless tests stabilized.

@MadameSheema MadameSheema self-assigned this Sep 28, 2023
@MadameSheema MadameSheema added release_note:skip Skip the PR/issue when compiling release notes Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. v8.11.0 labels Sep 28, 2023
@MadameSheema MadameSheema marked this pull request as ready for review September 28, 2023 12:37
@MadameSheema MadameSheema requested review from a team as code owners September 28, 2023 12:37
@elasticmachine
Copy link
Contributor

Pinging @elastic/security-solution (Team: SecuritySolution)

stephmilovic
stephmilovic approved these changes Sep 28, 2023
View reviewed changes
Copy link
Contributor

@stephmilovic stephmilovic left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm for explore, thanks glo!

banderror
banderror approved these changes Sep 29, 2023
View reviewed changes
Copy link
Contributor

@banderror banderror left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, just one small comment: #167494 (comment)

Thank you @MadameSheema and @dkirchan!

@dkirchan dkirchan force-pushed the cypress/serverless-qa branch from da8eff1 to 93465b8 Compare October 13, 2023 10:55
@kibana-ci
Copy link
Collaborator

💛 Build succeeded, but was flaky

Failed CI Steps

Test Failures

  • [job] [logs] FTR Configs #60 / Cases - group 1 cases deletion sub privilege "after all" hook in "cases deletion sub privilege"
  • [job] [logs] FTR Configs #60 / Cases - group 1 cases deletion sub privilege create two cases logging in with user cases_read_delete_user "after all" hook in "logging in with user cases_read_delete_user"
  • [job] [logs] FTR Configs #60 / Cases - group 1 cases deletion sub privilege create two cases logging in with user cases_read_delete_user "before each" hook for "User cases_read_delete_user can delete a case while on a specific case page"
  • [job] [logs] Explore - Security Solution Cypress Tests #2 / risk tab with new risk score renders the table renders the table

Metrics [docs]

✅ unchanged

History

  • 💔 Build #167406 failed aa3547063330ab0154a7cbcc27b5b6a55fc7a6f1
  • 💔 Build #167339 failed 21d1890b5846eeef24563c31e872e3c044558994
  • 💛 Build #166207 was flaky 33c9c010976f8019bd5f4db6ce11b80ac3bfad96
  • 💔 Build #165416 failed 728d091dbea3b8378b038402f3522ec8827c299c
  • 💔 Build #165291 failed 1a2d50af1548b929234e79921de55dea69c0e040

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @MadameSheema

@dkirchan dkirchan merged commit aa27079 into elastic:main Oct 13, 2023
@kibanamachine
Copy link
Contributor

💔 All backports failed

Status Branch Result
❌ 8.11 An unhandled error occurred. Please see the logs for details

Manual backport

To create the backport manually run:

node scripts/backport --pr 167494

Questions ?

Please refer to the Backport tool documentation

@banderror
Copy link
Contributor

@MadameSheema FYI backport failed

@lukeelmers
Copy link
Member

JFYI, this QA quality gate failed with what seems to be an issue with the cypress script: https://buildkite.com/elastic/kibana-tests/builds/68

$ ./.buildkite/scripts/pipelines/security_solution_quality_gate/pipeline.sh
.buildkite/scripts/steps/functional/common_cypress.sh: line 3: Xvfb: command not found
[bazel] writing .bazelrc
.buildkite/scripts/common/setup_bazel.sh: line 6: KIBANA_DIR: unbound variable
:rotating_light: Error: The command exited with status 1

This is going to be a blocker for subsequent serverless releases, we'll probably need to either revert/disable this quality gate, allow it to soft fail, or find a way to fix the script quickly. I'm not at all familiar with this QG or the cypress tests setup, but hopefully @elastic/kibana-operations can help advise. cc @watson

@lukeelmers lukeelmers mentioned this pull request Oct 13, 2023
Merged
@lukeelmers
Copy link
Member

I've raised a PR to allow these QGs to soft fail so we can be unblocked: #168898

Ikuni17 pushed a commit that referenced this pull request Oct 13, 2023
@lukeelmers
Allow security solution QA QGs to soft fail. (#168898)
492a074 
Security solution's serverless QA quality gates which were added in
#167494 are failing when trying to
run cypress tests. I've marked them to soft fail here so we can unblock
the serverless release pipeline.

cc @watson @elastic/kibana-operations @MadameSheema @charlie-pichette
@YulNaumenko
@kibanamachine kibanamachine mentioned this pull request Oct 17, 2023
Merged
dej611 pushed a commit to dej611/kibana that referenced this pull request Oct 17, 2023
@MadameSheema @banderror
@dkirchan @dej611
[Security Solution] Adding serverlessQA tag (elastic#167494)
ee58fdb 
## Summary

Currently, we have our cypress tests properly integrated with the first
quality gate. 


In this quality gate, our tests are executed as part of the PR process
using the serverless FTR environment. This environment uses the latest
elasticsearch snapshot and the kibana source code, but it is not a real
serverless environment because is not a deployed project on the cloud.

In order to continue assuring that we don’t introduce new issues, we
want to execute automated tests in the second quality gate as well. The
second quality gate is a deployed serverless project in the QA
environment.

We want to start moving slowly in that direction and at the same time
make sure we don't introduce flakiness, and the tests perform well.

In this PR we are creating the `@serverlessQA` tag. Tests including that
label will be executed on the second quality gate. 
We are also adding
the label to one of the tests we know that is stable in serverless to
test the full integration.

Note that currently, we have some known limitations in this gate pending
to be solved and we are working on it:
- Execution of the tests is sequential
- No differentiation between teams during the execution is done over the
AET umbrella
- The environment is set in `complete` so tests that exercise the
`essential` behaviour should not be added.
- Please ping me in case you want to add more tests into that gate to
asses the risk before we have the gate tested and all the serverless
tests stabilized.

---------

Co-authored-by: Georgii Gorbachev <[email protected]>
Co-authored-by: dkirchan <[email protected]>
dej611 pushed a commit to dej611/kibana that referenced this pull request Oct 17, 2023
@lukeelmers @dej611
Allow security solution QA QGs to soft fail. (elastic#168898)
fc79f28 
Security solution's serverless QA quality gates which were added in
elastic#167494 are failing when trying to
run cypress tests. I've marked them to soft fail here so we can unblock
the serverless release pipeline.

cc @watson @elastic/kibana-operations @MadameSheema @charlie-pichette
@YulNaumenko
This was referenced Oct 26, 2023
Merged
Merged
This was referenced Nov 3, 2023
Merged
Merged
This was referenced Nov 28, 2023
Merged
Merged
@kibanamachine
Copy link
Contributor

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create automatically backports add the label auto-backport or prevent reminders by adding the backport:skip label.
You can also create backports manually by running node scripts/backport --pr 167494 locally

@kibanamachine kibanamachine added the backport missing Added to PRs automatically when the are determined to be missing a backport. label Jun 9, 2024
@kibanamachine
Copy link
Contributor

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create automatically backports add the label auto-backport or prevent reminders by adding the backport:skip label.
You can also create backports manually by running node scripts/backport --pr 167494 locally

4 similar comments
@kibanamachine
Copy link
Contributor

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create automatically backports add the label auto-backport or prevent reminders by adding the backport:skip label.
You can also create backports manually by running node scripts/backport --pr 167494 locally

@kibanamachine
Copy link
Contributor

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create automatically backports add the label auto-backport or prevent reminders by adding the backport:skip label.
You can also create backports manually by running node scripts/backport --pr 167494 locally

@kibanamachine
Copy link
Contributor

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create automatically backports add the label auto-backport or prevent reminders by adding the backport:skip label.
You can also create backports manually by running node scripts/backport --pr 167494 locally

@kibanamachine
Copy link
Contributor

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create automatically backports add the label auto-backport or prevent reminders by adding the backport:skip label.
You can also create backports manually by running node scripts/backport --pr 167494 locally

@delanni delanni added the backport:skip This commit does not require backporting label Jun 24, 2024
@kibanamachine kibanamachine removed the backport missing Added to PRs automatically when the are determined to be missing a backport. label Jun 24, 2024
@MadameSheema MadameSheema deleted the cypress/serverless-qa branch June 28, 2024 10:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dkirchan dkirchan dkirchan approved these changes

@jbudz jbudz jbudz left review comments

@Ikuni17 Ikuni17 Ikuni17 approved these changes

@banderror banderror banderror approved these changes

@stephmilovic stephmilovic stephmilovic approved these changes

Assignees

@MadameSheema MadameSheema

Labels
backport:skip This commit does not require backporting release_note:skip Skip the PR/issue when compiling release notes Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. v8.11.0 v8.12.0
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.