[Security Solution] Adds serverlessQA tag to the Cypress tests
#179737
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
MadameSheema
requested review from
patrykkopycinski,
oatkiller,
maximpn,
banderror and
a team
as code owners
|
/ci |
MadameSheema
added
the
Team: SecuritySolution
|
Pinging @elastic/security-solution (Team: SecuritySolution) |
MadameSheema
added
release_note:skip
dkirchan
approved these changes
Apr 1, 2024
dkirchan
approved these changes
Apr 1, 2024
yctercero
reviewed
Apr 1, 2024
maximpn
reviewed
Apr 1, 2024
MadameSheema
added a commit
that referenced
this pull request
Apr 4, 2024
This is an intermediate PR before continuing working with #179737 ## Summary When I started to implement the new solution for the labeling in serverless, I saw we had lots of tests with `@brokenInServerlessQA` without any linked ticket. Before proceeding with the implementation of the new solution I wanted to check wich was the current status of the tests since once the new implementation is ready, `brokenInServerlessQA` will not exist anymore and a test skipped in serverless is going to be skipped in both PR and MKI environments. As an output of this PR: - The `@brokenInServerlessQA` tag has been removed on those tests that currently work on MKI. - For the non-working tests on MKI a link to a github ticket has been added. - The `alerts_details` tests have been fixed in MKI. With the previous implementation, we were not able to access the local storage on MKI. - The tests using `createEmailConnector` method have been fixed. Once this PR is merged I'll continue working on: #179737 and the `@brokenInServerlessQA` tag will be removed there. --------- Co-authored-by: kibanamachine <[email protected]>
6 tasks
angorayc
approved these changes
Apr 5, 2024
banderror
approved these changes
Apr 5, 2024
There was a problem hiding this comment.
Thank you @MadameSheema! LGTM 👍
We'll track the skipped Rule Management tests and think about what tests we should enable in the 2nd quality gate. This will be done as part of https://github.com/elastic/kibana/issues/179264.
yctercero
approved these changes
Apr 8, 2024
|
In my last commit skipped the following serverless tests without a ticket |
💚 Build Succeeded
Metrics [docs]
History
To update your PR or re-run it, just comment with: |
This was referenced Apr 9, 2024
MadameSheema
added a commit
that referenced
this pull request
Apr 23, 2024
Continues #179737 effort we are aligning the tags on Cypress and API to have a unified experience. ## Summary We want to start integrating our Cypress tests with the serverless Kibana quality gate. However, not all the teams feel comfortable enabling all the tests, to facilitate the effort of enabling tests in the quality gate we are adding the `@serverlessQA` tag, now on API tests as well. We use tags to select which tests we want to execute on each environment and pipeline. `ess` - runs in ESS env `serverless` - runs in serverless env and periodic pipeline (failures don't block release) `serverlessQA` - runs in kibana release process (failures block release) `skipInEss` - skipped for ESS env `skipInServerless` - skipped for all serverless related environments `skipInServerlessMKI` - skipped for MKI environments ### Description **Tests tagged as `@serverless`** All the tests tagged as `@serverless` will be executed as part of the PR validation process using the serverless FTR environment (not a real one). That tests will be executed as well as part of the periodic pipeline using a real serverless project. QA environment is used to do so using the latest available commit in main at the time of the execution. **Tests tagged as `@serverlessQA`** All the tests tagged as `@serverlessQA` will be executed as part of the kibana release process using a real serverless project with the latest image available in the QA environment **Tests tagged as `@ess`** All the tests tagged as `@ess` will be executed as part of the PR validation process using an on-prem ESS environment. **Tests tagged as `@skipInServerless`** All the tests tagged as `@skipInServerless` will be excluded from the PR validation process using the serverless FTR environment, the periodic pipeline and kibana release process for Serverless. **Tests tagged as `@skipInEss`** All the tests tagged as `skipInEss` will be excluded from the PR validation process using an on-prem ESS environment. --------- Co-authored-by: Ryland Herrick <[email protected]>
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this pull request
Apr 23, 2024
…80773) Continues elastic#179737 effort we are aligning the tags on Cypress and API to have a unified experience. ## Summary We want to start integrating our Cypress tests with the serverless Kibana quality gate. However, not all the teams feel comfortable enabling all the tests, to facilitate the effort of enabling tests in the quality gate we are adding the `@serverlessQA` tag, now on API tests as well. We use tags to select which tests we want to execute on each environment and pipeline. `ess` - runs in ESS env `serverless` - runs in serverless env and periodic pipeline (failures don't block release) `serverlessQA` - runs in kibana release process (failures block release) `skipInEss` - skipped for ESS env `skipInServerless` - skipped for all serverless related environments `skipInServerlessMKI` - skipped for MKI environments ### Description **Tests tagged as `@serverless`** All the tests tagged as `@serverless` will be executed as part of the PR validation process using the serverless FTR environment (not a real one). That tests will be executed as well as part of the periodic pipeline using a real serverless project. QA environment is used to do so using the latest available commit in main at the time of the execution. **Tests tagged as `@serverlessQA`** All the tests tagged as `@serverlessQA` will be executed as part of the kibana release process using a real serverless project with the latest image available in the QA environment **Tests tagged as `@ess`** All the tests tagged as `@ess` will be executed as part of the PR validation process using an on-prem ESS environment. **Tests tagged as `@skipInServerless`** All the tests tagged as `@skipInServerless` will be excluded from the PR validation process using the serverless FTR environment, the periodic pipeline and kibana release process for Serverless. **Tests tagged as `@skipInEss`** All the tests tagged as `skipInEss` will be excluded from the PR validation process using an on-prem ESS environment. --------- Co-authored-by: Ryland Herrick <[email protected]> (cherry picked from commit c351e14)
kibanamachine
added a commit
that referenced
this pull request
Apr 23, 2024
…80773) (#181468) # Backport This will backport the following commits from `main` to `8.14`: - [[Security Solution] Adds serverlessQA tag to the API tests (#180773)](#180773) <!--- Backport version: 9.4.3 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Gloria Hornero","email":"[email protected]"},"sourceCommit":{"committedDate":"2024-04-23T16:21:13Z","message":"[Security Solution] Adds serverlessQA tag to the API tests (#180773)\n\nContinues #179737 effort we are\r\naligning the tags on Cypress and API to have a unified experience.\r\n\r\n## Summary\r\n\r\nWe want to start integrating our Cypress tests with the serverless\r\nKibana quality gate. However, not all the teams feel comfortable\r\nenabling all the tests, to facilitate the effort of enabling tests in\r\nthe quality gate we are adding the `@serverlessQA` tag, now on API tests\r\nas well.\r\n\r\nWe use tags to select which tests we want to execute on each environment\r\nand pipeline.\r\n\r\n`ess` - runs in ESS env\r\n`serverless` - runs in serverless env and periodic pipeline (failures\r\ndon't block release)\r\n`serverlessQA` - runs in kibana release process (failures block release)\r\n`skipInEss` - skipped for ESS env\r\n`skipInServerless` - skipped for all serverless related environments\r\n`skipInServerlessMKI` - skipped for MKI environments\r\n\r\n### Description\r\n\r\n**Tests tagged as `@serverless`**\r\n\r\nAll the tests tagged as `@serverless` will be executed as part of the PR\r\nvalidation process using the serverless FTR environment (not a real\r\none). That tests will be executed as well as part of the periodic\r\npipeline using a real serverless project. QA environment is used to do\r\nso using the latest available commit in main at the time of the\r\nexecution.\r\n\r\n**Tests tagged as `@serverlessQA`**\r\n\r\nAll the tests tagged as `@serverlessQA` will be executed as part of the\r\nkibana release process using a real serverless project with the latest\r\nimage available in the QA environment\r\n\r\n**Tests tagged as `@ess`**\r\n\r\nAll the tests tagged as `@ess` will be executed as part of the PR\r\nvalidation process using an on-prem ESS environment.\r\n\r\n**Tests tagged as `@skipInServerless`**\r\n\r\nAll the tests tagged as `@skipInServerless` will be excluded from the PR\r\nvalidation process using the serverless FTR environment, the periodic\r\npipeline and kibana release process for Serverless.\r\n\r\n**Tests tagged as `@skipInEss`**\r\n\r\nAll the tests tagged as `skipInEss` will be excluded from the PR\r\nvalidation process using an on-prem ESS environment.\r\n\r\n---------\r\n\r\nCo-authored-by: Ryland Herrick <[email protected]>","sha":"c351e14e83552262a25488104d207ce70e1d0b47","branchLabelMapping":{"^v8.15.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","Team: SecuritySolution","v8.14.0","v8.15.0"],"title":"[Security Solution] Adds serverlessQA tag to the API tests","number":180773,"url":"https://github.com/elastic/kibana/pull/180773","mergeCommit":{"message":"[Security Solution] Adds serverlessQA tag to the API tests (#180773)\n\nContinues #179737 effort we are\r\naligning the tags on Cypress and API to have a unified experience.\r\n\r\n## Summary\r\n\r\nWe want to start integrating our Cypress tests with the serverless\r\nKibana quality gate. However, not all the teams feel comfortable\r\nenabling all the tests, to facilitate the effort of enabling tests in\r\nthe quality gate we are adding the `@serverlessQA` tag, now on API tests\r\nas well.\r\n\r\nWe use tags to select which tests we want to execute on each environment\r\nand pipeline.\r\n\r\n`ess` - runs in ESS env\r\n`serverless` - runs in serverless env and periodic pipeline (failures\r\ndon't block release)\r\n`serverlessQA` - runs in kibana release process (failures block release)\r\n`skipInEss` - skipped for ESS env\r\n`skipInServerless` - skipped for all serverless related environments\r\n`skipInServerlessMKI` - skipped for MKI environments\r\n\r\n### Description\r\n\r\n**Tests tagged as `@serverless`**\r\n\r\nAll the tests tagged as `@serverless` will be executed as part of the PR\r\nvalidation process using the serverless FTR environment (not a real\r\none). That tests will be executed as well as part of the periodic\r\npipeline using a real serverless project. QA environment is used to do\r\nso using the latest available commit in main at the time of the\r\nexecution.\r\n\r\n**Tests tagged as `@serverlessQA`**\r\n\r\nAll the tests tagged as `@serverlessQA` will be executed as part of the\r\nkibana release process using a real serverless project with the latest\r\nimage available in the QA environment\r\n\r\n**Tests tagged as `@ess`**\r\n\r\nAll the tests tagged as `@ess` will be executed as part of the PR\r\nvalidation process using an on-prem ESS environment.\r\n\r\n**Tests tagged as `@skipInServerless`**\r\n\r\nAll the tests tagged as `@skipInServerless` will be excluded from the PR\r\nvalidation process using the serverless FTR environment, the periodic\r\npipeline and kibana release process for Serverless.\r\n\r\n**Tests tagged as `@skipInEss`**\r\n\r\nAll the tests tagged as `skipInEss` will be excluded from the PR\r\nvalidation process using an on-prem ESS environment.\r\n\r\n---------\r\n\r\nCo-authored-by: Ryland Herrick <[email protected]>","sha":"c351e14e83552262a25488104d207ce70e1d0b47"}},"sourceBranch":"main","suggestedTargetBranches":["8.14"],"targetPullRequestStates":[{"branch":"8.14","label":"v8.14.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v8.15.0","branchLabelMappingKey":"^v8.15.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/180773","number":180773,"mergeCommit":{"message":"[Security Solution] Adds serverlessQA tag to the API tests (#180773)\n\nContinues #179737 effort we are\r\naligning the tags on Cypress and API to have a unified experience.\r\n\r\n## Summary\r\n\r\nWe want to start integrating our Cypress tests with the serverless\r\nKibana quality gate. However, not all the teams feel comfortable\r\nenabling all the tests, to facilitate the effort of enabling tests in\r\nthe quality gate we are adding the `@serverlessQA` tag, now on API tests\r\nas well.\r\n\r\nWe use tags to select which tests we want to execute on each environment\r\nand pipeline.\r\n\r\n`ess` - runs in ESS env\r\n`serverless` - runs in serverless env and periodic pipeline (failures\r\ndon't block release)\r\n`serverlessQA` - runs in kibana release process (failures block release)\r\n`skipInEss` - skipped for ESS env\r\n`skipInServerless` - skipped for all serverless related environments\r\n`skipInServerlessMKI` - skipped for MKI environments\r\n\r\n### Description\r\n\r\n**Tests tagged as `@serverless`**\r\n\r\nAll the tests tagged as `@serverless` will be executed as part of the PR\r\nvalidation process using the serverless FTR environment (not a real\r\none). That tests will be executed as well as part of the periodic\r\npipeline using a real serverless project. QA environment is used to do\r\nso using the latest available commit in main at the time of the\r\nexecution.\r\n\r\n**Tests tagged as `@serverlessQA`**\r\n\r\nAll the tests tagged as `@serverlessQA` will be executed as part of the\r\nkibana release process using a real serverless project with the latest\r\nimage available in the QA environment\r\n\r\n**Tests tagged as `@ess`**\r\n\r\nAll the tests tagged as `@ess` will be executed as part of the PR\r\nvalidation process using an on-prem ESS environment.\r\n\r\n**Tests tagged as `@skipInServerless`**\r\n\r\nAll the tests tagged as `@skipInServerless` will be excluded from the PR\r\nvalidation process using the serverless FTR environment, the periodic\r\npipeline and kibana release process for Serverless.\r\n\r\n**Tests tagged as `@skipInEss`**\r\n\r\nAll the tests tagged as `skipInEss` will be excluded from the PR\r\nvalidation process using an on-prem ESS environment.\r\n\r\n---------\r\n\r\nCo-authored-by: Ryland Herrick <[email protected]>","sha":"c351e14e83552262a25488104d207ce70e1d0b47"}}]}] BACKPORT--> Co-authored-by: Gloria Hornero <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
We want to start integrating our Cypress tests with the serverless Kibana quality gate. However, not all the teams feel comfortable enabling all the tests, to facilitate the effort of enabling tests in the quality gate we are adding the
@serverlessQAtag.Once this PR is merged, the behavior will be the following:
Regular PR flowAll the tests tagged as@serverlesswill be executed as part of the PR validation process using the serverless FTR environment (not a real one).If you want to skip a test to be executed in this flow use@brokenInServerlessor@skipInServerless.Periodic pipelineAll the tests tagged as@serverlesswill be executed as part of the periodic pipeline using a real serverless project. QA environment is used to do so using the latest available commit in main at the time of the execution.If you want to skip a test to be executed in this flow use@brokenInServerlessQA.Kibana second quality gateAll the tests tagged as@serverlessQAwill be executed as part of the kibana release process using a real serverless project with the latest image available in the QA environment.If you want to skip a test to be executed in this flow use@brokenInServerlessQA.Update
After addressing the following thread: #179737 (comment) these is going to be the new behaviour:
We use tags to select which tests we want to execute on each environment and pipeline.
Summary
ess- runs in ESS envserverless- runs in serverles env and periodic pipeline (failures don't block release)serverlessQA- runs in kibana release process (failures block release)skipInEss- skipped for ESS envskipInServerless- skipped for all serverless related environmentsDescription
Tests tagged as
@serverlessAll the tests tagged as
@serverlesswill be executed as part of the PR validation process using the serverless FTR environment (not a real one). That tests will be executed as well as part of the periodic pipeline using a real serverless project. QA environment is used to do so using the latest available commit in main at the time of the execution.Tests tagged as
@serverlessQAAll the tests tagged as
@serverlessQAwill be executed as part of the kibana release process using a real serverless project with the latest image available in the QA environmentTests tagged as
@essAll the tests tagged as
@esswill be executed as part of the PR validation process using an on-prem ESS environment.Tests tagged as
@skipInServerlessAll the tests tagged as
@skipInServerlesswill be excluded from the PR validation process using the serverless FTR environment, the periodic pipeline and kibana release process for Serverless.Tests tagged as
@skipInEssAll the tests tagged as
skipInEsswill be excluded from the PR validation process using an on-prem ESS environment.