-
Notifications
You must be signed in to change notification settings - Fork 8.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Security Solution] Adds serverlessQA tag to the API tests #180773
Conversation
/ci |
Pinging @elastic/security-solution (Team: SecuritySolution) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Entity analytics changes LGTM 🚀
/ci |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Security GenAI tag updates LGTM! 👍
Thanks for the clear communication and update here @MadameSheema!
/ci |
/ci |
/ci |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I had a few nits about definitions/discoverability, and it looks like a few of the build commands are missing their kibana commands (which needs to be fixed), but otherwise this LGTM.
.buildkite/pipelines/security_solution/api_integration_serverless_release.yml
Outdated
Show resolved
Hide resolved
.buildkite/pipelines/security_solution/api_integration_serverless_release.yml
Outdated
Show resolved
Hide resolved
limit: 2 | ||
|
||
- label: Running exception_operators_date_numeric_types:qa:serverless:release | ||
command: .buildkite/scripts/pipelines/security_solution_quality_gate/api_integration/api-integration-tests.sh exception_operators_date_numeric_types:qa:serverless:release |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
How does one trace this command to the tests/code that it executes?
The key exception_operators_date_numeric_types:qa:serverless:release
is the only argument to the test script, but that key only exists in this file. The test script seems to just call yarn run [key]
, but I don't see how yarn run exception_operators_date_numeric_types:qa:serverless:release
works if that command isn't defined somewhere.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Update: I see that I chose a key that was (accidentally?) omitted from x-pack/test/security_solution_api_integration/package.json
.
That answers my question about discoverability, but means that some of these are going to fail until the corresponding commands are defined.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It looks like either a wrong key usage or a lack of update in the branch with the latest code. @MadameSheema
- We are using the following tags: | ||
* `@ess`: Runs in an ESS environment (on-prem installation) as part of the CI validation on PRs. | ||
|
||
* `@serverless`: Runs in the first quality gate and in the periodic pipeline. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The only other mention of "first quality gate" is in the cypress README; I think it's worth defining these terms in some detail since they're not going to be self-evident to everyone.
Same thing for "periodic pipeline," "second quality gate," and "all quality gates."
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hey @rylnd everything is already documented.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@MadameSheema it's wonderful that it's documented, but how does someone reading this README find those definitions? You shared links to quality gates and periodic pipeline with me individually, but that process does not scale, so I'm proposing that we somehow reference that documentation here.
If we are reluctant/unable to share those docs because they're internal, I think that might be indication that this README either covers too much or is in the wrong place; our public documentation shouldn't reference internal processes that the reader has no insight on.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
++ to @rylnd's comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yup, share it with you in private due to being internal documentation, tbh I don't know what is best, thoughts? What do you prefer?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think the ideal approach in my mind would be:
- Internal documentation associating the cypress tags to our CI/QA infrastructure/processes
- Public documentation with definitions appropriate to those not involved in our operations
I'm imagining a contributor who's written a cypress test for a feature they'd like to add; I would not expect them to know/care about any of these tags. As part of the review process, an Elastician would tell them which tags need to be added to their tests.
Generally: if we can't provide the reader with enough information to determine whether they should use the tag, I don't think that it's helpful to mention it at all.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe a passing mention of the tags as "related to internal release processes" would be a way to both define the tags but also deemphasize them, here?
|
||
* `@serverless`: Runs in the first quality gate and in the periodic pipeline. | ||
|
||
* `@serverlessQA`: Runs in the second quality gate. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This tag seems to be strongly tied to the concept of a "release" (as evidenced in the command names, and this PR's description), but that isn't really reflected in either the README here nor in the tag name itself.
Would @serverlessRelease
be a more accurate and descriptive tag name?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
As part of the release, the tests might be executed as part of 2 other quality gates and we might have some on the second one that we don't want to execute it on the third one.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
IMO referencing the "gate order" might be the most straightforward way to express this; it's not intuitive that QA
represents "second gate," nor what the other gates are.
…ess_release.yml Co-authored-by: Ryland Herrick <[email protected]>
Co-authored-by: Ryland Herrick <[email protected]>
…ess_release.yml Co-authored-by: Ryland Herrick <[email protected]>
@@ -31,7 +31,7 @@ export default ({ getService }: FtrProviderContext): void => { | |||
/* from a package that was bundled with Kibana */ | |||
// | |||
// FLAKY: https://github.com/elastic/kibana/issues/180087 | |||
describe.skip('@ess @serverless @skipInQA install_bundled_prebuilt_rules', () => { | |||
describe.skip('@ess @serverless @skipInServerless install_bundled_prebuilt_rules', () => { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@MadameSheema We don't want all the tests for prebuilt rules to be skipped in all quality gates. We need them to run on CI, where, as far as I'm aware, they are not currently skipped and have been passing.
This also applies to some of the other rule management tests that are being skipped in CI by this PR.
I would suggest to:
- Either introduce an additional
@skipInServerlessMKI
tag and use it for these tests instead of@skipInServerless
. This tag would skip a test in the periodic pipeline + 2nd quality gate + 3rd and 4th quality gates in the future, but keep the test running on CI in the Serverless mode. - Or to rename the
@serverless
tag to@serverlessCI
and introduce:- an additional tag
@serverlessRelease
that would add a test to all the release pipelines: the 2nd quality gate + 3rd and 4th quality gates in the future - an additional tag
@serverlessPeriodic
that would add a test only to the periodic pipeline
- an additional tag
I'd personally choose the 2nd option as being more explicit about where a test should run.
Sorry if I missed this in #179737 where Cypress tests for prebuilt rules and rule management have been skipped in Serverless CI. We'd need to run those Cypress tests in CI too, so I'd propose the same changes for Cypress tests.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@maximpn @yctercero since you were the ones against having extra labels, can you please arrive to an agreement about how to proceed with it?
This is my proposal:
@serverless
-> Executes tests on CI and periodic pipeline (MKI)
@serverlessQA
-> Executes tests on the second quality gate (QA environment)
@skipServerless-> Does not execute the test in any serverless environment
@skipServerlessMKI
-> Does not execute the test in any MKI environment (periodic pipeline and quality gate)
If you don’t want a test to be executed as part of the second quality gate you don’t add the @serverlessQA
tag to it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@MadameSheema do I get it correctly a test which should be skipped in MKI env but run everywhere else will have the following tags
describe('@ess @serverless @serverlessQA @skipServerlessMKI install_bundled_prebuilt_rules', () => {
Here we return to the situation when it equals to
describe('@ess @serverless install_bundled_prebuilt_rules', () => {
Here we have two options
- Have only
@ess
,@skipInEss
,@serverless
,@serverlessQA
and@skipInServerless
tags. When we need to run a test in another MKI end we would add another@serverless<name>
tag. By default we just don't add tags so the test isn't enabled automatically for any MKI environment. There is an assumption MKI env is less stable than mocked Serverless env. It's impossible to disabled a test for only mocked Serverless env but still run it against some MKI env. But it doesn't look like an issue. - Have only
@ess
,@skipInEss
,@serverless
,@skipInServerlessMock
(what's the best name for local/CI Serverless instead of serverless mock?) and@skipInServerlessQA
tags. In this case tests tagged by@serverless
will run against everything (besides ESS) and adding a corresponding@skipInServerless<name>
tag skips a test in a proper env.
The second approach looks better for automatic skipping script. When a test fails in some env or becomes flaky it could be skipped automatically.
I checked the PR's diff once more time and see that it's more convenient to use the second approach. Quite often describe
section has permissive tags @ess
and @serverless
while individual tests have may be skipped via @skipInServerless
. It will require manually skip all the tests in MKI env via @skipInServerlessQA
but it should be simpler to maintain the tests.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Have only
@ess
,@skipInEss
,@serverless
,@serverlessQA
and@skipInServerless
tags.
This set of tags doesn't allow to run a test on CI in ESS + on CI in Serverless, but not run it in the periodic pipeline + release gates.
At this point, we need to be able to run many of our tests ONLY on CI in ESS + on CI in Serverless, because they are broken in MKI environments. Fixing this requires time and effort we don't have at the moment.
Have only
@ess
,@skipInEss
,@serverless
,@skipInServerlessMock
(what's the best name for local/CI Serverless instead of serverless mock?) and@skipInServerlessQA
tags
This set of tags also doesn't allow to run a test on CI in ESS + on CI in Serverless, but not run it in the periodic pipeline + release gates.
We don't need to be able to skip a test on CI in Serverless, but run it in MKI pipelines. We don't have such class of tests as of now.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Could you clarify the requirements to running tests in periodic pipeline? Do we need a separate way to skip tests for periodic pipeline?
In my understanding if a test is broken in MKI env it will be broken in periodic pipeline since it's an MKI env as well. Taking this into account a test shouldn't have @serverlessQA
tag (option 1) or have @skipInServerlessQA
tag (option 2). It won't run against any MKI env until it's fixed, tested manually and tag manipulation is performed to run the test against MKI envs.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Could you clarify the requirements to running tests in periodic pipeline? Do we need a separate way to skip tests for periodic pipeline?
In my understanding if a test is broken in MKI env it will be broken in periodic pipeline since it's an MKI env as well. Taking this into account a test shouldn't have @serverlessQAtag (option 1) or have @skipInServerlessQA tag (option 2). It won't run against any MKI env until it's fixed, tested manually and tag manipulation is performed to run the test against MKI envs.
As there are differences between the serverless environment we use on CI and a real one but at the same time, we don't want to execute all the tests in the quality gate, is a way for us to determine the health of our application in a real serverless project environment.
We need a separate way of skipping tests from MKI but not CI.
A test with just @serverless
is going to be executed in CI and periodic pipeline. If you add to the test @serverlessQA
that test will be executed in the second quality gate as part of the release process. If a test is broken on MKI is going to fail in both on periodic and quality gate, so you can skip it with @serverlessMKI
but it will still be executed on CI as part of the PRs so I strongly believe that right now what it makes sense for us to have as labels is:
@ess: Test is executed on CI ess environment
@skipEss: Test is excluded fom the CI ess environment execution
@serverless: Test is executed on CI serverless and periodic pipeline (MKI environment)
@serverlessQA: Test is executed as part of the release process on the second quality gate which is the QA environment (MKI environment)
@skipServerlessMKI: Test is excluded from being executed in any MKI environment, meaning, is not executed as part of the periodic pipeline and is not executed as part of the release process
@skipServerless: Tests is excluded from being executed as part from the CI and is not executed in any MKI environment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@MadameSheema Your proposal works for me 👍
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Above proposal looks great to me @MadameSheema
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Me and @banderror discussed the approach on a zoom call. Tags you suggest in the last comment look good to me. I have some comments though
ESS tags are pretty clear. We need to run tests against ESS env and skip broken tests.
@ess
@skipEss
Serverless in general is pretty clear as well. We need to run test against Serverless env (abstractly it doesn't matter which one) and skip broken tests
@serverless
@skipServerless
If we don't need to run tests against some env an appropriate tag @ess
or @serverless
isn't added.
The most interesting part start from the fact that Serverless envs are different. Not all tests work in Serverless MKI env currently. Taking into account this fact we need an ability to skip such tests so we can use the following tag
@skipServerlessMKI
And the last part is that we need some test to block Serverless release when critical functionality doesn't work. In this case something explicit like @blockServerlessRelease
or @serverlessRelease
would work the best
@serverlessRelease
I have a question about your suggestion. Why do you want to use explicit @serverlessQA
. Is it expected we are gonna run different test sets/subsets in QA
, Staging
etc Serverless MKI envs? It looks like if a test work in one Serverless MKI env it should work in the other one as well. Is there some difference between QA
and Staging
?
cc @rylnd
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe at this point it's best to jump on a Zoom. I'm a bit confused as to what's being asked here anymore.
@MadameSheema worked on this PR after we chatted and decided it'd be great for our API and cypress tags to match so this PR brought them into parity. We had discussed and decided on the tags during the first PR that changed the Cypress tags. It would be great to finalize the discussion so we don't keep rehashing the discussion in each PR.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I agree with your notes here: #180773 (comment) Thank you for making these changes @MadameSheema !
/ci |
💛 Build succeeded, but was flaky
Failed CI StepsMetrics [docs]
History
To update your PR or re-run it, just comment with: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you for addressing my concern @MadameSheema, LGTM 👍
Could we please have the same set of tags for Cypress tests?
- We are using the following tags: | ||
* `@ess`: Runs in an ESS environment (on-prem installation) as part of the CI validation on PRs. | ||
|
||
* `@serverless`: Runs in the first quality gate and in the periodic pipeline. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
++ to @rylnd's comment
…80773) Continues elastic#179737 effort we are aligning the tags on Cypress and API to have a unified experience. ## Summary We want to start integrating our Cypress tests with the serverless Kibana quality gate. However, not all the teams feel comfortable enabling all the tests, to facilitate the effort of enabling tests in the quality gate we are adding the `@serverlessQA` tag, now on API tests as well. We use tags to select which tests we want to execute on each environment and pipeline. `ess` - runs in ESS env `serverless` - runs in serverless env and periodic pipeline (failures don't block release) `serverlessQA` - runs in kibana release process (failures block release) `skipInEss` - skipped for ESS env `skipInServerless` - skipped for all serverless related environments `skipInServerlessMKI` - skipped for MKI environments ### Description **Tests tagged as `@serverless`** All the tests tagged as `@serverless` will be executed as part of the PR validation process using the serverless FTR environment (not a real one). That tests will be executed as well as part of the periodic pipeline using a real serverless project. QA environment is used to do so using the latest available commit in main at the time of the execution. **Tests tagged as `@serverlessQA`** All the tests tagged as `@serverlessQA` will be executed as part of the kibana release process using a real serverless project with the latest image available in the QA environment **Tests tagged as `@ess`** All the tests tagged as `@ess` will be executed as part of the PR validation process using an on-prem ESS environment. **Tests tagged as `@skipInServerless`** All the tests tagged as `@skipInServerless` will be excluded from the PR validation process using the serverless FTR environment, the periodic pipeline and kibana release process for Serverless. **Tests tagged as `@skipInEss`** All the tests tagged as `skipInEss` will be excluded from the PR validation process using an on-prem ESS environment. --------- Co-authored-by: Ryland Herrick <[email protected]> (cherry picked from commit c351e14)
💚 All backports created successfully
Note: Successful backport PRs will be merged automatically after passing CI. Questions ?Please refer to the Backport tool documentation |
…80773) (#181468) # Backport This will backport the following commits from `main` to `8.14`: - [[Security Solution] Adds serverlessQA tag to the API tests (#180773)](#180773) <!--- Backport version: 9.4.3 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Gloria Hornero","email":"[email protected]"},"sourceCommit":{"committedDate":"2024-04-23T16:21:13Z","message":"[Security Solution] Adds serverlessQA tag to the API tests (#180773)\n\nContinues #179737 effort we are\r\naligning the tags on Cypress and API to have a unified experience.\r\n\r\n## Summary\r\n\r\nWe want to start integrating our Cypress tests with the serverless\r\nKibana quality gate. However, not all the teams feel comfortable\r\nenabling all the tests, to facilitate the effort of enabling tests in\r\nthe quality gate we are adding the `@serverlessQA` tag, now on API tests\r\nas well.\r\n\r\nWe use tags to select which tests we want to execute on each environment\r\nand pipeline.\r\n\r\n`ess` - runs in ESS env\r\n`serverless` - runs in serverless env and periodic pipeline (failures\r\ndon't block release)\r\n`serverlessQA` - runs in kibana release process (failures block release)\r\n`skipInEss` - skipped for ESS env\r\n`skipInServerless` - skipped for all serverless related environments\r\n`skipInServerlessMKI` - skipped for MKI environments\r\n\r\n### Description\r\n\r\n**Tests tagged as `@serverless`**\r\n\r\nAll the tests tagged as `@serverless` will be executed as part of the PR\r\nvalidation process using the serverless FTR environment (not a real\r\none). That tests will be executed as well as part of the periodic\r\npipeline using a real serverless project. QA environment is used to do\r\nso using the latest available commit in main at the time of the\r\nexecution.\r\n\r\n**Tests tagged as `@serverlessQA`**\r\n\r\nAll the tests tagged as `@serverlessQA` will be executed as part of the\r\nkibana release process using a real serverless project with the latest\r\nimage available in the QA environment\r\n\r\n**Tests tagged as `@ess`**\r\n\r\nAll the tests tagged as `@ess` will be executed as part of the PR\r\nvalidation process using an on-prem ESS environment.\r\n\r\n**Tests tagged as `@skipInServerless`**\r\n\r\nAll the tests tagged as `@skipInServerless` will be excluded from the PR\r\nvalidation process using the serverless FTR environment, the periodic\r\npipeline and kibana release process for Serverless.\r\n\r\n**Tests tagged as `@skipInEss`**\r\n\r\nAll the tests tagged as `skipInEss` will be excluded from the PR\r\nvalidation process using an on-prem ESS environment.\r\n\r\n---------\r\n\r\nCo-authored-by: Ryland Herrick <[email protected]>","sha":"c351e14e83552262a25488104d207ce70e1d0b47","branchLabelMapping":{"^v8.15.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","Team: SecuritySolution","v8.14.0","v8.15.0"],"title":"[Security Solution] Adds serverlessQA tag to the API tests","number":180773,"url":"https://github.com/elastic/kibana/pull/180773","mergeCommit":{"message":"[Security Solution] Adds serverlessQA tag to the API tests (#180773)\n\nContinues #179737 effort we are\r\naligning the tags on Cypress and API to have a unified experience.\r\n\r\n## Summary\r\n\r\nWe want to start integrating our Cypress tests with the serverless\r\nKibana quality gate. However, not all the teams feel comfortable\r\nenabling all the tests, to facilitate the effort of enabling tests in\r\nthe quality gate we are adding the `@serverlessQA` tag, now on API tests\r\nas well.\r\n\r\nWe use tags to select which tests we want to execute on each environment\r\nand pipeline.\r\n\r\n`ess` - runs in ESS env\r\n`serverless` - runs in serverless env and periodic pipeline (failures\r\ndon't block release)\r\n`serverlessQA` - runs in kibana release process (failures block release)\r\n`skipInEss` - skipped for ESS env\r\n`skipInServerless` - skipped for all serverless related environments\r\n`skipInServerlessMKI` - skipped for MKI environments\r\n\r\n### Description\r\n\r\n**Tests tagged as `@serverless`**\r\n\r\nAll the tests tagged as `@serverless` will be executed as part of the PR\r\nvalidation process using the serverless FTR environment (not a real\r\none). That tests will be executed as well as part of the periodic\r\npipeline using a real serverless project. QA environment is used to do\r\nso using the latest available commit in main at the time of the\r\nexecution.\r\n\r\n**Tests tagged as `@serverlessQA`**\r\n\r\nAll the tests tagged as `@serverlessQA` will be executed as part of the\r\nkibana release process using a real serverless project with the latest\r\nimage available in the QA environment\r\n\r\n**Tests tagged as `@ess`**\r\n\r\nAll the tests tagged as `@ess` will be executed as part of the PR\r\nvalidation process using an on-prem ESS environment.\r\n\r\n**Tests tagged as `@skipInServerless`**\r\n\r\nAll the tests tagged as `@skipInServerless` will be excluded from the PR\r\nvalidation process using the serverless FTR environment, the periodic\r\npipeline and kibana release process for Serverless.\r\n\r\n**Tests tagged as `@skipInEss`**\r\n\r\nAll the tests tagged as `skipInEss` will be excluded from the PR\r\nvalidation process using an on-prem ESS environment.\r\n\r\n---------\r\n\r\nCo-authored-by: Ryland Herrick <[email protected]>","sha":"c351e14e83552262a25488104d207ce70e1d0b47"}},"sourceBranch":"main","suggestedTargetBranches":["8.14"],"targetPullRequestStates":[{"branch":"8.14","label":"v8.14.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v8.15.0","branchLabelMappingKey":"^v8.15.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/180773","number":180773,"mergeCommit":{"message":"[Security Solution] Adds serverlessQA tag to the API tests (#180773)\n\nContinues #179737 effort we are\r\naligning the tags on Cypress and API to have a unified experience.\r\n\r\n## Summary\r\n\r\nWe want to start integrating our Cypress tests with the serverless\r\nKibana quality gate. However, not all the teams feel comfortable\r\nenabling all the tests, to facilitate the effort of enabling tests in\r\nthe quality gate we are adding the `@serverlessQA` tag, now on API tests\r\nas well.\r\n\r\nWe use tags to select which tests we want to execute on each environment\r\nand pipeline.\r\n\r\n`ess` - runs in ESS env\r\n`serverless` - runs in serverless env and periodic pipeline (failures\r\ndon't block release)\r\n`serverlessQA` - runs in kibana release process (failures block release)\r\n`skipInEss` - skipped for ESS env\r\n`skipInServerless` - skipped for all serverless related environments\r\n`skipInServerlessMKI` - skipped for MKI environments\r\n\r\n### Description\r\n\r\n**Tests tagged as `@serverless`**\r\n\r\nAll the tests tagged as `@serverless` will be executed as part of the PR\r\nvalidation process using the serverless FTR environment (not a real\r\none). That tests will be executed as well as part of the periodic\r\npipeline using a real serverless project. QA environment is used to do\r\nso using the latest available commit in main at the time of the\r\nexecution.\r\n\r\n**Tests tagged as `@serverlessQA`**\r\n\r\nAll the tests tagged as `@serverlessQA` will be executed as part of the\r\nkibana release process using a real serverless project with the latest\r\nimage available in the QA environment\r\n\r\n**Tests tagged as `@ess`**\r\n\r\nAll the tests tagged as `@ess` will be executed as part of the PR\r\nvalidation process using an on-prem ESS environment.\r\n\r\n**Tests tagged as `@skipInServerless`**\r\n\r\nAll the tests tagged as `@skipInServerless` will be excluded from the PR\r\nvalidation process using the serverless FTR environment, the periodic\r\npipeline and kibana release process for Serverless.\r\n\r\n**Tests tagged as `@skipInEss`**\r\n\r\nAll the tests tagged as `skipInEss` will be excluded from the PR\r\nvalidation process using an on-prem ESS environment.\r\n\r\n---------\r\n\r\nCo-authored-by: Ryland Herrick <[email protected]>","sha":"c351e14e83552262a25488104d207ce70e1d0b47"}}]}] BACKPORT--> Co-authored-by: Gloria Hornero <[email protected]>
Following up from #180773 ## Summary In this PR we are introducing the `@skipServerlessMKI` label, with it, a test will be excluded from the execution on any MKI environment but it will be executed as part of the CI check if the `@serverless` tag is present. With the new changes the serverless labels will work as follows: * `@serverless`: The test is executed as part of the PR check process and in the periodic pipeline (MKI environment). * `@serverlessQA`: The test is executed as part of the kibana release process in the QA environment (MKI environment). * `@skipInServerless`: The test is skipped from being executed in CI as part of the PR check and is skipped from being executed in any MKI environment. `@skipInServerlessMKI`: The test is skipped from being executed in any MKI environment but it will continue being executed as part of the PR process if the `@serverless` tag is present. **IMPORTANT:** The skip labels have been created for those tests that use `@serverless` or `@serverlessQA` labels. The absence of them (`@serverless` or `@serverlessQA`) will exclude automatically the execution of the test in the targeted environments. I.E: A test without `@serverlessQA` will never be executed as part of the Kibana release process. A test without `@serverless` will never be executed as part of the PR CI check neither the periodic pipeline.
Following up from elastic#180773 ## Summary In this PR we are introducing the `@skipServerlessMKI` label, with it, a test will be excluded from the execution on any MKI environment but it will be executed as part of the CI check if the `@serverless` tag is present. With the new changes the serverless labels will work as follows: * `@serverless`: The test is executed as part of the PR check process and in the periodic pipeline (MKI environment). * `@serverlessQA`: The test is executed as part of the kibana release process in the QA environment (MKI environment). * `@skipInServerless`: The test is skipped from being executed in CI as part of the PR check and is skipped from being executed in any MKI environment. `@skipInServerlessMKI`: The test is skipped from being executed in any MKI environment but it will continue being executed as part of the PR process if the `@serverless` tag is present. **IMPORTANT:** The skip labels have been created for those tests that use `@serverless` or `@serverlessQA` labels. The absence of them (`@serverless` or `@serverlessQA`) will exclude automatically the execution of the test in the targeted environments. I.E: A test without `@serverlessQA` will never be executed as part of the Kibana release process. A test without `@serverless` will never be executed as part of the PR CI check neither the periodic pipeline. (cherry picked from commit 37d47da)
# Backport This will backport the following commits from `main` to `8.14`: - [[Security Solution] Labels adjustments (#181371)](#181371) <!--- Backport version: 9.4.3 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Gloria Hornero","email":"[email protected]"},"sourceCommit":{"committedDate":"2024-04-24T08:48:30Z","message":"[Security Solution] Labels adjustments (#181371)\n\nFollowing up from https://github.com/elastic/kibana/pull/180773\r\n\r\n## Summary\r\n \r\nIn this PR we are introducing the `@skipServerlessMKI` label, with it, a\r\ntest will be excluded from the execution on any MKI environment but it\r\nwill be executed as part of the CI check if the `@serverless` tag is\r\npresent.\r\n \r\n With the new changes the serverless labels will work as follows:\r\n \r\n* `@serverless`: The test is executed as part of the PR check process\r\nand in the periodic pipeline (MKI environment).\r\n \r\n* `@serverlessQA`: The test is executed as part of the kibana release\r\nprocess in the QA environment (MKI environment).\r\n \r\n* `@skipInServerless`: The test is skipped from being executed in CI as\r\npart of the PR check and is skipped from being executed in any MKI\r\nenvironment.\r\n \r\n`@skipInServerlessMKI`: The test is skipped from being executed in any\r\nMKI environment but it will continue being executed as part of the PR\r\nprocess if the `@serverless` tag is present.\r\n \r\n **IMPORTANT:**\r\n \r\nThe skip labels have been created for those tests that use `@serverless`\r\nor `@serverlessQA` labels. The absence of them (`@serverless` or\r\n`@serverlessQA`) will exclude automatically the execution of the test in\r\nthe targeted environments.\r\n \r\nI.E: A test without `@serverlessQA` will never be executed as part of\r\nthe Kibana release process. A test without `@serverless` will never be\r\nexecuted as part of the PR CI check neither the periodic pipeline.","sha":"37d47da7711710ded8d30f0514f4da2e1810dc8f","branchLabelMapping":{"^v8.15.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","Team: SecuritySolution","v8.14.0","v8.15.0"],"title":"[Security Solution] Labels adjustments","number":181371,"url":"https://github.com/elastic/kibana/pull/181371","mergeCommit":{"message":"[Security Solution] Labels adjustments (#181371)\n\nFollowing up from https://github.com/elastic/kibana/pull/180773\r\n\r\n## Summary\r\n \r\nIn this PR we are introducing the `@skipServerlessMKI` label, with it, a\r\ntest will be excluded from the execution on any MKI environment but it\r\nwill be executed as part of the CI check if the `@serverless` tag is\r\npresent.\r\n \r\n With the new changes the serverless labels will work as follows:\r\n \r\n* `@serverless`: The test is executed as part of the PR check process\r\nand in the periodic pipeline (MKI environment).\r\n \r\n* `@serverlessQA`: The test is executed as part of the kibana release\r\nprocess in the QA environment (MKI environment).\r\n \r\n* `@skipInServerless`: The test is skipped from being executed in CI as\r\npart of the PR check and is skipped from being executed in any MKI\r\nenvironment.\r\n \r\n`@skipInServerlessMKI`: The test is skipped from being executed in any\r\nMKI environment but it will continue being executed as part of the PR\r\nprocess if the `@serverless` tag is present.\r\n \r\n **IMPORTANT:**\r\n \r\nThe skip labels have been created for those tests that use `@serverless`\r\nor `@serverlessQA` labels. The absence of them (`@serverless` or\r\n`@serverlessQA`) will exclude automatically the execution of the test in\r\nthe targeted environments.\r\n \r\nI.E: A test without `@serverlessQA` will never be executed as part of\r\nthe Kibana release process. A test without `@serverless` will never be\r\nexecuted as part of the PR CI check neither the periodic pipeline.","sha":"37d47da7711710ded8d30f0514f4da2e1810dc8f"}},"sourceBranch":"main","suggestedTargetBranches":["8.14"],"targetPullRequestStates":[{"branch":"8.14","label":"v8.14.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v8.15.0","branchLabelMappingKey":"^v8.15.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/181371","number":181371,"mergeCommit":{"message":"[Security Solution] Labels adjustments (#181371)\n\nFollowing up from https://github.com/elastic/kibana/pull/180773\r\n\r\n## Summary\r\n \r\nIn this PR we are introducing the `@skipServerlessMKI` label, with it, a\r\ntest will be excluded from the execution on any MKI environment but it\r\nwill be executed as part of the CI check if the `@serverless` tag is\r\npresent.\r\n \r\n With the new changes the serverless labels will work as follows:\r\n \r\n* `@serverless`: The test is executed as part of the PR check process\r\nand in the periodic pipeline (MKI environment).\r\n \r\n* `@serverlessQA`: The test is executed as part of the kibana release\r\nprocess in the QA environment (MKI environment).\r\n \r\n* `@skipInServerless`: The test is skipped from being executed in CI as\r\npart of the PR check and is skipped from being executed in any MKI\r\nenvironment.\r\n \r\n`@skipInServerlessMKI`: The test is skipped from being executed in any\r\nMKI environment but it will continue being executed as part of the PR\r\nprocess if the `@serverless` tag is present.\r\n \r\n **IMPORTANT:**\r\n \r\nThe skip labels have been created for those tests that use `@serverless`\r\nor `@serverlessQA` labels. The absence of them (`@serverless` or\r\n`@serverlessQA`) will exclude automatically the execution of the test in\r\nthe targeted environments.\r\n \r\nI.E: A test without `@serverlessQA` will never be executed as part of\r\nthe Kibana release process. A test without `@serverless` will never be\r\nexecuted as part of the PR CI check neither the periodic pipeline.","sha":"37d47da7711710ded8d30f0514f4da2e1810dc8f"}}]}] BACKPORT--> Co-authored-by: Gloria Hornero <[email protected]>
Continues #179737 effort we are aligning the tags on Cypress and API to have a unified experience.
Summary
We want to start integrating our Cypress tests with the serverless Kibana quality gate. However, not all the teams feel comfortable enabling all the tests, to facilitate the effort of enabling tests in the quality gate we are adding the
@serverlessQA
tag, now on API tests as well.We use tags to select which tests we want to execute on each environment and pipeline.
ess
- runs in ESS envserverless
- runs in serverless env and periodic pipeline (failures don't block release)serverlessQA
- runs in kibana release process (failures block release)skipInEss
- skipped for ESS envskipInServerless
- skipped for all serverless related environmentsskipInServerlessMKI
- skipped for MKI environmentsDescription
Tests tagged as
@serverless
All the tests tagged as
@serverless
will be executed as part of the PR validation process using the serverless FTR environment (not a real one). That tests will be executed as well as part of the periodic pipeline using a real serverless project. QA environment is used to do so using the latest available commit in main at the time of the execution.Tests tagged as
@serverlessQA
All the tests tagged as
@serverlessQA
will be executed as part of the kibana release process using a real serverless project with the latest image available in the QA environmentTests tagged as
@ess
All the tests tagged as
@ess
will be executed as part of the PR validation process using an on-prem ESS environment.Tests tagged as
@skipInServerless
All the tests tagged as
@skipInServerless
will be excluded from the PR validation process using the serverless FTR environment, the periodic pipeline and kibana release process for Serverless.Tests tagged as
@skipInEss
All the tests tagged as
skipInEss
will be excluded from the PR validation process using an on-prem ESS environment.