Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Security Solution] Adds serverlessQA tag to the API tests #180773

Merged
merged 19 commits into from
Apr 23, 2024

Conversation

MadameSheema
Copy link
Member

@MadameSheema MadameSheema commented Apr 15, 2024

Continues #179737 effort we are aligning the tags on Cypress and API to have a unified experience.

Summary

We want to start integrating our Cypress tests with the serverless Kibana quality gate. However, not all the teams feel comfortable enabling all the tests, to facilitate the effort of enabling tests in the quality gate we are adding the @serverlessQA tag, now on API tests as well.

We use tags to select which tests we want to execute on each environment and pipeline.

ess - runs in ESS env
serverless - runs in serverless env and periodic pipeline (failures don't block release)
serverlessQA - runs in kibana release process (failures block release)
skipInEss - skipped for ESS env
skipInServerless - skipped for all serverless related environments
skipInServerlessMKI - skipped for MKI environments

Description

Tests tagged as @serverless

All the tests tagged as @serverless will be executed as part of the PR validation process using the serverless FTR environment (not a real one). That tests will be executed as well as part of the periodic pipeline using a real serverless project. QA environment is used to do so using the latest available commit in main at the time of the execution.

Tests tagged as @serverlessQA

All the tests tagged as @serverlessQA will be executed as part of the kibana release process using a real serverless project with the latest image available in the QA environment

Tests tagged as @ess

All the tests tagged as @ess will be executed as part of the PR validation process using an on-prem ESS environment.

Tests tagged as @skipInServerless

All the tests tagged as @skipInServerless will be excluded from the PR validation process using the serverless FTR environment, the periodic pipeline and kibana release process for Serverless.

Tests tagged as @skipInEss

All the tests tagged as skipInEss will be excluded from the PR validation process using an on-prem ESS environment.

@MadameSheema
Copy link
Member Author

/ci

@MadameSheema MadameSheema added release_note:skip Skip the PR/issue when compiling release notes Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. v8.14.0 labels Apr 15, 2024
@MadameSheema MadameSheema self-assigned this Apr 15, 2024
@MadameSheema MadameSheema marked this pull request as ready for review April 15, 2024 10:22
@MadameSheema MadameSheema requested review from a team as code owners April 15, 2024 10:22
@elasticmachine
Copy link
Contributor

Pinging @elastic/security-solution (Team: SecuritySolution)

Copy link
Contributor

@hop-dev hop-dev left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Entity analytics changes LGTM 🚀

@MadameSheema
Copy link
Member Author

/ci

Copy link
Member

@spong spong left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Security GenAI tag updates LGTM! 👍

Thanks for the clear communication and update here @MadameSheema!

@MadameSheema
Copy link
Member Author

/ci

@MadameSheema
Copy link
Member Author

/ci

@MadameSheema
Copy link
Member Author

/ci

Copy link
Contributor

@rylnd rylnd left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had a few nits about definitions/discoverability, and it looks like a few of the build commands are missing their kibana commands (which needs to be fixed), but otherwise this LGTM.

limit: 2

- label: Running exception_operators_date_numeric_types:qa:serverless:release
command: .buildkite/scripts/pipelines/security_solution_quality_gate/api_integration/api-integration-tests.sh exception_operators_date_numeric_types:qa:serverless:release
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How does one trace this command to the tests/code that it executes?

The key exception_operators_date_numeric_types:qa:serverless:release is the only argument to the test script, but that key only exists in this file. The test script seems to just call yarn run [key], but I don't see how yarn run exception_operators_date_numeric_types:qa:serverless:release works if that command isn't defined somewhere.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Update: I see that I chose a key that was (accidentally?) omitted from x-pack/test/security_solution_api_integration/package.json.

That answers my question about discoverability, but means that some of these are going to fail until the corresponding commands are defined.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like either a wrong key usage or a lack of update in the branch with the latest code. @MadameSheema

- We are using the following tags:
* `@ess`: Runs in an ESS environment (on-prem installation) as part of the CI validation on PRs.

* `@serverless`: Runs in the first quality gate and in the periodic pipeline.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The only other mention of "first quality gate" is in the cypress README; I think it's worth defining these terms in some detail since they're not going to be self-evident to everyone.

Same thing for "periodic pipeline," "second quality gate," and "all quality gates."

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey @rylnd everything is already documented.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@MadameSheema it's wonderful that it's documented, but how does someone reading this README find those definitions? You shared links to quality gates and periodic pipeline with me individually, but that process does not scale, so I'm proposing that we somehow reference that documentation here.

If we are reluctant/unable to share those docs because they're internal, I think that might be indication that this README either covers too much or is in the wrong place; our public documentation shouldn't reference internal processes that the reader has no insight on.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

++ to @rylnd's comment

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yup, share it with you in private due to being internal documentation, tbh I don't know what is best, thoughts? What do you prefer?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the ideal approach in my mind would be:

  1. Internal documentation associating the cypress tags to our CI/QA infrastructure/processes
  2. Public documentation with definitions appropriate to those not involved in our operations

I'm imagining a contributor who's written a cypress test for a feature they'd like to add; I would not expect them to know/care about any of these tags. As part of the review process, an Elastician would tell them which tags need to be added to their tests.

Generally: if we can't provide the reader with enough information to determine whether they should use the tag, I don't think that it's helpful to mention it at all.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe a passing mention of the tags as "related to internal release processes" would be a way to both define the tags but also deemphasize them, here?


* `@serverless`: Runs in the first quality gate and in the periodic pipeline.

* `@serverlessQA`: Runs in the second quality gate.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This tag seems to be strongly tied to the concept of a "release" (as evidenced in the command names, and this PR's description), but that isn't really reflected in either the README here nor in the tag name itself.

Would @serverlessRelease be a more accurate and descriptive tag name?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As part of the release, the tests might be executed as part of 2 other quality gates and we might have some on the second one that we don't want to execute it on the third one.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IMO referencing the "gate order" might be the most straightforward way to express this; it's not intuitive that QA represents "second gate," nor what the other gates are.

@@ -31,7 +31,7 @@ export default ({ getService }: FtrProviderContext): void => {
/* from a package that was bundled with Kibana */
//
// FLAKY: https://github.com/elastic/kibana/issues/180087
describe.skip('@ess @serverless @skipInQA install_bundled_prebuilt_rules', () => {
describe.skip('@ess @serverless @skipInServerless install_bundled_prebuilt_rules', () => {
Copy link
Contributor

@banderror banderror Apr 18, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@MadameSheema We don't want all the tests for prebuilt rules to be skipped in all quality gates. We need them to run on CI, where, as far as I'm aware, they are not currently skipped and have been passing.

This also applies to some of the other rule management tests that are being skipped in CI by this PR.

I would suggest to:

  1. Either introduce an additional @skipInServerlessMKI tag and use it for these tests instead of @skipInServerless. This tag would skip a test in the periodic pipeline + 2nd quality gate + 3rd and 4th quality gates in the future, but keep the test running on CI in the Serverless mode.
  2. Or to rename the @serverless tag to @serverlessCI and introduce:
    • an additional tag @serverlessRelease that would add a test to all the release pipelines: the 2nd quality gate + 3rd and 4th quality gates in the future
    • an additional tag @serverlessPeriodic that would add a test only to the periodic pipeline

I'd personally choose the 2nd option as being more explicit about where a test should run.

Sorry if I missed this in #179737 where Cypress tests for prebuilt rules and rule management have been skipped in Serverless CI. We'd need to run those Cypress tests in CI too, so I'd propose the same changes for Cypress tests.

cc @jpdjere @maximpn @xcrzx

Copy link
Member Author

@MadameSheema MadameSheema Apr 18, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@maximpn @yctercero since you were the ones against having extra labels, can you please arrive to an agreement about how to proceed with it?

This is my proposal:

@serverless -> Executes tests on CI and periodic pipeline (MKI)


@serverlessQA -> Executes tests on the second quality gate (QA environment)

@skipServerless-> Does not execute the test in any serverless environment

@skipServerlessMKI -> Does not execute the test in any MKI environment (periodic pipeline and quality gate)



If you don’t want a test to be executed as part of the second quality gate you don’t add the @serverlessQA tag to it.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@MadameSheema do I get it correctly a test which should be skipped in MKI env but run everywhere else will have the following tags

describe('@ess @serverless @serverlessQA @skipServerlessMKI install_bundled_prebuilt_rules', () => {

Here we return to the situation when it equals to

describe('@ess @serverless install_bundled_prebuilt_rules', () => {

Here we have two options

  • Have only @ess, @skipInEss, @serverless, @serverlessQA and @skipInServerless tags. When we need to run a test in another MKI end we would add another @serverless<name> tag. By default we just don't add tags so the test isn't enabled automatically for any MKI environment. There is an assumption MKI env is less stable than mocked Serverless env. It's impossible to disabled a test for only mocked Serverless env but still run it against some MKI env. But it doesn't look like an issue.
  • Have only @ess, @skipInEss, @serverless, @skipInServerlessMock (what's the best name for local/CI Serverless instead of serverless mock?) and @skipInServerlessQA tags. In this case tests tagged by @serverless will run against everything (besides ESS) and adding a corresponding @skipInServerless<name> tag skips a test in a proper env.

The second approach looks better for automatic skipping script. When a test fails in some env or becomes flaky it could be skipped automatically.

I checked the PR's diff once more time and see that it's more convenient to use the second approach. Quite often describe section has permissive tags @ess and @serverless while individual tests have may be skipped via @skipInServerless. It will require manually skip all the tests in MKI env via @skipInServerlessQA but it should be simpler to maintain the tests.

Copy link
Contributor

@banderror banderror Apr 18, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@maximpn

Have only @ess, @skipInEss, @serverless, @serverlessQA and @skipInServerless tags.

This set of tags doesn't allow to run a test on CI in ESS + on CI in Serverless, but not run it in the periodic pipeline + release gates.

At this point, we need to be able to run many of our tests ONLY on CI in ESS + on CI in Serverless, because they are broken in MKI environments. Fixing this requires time and effort we don't have at the moment.

Have only @ess, @skipInEss, @serverless, @skipInServerlessMock (what's the best name for local/CI Serverless instead of serverless mock?) and @skipInServerlessQA tags

This set of tags also doesn't allow to run a test on CI in ESS + on CI in Serverless, but not run it in the periodic pipeline + release gates.

We don't need to be able to skip a test on CI in Serverless, but run it in MKI pipelines. We don't have such class of tests as of now.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@banderror @MadameSheema

Could you clarify the requirements to running tests in periodic pipeline? Do we need a separate way to skip tests for periodic pipeline?

In my understanding if a test is broken in MKI env it will be broken in periodic pipeline since it's an MKI env as well. Taking this into account a test shouldn't have @serverlessQAtag (option 1) or have @skipInServerlessQA tag (option 2). It won't run against any MKI env until it's fixed, tested manually and tag manipulation is performed to run the test against MKI envs.

Copy link
Member Author

@MadameSheema MadameSheema Apr 18, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@banderror @MadameSheema

Could you clarify the requirements to running tests in periodic pipeline? Do we need a separate way to skip tests for periodic pipeline?

In my understanding if a test is broken in MKI env it will be broken in periodic pipeline since it's an MKI env as well. Taking this into account a test shouldn't have @serverlessQAtag (option 1) or have @skipInServerlessQA tag (option 2). It won't run against any MKI env until it's fixed, tested manually and tag manipulation is performed to run the test against MKI envs.

As there are differences between the serverless environment we use on CI and a real one but at the same time, we don't want to execute all the tests in the quality gate, is a way for us to determine the health of our application in a real serverless project environment.

We need a separate way of skipping tests from MKI but not CI.

A test with just @serverless is going to be executed in CI and periodic pipeline. If you add to the test @serverlessQA that test will be executed in the second quality gate as part of the release process. If a test is broken on MKI is going to fail in both on periodic and quality gate, so you can skip it with @serverlessMKI but it will still be executed on CI as part of the PRs so I strongly believe that right now what it makes sense for us to have as labels is:

@ess: Test is executed on CI ess environment
@skipEss: Test is excluded fom the CI ess environment execution
@serverless: Test is executed on CI serverless and periodic pipeline (MKI environment)
@serverlessQA: Test is executed as part of the release process on the second quality gate which is the QA environment (MKI environment)
@skipServerlessMKI: Test is excluded from being executed in any MKI environment, meaning, is not executed as part of the periodic pipeline and is not executed as part of the release process
@skipServerless: Tests is excluded from being executed as part from the CI and is not executed in any MKI environment

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@MadameSheema Your proposal works for me 👍

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Above proposal looks great to me @MadameSheema

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@MadameSheema

Me and @banderror discussed the approach on a zoom call. Tags you suggest in the last comment look good to me. I have some comments though

ESS tags are pretty clear. We need to run tests against ESS env and skip broken tests.

  • @ess
  • @skipEss

Serverless in general is pretty clear as well. We need to run test against Serverless env (abstractly it doesn't matter which one) and skip broken tests

  • @serverless
  • @skipServerless

If we don't need to run tests against some env an appropriate tag @ess or @serverless isn't added.

The most interesting part start from the fact that Serverless envs are different. Not all tests work in Serverless MKI env currently. Taking into account this fact we need an ability to skip such tests so we can use the following tag

  • @skipServerlessMKI

And the last part is that we need some test to block Serverless release when critical functionality doesn't work. In this case something explicit like @blockServerlessRelease or @serverlessRelease would work the best

  • @serverlessRelease

I have a question about your suggestion. Why do you want to use explicit @serverlessQA. Is it expected we are gonna run different test sets/subsets in QA, Staging etc Serverless MKI envs? It looks like if a test work in one Serverless MKI env it should work in the other one as well. Is there some difference between QA and Staging?

cc @rylnd

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe at this point it's best to jump on a Zoom. I'm a bit confused as to what's being asked here anymore.

@MadameSheema worked on this PR after we chatted and decided it'd be great for our API and cypress tags to match so this PR brought them into parity. We had discussed and decided on the tags during the first PR that changed the Cypress tags. It would be great to finalize the discussion so we don't keep rehashing the discussion in each PR.

Copy link
Contributor

@michaelolo24 michaelolo24 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree with your notes here: #180773 (comment) Thank you for making these changes @MadameSheema !

@MadameSheema
Copy link
Member Author

/ci

@kibana-ci
Copy link
Collaborator

💛 Build succeeded, but was flaky

Failed CI Steps

Metrics [docs]

✅ unchanged

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @MadameSheema

Copy link
Contributor

@banderror banderror left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for addressing my concern @MadameSheema, LGTM 👍

Could we please have the same set of tags for Cypress tests?

- We are using the following tags:
* `@ess`: Runs in an ESS environment (on-prem installation) as part of the CI validation on PRs.

* `@serverless`: Runs in the first quality gate and in the periodic pipeline.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

++ to @rylnd's comment

@MadameSheema MadameSheema merged commit c351e14 into elastic:main Apr 23, 2024
20 checks passed
@MadameSheema MadameSheema deleted the api/serverlessQA branch April 23, 2024 16:21
kibanamachine pushed a commit to kibanamachine/kibana that referenced this pull request Apr 23, 2024
…80773)

Continues elastic#179737 effort we are
aligning the tags on Cypress and API to have a unified experience.

## Summary

We want to start integrating our Cypress tests with the serverless
Kibana quality gate. However, not all the teams feel comfortable
enabling all the tests, to facilitate the effort of enabling tests in
the quality gate we are adding the `@serverlessQA` tag, now on API tests
as well.

We use tags to select which tests we want to execute on each environment
and pipeline.

`ess` - runs in ESS env
`serverless` - runs in serverless env and periodic pipeline (failures
don't block release)
`serverlessQA` - runs in kibana release process (failures block release)
`skipInEss` - skipped for ESS env
`skipInServerless` - skipped for all serverless related environments
`skipInServerlessMKI` - skipped for MKI environments

### Description

**Tests tagged as `@serverless`**

All the tests tagged as `@serverless` will be executed as part of the PR
validation process using the serverless FTR environment (not a real
one). That tests will be executed as well as part of the periodic
pipeline using a real serverless project. QA environment is used to do
so using the latest available commit in main at the time of the
execution.

**Tests tagged as `@serverlessQA`**

All the tests tagged as `@serverlessQA` will be executed as part of the
kibana release process using a real serverless project with the latest
image available in the QA environment

**Tests tagged as `@ess`**

All the tests tagged as `@ess` will be executed as part of the PR
validation process using an on-prem ESS environment.

**Tests tagged as `@skipInServerless`**

All the tests tagged as `@skipInServerless` will be excluded from the PR
validation process using the serverless FTR environment, the periodic
pipeline and kibana release process for Serverless.

**Tests tagged as `@skipInEss`**

All the tests tagged as `skipInEss` will be excluded from the PR
validation process using an on-prem ESS environment.

---------

Co-authored-by: Ryland Herrick <[email protected]>
(cherry picked from commit c351e14)
@kibanamachine
Copy link
Contributor

💚 All backports created successfully

Status Branch Result
8.14

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

kibanamachine added a commit that referenced this pull request Apr 23, 2024
…80773) (#181468)

# Backport

This will backport the following commits from `main` to `8.14`:
- [[Security Solution] Adds serverlessQA tag to the API tests
(#180773)](#180773)

<!--- Backport version: 9.4.3 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Gloria
Hornero","email":"[email protected]"},"sourceCommit":{"committedDate":"2024-04-23T16:21:13Z","message":"[Security
Solution] Adds serverlessQA tag to the API tests (#180773)\n\nContinues
#179737 effort we are\r\naligning
the tags on Cypress and API to have a unified experience.\r\n\r\n##
Summary\r\n\r\nWe want to start integrating our Cypress tests with the
serverless\r\nKibana quality gate. However, not all the teams feel
comfortable\r\nenabling all the tests, to facilitate the effort of
enabling tests in\r\nthe quality gate we are adding the `@serverlessQA`
tag, now on API tests\r\nas well.\r\n\r\nWe use tags to select which
tests we want to execute on each environment\r\nand
pipeline.\r\n\r\n`ess` - runs in ESS env\r\n`serverless` - runs in
serverless env and periodic pipeline (failures\r\ndon't block
release)\r\n`serverlessQA` - runs in kibana release process (failures
block release)\r\n`skipInEss` - skipped for ESS
env\r\n`skipInServerless` - skipped for all serverless related
environments\r\n`skipInServerlessMKI` - skipped for MKI
environments\r\n\r\n### Description\r\n\r\n**Tests tagged as
`@serverless`**\r\n\r\nAll the tests tagged as `@serverless` will be
executed as part of the PR\r\nvalidation process using the serverless
FTR environment (not a real\r\none). That tests will be executed as well
as part of the periodic\r\npipeline using a real serverless project. QA
environment is used to do\r\nso using the latest available commit in
main at the time of the\r\nexecution.\r\n\r\n**Tests tagged as
`@serverlessQA`**\r\n\r\nAll the tests tagged as `@serverlessQA` will be
executed as part of the\r\nkibana release process using a real
serverless project with the latest\r\nimage available in the QA
environment\r\n\r\n**Tests tagged as `@ess`**\r\n\r\nAll the tests
tagged as `@ess` will be executed as part of the PR\r\nvalidation
process using an on-prem ESS environment.\r\n\r\n**Tests tagged as
`@skipInServerless`**\r\n\r\nAll the tests tagged as `@skipInServerless`
will be excluded from the PR\r\nvalidation process using the serverless
FTR environment, the periodic\r\npipeline and kibana release process for
Serverless.\r\n\r\n**Tests tagged as `@skipInEss`**\r\n\r\nAll the tests
tagged as `skipInEss` will be excluded from the PR\r\nvalidation process
using an on-prem ESS
environment.\r\n\r\n---------\r\n\r\nCo-authored-by: Ryland Herrick
<[email protected]>","sha":"c351e14e83552262a25488104d207ce70e1d0b47","branchLabelMapping":{"^v8.15.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","Team:
SecuritySolution","v8.14.0","v8.15.0"],"title":"[Security Solution] Adds
serverlessQA tag to the API
tests","number":180773,"url":"https://github.com/elastic/kibana/pull/180773","mergeCommit":{"message":"[Security
Solution] Adds serverlessQA tag to the API tests (#180773)\n\nContinues
#179737 effort we are\r\naligning
the tags on Cypress and API to have a unified experience.\r\n\r\n##
Summary\r\n\r\nWe want to start integrating our Cypress tests with the
serverless\r\nKibana quality gate. However, not all the teams feel
comfortable\r\nenabling all the tests, to facilitate the effort of
enabling tests in\r\nthe quality gate we are adding the `@serverlessQA`
tag, now on API tests\r\nas well.\r\n\r\nWe use tags to select which
tests we want to execute on each environment\r\nand
pipeline.\r\n\r\n`ess` - runs in ESS env\r\n`serverless` - runs in
serverless env and periodic pipeline (failures\r\ndon't block
release)\r\n`serverlessQA` - runs in kibana release process (failures
block release)\r\n`skipInEss` - skipped for ESS
env\r\n`skipInServerless` - skipped for all serverless related
environments\r\n`skipInServerlessMKI` - skipped for MKI
environments\r\n\r\n### Description\r\n\r\n**Tests tagged as
`@serverless`**\r\n\r\nAll the tests tagged as `@serverless` will be
executed as part of the PR\r\nvalidation process using the serverless
FTR environment (not a real\r\none). That tests will be executed as well
as part of the periodic\r\npipeline using a real serverless project. QA
environment is used to do\r\nso using the latest available commit in
main at the time of the\r\nexecution.\r\n\r\n**Tests tagged as
`@serverlessQA`**\r\n\r\nAll the tests tagged as `@serverlessQA` will be
executed as part of the\r\nkibana release process using a real
serverless project with the latest\r\nimage available in the QA
environment\r\n\r\n**Tests tagged as `@ess`**\r\n\r\nAll the tests
tagged as `@ess` will be executed as part of the PR\r\nvalidation
process using an on-prem ESS environment.\r\n\r\n**Tests tagged as
`@skipInServerless`**\r\n\r\nAll the tests tagged as `@skipInServerless`
will be excluded from the PR\r\nvalidation process using the serverless
FTR environment, the periodic\r\npipeline and kibana release process for
Serverless.\r\n\r\n**Tests tagged as `@skipInEss`**\r\n\r\nAll the tests
tagged as `skipInEss` will be excluded from the PR\r\nvalidation process
using an on-prem ESS
environment.\r\n\r\n---------\r\n\r\nCo-authored-by: Ryland Herrick
<[email protected]>","sha":"c351e14e83552262a25488104d207ce70e1d0b47"}},"sourceBranch":"main","suggestedTargetBranches":["8.14"],"targetPullRequestStates":[{"branch":"8.14","label":"v8.14.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v8.15.0","branchLabelMappingKey":"^v8.15.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/180773","number":180773,"mergeCommit":{"message":"[Security
Solution] Adds serverlessQA tag to the API tests (#180773)\n\nContinues
#179737 effort we are\r\naligning
the tags on Cypress and API to have a unified experience.\r\n\r\n##
Summary\r\n\r\nWe want to start integrating our Cypress tests with the
serverless\r\nKibana quality gate. However, not all the teams feel
comfortable\r\nenabling all the tests, to facilitate the effort of
enabling tests in\r\nthe quality gate we are adding the `@serverlessQA`
tag, now on API tests\r\nas well.\r\n\r\nWe use tags to select which
tests we want to execute on each environment\r\nand
pipeline.\r\n\r\n`ess` - runs in ESS env\r\n`serverless` - runs in
serverless env and periodic pipeline (failures\r\ndon't block
release)\r\n`serverlessQA` - runs in kibana release process (failures
block release)\r\n`skipInEss` - skipped for ESS
env\r\n`skipInServerless` - skipped for all serverless related
environments\r\n`skipInServerlessMKI` - skipped for MKI
environments\r\n\r\n### Description\r\n\r\n**Tests tagged as
`@serverless`**\r\n\r\nAll the tests tagged as `@serverless` will be
executed as part of the PR\r\nvalidation process using the serverless
FTR environment (not a real\r\none). That tests will be executed as well
as part of the periodic\r\npipeline using a real serverless project. QA
environment is used to do\r\nso using the latest available commit in
main at the time of the\r\nexecution.\r\n\r\n**Tests tagged as
`@serverlessQA`**\r\n\r\nAll the tests tagged as `@serverlessQA` will be
executed as part of the\r\nkibana release process using a real
serverless project with the latest\r\nimage available in the QA
environment\r\n\r\n**Tests tagged as `@ess`**\r\n\r\nAll the tests
tagged as `@ess` will be executed as part of the PR\r\nvalidation
process using an on-prem ESS environment.\r\n\r\n**Tests tagged as
`@skipInServerless`**\r\n\r\nAll the tests tagged as `@skipInServerless`
will be excluded from the PR\r\nvalidation process using the serverless
FTR environment, the periodic\r\npipeline and kibana release process for
Serverless.\r\n\r\n**Tests tagged as `@skipInEss`**\r\n\r\nAll the tests
tagged as `skipInEss` will be excluded from the PR\r\nvalidation process
using an on-prem ESS
environment.\r\n\r\n---------\r\n\r\nCo-authored-by: Ryland Herrick
<[email protected]>","sha":"c351e14e83552262a25488104d207ce70e1d0b47"}}]}]
BACKPORT-->

Co-authored-by: Gloria Hornero <[email protected]>
MadameSheema added a commit that referenced this pull request Apr 24, 2024
Following up from #180773

## Summary
 
In this PR we are introducing the `@skipServerlessMKI` label, with it, a
test will be excluded from the execution on any MKI environment but it
will be executed as part of the CI check if the `@serverless` tag is
present.
 
 With the new changes the serverless labels will work as follows:
 
* `@serverless`: The test is executed as part of the PR check process
and in the periodic pipeline (MKI environment).
 
* `@serverlessQA`: The test is executed as part of the kibana release
process in the QA environment (MKI environment).
 
* `@skipInServerless`: The test is skipped from being executed in CI as
part of the PR check and is skipped from being executed in any MKI
environment.
 
`@skipInServerlessMKI`: The test is skipped from being executed in any
MKI environment but it will continue being executed as part of the PR
process if the `@serverless` tag is present.
 
 **IMPORTANT:**
 
The skip labels have been created for those tests that use `@serverless`
or `@serverlessQA` labels. The absence of them (`@serverless` or
`@serverlessQA`) will exclude automatically the execution of the test in
the targeted environments.
 
I.E: A test without `@serverlessQA` will never be executed as part of
the Kibana release process. A test without `@serverless` will never be
executed as part of the PR CI check neither the periodic pipeline.
kibanamachine pushed a commit to kibanamachine/kibana that referenced this pull request Apr 24, 2024
Following up from elastic#180773

## Summary

In this PR we are introducing the `@skipServerlessMKI` label, with it, a
test will be excluded from the execution on any MKI environment but it
will be executed as part of the CI check if the `@serverless` tag is
present.

 With the new changes the serverless labels will work as follows:

* `@serverless`: The test is executed as part of the PR check process
and in the periodic pipeline (MKI environment).

* `@serverlessQA`: The test is executed as part of the kibana release
process in the QA environment (MKI environment).

* `@skipInServerless`: The test is skipped from being executed in CI as
part of the PR check and is skipped from being executed in any MKI
environment.

`@skipInServerlessMKI`: The test is skipped from being executed in any
MKI environment but it will continue being executed as part of the PR
process if the `@serverless` tag is present.

 **IMPORTANT:**

The skip labels have been created for those tests that use `@serverless`
or `@serverlessQA` labels. The absence of them (`@serverless` or
`@serverlessQA`) will exclude automatically the execution of the test in
the targeted environments.

I.E: A test without `@serverlessQA` will never be executed as part of
the Kibana release process. A test without `@serverless` will never be
executed as part of the PR CI check neither the periodic pipeline.

(cherry picked from commit 37d47da)
kibanamachine referenced this pull request Apr 24, 2024
# Backport

This will backport the following commits from `main` to `8.14`:
- [[Security Solution] Labels adjustments
(#181371)](#181371)

<!--- Backport version: 9.4.3 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Gloria
Hornero","email":"[email protected]"},"sourceCommit":{"committedDate":"2024-04-24T08:48:30Z","message":"[Security
Solution] Labels adjustments (#181371)\n\nFollowing up from
https://github.com/elastic/kibana/pull/180773\r\n\r\n## Summary\r\n
\r\nIn this PR we are introducing the `@skipServerlessMKI` label, with
it, a\r\ntest will be excluded from the execution on any MKI environment
but it\r\nwill be executed as part of the CI check if the `@serverless`
tag is\r\npresent.\r\n \r\n With the new changes the serverless labels
will work as follows:\r\n \r\n* `@serverless`: The test is executed as
part of the PR check process\r\nand in the periodic pipeline (MKI
environment).\r\n \r\n* `@serverlessQA`: The test is executed as part of
the kibana release\r\nprocess in the QA environment (MKI
environment).\r\n \r\n* `@skipInServerless`: The test is skipped from
being executed in CI as\r\npart of the PR check and is skipped from
being executed in any MKI\r\nenvironment.\r\n
\r\n`@skipInServerlessMKI`: The test is skipped from being executed in
any\r\nMKI environment but it will continue being executed as part of
the PR\r\nprocess if the `@serverless` tag is present.\r\n \r\n
**IMPORTANT:**\r\n \r\nThe skip labels have been created for those tests
that use `@serverless`\r\nor `@serverlessQA` labels. The absence of them
(`@serverless` or\r\n`@serverlessQA`) will exclude automatically the
execution of the test in\r\nthe targeted environments.\r\n \r\nI.E: A
test without `@serverlessQA` will never be executed as part of\r\nthe
Kibana release process. A test without `@serverless` will never
be\r\nexecuted as part of the PR CI check neither the periodic
pipeline.","sha":"37d47da7711710ded8d30f0514f4da2e1810dc8f","branchLabelMapping":{"^v8.15.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","Team:
SecuritySolution","v8.14.0","v8.15.0"],"title":"[Security Solution]
Labels
adjustments","number":181371,"url":"https://github.com/elastic/kibana/pull/181371","mergeCommit":{"message":"[Security
Solution] Labels adjustments (#181371)\n\nFollowing up from
https://github.com/elastic/kibana/pull/180773\r\n\r\n## Summary\r\n
\r\nIn this PR we are introducing the `@skipServerlessMKI` label, with
it, a\r\ntest will be excluded from the execution on any MKI environment
but it\r\nwill be executed as part of the CI check if the `@serverless`
tag is\r\npresent.\r\n \r\n With the new changes the serverless labels
will work as follows:\r\n \r\n* `@serverless`: The test is executed as
part of the PR check process\r\nand in the periodic pipeline (MKI
environment).\r\n \r\n* `@serverlessQA`: The test is executed as part of
the kibana release\r\nprocess in the QA environment (MKI
environment).\r\n \r\n* `@skipInServerless`: The test is skipped from
being executed in CI as\r\npart of the PR check and is skipped from
being executed in any MKI\r\nenvironment.\r\n
\r\n`@skipInServerlessMKI`: The test is skipped from being executed in
any\r\nMKI environment but it will continue being executed as part of
the PR\r\nprocess if the `@serverless` tag is present.\r\n \r\n
**IMPORTANT:**\r\n \r\nThe skip labels have been created for those tests
that use `@serverless`\r\nor `@serverlessQA` labels. The absence of them
(`@serverless` or\r\n`@serverlessQA`) will exclude automatically the
execution of the test in\r\nthe targeted environments.\r\n \r\nI.E: A
test without `@serverlessQA` will never be executed as part of\r\nthe
Kibana release process. A test without `@serverless` will never
be\r\nexecuted as part of the PR CI check neither the periodic
pipeline.","sha":"37d47da7711710ded8d30f0514f4da2e1810dc8f"}},"sourceBranch":"main","suggestedTargetBranches":["8.14"],"targetPullRequestStates":[{"branch":"8.14","label":"v8.14.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v8.15.0","branchLabelMappingKey":"^v8.15.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/181371","number":181371,"mergeCommit":{"message":"[Security
Solution] Labels adjustments (#181371)\n\nFollowing up from
https://github.com/elastic/kibana/pull/180773\r\n\r\n## Summary\r\n
\r\nIn this PR we are introducing the `@skipServerlessMKI` label, with
it, a\r\ntest will be excluded from the execution on any MKI environment
but it\r\nwill be executed as part of the CI check if the `@serverless`
tag is\r\npresent.\r\n \r\n With the new changes the serverless labels
will work as follows:\r\n \r\n* `@serverless`: The test is executed as
part of the PR check process\r\nand in the periodic pipeline (MKI
environment).\r\n \r\n* `@serverlessQA`: The test is executed as part of
the kibana release\r\nprocess in the QA environment (MKI
environment).\r\n \r\n* `@skipInServerless`: The test is skipped from
being executed in CI as\r\npart of the PR check and is skipped from
being executed in any MKI\r\nenvironment.\r\n
\r\n`@skipInServerlessMKI`: The test is skipped from being executed in
any\r\nMKI environment but it will continue being executed as part of
the PR\r\nprocess if the `@serverless` tag is present.\r\n \r\n
**IMPORTANT:**\r\n \r\nThe skip labels have been created for those tests
that use `@serverless`\r\nor `@serverlessQA` labels. The absence of them
(`@serverless` or\r\n`@serverlessQA`) will exclude automatically the
execution of the test in\r\nthe targeted environments.\r\n \r\nI.E: A
test without `@serverlessQA` will never be executed as part of\r\nthe
Kibana release process. A test without `@serverless` will never
be\r\nexecuted as part of the PR CI check neither the periodic
pipeline.","sha":"37d47da7711710ded8d30f0514f4da2e1810dc8f"}}]}]
BACKPORT-->

Co-authored-by: Gloria Hornero <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
release_note:skip Skip the PR/issue when compiling release notes Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. v8.14.0 v8.15.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.