Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Security Solution] Adds serverlessQA tag to the API tests #180773

Merged
merged 19 commits into from
Apr 23, 2024
Merged

[Security Solution] Adds serverlessQA tag to the API tests #180773

Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
973d5da
adds serverlessQA tag to API tests
MadameSheema Apr 15, 2024
ec2c07b
aling tags with Cypress tests
MadameSheema Apr 15, 2024
77fa125
Merge branch 'main' into api/serverlessQA
MadameSheema Apr 15, 2024
08b8ebc
Merge branch 'main' into api/serverlessQA
MadameSheema Apr 15, 2024
b052b7f
adds missing file
MadameSheema Apr 15, 2024
dae1ce9
fixes
MadameSheema Apr 16, 2024
2137e9e
more fixes
MadameSheema Apr 16, 2024
3980615
adjustments
MadameSheema Apr 16, 2024
39270f7
adds tags information to the readme
MadameSheema Apr 17, 2024
0cc8b13
Applying Yara's comment: we need to add serverlessQA to the suites t…
MadameSheema Apr 17, 2024
37d3b3a
Merge branch 'main' into api/serverlessQA
MadameSheema Apr 17, 2024
cfdaefb
Update .buildkite/pipelines/security_solution/api_integration_serverl…
MadameSheema Apr 18, 2024
dc14e41
Update x-pack/test/security_solution_api_integration/README.md
MadameSheema Apr 18, 2024
6ff7457
Update .buildkite/pipelines/security_solution/api_integration_serverl…
MadameSheema Apr 18, 2024
78760a6
Merge branch 'main' into api/serverlessQA
MadameSheema Apr 18, 2024
1a6dc9a
updates readme
MadameSheema Apr 22, 2024
7dae577
updates package.json
MadameSheema Apr 22, 2024
3e1dd97
updating tags
MadameSheema Apr 22, 2024
0ac31e6
Merge branch 'main' into api/serverlessQA
MadameSheema Apr 22, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
210 changes: 105 additions & 105 deletions .buildkite/pipelines/security_solution/api_integration.yml
View file
Open in desktop

Large diffs are not rendered by default.

389 changes: 389 additions & 0 deletions .buildkite/pipelines/security_solution/api_integration_serverless_release.yml
View file
Open in desktop

Large diffs are not rendered by default.

4 changes: 2 additions & 2 deletions x-pack/test/security_solution_api_integration/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ we have introduced the `detection_response` directory to consolidate all the int

- In this directory, Mocha tagging is utilized to assign tags to specific test suites and individual test cases. This tagging system enables the ability to selectively apply tags to test suites and test cases, facilitating the exclusion of specific test cases within a test suite as needed.

- There are three primary tags that have been defined: @ess, @serverless, and @brokenInServerless
- There are three primary tags that have been defined: @ess, @serverless, and @skipInServerless
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we add all tags to this list?


- Test suites and cases are prefixed with specific tags to determine their execution in particular environments or to exclude them from specific environments.

Expand All @@ -24,7 +24,7 @@ ex:
describe('@serverless @ess create_rules', () => { ==> tests in this suite will run in both Ess and Serverless
describe('creating rules', () => {});

describe('@brokenInServerless missing timestamps', () => {}); ==> tests in this suite will be excluded in Serverless
describe('@skipInServerless missing timestamps', () => {}); ==> tests in this suite will be excluded in Serverless
rylnd marked this conversation as resolved.
Show resolved Hide resolved

```

Expand Down
2 changes: 1 addition & 1 deletion x-pack/test/security_solution_api_integration/config/ess/config.base.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -95,7 +95,7 @@ export function createTestConfig(options: CreateTestConfigOptions, testFiles?: s
],
},
mochaOpts: {
grep: '/^(?!.*@brokenInEss).*@ess.*/',
grep: '/^(?!.*@skipInEss).*@ess.*/',
},
};
};
Expand Down
2 changes: 1 addition & 1 deletion x-pack/test/security_solution_api_integration/config/serverless/config.base.essentials.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@ export function createTestConfig(options: CreateTestConfigOptions) {

mochaOpts: {
...svlSharedConfig.get('mochaOpts'),
grep: '/^(?!.*@brokenInServerless).*@serverless.*/',
grep: '/^(?!.*@skipInServerless).*@serverless.*/',
},
};
};
Expand Down
2 changes: 1 addition & 1 deletion x-pack/test/security_solution_api_integration/config/serverless/config.base.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ export function createTestConfig(options: CreateTestConfigOptions) {

mochaOpts: {
...svlSharedConfig.get('mochaOpts'),
grep: '/^(?!.*@brokenInServerless).*@serverless.*/',
grep: '/^(?!.*@skipInServerless).*@serverless.*/',
},
};
};
Expand Down
38 changes: 38 additions & 0 deletions x-pack/test/security_solution_api_integration/package.json
View file
Open in desktop

Large diffs are not rendered by default.

6 changes: 3 additions & 3 deletions x-pack/test/security_solution_api_integration/scripts/index.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,15 +21,15 @@ let grepArgs = [];
if (type !== 'server') {
switch (environment) {
case 'serverlessEnv':
grepArgs = ['--grep', '/^(?!.*@brokenInServerless).*@serverless.*/'];
grepArgs = ['--grep', '/^(?!.*@skipInServerless).*@serverless.*/'];
break;

case 'essEnv':
grepArgs = ['--grep', '/^(?!.*@brokenInEss).*@ess.*/'];
grepArgs = ['--grep', '/^(?!.*@skipInEss).*@ess.*/'];
break;

case 'qaEnv':
grepArgs = ['--grep', '/^(?!.*@brokenInServerless|.*@skipInQA).*@serverless.*/'];
grepArgs = ['--grep', '/^(?!.*@skipInServerless).*@serverlessQA.*/'];
break;

default:
Expand Down
2 changes: 1 addition & 1 deletion ...ections_response/detection_engine/actions/trial_license_complete_tier/check_privileges.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ export default ({ getService }: FtrProviderContext) => {
const log = getService('log');
const es = getService('es');

describe('@ess @serverless @brokenInServerless check_privileges', () => {
describe('@ess @serverless @skipInServerless check_privileges', () => {
before(async () => {
await esArchiver.load('x-pack/test/functional/es_archives/auditbeat/hosts');
await esArchiver.load('x-pack/test/functional/es_archives/security_solution/alias');
Expand Down
8 changes: 4 additions & 4 deletions ...etections_response/detection_engine/actions/trial_license_complete_tier/update_actions.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -147,7 +147,7 @@ export default ({ getService }: FtrProviderContext) => {
expect(body?.execution_summary?.last_execution?.status).toBe('succeeded');
});

it('@skipInQA expects an updated rule with a webhook action and meta field runs successfully', async () => {
it('@skipInServerless expects an updated rule with a webhook action and meta field runs successfully', async () => {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we add a comment here:

// Broken in MKI environment, needs triage

Just so we know it needs follow up.

const webhookAction = await createWebHookRuleAction(supertest);

await supertest
Expand Down Expand Up @@ -187,7 +187,7 @@ export default ({ getService }: FtrProviderContext) => {
expect(body?.execution_summary?.last_execution?.status).toBe('succeeded');
});

it('@skipInQA adds a webhook to an immutable rule', async () => {
it('@skipInServerless adds a webhook to an immutable rule', async () => {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we add a comment here:

// Broken in MKI environment, needs triage

Just so we know it needs follow up.

const immutableRule = await getImmutableRule();
const webhookAction = await createWebHookRuleAction(supertest);
const ruleAction = {
Expand All @@ -210,7 +210,7 @@ export default ({ getService }: FtrProviderContext) => {
expect(updatedRule.throttle).toEqual(immutableRule.throttle);
});

it('@skipInQA should be able to create a new webhook action, attach it to an immutable rule and the count of prepackaged rules should not increase. If this fails, suspect the immutable tags are not staying on the rule correctly.', async () => {
it('@skipInServerless should be able to create a new webhook action, attach it to an immutable rule and the count of prepackaged rules should not increase. If this fails, suspect the immutable tags are not staying on the rule correctly.', async () => {
const immutableRule = await getImmutableRule();
const hookAction = await createWebHookRuleAction(supertest);
const ruleToUpdate = getRuleWithWebHookAction(
Expand All @@ -224,7 +224,7 @@ export default ({ getService }: FtrProviderContext) => {
expect(status.rules_not_installed).toBe(0);
});

it('@skipInQA should be able to create a new webhook action, attach it to an immutable rule and the rule should stay immutable when searching against immutable tags', async () => {
it('@skipInServerless should be able to create a new webhook action, attach it to an immutable rule and the rule should stay immutable when searching against immutable tags', async () => {
const immutableRule = await getImmutableRule();
const webhookAction = await createWebHookRuleAction(supertest);
const ruleAction = {
Expand Down
4 changes: 2 additions & 2 deletions ...tections_response/detection_engine/alerts/basic_license_essentials_tier/set_alert_tags.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,8 +40,8 @@ export default ({ getService }: FtrProviderContext) => {
const dataPathBuilder = new EsArchivePathBuilder(isServerless);
const path = dataPathBuilder.getPath('auditbeat/hosts');

// Intentionally setting as @skipInQA, keeping tests running in MKI that should block release
describe('@ess @serverless @skipInQA set_alert_tags', () => {
// Intentionally setting as @skipInServerless, keeping tests running in MKI that should block release
describe('@ess @serverless @skipInServerless set_alert_tags', () => {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This one's a bit tricky - we technically need to add serverlessQA to any test suites that do not have @skipInQA and remove @skipInQA from test suites that have this comment about intentionally setting @skipInQA, because we do want those run in serverless, just not in MKI.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

because we do want those run in serverless, just not in MKI.

Per the agreement we did on the Cypress PR: #179737 this is not possible without having an extra tag. Currently for both API and cypress:

  • A serverless tests is executed in CI, periodic pipeline (MKI) and release process (MKI)
  • A serverless tests is executed in CI and periodic pipeline (MKI)
  • A serverless test is not executed

describe('validation checks', () => {
it('should give errors when no alert ids are provided', async () => {
const { body } = await supertest
Expand Down
4 changes: 2 additions & 2 deletions ...s_response/detection_engine/alerts/trial_license_complete_tier/assignments/assignments.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,8 +39,8 @@ export default ({ getService }: FtrProviderContext) => {
const dataPathBuilder = new EsArchivePathBuilder(isServerless);
const path = dataPathBuilder.getPath('auditbeat/hosts');

// Intentionally setting as @skipInQA, keeping tests running in MKI that should block release
describe('@ess @serverless @skipInQA Alert User Assignment - ESS & Serverless', () => {
// Intentionally setting as @skipInServerless, keeping tests running in MKI that should block release
describe('@ess @serverless @skipInServerless Alert User Assignment - ESS & Serverless', () => {
describe('validation checks', () => {
it('should give errors when no alert ids are provided', async () => {
const { body } = await supertest
Expand Down
4 changes: 2 additions & 2 deletions ...detection_engine/alerts/trial_license_complete_tier/assignments/assignments_serverless.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,8 +33,8 @@ export default ({ getService }: FtrProviderContext) => {
const dataPathBuilder = new EsArchivePathBuilder(isServerless);
const path = dataPathBuilder.getPath('auditbeat/hosts');

// Intentionally setting as @skipInQA, keeping tests running in MKI that should block release
describe('@serverless @skipInQA Alert User Assignment - Serverless', () => {
// Intentionally setting as @skipInServerless, keeping tests running in MKI that should block release
describe('@serverless @skipInServerless Alert User Assignment - Serverless', () => {
before(async () => {
await esArchiver.load(path);
});
Expand Down
2 changes: 1 addition & 1 deletion ...s_response/detection_engine/alerts/trial_license_complete_tier/document_level_security.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -74,7 +74,7 @@ export default ({ getService }: FtrProviderContext) => {
const esArchiver = getService('esArchiver');
const security = getService('security');

describe('@ess @serverless @brokenInServerless find alert with/without doc level security', () => {
describe('@ess @serverless @skipInServerless find alert with/without doc level security', () => {
before(async () => {
await security.role.create(
roleToAccessSecuritySolution.name,
Expand Down
4 changes: 2 additions & 2 deletions ...ngine/exceptions/workflows/basic_license_essentials_tier/exception_comments_serverless.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,8 +26,8 @@ export default ({ getService }: FtrProviderContext) => {
const log = getService('log');
const supertestWithoutAuth = getService('supertestWithoutAuth');

// @skipInQA purposefully - only running tests in MKI whose failure should block release
describe('@serverless @skipInQA exception item comments - serverless specific behavior', () => {
// @skipInServerless purposefully - only running tests in MKI whose failure should block release
describe('@serverless @skipInServerless exception item comments - serverless specific behavior', () => {
describe('Rule Exceptions', () => {
afterEach(async () => {
await deleteAllExceptions(supertest, log);
Expand Down
4 changes: 2 additions & 2 deletions ...on_engine/exceptions/workflows/basic_license_essentials_tier/exceptions_data_integrity.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,8 +37,8 @@ export default ({ getService }: FtrProviderContext) => {
const log = getService('log');
const es = getService('es');

// @skipInQA purposefully - only running tests in MKI whose failure should block release
describe('@serverless @ess @skipInQA exceptions data integrity', () => {
// @skipInServerless purposefully - only running tests in MKI whose failure should block release
describe('@serverless @ess @skipInServerless exceptions data integrity', () => {
afterEach(async () => {
await deleteAllAlerts(supertest, log, es);
await deleteAllRules(supertest, log);
Expand Down
4 changes: 2 additions & 2 deletions ...nse/detection_engine/exceptions/workflows/basic_license_essentials_tier/prebuilt_rules.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,8 +37,8 @@ export default ({ getService }: FtrProviderContext) => {
const log = getService('log');
const es = getService('es');

// @skipInQA purposefully - only running tests in MKI whose failure should block release
describe('@serverless @ess @skipInQA exceptions workflows for prebuilt rules', () => {
// @skipInServerless purposefully - only running tests in MKI whose failure should block release
describe('@serverless @ess @skipInServerless exceptions workflows for prebuilt rules', () => {
describe('creating rules with exceptions', () => {
beforeEach(async () => {
await createAlertsIndex(supertest, log);
Expand Down
6 changes: 3 additions & 3 deletions ...gine/rule_execution_logic/trial_license_complete_tier/execution_logic/machine_learning.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -216,7 +216,7 @@ export default ({ getService }: FtrProviderContext) => {
).toEqual(1);
});

it('@skipInQA generates max alerts warning when circuit breaker is exceeded', async () => {
it('@skipInServerless generates max alerts warning when circuit breaker is exceeded', async () => {
const { logs } = await previewRule({
supertest,
rule: { ...rule, anomaly_threshold: 1, max_signals: 5 }, // This threshold generates 10 alerts with the current esArchive
Expand All @@ -232,7 +232,7 @@ export default ({ getService }: FtrProviderContext) => {
expect(logs[0].warnings).not.toContain(getMaxAlertsWarning());
});

it('@skipInQA should create 7 alerts from ML rule when records meet anomaly_threshold', async () => {
it('@skipInServerless should create 7 alerts from ML rule when records meet anomaly_threshold', async () => {
const { previewId } = await previewRule({
supertest,
rule: { ...rule, anomaly_threshold: 20 },
Expand Down Expand Up @@ -311,7 +311,7 @@ export default ({ getService }: FtrProviderContext) => {
await esArchiver.unload('x-pack/test/functional/es_archives/entity/risks');
});

it('@skipInQA should be enriched with host risk score', async () => {
it('@skipInServerless should be enriched with host risk score', async () => {
const { previewId } = await previewRule({ supertest, rule });
const previewAlerts = await getPreviewAlerts({ es, previewId });
expect(previewAlerts.length).toBe(1);
Expand Down
2 changes: 1 addition & 1 deletion ...etection_engine/rule_execution_logic/trial_license_complete_tier/execution_logic/query.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2373,7 +2373,7 @@ export default ({ getService }: FtrProviderContext) => {
});

// TODO: Ask YARA
describe('@brokenInServerless legacy investigation_fields', () => {
describe('@skipInServerless legacy investigation_fields', () => {
let ruleWithLegacyInvestigationField: Rule<BaseRuleParams>;

beforeEach(async () => {
Expand Down
4 changes: 2 additions & 2 deletions ...sponse/detection_engine/rule_execution_logic/trial_license_complete_tier/ignore_fields.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,7 @@ export default ({ getService }: FtrProviderContext): void => {
await deleteAllRules(supertest, log);
});

it('@skipInQA should ignore the field of "testing_ignored"', async () => {
it('@skipInServerless should ignore the field of "testing_ignored"', async () => {
const rule = getEqlRuleForAlertTesting(['ignore_fields']);

const { id } = await createRule(supertest, log, rule);
Expand All @@ -84,7 +84,7 @@ export default ({ getService }: FtrProviderContext): void => {
expect(hits).to.eql([undefined, undefined, undefined, undefined]);
});

it('@skipInQA should ignore the field of "testing_regex"', async () => {
it('@skipInServerless should ignore the field of "testing_regex"', async () => {
const rule = getEqlRuleForAlertTesting(['ignore_fields']);

const { id } = await createRule(supertest, log, rule);
Expand Down
2 changes: 1 addition & 1 deletion ...ebuilt_rules_package/trial_license_complete_tier/install_latest_bundled_prebuilt_rules.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ export default ({ getService }: FtrProviderContext): void => {
/* from a package that was bundled with Kibana */
//
// FLAKY: https://github.com/elastic/kibana/issues/180087
describe.skip('@ess @serverless @skipInQA install_bundled_prebuilt_rules', () => {
describe.skip('@ess @serverless @skipInServerless install_bundled_prebuilt_rules', () => {
Copy link
Contributor

@banderror banderror Apr 18, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@MadameSheema We don't want all the tests for prebuilt rules to be skipped in all quality gates. We need them to run on CI, where, as far as I'm aware, they are not currently skipped and have been passing.

This also applies to some of the other rule management tests that are being skipped in CI by this PR.

I would suggest to:

  1. Either introduce an additional @skipInServerlessMKI tag and use it for these tests instead of @skipInServerless. This tag would skip a test in the periodic pipeline + 2nd quality gate + 3rd and 4th quality gates in the future, but keep the test running on CI in the Serverless mode.
  2. Or to rename the @serverless tag to @serverlessCI and introduce:
    • an additional tag @serverlessRelease that would add a test to all the release pipelines: the 2nd quality gate + 3rd and 4th quality gates in the future
    • an additional tag @serverlessPeriodic that would add a test only to the periodic pipeline

I'd personally choose the 2nd option as being more explicit about where a test should run.

Sorry if I missed this in #179737 where Cypress tests for prebuilt rules and rule management have been skipped in Serverless CI. We'd need to run those Cypress tests in CI too, so I'd propose the same changes for Cypress tests.

cc @jpdjere @maximpn @xcrzx

Copy link
Member Author

@MadameSheema MadameSheema Apr 18, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@maximpn @yctercero since you were the ones against having extra labels, can you please arrive to an agreement about how to proceed with it?

This is my proposal:

@serverless -> Executes tests on CI and periodic pipeline (MKI)


@serverlessQA -> Executes tests on the second quality gate (QA environment)

@skipServerless-> Does not execute the test in any serverless environment

@skipServerlessMKI -> Does not execute the test in any MKI environment (periodic pipeline and quality gate)



If you don’t want a test to be executed as part of the second quality gate you don’t add the @serverlessQA tag to it.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@MadameSheema do I get it correctly a test which should be skipped in MKI env but run everywhere else will have the following tags

describe('@ess @serverless @serverlessQA @skipServerlessMKI install_bundled_prebuilt_rules', () => {

Here we return to the situation when it equals to

describe('@ess @serverless install_bundled_prebuilt_rules', () => {

Here we have two options

  • Have only @ess, @skipInEss, @serverless, @serverlessQA and @skipInServerless tags. When we need to run a test in another MKI end we would add another @serverless<name> tag. By default we just don't add tags so the test isn't enabled automatically for any MKI environment. There is an assumption MKI env is less stable than mocked Serverless env. It's impossible to disabled a test for only mocked Serverless env but still run it against some MKI env. But it doesn't look like an issue.
  • Have only @ess, @skipInEss, @serverless, @skipInServerlessMock (what's the best name for local/CI Serverless instead of serverless mock?) and @skipInServerlessQA tags. In this case tests tagged by @serverless will run against everything (besides ESS) and adding a corresponding @skipInServerless<name> tag skips a test in a proper env.

The second approach looks better for automatic skipping script. When a test fails in some env or becomes flaky it could be skipped automatically.

I checked the PR's diff once more time and see that it's more convenient to use the second approach. Quite often describe section has permissive tags @ess and @serverless while individual tests have may be skipped via @skipInServerless. It will require manually skip all the tests in MKI env via @skipInServerlessQA but it should be simpler to maintain the tests.

Copy link
Contributor

@banderror banderror Apr 18, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@maximpn

Have only @ess, @skipInEss, @serverless, @serverlessQA and @skipInServerless tags.

This set of tags doesn't allow to run a test on CI in ESS + on CI in Serverless, but not run it in the periodic pipeline + release gates.

At this point, we need to be able to run many of our tests ONLY on CI in ESS + on CI in Serverless, because they are broken in MKI environments. Fixing this requires time and effort we don't have at the moment.

Have only @ess, @skipInEss, @serverless, @skipInServerlessMock (what's the best name for local/CI Serverless instead of serverless mock?) and @skipInServerlessQA tags

This set of tags also doesn't allow to run a test on CI in ESS + on CI in Serverless, but not run it in the periodic pipeline + release gates.

We don't need to be able to skip a test on CI in Serverless, but run it in MKI pipelines. We don't have such class of tests as of now.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@banderror @MadameSheema

Could you clarify the requirements to running tests in periodic pipeline? Do we need a separate way to skip tests for periodic pipeline?

In my understanding if a test is broken in MKI env it will be broken in periodic pipeline since it's an MKI env as well. Taking this into account a test shouldn't have @serverlessQAtag (option 1) or have @skipInServerlessQA tag (option 2). It won't run against any MKI env until it's fixed, tested manually and tag manipulation is performed to run the test against MKI envs.

Copy link
Member Author

@MadameSheema MadameSheema Apr 18, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@banderror @MadameSheema

Could you clarify the requirements to running tests in periodic pipeline? Do we need a separate way to skip tests for periodic pipeline?

In my understanding if a test is broken in MKI env it will be broken in periodic pipeline since it's an MKI env as well. Taking this into account a test shouldn't have @serverlessQAtag (option 1) or have @skipInServerlessQA tag (option 2). It won't run against any MKI env until it's fixed, tested manually and tag manipulation is performed to run the test against MKI envs.

As there are differences between the serverless environment we use on CI and a real one but at the same time, we don't want to execute all the tests in the quality gate, is a way for us to determine the health of our application in a real serverless project environment.

We need a separate way of skipping tests from MKI but not CI.

A test with just @serverless is going to be executed in CI and periodic pipeline. If you add to the test @serverlessQA that test will be executed in the second quality gate as part of the release process. If a test is broken on MKI is going to fail in both on periodic and quality gate, so you can skip it with @serverlessMKI but it will still be executed on CI as part of the PRs so I strongly believe that right now what it makes sense for us to have as labels is:

@ess: Test is executed on CI ess environment
@skipEss: Test is excluded fom the CI ess environment execution
@serverless: Test is executed on CI serverless and periodic pipeline (MKI environment)
@serverlessQA: Test is executed as part of the release process on the second quality gate which is the QA environment (MKI environment)
@skipServerlessMKI: Test is excluded from being executed in any MKI environment, meaning, is not executed as part of the periodic pipeline and is not executed as part of the release process
@skipServerless: Tests is excluded from being executed as part from the CI and is not executed in any MKI environment

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@MadameSheema Your proposal works for me 👍

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Above proposal looks great to me @MadameSheema

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@MadameSheema

Me and @banderror discussed the approach on a zoom call. Tags you suggest in the last comment look good to me. I have some comments though

ESS tags are pretty clear. We need to run tests against ESS env and skip broken tests.

  • @ess
  • @skipEss

Serverless in general is pretty clear as well. We need to run test against Serverless env (abstractly it doesn't matter which one) and skip broken tests

  • @serverless
  • @skipServerless

If we don't need to run tests against some env an appropriate tag @ess or @serverless isn't added.

The most interesting part start from the fact that Serverless envs are different. Not all tests work in Serverless MKI env currently. Taking into account this fact we need an ability to skip such tests so we can use the following tag

  • @skipServerlessMKI

And the last part is that we need some test to block Serverless release when critical functionality doesn't work. In this case something explicit like @blockServerlessRelease or @serverlessRelease would work the best

  • @serverlessRelease

I have a question about your suggestion. Why do you want to use explicit @serverlessQA. Is it expected we are gonna run different test sets/subsets in QA, Staging etc Serverless MKI envs? It looks like if a test work in one Serverless MKI env it should work in the other one as well. Is there some difference between QA and Staging?

cc @rylnd

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe at this point it's best to jump on a Zoom. I'm a bit confused as to what's being asked here anymore.

@MadameSheema worked on this PR after we chatted and decided it'd be great for our API and cypress tags to match so this PR brought them into parity. We had discussed and decided on the tags during the first PR that changed the Cypress tags. It would be great to finalize the discussion so we don't keep rehashing the discussion in each PR.

beforeEach(async () => {
await deleteAllRules(supertest, log);
await deleteAllPrebuiltRuleAssets(es, log);
Expand Down
2 changes: 1 addition & 1 deletion ...t_rules/bundled_prebuilt_rules_package/trial_license_complete_tier/prerelease_packages.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ export default ({ getService }: FtrProviderContext): void => {
/* (We use high mock version numbers to prevent clashes with real packages downloaded in other tests.)
/* To do assertions on which packages have been installed, 99.0.0 has a single rule to install,
/* while 99.0.1-beta.1 has 2 rules to install. Also, both packages have the version as part of the rule names. */
describe('@ess @serverless @skipInQA prerelease_packages', () => {
describe('@ess @serverless @skipInServerless prerelease_packages', () => {
beforeEach(async () => {
await deleteAllRules(supertest, log);
await deleteAllPrebuiltRuleAssets(es, log);
Expand Down
2 changes: 1 addition & 1 deletion ...rebuilt_rules_package/trial_license_complete_tier/install_large_prebuilt_rules_package.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ export default ({ getService }: FtrProviderContext): void => {
const supertest = getService('supertest');
const log = getService('log');

describe('@ess @serverless @skipInQA install_large_prebuilt_rules_package', () => {
describe('@ess @serverless @skipInServerless install_large_prebuilt_rules_package', () => {
beforeEach(async () => {
await deleteAllRules(supertest, log);
await deleteAllPrebuiltRuleAssets(es, log);
Expand Down
2 changes: 1 addition & 1 deletion ...les_management/prebuilt_rules/management/trial_license_complete_tier/fleet_integration.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ export default ({ getService }: FtrProviderContext): void => {
const log = getService('log');
const retry = getService('retry');

describe('@ess @serverless @skipInQA install_prebuilt_rules_from_real_package', () => {
describe('@ess @serverless @skipInServerless install_prebuilt_rules_from_real_package', () => {
beforeEach(async () => {
await deletePrebuiltRulesFleetPackage(supertest);
await deleteAllRules(supertest, log);
Expand Down
2 changes: 1 addition & 1 deletion ...gement/prebuilt_rules/management/trial_license_complete_tier/get_prebuilt_rules_status.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ export default ({ getService }: FtrProviderContext): void => {
const es = getService('es');
const log = getService('log');

describe('@ess @serverless @skipInQA Prebuilt Rules status', () => {
describe('@ess @serverless @skipInServerless Prebuilt Rules status', () => {
describe('get_prebuilt_rules_status', () => {
beforeEach(async () => {
await deleteAllPrebuiltRuleAssets(es, log);
Expand Down
2 changes: 1 addition & 1 deletion ...nt/prebuilt_rules/management/trial_license_complete_tier/get_prebuilt_timelines_status.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ export default ({ getService }: FtrProviderContext): void => {
const es = getService('es');
const log = getService('log');

describe('@ess @serverless @skipInQA get_prebuilt_timelines_status', () => {
describe('@ess @serverless @skipInServerless get_prebuilt_timelines_status', () => {
beforeEach(async () => {
await deleteAllTimelines(es, log);
});
Expand Down
2 changes: 1 addition & 1 deletion ...anagement/prebuilt_rules/management/trial_license_complete_tier/install_prebuilt_rules.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ export default ({ getService }: FtrProviderContext): void => {
const supertest = getService('supertest');
const log = getService('log');

describe('@ess @serverless @skipInQA install prebuilt rules from package without historical versions with mock rule assets', () => {
describe('@ess @serverless @skipInServerless install prebuilt rules from package without historical versions with mock rule assets', () => {
const getRuleAssetSavedObjects = () => [
createRuleAssetSavedObject({ rule_id: 'rule-1', version: 1 }),
createRuleAssetSavedObject({ rule_id: 'rule-2', version: 2 }),
Expand Down
2 changes: 1 addition & 1 deletion ...management/trial_license_complete_tier/install_prebuilt_rules_with_historical_versions.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ export default ({ getService }: FtrProviderContext): void => {
const supertest = getService('supertest');
const log = getService('log');

describe('@ess @serverless @skipInQA install prebuilt rules from package with historical versions with mock rule assets', () => {
describe('@ess @serverless @skipInServerless install prebuilt rules from package with historical versions with mock rule assets', () => {
const getRuleAssetSavedObjects = () => [
createRuleAssetSavedObject({ rule_id: 'rule-1', version: 1 }),
createRuleAssetSavedObject({ rule_id: 'rule-1', version: 2 }),
Expand Down
2 changes: 1 addition & 1 deletion ...anagement/prebuilt_rules/management/trial_license_complete_tier/upgrade_prebuilt_rules.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ export default ({ getService }: FtrProviderContext): void => {
const supertest = getService('supertest');
const log = getService('log');

describe('@ess @serverless @skipInQA upgrade prebuilt rules from package without historical versions with mock rule assets', () => {
describe('@ess @serverless @skipInServerless upgrade prebuilt rules from package without historical versions with mock rule assets', () => {
const getRuleAssetSavedObjects = () => [
createRuleAssetSavedObject({ rule_id: 'rule-1', version: 1 }),
createRuleAssetSavedObject({ rule_id: 'rule-2', version: 2 }),
Expand Down
2 changes: 1 addition & 1 deletion ...management/trial_license_complete_tier/upgrade_prebuilt_rules_with_historical_versions.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ export default ({ getService }: FtrProviderContext): void => {
const supertest = getService('supertest');
const log = getService('log');

describe('@ess @serverless @skipInQA upgrade prebuilt rules from package with historical versions with mock rule assets', () => {
describe('@ess @serverless @skipInServerless upgrade prebuilt rules from package with historical versions with mock rule assets', () => {
beforeEach(async () => {
await deleteAllRules(supertest, log);
await deleteAllTimelines(es, log);
Expand Down
2 changes: 1 addition & 1 deletion ...pdate_prebuilt_rules_package/trial_license_complete_tier/update_prebuilt_rules_package.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,7 @@ export default ({ getService }: FtrProviderContext): void => {
return getPackageResponse.body.item.version ?? '';
};

describe('@ess @serverless @skipInQA update_prebuilt_rules_package', () => {
describe('@ess @serverless @skipInServerless update_prebuilt_rules_package', () => {
before(async () => {
const configFilePath = path.resolve(REPO_ROOT, 'fleet_packages.json');
const fleetPackages = await fs.readFile(configFilePath, 'utf8');
Expand Down
2 changes: 1 addition & 1 deletion ...nse/rules_management/rule_bulk_actions/trial_license_complete_tier/perform_bulk_action.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -85,7 +85,7 @@ export default ({ getService }: FtrProviderContext): void => {
const createWebHookConnector = () => createConnector(getWebHookAction());
const createSlackConnector = () => createConnector(getSlackAction());

describe('@ess @serverless @brokenInServerless @skipInQA perform_bulk_action', () => {
describe('@ess @serverless @skipInServerless perform_bulk_action', () => {
beforeEach(async () => {
await createAlertsIndex(supertest, log);
await esArchiver.load('x-pack/test/functional/es_archives/auditbeat/hosts');
Expand Down
2 changes: 1 addition & 1 deletion ...s_management/rule_bulk_actions/trial_license_complete_tier/perform_bulk_action_dry_run.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ export default ({ getService }: FtrProviderContext): void => {
const log = getService('log');
const es = getService('es');

describe('@ess @serverless @skipInQA perform_bulk_action dry_run', () => {
describe('@ess @serverless @skipInServerless perform_bulk_action dry_run', () => {
beforeEach(async () => {
await createAlertsIndex(supertest, log);
});
Expand Down
Loading