[Security Solution] [Serverless] Integrates Cypress in visual mode with QA environment #171107

MadameSheema · 2023-11-13T15:33:16Z

Summary

We are working in order to have our Cypress tests ready to be executed on MKI projects. Our first goal is to be able to execute the tests in the QA environment.

So far we have created all the code needed to be able to execute them in headless mode:

With all the above code, what we are doing is mimicking what we currently have for ESS and serverless on the first quality gate but for MKI projects in the QA environment. We create a project for each spec file following the indications of the Cypress spec and we execute the tests on that specific project.

That is really convenient for CI, but what happens when a test fails? How can we debug what is happening there?

In this PR

Continue mimicking our current flows, in this PR we have created the cypress:open:qa:serverless command.

When the command is executed:

A security project will be created in QA environment with the following characteristics:

   {
      "product_line":"security",
      "product_tier":"complete"
   },
   {
      "product_line":"endpoint",
      "product_tier":"complete"
   },
   {
      "product_line":"cloud",
      "product_tier":"complete"
   }

Opens the Cypress UI with all tests in the e2e directory tagged as @serverless.
Tests are executed using the recently created environment
The project is automatically deleted when:

fn resolve/rejects
Something calls process.exit()
ctrl+c is used in the terminal causing the SIGINT signal to be sent to the process

How to test it

Setup a valid Elastic Cloud API key for QA environment:

Navigate to QA environment.
Click on the User menu button located on the top right of the header.
Click on Organization.
Click on the API keys tab.
Click on Create API key button.
Add a name, set an expiration date, assign an organization owner role.
Click on Create API key
Save the value of the key
Store the saved key on ~/.elastic/cloud.json using the following format:

{
  "api_key": {
    "qa": "<API_KEY>"
  }
}

From the x-pack/test/security_solution_cypress you can execute the following commands:

yarn cypress:open:qa:serverless

The above command will create a project with the following characteristics

   {
      "product_line":"security",
      "product_tier":"complete"
   },
   {
      "product_line":"endpoint",
      "product_tier":"complete"
   },
   {
      "product_line":"cloud",
      "product_tier":"complete"
   }

yarn cypress:open:qa:serverless --tier=essentials

OR

yarn cypress:open:qa:serverless -t essentials

The above commands will create a project with the following characteristics

   {
      "product_line":"security",
      "product_tier":"essentials"
   },
   {
      "product_line":"endpoint",
      "product_tier":"essentials"
   },
   {
      "product_line":"cloud",
      "product_tier":"essentials"
   }

yarn cypress:open:qa:serverless --no-endpoint-addon

OR

yarn cypress:open:qa:serverless --endpoint-addon=false

The above command will create a project with the following characteristics

   {
      "product_line":"security",
      "product_tier":"complete"
   },
   {
      "product_line":"cloud",
      "product_tier":"complete"
   }

yarn cypress:open:qa:serverless --no-cloud-addon

OR

yarn cypress:open:qa:serverless --cloud-addon=false

The above commands will create a project with the following characteristics

   {
      "product_line":"security",
      "product_tier":"complete"
   },
   {
      "product_line":"endpoint",
      "product_tier":"complete"
   }

Note that tier and addons flags can be combined, for instance:

yarn cypress:open:qa:serverless --tier=essentials --no-endpoint-addon

The above command will create a project with the following characteristics

   {
      "product_line":"security",
      "product_tier":"essentials"
   },
   {
      "product_line":"cloud",
      "product_tier":"essentials"
   }

To take into consideration

RBAC cannot be currently tested on QA
Cypress in visual mode does not have the logic to modify/create a project with the product tier and line(s) needed, you should be the one providing that information when executing the command
The code available is the one deployed in QA environment
The setup of the project takes time, there is nothing we can do to speed-up the process since is outside our control

… visual mode

elasticmachine · 2023-11-15T13:46:41Z

Pinging @elastic/security-solution (Team: SecuritySolution)

x-pack/plugins/security_solution/scripts/run_cypress/parallel_serverless.ts

cavokz

As far as I can understand, LGTM.

x-pack/plugins/security_solution/scripts/run_cypress/parallel_serverless.ts

…serverless.ts Co-authored-by: Sergi Massaneda <[email protected]>

semd

LGTM! Thanks for doing this Glo 💯

kibana-ci · 2023-11-15T16:50:17Z

💚 Build Succeeded

Buildkite Build
Commit: 7156604

Metrics [docs]

✅ unchanged

History

💔 Build #176164 failed 073d7ec
💔 Build #176061 failed 5a6e17d
💚 Build #175869 succeeded 7e2f4e0
💚 Build #175561 succeeded eb0de0f
💚 Build #175447 succeeded dabe482

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @MadameSheema

maximpn

@MadameSheema thank you for adding an ability to run Cypress against QA in visual mode 👍

The changes look good overall. I left some comments regarding code structure and naming. On top of this I believe we should stick to first capital letter in texts for Kibana and Serverless but it's a minor note.

maximpn · 2023-11-16T08:40:42Z

x-pack/plugins/security_solution/scripts/run_cypress/parallel_serverless.ts

+  endpointAddon: boolean,
+  cloudAddon: boolean
+): ProductType[] => {
+  let productTypes: ProductType[] = [...DEFAULT_CONFIGURATION];


nit: Instead of filtering out product types I'd recommend to add product types if a corresponding flag is set. It will make the implementation clearer.

maximpn · 2023-11-16T08:46:32Z

x-pack/plugins/security_solution/scripts/run_cypress/parallel_serverless.ts

@@ -371,7 +395,22 @@ export const cli = () => {
            }
            return acc;
          }, {} as Record<string, string | number>)
-        );
+        )
+        .option('tier', {


nit: tier, endpointAddon and cloudAddon are only specific for running Cypress against QA env in visual mode but script consumer can specify these props in the other modes as well though it has no effect. Since yargs supports commands clearer solution would to split functionality in commands with unique params.

maximpn · 2023-11-16T08:49:22Z

x-pack/plugins/security_solution/scripts/run_cypress/parallel_serverless.ts

@@ -325,9 +326,32 @@ function waitForKibanaLogin(kbUrl: string, credentials: Credentials): Promise<vo
  return pRetry(fetchLoginStatusAttempt, retryOptions);
 }

+const getProductTypes = (


getProductTypes sounds too generic and doesn't reflect its purpose. Something like getVisualModeProductTypes should work better.

maximpn · 2023-11-16T08:51:56Z

x-pack/plugins/security_solution/scripts/run_cypress/parallel_serverless.ts


            if (!project) {
              log.info('Failed to create project.');
              // eslint-disable-next-line no-process-exit
              return process.exit(1);
            }

+            context.addCleanupTask(() => {
+              const command = `curl -X DELETE ${BASE_ENV_URL}/api/v1/serverless/projects/security/${project.id} -H "Authorization: ApiKey ${API_KEY}"`;


Just curious. Why did you decide to run curl command instead of axios?

@maximpn this addCleanupTask() didn't accept async, that was the reason behind if there is any other solution/way, more than happy to implement it :)

maximpn · 2023-11-16T08:53:41Z

x-pack/test/security_solution_cypress/cypress/README.md

@@ -72,6 +71,10 @@ Run the tests with the following yarn scripts from `x-pack/test/security_solutio
 | cypress:explore:run:ess | Runs all tests tagged as ESS in the `e2e/explore` directory in headless mode |
 | cypress:investigations:run:serverless | Runs all tests tagged as SERVERLESS in the `e2e/investigations` directory in headless mode |
 | cypress:explore:run:serverless | Runs all tests tagged as SERVERLESS in the `e2e/explore` directory in headless mode |
+| cypress:open:qa:serverless | Opens the Cypress UI with all tests in the `e2e` directory tagged as SERVERLESS. This also creates an MKI project in console.qa enviornment. The kibana instance will reload when you make code changes. This is the recommended way to debug tests in QA. Follow the readme in order to learn about the known limitations. |


kibana -> Kibana

maximpn · 2023-11-16T08:54:53Z

x-pack/test/security_solution_cypress/cypress/README.md

@@ -190,7 +193,7 @@ Task [cypress/support/es_archiver.ts](https://github.com/elastic/kibana/blob/mai
 Note that we use tags in order to select which tests we want to execute, if you want a test to be executed on serverless you need to add @serverless tag to it.


-### Running the serverless tests locally
+### Running serverless tests locally pointing to FTR serverless (First Quality Gate) 


Suggested change

### Running serverless tests locally pointing to FTR serverless (First Quality Gate)

### Running serverless tests locally against FTR Serverless (First Quality Gate)

maximpn · 2023-11-16T08:56:29Z

x-pack/test/security_solution_cypress/cypress/README.md

@@ -234,6 +237,97 @@ Per the way we set the environment during the execution process on CI, the above

 For test developing or test debugging purposes, you need to modify the configuration but without committing and pushing the changes in `x-pack/test/security_solution_cypress/serverless_config.ts`.

+
+### Running serverless tests locally pointing to a MKI project created in QA environment (Second Quality Gate)


Suggested change

### Running serverless tests locally pointing to a MKI project created in QA environment (Second Quality Gate)

### Running serverless tests locally against a MKI project created in QA environment (Second Quality Gate)

maximpn · 2023-11-16T08:56:52Z

x-pack/test/security_solution_cypress/cypress/README.md

+
+| Script Name | Description |
+| ----------- | ----------- |
+| cypress:open:qa:serverless | Opens the Cypress UI with all tests in the `e2e` directory tagged as SERVERLESS. This also creates an MKI project in console.qa enviornment. The kibana instance will reload when you make code changes. This is the recommended way to debug tests in QA. Follow the readme in order to learn about the known limitations. |


kibana -> Kibana

maximpn · 2023-11-16T09:03:13Z

x-pack/test/security_solution_cypress/package.json

@@ -28,6 +28,7 @@
    "cypress:changed-specs-only:serverless": "yarn cypress:serverless --changed-specs-only --env burn=5",
    "cypress:burn:serverless": "yarn cypress:serverless --env burn=2",
    "cypress:qa:serverless": "TZ=UTC NODE_OPTIONS=--openssl-legacy-provider node ../../plugins/security_solution/scripts/start_cypress_parallel_serverless --config-file ../../test/security_solution_cypress/cypress/cypress_ci_serverless_qa.config.ts",
+    "cypress:open:qa:serverless": "yarn cypress:qa:serverless open",


I'm curious about command naming. We have only one separate QA environment we run tests against. It turns out qa is a subset of serverless in some sense. I can imagine we have ESS QA environment in the future and this naming will have more sense.

Have you considered the other options like cypress:open:serverless-qa?

Naming in this file isn't consistent but we can improve it by sticking to one format something like cypress:mode:env[:suites name].

MadameSheema added 14 commits November 13, 2023 16:32

adding the script

9b80e45

trying to delete the project when cypress is not executed anymore

9d12316

clean up task working solution

0bdc4e7

removes severlessQA tag from test

9925a08

updates tags section in readme

dabe482

adding the new scripts to readme

d45456a

removing log.info added for testing purposes

eb0de0f

extending readme

3ed0458

adding more details to readme

a7366d3

Merge branch 'main' into open-qa-serverless

ef368d1

adding node option to avoid es_archiver to fail when using cypress in…

eb188b5

… visual mode

parametrizes the script to force a specific tier or product line

8ec88f7

completing readme

57d1183

little refactor

7e2f4e0

MadameSheema changed the title ~~adding the script~~ [Security Solution] Integrates Cypress in visual mode with QA environment Nov 14, 2023

MadameSheema changed the title ~~[Security Solution] Integrates Cypress in visual mode with QA environment~~ [Security Solution] [Serverless] Integrates Cypress in visual mode with QA environment Nov 14, 2023

MadameSheema added 5 commits November 15, 2023 10:53

last adjusments

5a6e17d

fixes

2facb09

refactor

b48532d

last adjustments

57923b2

updates readme

073d7ec

MadameSheema self-assigned this Nov 15, 2023

MadameSheema added release_note:skip Skip the PR/issue when compiling release notes Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. v8.12.0 labels Nov 15, 2023

MadameSheema marked this pull request as ready for review November 15, 2023 13:46

MadameSheema requested review from patrykkopycinski, oatkiller, maximpn and banderror as code owners November 15, 2023 13:46

MadameSheema requested review from a team as code owners November 15, 2023 13:46

MadameSheema enabled auto-merge (squash) November 15, 2023 13:52

semd reviewed Nov 15, 2023

View reviewed changes

x-pack/plugins/security_solution/scripts/run_cypress/parallel_serverless.ts Outdated Show resolved Hide resolved

semd reviewed Nov 15, 2023

View reviewed changes

x-pack/plugins/security_solution/scripts/run_cypress/parallel_serverless.ts Outdated Show resolved Hide resolved

cavokz approved these changes Nov 15, 2023

View reviewed changes

semd reviewed Nov 15, 2023

View reviewed changes

x-pack/plugins/security_solution/scripts/run_cypress/parallel_serverless.ts Outdated Show resolved Hide resolved

MadameSheema and others added 4 commits November 15, 2023 15:49

Update x-pack/plugins/security_solution/scripts/run_cypress/parallel_…

b48924c

…serverless.ts Co-authored-by: Sergi Massaneda <[email protected]>

addressing comments

6fbfbae

Merge branch 'main' into open-qa-serverless

7d0b6aa

refactor

7156604

semd approved these changes Nov 15, 2023

View reviewed changes

maximpn approved these changes Nov 16, 2023

View reviewed changes

MadameSheema merged commit b47c793 into elastic:main Nov 16, 2023

kibanamachine added the backport:skip This commit does not require backporting label Nov 16, 2023

MadameSheema mentioned this pull request Dec 13, 2023

[Security Solution] Preparing Cypress for the second quality gate - latest steps #173327

Open

43 tasks

MadameSheema deleted the open-qa-serverless branch June 28, 2024 10:54

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Security Solution] [Serverless] Integrates Cypress in visual mode with QA environment #171107

[Security Solution] [Serverless] Integrates Cypress in visual mode with QA environment #171107

MadameSheema commented Nov 13, 2023 •

edited

Loading

elasticmachine commented Nov 15, 2023

cavokz left a comment

semd left a comment

kibana-ci commented Nov 15, 2023

maximpn left a comment

maximpn Nov 16, 2023

maximpn Nov 16, 2023

maximpn Nov 16, 2023

maximpn Nov 16, 2023

MadameSheema Nov 16, 2023

maximpn Nov 16, 2023

maximpn Nov 16, 2023

maximpn Nov 16, 2023

maximpn Nov 16, 2023

maximpn Nov 16, 2023

	### Running serverless tests locally pointing to FTR serverless (First Quality Gate)
	### Running serverless tests locally against FTR Serverless (First Quality Gate)

		@@ -234,6 +237,97 @@ Per the way we set the environment during the execution process on CI, the above

		For test developing or test debugging purposes, you need to modify the configuration but without committing and pushing the changes in `x-pack/test/security_solution_cypress/serverless_config.ts`.


		### Running serverless tests locally pointing to a MKI project created in QA environment (Second Quality Gate)

[Security Solution] [Serverless] Integrates Cypress in visual mode with QA environment #171107

[Security Solution] [Serverless] Integrates Cypress in visual mode with QA environment #171107

Conversation

MadameSheema commented Nov 13, 2023 • edited Loading

Summary

In this PR

How to test it

To take into consideration

elasticmachine commented Nov 15, 2023

cavokz left a comment

Choose a reason for hiding this comment

semd left a comment

Choose a reason for hiding this comment

kibana-ci commented Nov 15, 2023

💚 Build Succeeded

Metrics [docs]

History

maximpn left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MadameSheema commented Nov 13, 2023 •

edited

Loading