[Security Solution][Serverless] Reusing Cypress tests for Serverless infrastructure #162698

MadameSheema · 2023-07-28T11:03:58Z

UPDATE!

It has been finally decided to move all the Security Solution Cypress tests to the x-pack/test/security_solution_cypress folder in order to avoid circular dependency.

AFTER MERGING THIS PR:

All the existing and new tests needs to have a tag to be executed on the environment you expect, if not, the test is not going to be executed
The tests marked as serverless are going to be executed on the FTR serverless environment when a PR is opened
Teams should be extremely careful when adding new tests to make sure we are not introducing flakiness. Note that our serverless tests are initially executed on ALL the PRs, so we should not create blockers to other teams and our tests are not executed just when something is changed in our code.
It is extremely forbidden to add conditionals on assertions or actions to make a test work in both environments. That is a red flag warning you about having to create a specific test for it.
Specific serverless tests should be written on security_solution_cypress/cypress/e2e folder of each specific team, and should be tagged just with @serverless
Teams are responsible to investigate the tests marked as @brokenInServerless, I can give support and advice.
Use yarn cypress:open:ess to develop and debug tests on ESS environment
User yarn cypress:open:serverless to develop and debug tests on FTR serverless environment

Summary

As mentioned on the ticket #161537, Analyst Experience Teams are going to need to start adding serverless tests.
We already have a lot of existing tests so taking into consideration the time restrictions and the head-ache of having to maintain duplicated code, we’d love to reuse the ones we can.

For Cypress specific, we explored two different ideas given by @semd and @patrykkopycinski with their support and advice (Thanks ❤️): 

After exploring both options, finally it was decided to implement second one, because:

It is fast to setup
Easy to reuse tests
The impact on the development flow is minimal
Taking into consideration all the above, we are not going to lose development engagement around adding ore creating new tests.

In this PR we are implementing the solution:

~~- We are going to maintain all the tests on the same folder to don’t disrupt developers flow, so specific Cypress serverless tests should be added to the regular folder.~~

Cypress tests are placed now on x-pack/test/security_solution_cypress folder
We are adding cypress grep (https://github.com/cypress-io/cypress/tree/develop/npm/grep) to enable tagging inside Cypress.
We are adding two new cypress configurations for serverless-specific executions
We are adding new scripts for serverless executions
We are adding the new executions to buildkite so they are part of the CI process
We are removing the waitForPageToBeLoaded method. This is because cannot be reused for both ESS and Serverless and it has been demonstrated to not be useful since Cypress waits for an element to be present before performing an action.
We are tagging with @ess the tests that are executed on ESS environment.
We are tagging with @serverless the tests that are executed on serverless environment.
We are tagging with @brokenInServerless the tests that should be executed on serverless but currently are failing on that enviroment.
We are updating the readme.

IMPORTANT!!!!!

 Once this PR is merged:

All the existing and new tests needs to have a tag to be executed on the environment you expect, if not, the test is not going to be executed
The tests marked as serverless are going to be executed on the FTR serverless environment when a PR is opened
Teams should be extremely careful when adding new tests to make sure we are not introducing flakiness. Note that our serverless tests are initially executed on ALL the PRs, so we should not create blockers to other teams and our tests are not executed just when something is changed in our code.
It is extremely forbidden to add conditionals on assertions or actions to make a test work in both environments. That is a red flag warning you about having to create a specific test for it.
Specific serverless tests should be written on the regular cypress folder and should be tagged just with @serverless
Teams are responsible to investigate the tests marked as @brokenInServerless, I can give support and advice.
Use ~~yarn cypress:open~~ yarn cypress:open:serverless to develop and debug tests on ESS environment
User yarn cypress:open:serverless to develop and debug tests on FTR serverless environment

# Conflicts: # x-pack/plugins/security_solution/cypress/e2e/exceptions/alerts_table_flow/endpoint_exceptions.cy.ts # x-pack/plugins/security_solution/cypress/e2e/exceptions/alerts_table_flow/rule_exceptions/auto_populate_with_alert_data.cy.ts # x-pack/plugins/security_solution/cypress/e2e/exceptions/entry/comments.cy.ts

banderror

I couldn't run Cypress tests locally for ESS.

I'm in x-pack/test/security_solution_cypress, I run cypress:open:ess. Then ES and Kibana start apparently successfully, after that it crashes with the following error, and the test runner doesn't start:

 warn Failure loading service "security"
ERROR TimeoutError: Request timed out
          at SniffingTransport.request (/Users/georgii/Code/elastic/kibana-pr1/node_modules/@elastic/transport/src/Transport.ts:584:17)
          at processTicksAndRejections (node:internal/process/task_queues:95:5)
          at Security.putRole (/Users/georgii/Code/elastic/kibana-pr1/node_modules/@elastic/elasticsearch/src/api/api/security.ts:1391:12)
          at ensureSystemIndicesUser (system_indices_user.ts:33:3)
          at createSystemIndicesUser (system_indices_user.ts:78:3)
          at SecurityServiceProvider (security.ts:32:3)
          at provider_collection.ts:53:13
          at async Promise.all (index 9)
          at ProviderCollection.loadAll (provider_collection.ts:48:5)
          at parallel.ts:347:13
          at withProcRunner (with_proc_runner.ts:29:5)
          at concurrency (parallel.ts:181:11)
          at /Users/georgii/Code/elastic/kibana-pr1/node_modules/p-map/index.js:57:22
 info [kibana] exited with null after 156.8 seconds
ERROR UNHANDLED ERROR
ERROR Error: Failure initializing 1 service(s)
          at ProviderCollection.loadAll (provider_collection.ts:64:13)
          at processTicksAndRejections (node:internal/process/task_queues:95:5)
          at parallel.ts:347:13
          at withProcRunner (with_proc_runner.ts:29:5)
          at concurrency (parallel.ts:181:11)
          at /Users/georgii/Code/elastic/kibana-pr1/node_modules/p-map/index.js:57:22
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

I also noticed that you deleted the Cypress burn CI job, what was the reason for doing this?

banderror · 2023-08-15T11:57:34Z

.buildkite/pipelines/pull_request/security_solution.yml

-  - command: .buildkite/scripts/steps/functional/security_solution_burn.sh
-    label: 'Security Solution Cypress tests, burning changed specs'
-    agents:
-      queue: n2-4-spot
-    depends_on: build
-    timeout_in_minutes: 120
-    parallelism: 1
-    soft_fail: true
-    artifact_paths:
-      - "target/kibana-security-solution/**/*"


Why deleted? This job was just recently added by @patrykkopycinski to help us reveal flakiness

As I changed all the files the execution was taking extremely long, cancel it was not helping either. The idea is to enable it again on a follow-up PR.

Okay, sounds good 👍

banderror · 2023-08-15T12:10:06Z

x-pack/plugins/threat_intelligence/cypress/tasks/login.ts

-} from '@kbn/security-solution-plugin/cypress/screens/security_header';
-import { encode } from '@kbn/rison';
-import { NEW_FEATURES_TOUR_STORAGE_KEYS } from '@kbn/security-solution-plugin/common/constants';
+} from '../../../../test/security_solution_cypress/cypress/screens/security_header';


This is a bit suspicious dependency from a plugin to test code outside of plugins. cc @PhilippeOberti

banderror

The issue mentioned in #162698 (review) appeared to happen with a few folks who are on Intel-based Macs (internal slack thread). The suggested workaround is to run tests against manually started ES and Kibana instances (slack):

Hey, one workaround, it’s to run cypress tests agains you local es/kibana
So if you have es/kibana running locally, you can run cypress tests with this command

CYPRESS_CYPRESS_BASE_URL="http://elastic:changeme@localhost:5601/"  CYPRESS_BASE_URL="http://elastic:changeme@localhost:5601/" CYPRESS_ELASTICSEARCH_URL="http://system_indices_superuser:changeme@localhost:9200/" CYPRESS_ELASTICSEARCH_USERNAME="system_indices_superuser" CYPRESS_ELASTICSEARCH_PASSWORD="changeme" baseUrl="http://elastic:changeme@localhost:5601/" BASE_URL="http://elastic:changeme@localhost:5601/" ELASTICSEARCH_URL="http://system_indices_superuser:changeme@localhost:9200/" ELASTICSEARCH_USERNAME="system_indices_superuser" ELASTICSEARCH_PASSWORD="changeme" ELASTICSEARCH_URL_WITH_AUTH="http://system_indices_superuser:changeme@localhost:9200/" FLEET_SERVER_URL="http://localhost:8220/" KIBANA_URL="http://localhost:5601/" KIBANA_URL_WITH_AUTH="http://elastic:changeme@localhost:5601/" KIBANA_USERNAME="elastic" KIBANA_PASSWORD="changeme" yarn cypress open --config-file ./cypress/cypress.config.ts

Also this command don’t work for me until I commented this in kibana.dev.yaml

# server.name: kibana
# server.host: "0.0.0.0"

I'll follow up in those threads to see if we can fix this.

I also added a few fixes to the branch addressing my previously posted comments.

Thank you @MadameSheema. Go reusable Cypress tests! 🤘 🤘

gergoabraham

change in 24da233 looks good! 👍

kibana-ci · 2023-08-15T17:49:56Z

💛 Build succeeded, but was flaky

Buildkite Build
Commit: 47d9dd6

Failed CI Steps

Test Failures

[job] [logs] Defend Workflows Cypress Tests #1 / Endpoint Policy Response from Endpoint List page should display policy response with errors should display policy response with errors

Metrics [docs]

Unknown metric groups

ESLint disabled line counts

id	before	after	diff
`securitySolution`	466	449	-17

Total ESLint disabled count

id	before	after	diff
`securitySolution`	532	515	-17

History

💔 Build #150568 failed 24da233
💔 Build #150557 failed f25e362
💛 Build #150446 was flaky 94c9a45
💔 Build #150429 failed 1fbeca3
💔 Build #150418 failed 835a35d
💔 Build #150248 failed d36e8ba

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @MadameSheema

…infrastructure (elastic#162698) Co-authored-by: kibanamachine <[email protected]> Co-authored-by: Patryk Kopycinski <[email protected]> Co-authored-by: Georgii Gorbachev <[email protected]>

## Summary closes: #157847 The new links and pages in Security Solution for Serverless: - `Investigations` - `Timelines` - `Osquery` - `Assets` - `Fleet` (and all its sub-links) - `Endpoints` (and all its sub-links) - `Cloud defend` (and all its sub-links) - Callout with button linking `Integrations` in Project Setting - `Project settings` - `Users & roles` (Cloud UI) - `Billing & consumption` (Cloud UI) - `Integrations` (link to integrations with _/browse/security_ path parameter) - `Entity risk score` (link currently under the `riskScoringRoutesEnabled` experimental flag) - `Management` accordion with a set of (stack) management categories and pages links Sections updated: (ESS & Serverless) `Rules` links have been updated according to new specs. (ESS) The `Settings` page was renamed back to `Manage`. (Serverless) The `Dev tools` link was moved to the bottom of the side navigation. #### Cypress tests for serverless: They will be implemented in a follow-up PR when the infrastructure is ready #162698 ## Screenshots ### Serverless Investigations: ![investigations](https://github.com/elastic/kibana/assets/17747913/19b602ab-53bb-4280-b0c3-dc69255ea3ab) ![investigations_panel](https://github.com/elastic/kibana/assets/17747913/8be036fc-6095-4234-8b07-1a7149c8a92b) Assets: ![assets](https://github.com/elastic/kibana/assets/17747913/963723d5-16da-45dd-955c-a7d7b61099e6) ![assets_panel](https://github.com/elastic/kibana/assets/17747913/977c0716-7cc3-4369-acdf-0787e264a912) Rules: ![rules](https://github.com/elastic/kibana/assets/17747913/5bafbd08-508c-4b50-bd28-dc052371e76a) ![rules_panel](https://github.com/elastic/kibana/assets/17747913/ea47f8d9-4b68-44f3-8a42-6f795d05b982) Project Settings: ![project_settings](https://github.com/elastic/kibana/assets/17747913/36dbb446-3354-4519-b59c-e8b5005fbb71) ![project_settings_panel](https://github.com/elastic/kibana/assets/17747913/1315b54b-4423-41a2-b0ed-997ee77c1f6d) ## ESS Side Navigation: ![ess_side_nav](https://github.com/elastic/kibana/assets/17747913/a3687a47-3ccb-4ce8-aa53-9e962a6ef1cf) --------- Co-authored-by: kibanamachine <[email protected]> Co-authored-by: Tomasz Ciecierski <[email protected]> Co-authored-by: Tomasz Ciecierski <[email protected]>

…infrastructure (#162698) Co-authored-by: kibanamachine <[email protected]> Co-authored-by: Patryk Kopycinski <[email protected]> Co-authored-by: Georgii Gorbachev <[email protected]>

…s for Serverless (#165094) ## Summary - introduces tags for [Defend Workflows] cypress tests (similarly to #162698) - adds scripts to Security Solution: - `cypress:dw:serverless:open` and `:run` - `cypress:dw:endpoint:serverless:open` and `:run` - adds CI jobs to run these scripts - so far most of the expected tests got both `@serverless` and `@brokenInServerless` tests, because of other issues to be solved, - one test is able to run against serverless: `x-pack/plugins/security_solution/public/management/cypress/e2e/mocked_data/policy_details.cy.ts`

MadameSheema added 5 commits July 28, 2023 11:27

adding cypress grep

9901c5a

added the serverless environment

de08144

adding specific serverless executions

7b8d4fd

removing 'waitForPageToBeLoaded'

3974e6a

modifying execution on buildkite

17be1a9

MadameSheema force-pushed the serverless-tags branch from 6dbef51 to 3eeb56b Compare July 28, 2023 11:07

adding executions to buildkite

8836fdc

MadameSheema force-pushed the serverless-tags branch from 3eeb56b to 8836fdc Compare July 28, 2023 11:20

MadameSheema added 13 commits July 28, 2023 14:31

updates scripts

acc58b6

updates server

9622dfd

adding tags

adfd9c0

adding all tags

1d5c905

Merge branch 'main' into serverless-tags

9faada6

fixes explore script

14081a9

script fixes

c97d940

tag refactor

680a4c1

fixes

20efcd2

adjustments

f2fdf2d

updating readme

5123731

identifying failures

a77d950

adjusting to failures

6b2bb1c

banderror mentioned this pull request Aug 1, 2023

[Security Solution] Find a way to reuse tests between ESS and Serverless suites #161537

Closed

7 tasks

MadameSheema added 3 commits August 2, 2023 09:25

tagging broken tests in serverless

883572f

Merge branch 'main' into serverless-tags

11a9662

MadameSheema changed the title ~~adding cypress grep~~ [Security Solution][Serverless] Reusing Cypress tests for Serverless infrastructure Aug 2, 2023

MadameSheema self-assigned this Aug 2, 2023

MadameSheema added release_note:skip Skip the PR/issue when compiling release notes Team:Detections and Resp Security Detection Response Team Team:Threat Hunting Security Solution Threat Hunting Team labels Aug 2, 2023

MadameSheema added 2 commits August 15, 2023 10:44

typo

1fbeca3

Merge branch 'main' into serverless-tags

94c9a45

banderror requested changes Aug 15, 2023

View reviewed changes

Address own comments

04d49fb

banderror approved these changes Aug 15, 2023

View reviewed changes

MadameSheema added the ci:all-cypress-suites label Aug 15, 2023

MadameSheema added 2 commits August 15, 2023 16:30

Merge branch 'main' into serverless-tags

f25e362

fixing osquery scripts

24da233

MadameSheema requested a review from a team as a code owner August 15, 2023 15:18

MadameSheema requested review from pzl and ashokaditya August 15, 2023 15:18

patrykkopycinski approved these changes Aug 15, 2023

View reviewed changes

gergoabraham approved these changes Aug 15, 2023

View reviewed changes

patrykkopycinski added the ci:skip-cypress-osquery Skips osquery cypress checks label Aug 15, 2023

Merge branch 'main' into serverless-tags

47d9dd6

MadameSheema merged commit f9c9722 into elastic:main Aug 15, 2023

MadameSheema deleted the serverless-tags branch August 15, 2023 17:51

MadameSheema mentioned this pull request Aug 15, 2023

[Security Solution] Reenabling Cypress burn #163965

Merged

kibanamachine added the backport:skip This commit does not require backporting label Aug 15, 2023

This was referenced Aug 22, 2023

[Security Solution] Cypress test runner doesn't start locally on Intel-based Macs #164435

Closed

[Security Solution] Skip all existing Cypress tests in Serverless: Detection & Response area #164441

Closed

gergoabraham mentioned this pull request Aug 29, 2023

[Defend Workflows] Add tags for mocked and real Endpoint Cypress tests for Serverless #165094

Merged

gbamparop mentioned this pull request Dec 4, 2023

[Logs UX] Investigate options for running Cypress tests in a serverless environment #172492

Open

paulb-elastic mentioned this pull request Apr 4, 2024

[R&D] Stateful/stateless test harness options for observability #179549

Closed

13 tasks

mgiota mentioned this pull request May 14, 2024

[Observability solution] [SLO] Run burn rate api tests in serverless & ess using mocha tagging #183113

Closed

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Security Solution][Serverless] Reusing Cypress tests for Serverless infrastructure #162698

[Security Solution][Serverless] Reusing Cypress tests for Serverless infrastructure #162698

MadameSheema commented Jul 28, 2023 •

edited

Loading

banderror left a comment

banderror Aug 15, 2023

MadameSheema Aug 15, 2023

banderror Aug 15, 2023

banderror Aug 15, 2023

banderror left a comment

gergoabraham left a comment •

edited

Loading

kibana-ci commented Aug 15, 2023

ESLint disabled line counts

Total ESLint disabled count

[Security Solution][Serverless] Reusing Cypress tests for Serverless infrastructure #162698

[Security Solution][Serverless] Reusing Cypress tests for Serverless infrastructure #162698

Conversation

MadameSheema commented Jul 28, 2023 • edited Loading

UPDATE!

Summary

IMPORTANT!!!!!

banderror left a comment

Choose a reason for hiding this comment

banderror Aug 15, 2023

Choose a reason for hiding this comment

MadameSheema Aug 15, 2023

Choose a reason for hiding this comment

banderror Aug 15, 2023

Choose a reason for hiding this comment

banderror Aug 15, 2023

Choose a reason for hiding this comment

banderror left a comment

Choose a reason for hiding this comment

gergoabraham left a comment • edited Loading

Choose a reason for hiding this comment

kibana-ci commented Aug 15, 2023

💛 Build succeeded, but was flaky

Failed CI Steps

Test Failures

Metrics [docs]

ESLint disabled line counts

Total ESLint disabled count

History

MadameSheema commented Jul 28, 2023 •

edited

Loading

gergoabraham left a comment •

edited

Loading