Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Failing test: Security Solution Cypress.x-pack/test/security_solution_cypress/cypress/e2e/detection_response/detection_engine/detection_alerts/assignments/assignments·cy·ts - Alert user assignment - ESS & Serverless Basic rendering "before each" hook for "alert with no assignees in alerts table" "before each" hook for "alert with no assignees in alerts table" #173429

Closed
kibanamachine opened this issue Dec 14, 2023 · 17 comments
Assignees
Labels
failed-test A test failure on a tracked branch, potentially flaky-test Team:Detection Engine Security Solution Detection Engine Area Team:Detections and Resp Security Detection Response Team Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. triage_needed

Comments

@kibanamachine
Copy link
Contributor

kibanamachine commented Dec 14, 2023

A test failed on a tracked branch

CypressError: `cy.request()` failed on:

https://system_indices_superuser:changeme@localhost:9294/.lists-*,.items-*,.alerts-security.alerts-*/_delete_by_query?conflicts=proceed&scroll_size=10000&refresh

The response we received from your web server was:

  > 503: Service Unavailable

This was considered a failure because the status code was not `2xx` or `3xx`.

If you do not want status codes to cause failures pass the option: `failOnStatusCode: false`

-----------------------------------------------------------

The request we sent was:

Method: POST
URL: https://system_indices_superuser:changeme@localhost:9294/.lists-*,.items-*,.alerts-security.alerts-*/_delete_by_query?conflicts=proceed&scroll_size=10000&refresh
Headers: {
  "Connection": "keep-alive",
  "kbn-xsrf": "cypress-creds",
  "x-elastic-internal-origin": "security-solution",
  "elastic-api-version": [
    "2023-10-31"
  ],
  "user-agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Cypress/13.3.0 Chrome/114.0.5735.289 Electron/25.8.1 Safari/537.36",
  "accept": "*/*",
  "cookie": "sid=Fe26.2**25d0833c7fb417728948d6237005e42e364112bd22c86f7953e89f83cf40f7f7*xKCuD9jt0l5eCeiJvlwktw*U3SncmkaT4Kqi9mdE0Sg91OKLrzsQhEDhAMGuF8W2pYT8LZs64hAkP5BHg_WPFEnjJFcv2ipmuw7XRmLJE1_fW3cp3w0FOgUE-sK10OBltGhjomSgbkPoiKR88A7o9Jvubj3V8EuKQtP0Kq5a-4yAPZCA6iB37VQxSCZRw6cMHg8RiOOsl5OY3BeI7sU_qGnnrtnZF3XsYIVQBYibJT5QYZsDh2pgl4KQuvOymuLe-UfaUOZ8XZacSC7nr1WvYMh**575e20ad84b9d2957ed5bfb7d459476add5203020432edca0cc767a0bb7574ca*7XRoJUZZYNwbddfOKjRdQK4_eVJxvcr4azGY4tlAIHo",
  "authorization": "Basic c3lzdGVtX2luZGljZXNfc3VwZXJ1c2VyOmNoYW5nZW1l",
  "accept-encoding": "gzip, deflate",
  "content-type": "application/json",
  "content-length": 26
}
Body: {"query":{"match_all":{}}}

-----------------------------------------------------------

The response we got was:

Status: 503 - Service Unavailable
Headers: {
  "x-elastic-product": "Elasticsearch",
  "elastic-api-version": "2023-10-31",
  "content-type": "application/json",
  "content-length": "433"
}
Body: {
  "error": {
    "root_cause": [],
    "type": "search_phase_execution_exception",
    "reason": "start",
    "phase": "can_match",
    "grouped": true,
    "failed_shards": [],
    "caused_by": {
      "type": "search_phase_execution_exception",
      "reason": "Search rejected due to missing shards [[.ds-.lists-default-2023.12.14-000001][0]]. Consider using `allow_partial_search_results` setting to bypass this error.",
      "phase": "can_match",
      "grouped": true,
      "failed_shards": []
    }
  },
  "status": 503
}


https://on.cypress.io/request

Because this error occurred during a `before each` hook we are skipping the remaining tests in the current suite: `Alert user assignment - ESS...`
    at <unknown> (http://localhost:5673/__cypress/runner/cypress_runner.js:133043:72)
    at tryCatcher (http://localhost:5673/__cypress/runner/cypress_runner.js:1807:23)
    at Promise._settlePromiseFromHandler (http://localhost:5673/__cypress/runner/cypress_runner.js:1519:31)
    at Promise._settlePromise (http://localhost:5673/__cypress/runner/cypress_runner.js:1576:18)
    at Promise._settlePromise0 (http://localhost:5673/__cypress/runner/cypress_runner.js:1621:10)
    at Promise._settlePromises (http://localhost:5673/__cypress/runner/cypress_runner.js:1701:18)
    at _drainQueueStep (http://localhost:5673/__cypress/runner/cypress_runner.js:2407:12)
    at _drainQueue (http://localhost:5673/__cypress/runner/cypress_runner.js:2400:9)
    at Async._drainQueues (http://localhost:5673/__cypress/runner/cypress_runner.js:2416:5)
    at Async.drainQueues (http://localhost:5673/__cypress/runner/cypress_runner.js:2286:14)
From Your Spec Code:
    at rootRequest (webpack:///./tasks/api_calls/common.ts:29:5)
    at deleteAllDocuments (webpack:///./tasks/api_calls/elasticsearch.ts:28:14)
    at deleteAlertsAndRules (webpack:///./tasks/api_calls/common.ts:71:22)
    at Context.eval (webpack:///./e2e/detection_response/detection_engine/detection_alerts/assignments/assignments.cy.ts:64:25)

First failure: CI Build - main

@kibanamachine kibanamachine added the failed-test A test failure on a tracked branch, potentially flaky-test label Dec 14, 2023
@botelastic botelastic bot added the needs-team Issues missing a team label label Dec 14, 2023
@banderror banderror added Team:Detections and Resp Security Detection Response Team Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. Team:Detection Engine Security Solution Detection Engine Area labels Dec 15, 2023
@elasticmachine
Copy link
Contributor

Pinging @elastic/security-detections-response (Team:Detections and Resp)

@elasticmachine
Copy link
Contributor

Pinging @elastic/security-solution (Team: SecuritySolution)

@botelastic botelastic bot removed the needs-team Issues missing a team label label Dec 15, 2023
@banderror banderror added triage_needed needs-team Issues missing a team label labels Dec 15, 2023
@botelastic botelastic bot removed the needs-team Issues missing a team label label Dec 15, 2023
@yctercero yctercero removed their assignment Jan 2, 2024
@kibanamachine
Copy link
Contributor Author

New failure: CI Build - main

@kibanamachine
Copy link
Contributor Author

New failure: CI Build - main

@MadameSheema
Copy link
Member

/skip

@kibanamachine
Copy link
Contributor Author

Skipped

main: 1865d4d

e40pud added a commit to e40pud/kibana that referenced this issue Jan 31, 2024
@e40pud e40pud mentioned this issue Jan 31, 2024
1 task
e40pud added a commit that referenced this issue Feb 7, 2024
## Summary

These changes fix flaky alert assignments cypress tests:
- #173429
- #172520
- #172557

@MadameSheema mentioned that there were a lot of changes around login
functionality recently which could have fixed original issues. We agreed
that we will run a flaky test runner and if everything works fine we
would un-skip tests and monitor them after the merge.

### Checklist

Delete any items that are not applicable to this PR.

- [x] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
- [ESS 97
times](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/5064)
- [Serverless 97
times](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/5052)
@e40pud e40pud self-assigned this Feb 7, 2024
@e40pud
Copy link
Contributor

e40pud commented Feb 7, 2024

We enabled these tests in #175965 and will be monitoring for new flakes.

@kibanamachine kibanamachine reopened this Feb 12, 2024
@kibanamachine
Copy link
Contributor Author

New failure: CI Build - main

@yctercero
Copy link
Contributor

@e40pud would you mind checking if it's the same issue occurring? Should we see if we can move some of these into unit tests?

e40pud added a commit to e40pud/kibana that referenced this issue Feb 12, 2024
@e40pud
Copy link
Contributor

e40pud commented Feb 12, 2024

@yctercero yeah looks like it is the same issue. It happens in "before each" section, which means we would need to move all of them into unit tests.

I want to try to change the way we activate necessary accounts - instead of just login to all of them one by one, I will login and load the main security page. Here is the draft PR, once the flaky test runner passed successfully I will convert it into ready to review PR #176689

CoenWarmer pushed a commit to CoenWarmer/kibana that referenced this issue Feb 15, 2024
## Summary

These changes fix flaky alert assignments cypress tests:
- elastic#173429
- elastic#172520
- elastic#172557

@MadameSheema mentioned that there were a lot of changes around login
functionality recently which could have fixed original issues. We agreed
that we will run a flaky test runner and if everything works fine we
would un-skip tests and monitor them after the merge.

### Checklist

Delete any items that are not applicable to this PR.

- [x] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
- [ESS 97
times](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/5064)
- [Serverless 97
times](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/5052)
@kibanamachine
Copy link
Contributor Author

New failure: CI Build - main

@kibanamachine
Copy link
Contributor Author

New failure: CI Build - main

@yctercero
Copy link
Contributor

@e40pud is there a way to mock out these profile calls?

fkanout pushed a commit to fkanout/kibana that referenced this issue Mar 4, 2024
## Summary

These changes fix flaky alert assignments cypress tests:
- elastic#173429
- elastic#172520
- elastic#172557

@MadameSheema mentioned that there were a lot of changes around login
functionality recently which could have fixed original issues. We agreed
that we will run a flaky test runner and if everything works fine we
would un-skip tests and monitor them after the merge.

### Checklist

Delete any items that are not applicable to this PR.

- [x] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
- [ESS 97
times](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/5064)
- [Serverless 97
times](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/5052)
@kibanamachine
Copy link
Contributor Author

New failure: CI Build - 8.13

@kibanamachine
Copy link
Contributor Author

New failure: CI Build - main

mistic added a commit that referenced this issue Mar 5, 2024
mistic added a commit that referenced this issue Mar 5, 2024
@mistic
Copy link
Member

mistic commented Mar 5, 2024

Skipped.

main: f5bc6fb
8.13: 5a93652

e40pud added a commit that referenced this issue Mar 6, 2024
## Summary

Addresses:
* #176529
* #172557
* #177573
* #173429

Fix flaky alert assignments tests. I split assignments tests into two
groups: tests with one assignee available and tests with multiple
assignees.

Right now there is a flakiness in tests with multiple assignees. Most
probably it is happening because we do multiple login calls in a row to
make sure we activate different users to make them available for
assignments:

```
// Login into accounts so that they got activated and visible in user profiles list
       login(ROLES.t1_analyst);
       login(ROLES.t2_analyst);
       login(ROLES.t3_analyst);
       login(ROLES.soc_manager);
       login(ROLES.detections_admin);
       login(ROLES.platform_engineer);
```

These tests are tend to be flaky and it is possible that kibana
operations team will skip those. To make sure that we run basic cypress
verification of alert assignments feature we decided to add tests with
only one assignee available (current user) which allows us to avoid
multiple consecutive login calls.

Also, as part of these changes I removed unnecessary logins and
un-skipped #176529

## NOTE

After discussing these failure with the team, we decided to remove tests
which are covered by the integration and unit tests. While fixing the
flakiness we realised that we do unnecessary work trying to fight the
internal errors within elastic search on serverless when we do multiple
user logins in a row. Instead we will rely on:
* integration tests coverage of API related functionality including RBAC
* unit tests coverage of all assignments UI components
* cypress tests coverage of basic UI interaction with the alert
assignments with only one user available for the assignments

cc @yctercero 

### Checklist

Delete any items that are not applicable to this PR.

- [x] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
- [ESS 50
times](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/5181)
- [Serverless 97
times](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/5182)

---------

Co-authored-by: Kibana Machine <[email protected]>
@e40pud
Copy link
Contributor

e40pud commented Mar 6, 2024

We refactored and enabled these tests in #176930.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
failed-test A test failure on a tracked branch, potentially flaky-test Team:Detection Engine Security Solution Detection Engine Area Team:Detections and Resp Security Detection Response Team Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. triage_needed
Projects
None yet
Development

No branches or pull requests

7 participants