Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Failing test: Security Solution Cypress.x-pack/test/security_solution_cypress/cypress/e2e/detection_response/detection_engine/detection_alerts/assignments/assignments_serverless_complete·cy·ts - Alert user assignment - Serverless Complete Authorization / RBAC "before each" hook for "users with editing privileges should be able to update assignees" "before each" hook for "users with editing privileges should be able to update assignees" #177573

Closed
kibanamachine opened this issue Feb 22, 2024 · 8 comments
Closed

Failing test: Security Solution Cypress.x-pack/test/security_solution_cypress/cypress/e2e/detection_response/detection_engine/detection_alerts/assignments/assignments_serverless_complete·cy·ts - Alert user assignment - Serverless Complete Authorization / RBAC "before each" hook for "users with editing privileges should be able to update assignees" "before each" hook for "users with editing privileges should be able to update assignees" #177573

kibanamachine opened this issue Feb 22, 2024 · 8 comments
Assignees
@e40pud
Labels
failed-test A test failure on a tracked branch, potentially flaky-test Team:Detection Engine Security Solution Detection Engine Area Team:Detections and Resp Security Detection Response Team Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. triage_needed

Comments

@kibanamachine
Copy link
Contributor

kibanamachine commented Feb 22, 2024 •
edited
Loading

A test failed on a tracked branch

CypressError: `cy.request()` failed on:

https://system_indices_superuser:changeme@localhost:9214/.lists-*,.items-*,.alerts-security.alerts-*/_delete_by_query?conflicts=proceed&scroll_size=10000&refresh

The response we received from your web server was:

  > 503: Service Unavailable

This was considered a failure because the status code was not `2xx` or `3xx`.

If you do not want status codes to cause failures pass the option: `failOnStatusCode: false`

-----------------------------------------------------------

The request we sent was:

Method: POST
URL: https://system_indices_superuser:changeme@localhost:9214/.lists-*,.items-*,.alerts-security.alerts-*/_delete_by_query?conflicts=proceed&scroll_size=10000&refresh
Headers: {
  "Connection": "keep-alive",
  "kbn-xsrf": "cypress-creds",
  "x-elastic-internal-origin": "security-solution",
  "elastic-api-version": [
    "2023-10-31"
  ],
  "user-agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Cypress/13.6.3 Chrome/114.0.5735.289 Electron/25.8.4 Safari/537.36",
  "accept": "*/*",
  "cookie": "sid=Fe26.2**eea52ef222a0615627f5d45fd3c80bc4cf00d242eabf38731b350903aad61655*4scpB8awgREhZSWfcwll8A*rU1AsCqGNTmwERa_K56hhkNWh77M5F07kSMk7_CBfM3hEqYQFkHkFKpjMh-S6Y9ifGnkZjOblqbia9NAoreqFmPLkO7taRjh0rhbgUXEeWmLUQUQTYPHwENFyi1qvGpjO9jQUxt2t5-wwfPlPSBTTACallo9SAwk3crdZ5vSDVp6Q-hSoRI2scwNRgjBqrswqXZgXAu_W_95pQIdeMJBFe1XtzMlGlL6nnbrIZ29VBvyA8w-mg056fRDvP94OeY8**b5a75da54051ed22fed2799c57101724b31595792bb240a733732b176ad9376a*lHoDlv-dK5enK0mInKIo0vnk100QnOQ8vGR_lnigXmY",
  "authorization": "Basic c3lzdGVtX2luZGljZXNfc3VwZXJ1c2VyOmNoYW5nZW1l",
  "accept-encoding": "gzip, deflate",
  "content-type": "application/json",
  "content-length": 26
}
Body: {"query":{"match_all":{}}}

-----------------------------------------------------------

The response we got was:

Status: 503 - Service Unavailable
Headers: {
  "x-elastic-product": "Elasticsearch",
  "elastic-api-version": "2023-10-31",
  "content-type": "application/json",
  "content-length": "433"
}
Body: {
  "error": {
    "root_cause": [],
    "type": "search_phase_execution_exception",
    "reason": "start",
    "phase": "can_match",
    "grouped": true,
    "failed_shards": [],
    "caused_by": {
      "type": "search_phase_execution_exception",
      "reason": "Search rejected due to missing shards [[.ds-.lists-default-2024.02.22-000001][0]]. Consider using `allow_partial_search_results` setting to bypass this error.",
      "phase": "can_match",
      "grouped": true,
      "failed_shards": []
    }
  },
  "status": 503
}


https://on.cypress.io/request

Because this error occurred during a `before each` hook we are skipping the remaining tests in the current suite: `Alert user assignment - Ser...`
    at <unknown> (http://localhost:5652/__cypress/runner/cypress_runner.js:133030:72)
    at tryCatcher (http://localhost:5652/__cypress/runner/cypress_runner.js:1807:23)
    at Promise._settlePromiseFromHandler (http://localhost:5652/__cypress/runner/cypress_runner.js:1519:31)
    at Promise._settlePromise (http://localhost:5652/__cypress/runner/cypress_runner.js:1576:18)
    at Promise._settlePromise0 (http://localhost:5652/__cypress/runner/cypress_runner.js:1621:10)
    at Promise._settlePromises (http://localhost:5652/__cypress/runner/cypress_runner.js:1701:18)
    at _drainQueueStep (http://localhost:5652/__cypress/runner/cypress_runner.js:2407:12)
    at _drainQueue (http://localhost:5652/__cypress/runner/cypress_runner.js:2400:9)
    at Async._drainQueues (http://localhost:5652/__cypress/runner/cypress_runner.js:2416:5)
    at Async.drainQueues (http://localhost:5652/__cypress/runner/cypress_runner.js:2286:14)
From Your Spec Code:
    at rootRequest (webpack:///./tasks/api_calls/common.ts:29:5)
    at deleteAllDocuments (webpack:///./tasks/api_calls/elasticsearch.ts:28:14)
    at deleteAlertsAndRules (webpack:///./tasks/api_calls/common.ts:71:22)
    at Context.eval (webpack:///./e2e/detection_response/detection_engine/detection_alerts/assignments/assignments_serverless_complete.cy.ts:55:27)

First failure: CI Build - main
@kibanamachine kibanamachine added the failed-test A test failure on a tracked branch, potentially flaky-test label Feb 22, 2024
@botelastic botelastic bot added the needs-team Issues missing a team label label Feb 22, 2024
@kibanamachine kibanamachine added the Team:Detection Engine Security Solution Detection Engine Area label Feb 22, 2024
@elasticmachine
Copy link
Contributor

Pinging @elastic/security-detection-engine (Team:Detection Engine)

@botelastic botelastic bot removed the needs-team Issues missing a team label label Feb 22, 2024
@banderror banderror added triage_needed Team:Detections and Resp Security Detection Response Team Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. labels Feb 22, 2024
@elasticmachine
Copy link
Contributor

Pinging @elastic/security-detections-response (Team:Detections and Resp)

@elasticmachine
Copy link
Contributor

Pinging @elastic/security-solution (Team: SecuritySolution)

@kibanamachine
Copy link
Contributor Author

New failure: CI Build - main

@yctercero yctercero assigned e40pud and unassigned yctercero Mar 3, 2024
@kibanamachine
Copy link
Contributor Author

New failure: CI Build - 8.13

@e40pud e40pud mentioned this issue Mar 5, 2024
Merged
1 task
@kibanamachine
Copy link
Contributor Author

New failure: CI Build - main

kibanamachine added a commit that referenced this issue Mar 5, 2024
@kibanamachine
skip failing test suite (#177573)
3a695a7
kibanamachine added a commit that referenced this issue Mar 5, 2024
@kibanamachine
skip failing test suite (#177573)
55edda6
@kibanamachine
Copy link
Contributor Author

Skipped

8.13: 3a695a7
main: 55edda6

e40pud added a commit that referenced this issue Mar 6, 2024
@e40pud @kibanamachine
Flaky alert assignment tests (#176930)
31cd917 
## Summary

Addresses:
* #176529
* #172557
* #177573
* #173429

Fix flaky alert assignments tests. I split assignments tests into two
groups: tests with one assignee available and tests with multiple
assignees.

Right now there is a flakiness in tests with multiple assignees. Most
probably it is happening because we do multiple login calls in a row to
make sure we activate different users to make them available for
assignments:

```
// Login into accounts so that they got activated and visible in user profiles list
       login(ROLES.t1_analyst);
       login(ROLES.t2_analyst);
       login(ROLES.t3_analyst);
       login(ROLES.soc_manager);
       login(ROLES.detections_admin);
       login(ROLES.platform_engineer);
```

These tests are tend to be flaky and it is possible that kibana
operations team will skip those. To make sure that we run basic cypress
verification of alert assignments feature we decided to add tests with
only one assignee available (current user) which allows us to avoid
multiple consecutive login calls.

Also, as part of these changes I removed unnecessary logins and
un-skipped #176529

## NOTE

After discussing these failure with the team, we decided to remove tests
which are covered by the integration and unit tests. While fixing the
flakiness we realised that we do unnecessary work trying to fight the
internal errors within elastic search on serverless when we do multiple
user logins in a row. Instead we will rely on:
* integration tests coverage of API related functionality including RBAC
* unit tests coverage of all assignments UI components
* cypress tests coverage of basic UI interaction with the alert
assignments with only one user available for the assignments

cc @yctercero 

### Checklist

Delete any items that are not applicable to this PR.

- [x] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
- [ESS 50
times](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/5181)
- [Serverless 97
times](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/5182)

---------

Co-authored-by: Kibana Machine <[email protected]>
@e40pud
Copy link
Contributor

e40pud commented Mar 6, 2024

We removed these tests in #176930 in favour of combination of basic e2e and extensive integration + unit tests.

@e40pud e40pud closed this as completed Mar 6, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@e40pud e40pud

Labels
failed-test A test failure on a tracked branch, potentially flaky-test Team:Detection Engine Security Solution Detection Engine Area Team:Detections and Resp Security Detection Response Team Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. triage_needed
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@e40pud @banderror @yctercero @elasticmachine @kibanamachine