Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Failing test: Security Solution Cypress.x-pack/test/security_solution_cypress/cypress/e2e/detection_response/detection_engine/detection_alerts/assignments/assignments_serverless_essentials·cy·ts - Alert user assignment - Serverless Essentials Authorization / RBAC "before each" hook for "users with editing privileges should be able to update assignees" "before each" hook for "users with editing privileges should be able to update assignees" #176529

Closed
kibanamachine opened this issue Feb 8, 2024 · 6 comments
Assignees
Labels
failed-test A test failure on a tracked branch, potentially flaky-test Team:Detection Engine Security Solution Detection Engine Area Team:Detections and Resp Security Detection Response Team Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. triage_needed

Comments

@kibanamachine
Copy link
Contributor

kibanamachine commented Feb 8, 2024

A test failed on a tracked branch

CypressError: `cy.request()` failed on:

https://system_indices_superuser:changeme@localhost:9298/.lists-*,.items-*,.alerts-security.alerts-*/_delete_by_query?conflicts=proceed&scroll_size=10000&refresh

The response we received from your web server was:

  > 503: Service Unavailable

This was considered a failure because the status code was not `2xx` or `3xx`.

If you do not want status codes to cause failures pass the option: `failOnStatusCode: false`

-----------------------------------------------------------

The request we sent was:

Method: POST
URL: https://system_indices_superuser:changeme@localhost:9298/.lists-*,.items-*,.alerts-security.alerts-*/_delete_by_query?conflicts=proceed&scroll_size=10000&refresh
Headers: {
  "Connection": "keep-alive",
  "kbn-xsrf": "cypress-creds",
  "x-elastic-internal-origin": "security-solution",
  "elastic-api-version": [
    "2023-10-31"
  ],
  "user-agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Cypress/13.6.3 Chrome/114.0.5735.289 Electron/25.8.4 Safari/537.36",
  "accept": "*/*",
  "cookie": "sid=Fe26.2**44c19ff38c721768ed23b3c8d8aad6df08e3aec006e293a8a6ee0045829069e7*DR7_XT7zzj2e6p41xoFQRQ*IGAK-WIdEncgrV7-QutdgL8iPgz8psUFHswJOUP32LPWZ-2SYNFmQq5Qlvl6I8ziezeYNKnbBzRQ2vlGOCvn6syeob3kLqb1yBCFmJsQhrKBQ0JB2WPowfMjeEt5gW69knnax0oSgUryccND3uWD6Mgx_OOL74Kz8equ_m2SBxIowUIJLgiy6lxKwrWhd74df_zoMqby7n4moJ7h0ND02ab-IOiDwcLQ52d06_gcKFGS9ppfC-6016pWcVKbXHDl**c01b7e316a86275060ec3660d640ae508d3955f4482570e7497d2bdb43fb0b24*RYf7kdiReP14yehFvxU73cm4xuSIdAtnS448JbzBcnY",
  "authorization": "Basic c3lzdGVtX2luZGljZXNfc3VwZXJ1c2VyOmNoYW5nZW1l",
  "accept-encoding": "gzip, deflate",
  "content-type": "application/json",
  "content-length": 26
}
Body: {"query":{"match_all":{}}}

-----------------------------------------------------------

The response we got was:

Status: 503 - Service Unavailable
Headers: {
  "x-elastic-product": "Elasticsearch",
  "elastic-api-version": "2023-10-31",
  "content-type": "application/json",
  "content-length": "433"
}
Body: {
  "error": {
    "root_cause": [],
    "type": "search_phase_execution_exception",
    "reason": "start",
    "phase": "can_match",
    "grouped": true,
    "failed_shards": [],
    "caused_by": {
      "type": "search_phase_execution_exception",
      "reason": "Search rejected due to missing shards [[.ds-.lists-default-2024.02.08-000001][0]]. Consider using `allow_partial_search_results` setting to bypass this error.",
      "phase": "can_match",
      "grouped": true,
      "failed_shards": []
    }
  },
  "status": 503
}


https://on.cypress.io/request

Because this error occurred during a `before each` hook we are skipping the remaining tests in the current suite: `Alert user assignment - Ser...`
    at <unknown> (http://localhost:5676/__cypress/runner/cypress_runner.js:133030:72)
    at tryCatcher (http://localhost:5676/__cypress/runner/cypress_runner.js:1807:23)
    at Promise._settlePromiseFromHandler (http://localhost:5676/__cypress/runner/cypress_runner.js:1519:31)
    at Promise._settlePromise (http://localhost:5676/__cypress/runner/cypress_runner.js:1576:18)
    at Promise._settlePromise0 (http://localhost:5676/__cypress/runner/cypress_runner.js:1621:10)
    at Promise._settlePromises (http://localhost:5676/__cypress/runner/cypress_runner.js:1701:18)
    at _drainQueueStep (http://localhost:5676/__cypress/runner/cypress_runner.js:2407:12)
    at _drainQueue (http://localhost:5676/__cypress/runner/cypress_runner.js:2400:9)
    at Async._drainQueues (http://localhost:5676/__cypress/runner/cypress_runner.js:2416:5)
    at Async.drainQueues (http://localhost:5676/__cypress/runner/cypress_runner.js:2286:14)
From Your Spec Code:
    at rootRequest (webpack:///./tasks/api_calls/common.ts:29:5)
    at deleteAllDocuments (webpack:///./tasks/api_calls/elasticsearch.ts:28:14)
    at deleteAlertsAndRules (webpack:///./tasks/api_calls/common.ts:71:22)
    at Context.eval (webpack:///./e2e/detection_response/detection_engine/detection_alerts/assignments/assignments_serverless_essentials.cy.ts:55:27)

First failure: CI Build - main

@kibanamachine kibanamachine added the failed-test A test failure on a tracked branch, potentially flaky-test label Feb 8, 2024
@botelastic botelastic bot added the needs-team Issues missing a team label label Feb 8, 2024
@kibanamachine kibanamachine added the Team:Detection Engine Security Solution Detection Engine Area label Feb 8, 2024
@elasticmachine
Copy link
Contributor

Pinging @elastic/security-detection-engine (Team:Detection Engine)

@botelastic botelastic bot removed the needs-team Issues missing a team label label Feb 8, 2024
@banderror banderror added triage_needed Team:Detections and Resp Security Detection Response Team Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. labels Feb 9, 2024
@elasticmachine
Copy link
Contributor

Pinging @elastic/security-detections-response (Team:Detections and Resp)

@elasticmachine
Copy link
Contributor

Pinging @elastic/security-solution (Team: SecuritySolution)

@kibanamachine
Copy link
Contributor Author

New failure: CI Build - main

@mistic
Copy link
Member

mistic commented Feb 13, 2024

Skipped.

main: 557ab3c

CoenWarmer pushed a commit to CoenWarmer/kibana that referenced this issue Feb 15, 2024
@yctercero yctercero assigned e40pud and unassigned yctercero Feb 18, 2024
fkanout pushed a commit to fkanout/kibana that referenced this issue Mar 4, 2024
e40pud added a commit that referenced this issue Mar 6, 2024
## Summary

Addresses:
* #176529
* #172557
* #177573
* #173429

Fix flaky alert assignments tests. I split assignments tests into two
groups: tests with one assignee available and tests with multiple
assignees.

Right now there is a flakiness in tests with multiple assignees. Most
probably it is happening because we do multiple login calls in a row to
make sure we activate different users to make them available for
assignments:

```
// Login into accounts so that they got activated and visible in user profiles list
       login(ROLES.t1_analyst);
       login(ROLES.t2_analyst);
       login(ROLES.t3_analyst);
       login(ROLES.soc_manager);
       login(ROLES.detections_admin);
       login(ROLES.platform_engineer);
```

These tests are tend to be flaky and it is possible that kibana
operations team will skip those. To make sure that we run basic cypress
verification of alert assignments feature we decided to add tests with
only one assignee available (current user) which allows us to avoid
multiple consecutive login calls.

Also, as part of these changes I removed unnecessary logins and
un-skipped #176529

## NOTE

After discussing these failure with the team, we decided to remove tests
which are covered by the integration and unit tests. While fixing the
flakiness we realised that we do unnecessary work trying to fight the
internal errors within elastic search on serverless when we do multiple
user logins in a row. Instead we will rely on:
* integration tests coverage of API related functionality including RBAC
* unit tests coverage of all assignments UI components
* cypress tests coverage of basic UI interaction with the alert
assignments with only one user available for the assignments

cc @yctercero 

### Checklist

Delete any items that are not applicable to this PR.

- [x] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
- [ESS 50
times](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/5181)
- [Serverless 97
times](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/5182)

---------

Co-authored-by: Kibana Machine <[email protected]>
@e40pud
Copy link
Contributor

e40pud commented Mar 6, 2024

We removed these tests in #176930 in favour of combination of basic e2e and extensive integration + unit tests.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
failed-test A test failure on a tracked branch, potentially flaky-test Team:Detection Engine Security Solution Detection Engine Area Team:Detections and Resp Security Detection Response Team Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. triage_needed
Projects
None yet
Development

No branches or pull requests

6 participants