Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Failing test: Creates and activates a new ml rule - Detection rules, machine learning Creates and activates a new ml rule #80569

Closed
kibanamachine opened this issue Oct 14, 2020 · 33 comments
Closed

Failing test: Creates and activates a new ml rule - Detection rules, machine learning Creates and activates a new ml rule #80569

kibanamachine opened this issue Oct 14, 2020 · 33 comments
Assignees
@MadameSheema
Labels
failed-test A test failure on a tracked branch, potentially flaky-test Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. Team:SIEM

Comments

@kibanamachine
Copy link
Contributor

kibanamachine commented Oct 14, 2020 •
edited
Loading

A test failed on a tracked branch

AssertionError: Timed out retrying: expected 'Custom rules (3)' to deeply equal 'Custom rules (1)'
    at Context.eval (http://localhost:6111/__cypress/tests?p=cypress/integration/alerts_detection_rules_ml.spec.ts:13512:74)

First failure: Jenkins Build
@kibanamachine kibanamachine added the failed-test A test failure on a tracked branch, potentially flaky-test label Oct 14, 2020
@elasticmachine
Copy link
Contributor

Pinging @elastic/kibana-test-triage (failed-test)

@elasticmachine
Copy link
Contributor

Pinging @elastic/siem (Team:SIEM)

@rylnd
Copy link
Contributor

rylnd commented Oct 14, 2020

@MadameSheema I looked into this briefly; I was not able to reproduce the failure locally (tested on the 7.x branch) but I had a few thoughts/comments:

  1. Are these specs run in parallel? There is only one test in this spec and so I would not expect there to be extant rules (which is what Custom rules (3) indicates to me), which leads me to believe there is state leaking between these specs.
  2. The timeline is unclear to me; from the Test Result in the linked build, it appears that the spec passed once and failed once, presumably in that order, but both the order and the cause of the retry are unclear to me.

elastic_kibana_7_x__8805_Creates_and_activates_a_new_ml_rule__Jenkins_

@MadameSheema
Copy link
Member

@rylnd cypress tests are not executed in parallel. Right now the tests are configured in a way that if something fails on headless mode, the test is going to be re-executed 2 more times, what presumably means that we are not clearing the state of the test in a proper way.

@kibanamachine
Copy link
Contributor Author

New failure: Jenkins Build

@kibanamachine
Copy link
Contributor Author

New failure: Jenkins Build

@kibanamachine
Copy link
Contributor Author

New failure: Jenkins Build

@MadameSheema
Copy link
Member

These flakiness should be fixed with the following PR: #81040

@MindyRS MindyRS added the Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. label Oct 27, 2020
@kibanamachine
Copy link
Contributor Author

New failure: Jenkins Build

@kibanamachine kibanamachine reopened this Dec 8, 2020
@kibanamachine
Copy link
Contributor Author

New failure: Jenkins Build

@kibanamachine
Copy link
Contributor Author

New failure: Jenkins Build

1 similar comment
@kibanamachine
Copy link
Contributor Author

New failure: Jenkins Build

@patrykkopycinski
Copy link
Contributor

#85235

@kibanamachine kibanamachine reopened this Dec 10, 2020
@kibanamachine
Copy link
Contributor Author

New failure: Jenkins Build

@kibanamachine
Copy link
Contributor Author

New failure: Jenkins Build

@kibanamachine
Copy link
Contributor Author

New failure: Jenkins Build

@kibanamachine
Copy link
Contributor Author

New failure: Jenkins Build

@MadameSheema
Copy link
Member

Closing this issue since looks like was a problem with the kibana snapshot.

@kibanamachine kibanamachine reopened this Dec 12, 2020
@kibanamachine
Copy link
Contributor Author

New failure: Jenkins Build

@kibanamachine
Copy link
Contributor Author

New failure: Jenkins Build

@kibanamachine
Copy link
Contributor Author

New failure: Jenkins Build

@kibanamachine
Copy link
Contributor Author

New failure: Jenkins Build

@kibanamachine
Copy link
Contributor Author

New failure: Jenkins Build

@kibanamachine
Copy link
Contributor Author

New failure: Jenkins Build

@kibanamachine
Copy link
Contributor Author

New failure: Jenkins Build

@kibanamachine
Copy link
Contributor Author

New failure: Jenkins Build

@kibanamachine
Copy link
Contributor Author

New failure: Jenkins Build

@kibanamachine
Copy link
Contributor Author

New failure: Jenkins Build

@kibanamachine
Copy link
Contributor Author

New failure: Jenkins Build

@kibanamachine
Copy link
Contributor Author

New failure: Jenkins Build

@kibanamachine
Copy link
Contributor Author

New failure: Jenkins Build

@kibanamachine
Copy link
Contributor Author

New failure: Jenkins Build

@kibanamachine
Copy link
Contributor Author

New failure: Jenkins Build

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@MadameSheema MadameSheema

Labels
failed-test A test failure on a tracked branch, potentially flaky-test Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. Team:SIEM
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
7 participants
@rylnd @spalger @MindyRS @patrykkopycinski @elasticmachine @MadameSheema @kibanamachine