Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Flaky tests #7272

Closed
connorshea opened this issue May 8, 2019 · 14 comments
Closed

Flaky tests #7272

connorshea opened this issue May 8, 2019 · 14 comments
Labels
Type:Discussion Issues & PRs related to ongoing discussions

Comments

@connorshea
Copy link
Contributor

connorshea commented May 8, 2019 •
edited
Loading

This issue is for reporting any flaky tests people run into. Hopefully, if we start reporting flaky tests we can find common sources of flakiness and start getting rid of them.

Please comment with a link to the test case, the PR and/or commit the test was run on, and the Travis CI job where it failed if you run into a flaky test :)

Template:

- Commit/PR: LINK_TO_COMMIT_OR_PR 
- Test case: [`TEST_FILE_PATH:TEST_NAME`](https://github.com/salesagility/SuiteCRM/blob/COMMIT_SHA/TEST_FILE_PATH)
- Travis CI job: https://travis-ci.org/salesagility/SuiteCRM/jobs/JOB_ID
- Branch: BRANCH_NAME
- Details:

Inspired by the process described in https://samsaffron.com/archive/2019/05/15/tests-that-sometimes-fail
@connorshea
Copy link
Contributor Author

connorshea commented May 8, 2019 •
edited
Loading

@Dillon-Brown Dillon-Brown added Type: Bug Bugs within the core SuiteCRM codebase Priority:Moderate Issues & PRs that are minor; broken styling, cosmetic, warnings - there are practical workarounds labels May 10, 2019
@connorshea
Copy link
Contributor Author

connorshea commented Jun 7, 2019 •
edited
Loading

@connorshea
Copy link
Contributor Author

connorshea commented Jun 7, 2019 •
edited
Loading 

[Facebook\WebDriver\Exception\NoSuchElementException] no such element: Unable to locate element: {"method":"css selector","selector":".listViewBody"}

@connorshea
Copy link
Contributor Author

connorshea commented Jun 7, 2019 •
edited
Loading 

Element located either by name, CSS or XPath element with '.panel-heading' was not found.

@connorshea
Copy link
Contributor Author

connorshea commented Jun 7, 2019 •
edited
Loading 

CSS or XPath element with '#moduleBuilder' was not found.

@connorshea
Copy link
Contributor Author 

  [Facebook\WebDriver\Exception\NoSuchElementException] no such element: Unable to locate element: {"method":"css selector","selector":".listViewBody"}

@connorshea
Copy link
Contributor Author 

1) AccountsCest: Create an Account
 Test  tests/acceptance/modules/Accounts/AccountsCest.php:testScenarioCreateAccountChild
                                                                                                                                                                                                                                                                                                                                   
  [Facebook\WebDriver\Exception\NoSuchElementException] no such element: Unable to locate element: {"method":"css selector","selector":".listViewBody"}
  (Session info: headless chrome=75.0.3770.80)
  (Driver info: chromedriver=2.36.540471 (9c759b81a907e70363c6312294d30b6ccccc2752),platform=Linux 4.15.0-1028-gcp x86_64)  
                                                                                                                                                                                                                                                                                                                                   
Scenario Steps:
 11. $I->waitForElementVisible(".listViewBody",120) at tests/_support/Step/Acceptance/ListView.php:50
 10. $I->click("Accounts","#toolbar.desktop-toolbar  > ul.nav.navbar-nav > ...") at tests/_support/Step/Acceptance/NavigationBarTester.php:96
 9. $I->waitForElementVisible("#toolbar.desktop-toolbar  > ul.nav.navba...",120) at tests/_support/Step/Acceptance/NavigationBarTester.php:95
 8. $I->click("All","#toolbar.desktop-toolbar  > ul.nav.navbar-nav > li.top...") at tests/_support/Step/Acceptance/NavigationBarTester.php:93
 7. $I->wait(1) at tests/_support/Step/Acceptance/NavigationBarTester.php:92
 6. $I->waitForElementVisible("#toolbar.desktop-toolbar  > ul.nav.navbar...",30) at tests/_support/Step/Acceptance/NavigationBarTester.php:91

@connorshea
Copy link
Contributor Author

I think most of these should be fixed by #7417, except maybe the AORReportsCest failure. :D It's still useful to catalogue these if they happen in the future, though, so I don't know if we should close this issue when that PR is merged?

@Dillon-Brown Dillon-Brown added the Status:Fix Proposed A issue that has a PR related to it that provides a possible resolution label Jun 19, 2019
@connorshea
Copy link
Contributor Author

connorshea commented Jun 25, 2019 •
edited
Loading

NOTE: Fixed by #7473(?) Not fixed. Fixed by #7565.

Screenshot left by acceptance test: https://www.simpleupload.co.uk/uploads/YjbP0HomeCest.testCreateChartsDashlet.fail.png
image

It seems to be pretty rare.

Error message:

1) HomeCest: Create a chart dashlet on the dashboard
 Test  tests/acceptance/modules/Home/HomeCest.php:testCreateChartsDashlet
                                                                                                                                                                                                                                                              
  [Facebook\WebDriver\Exception\ElementNotVisibleException] element not visible
  (Session info: headless chrome=75.0.3770.100)
  (Driver info: chromedriver=2.36.540471 (9c759b81a907e70363c6312294d30b6ccccc2752),platform=Linux 4.4.0-101-generic x86_64)  
                                                                                                                                                                                                                                                              
Scenario Steps:
 18. $I->click("All Opportunities By Lead Source By Outcome") at tests/acceptance/modules/Home/HomeCest.php:52
 17. $I->click("#chartCategory") at tests/acceptance/modules/Home/HomeCest.php:51
 16. $I->waitForElementVisible("#chartCategory") at tests/acceptance/modules/Home/HomeCest.php:50
 15. $I->click("Add Dashlets","#tab-actions > .dropdown-menu") at tests/_support/Step/Acceptance/DetailView.php:26
 14. $I->waitForElementVisible("#tab-actions > .dropdown-menu") at tests/_support/Step/Acceptance/DetailView.php:24
 13. $I->click("ACTIONS","#tab-actions") at tests/_support/Step/Acceptance/DetailView.php:23
#1  /home/travis/build/salesagility/SuiteCRM/vendor/facebook/webdriver/lib/Exception/WebDriverException.php:110
#2  /home/travis/build/salesagility/SuiteCRM/vendor/facebook/webdriver/lib/Remote/HttpCommandExecutor.php:320
#3  /home/travis/build/salesagility/SuiteCRM/vendor/facebook/webdriver/lib/Remote/RemoteWebDriver.php:535
#4  /home/travis/build/salesagility/SuiteCRM/vendor/facebook/webdriver/lib/Remote/RemoteExecuteMethod.php:40
#5  /home/travis/build/salesagility/SuiteCRM/vendor/facebook/webdriver/lib/Remote/RemoteWebElement.php:82
#6  Codeception\Module\WebDriver->click
#7  /home/travis/build/salesagility/SuiteCRM/tests/_support/_generated/AcceptanceTesterActions.php:606
#8  /home/travis/build/salesagility/SuiteCRM/tests/acceptance/modules/Home/HomeCest.php:52
#9  HomeCest->testCreateChartsDashlet

@connorshea connorshea mentioned this issue Jun 25, 2019
Merged
6 tasks
@connorshea
Copy link
Contributor Author

connorshea commented Jul 2, 2019 •
edited
Loading

image

1) CasesCest: Create a case
 Test  tests/acceptance/modules/Cases/CasesCest.php:testScenarioCreateCase
                                                                                                                                                                                                                                                                                                                                     
  [Facebook\WebDriver\Exception\NoSuchElementException] no such element: Unable to locate element: {"method":"css selector","selector":".detail-view"}
  (Session info: headless chrome=75.0.3770.100)
  (Driver info: chromedriver=2.36.540471 (9c759b81a907e70363c6312294d30b6ccccc2752),platform=Linux 4.4.0-101-generic x86_64)  
                                                                                                                                                                                                                                                                                                                                     
Scenario Steps:
 55. $I->waitForElementVisible(".detail-view") at tests/_support/Step/Acceptance/DetailView.php:35
 54. $I->click("Save") at tests/_support/Step/Acceptance/EditView.php:22
 53. $I->executeJS("window.scrollTo(0,0); return true;") at tests/_support/Step/Acceptance/EditView.php:21
 52. $I->seeElement("#assigned_user_name") at tests/_support/Step/Acceptance/Cases.php:33
 51. $I->executeJS("tinyMCE.activeEditor.setContent("TinyMCE Content Test");") at tests/_support/Step/Acceptance/Cases.php:31
 50. $I->selectOption("#status","Duplicate") at tests/_support/Step/Acceptance/Cases.php:29
#1  /home/travis/build/salesagility/SuiteCRM/vendor/facebook/webdriver/lib/Exception/WebDriverException.php:102
#2  /home/travis/build/salesagility/SuiteCRM/vendor/facebook/webdriver/lib/Remote/HttpCommandExecutor.php:320
#3  /home/travis/build/salesagility/SuiteCRM/vendor/facebook/webdriver/lib/Remote/RemoteWebDriver.php:535
#4  /home/travis/build/salesagility/SuiteCRM/vendor/facebook/webdriver/lib/Remote/RemoteWebDriver.php:176
#5  /home/travis/build/salesagility/SuiteCRM/vendor/facebook/webdriver/lib/WebDriverExpectedCondition.php:186
#6  Facebook\WebDriver\WebDriverExpectedCondition::Facebook\WebDriver\{closure}
#7  /home/travis/build/salesagility/SuiteCRM/vendor/facebook/webdriver/lib/WebDriverWait.php:67
#8  Codeception\Module\WebDriver->waitForElementVisible
#9  /home/travis/build/salesagility/SuiteCRM/tests/_support/_generated/AcceptanceTesterActions.php:2386
#10 /home/travis/build/salesagility/SuiteCRM/tests/_support/Step/Acceptance/DetailView.php:35

Note: I tried reproducing this locally but I wasn't able to, even after a few dozen test runs. I suppose the failure could be reliant on some sort of state in Travis, or reliant on some tests that happen to be run before the CasesCest and cause state poisoning.

@connorshea
Copy link
Contributor Author

Another instance of the CasesCest failure, with the same details except it was on PHP 7.2 and this is the screenshot:

image

@connorshea connorshea mentioned this issue Jul 11, 2019
Merged
6 tasks
@pgorod
Copy link
Contributor

pgorod commented Jul 22, 2019

Every time I see this Issue I have a vision of this in my mind
image

@connorshea connorshea mentioned this issue Jul 31, 2019
Closed
@Dillon-Brown Dillon-Brown added Type:Discussion Issues & PRs related to ongoing discussions and removed Status:Fix Proposed A issue that has a PR related to it that provides a possible resolution Priority:Moderate Issues & PRs that are minor; broken styling, cosmetic, warnings - there are practical workarounds Type: Bug Bugs within the core SuiteCRM codebase labels Aug 2, 2019
@samus-aran samus-aran mentioned this issue Aug 9, 2019
Closed
@connorshea
Copy link
Contributor Author

@Dillon-Brown are you aware of any tests that are still flaky at the moment? I haven't seen any recently.

@Dillon-Brown
Copy link
Contributor

@connorshea Seems to be fairly stable, I also haven't seen any recently.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Type:Discussion Issues & PRs related to ongoing discussions
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@connorshea @pgorod @Dillon-Brown @SuiteBot