Functional test runner just stops in certain cases

[liza-mae]

Recently I am seeing the functional test runner just stop and there is no indication as to why. It is difficult to reproduce and it is blocking tests from running to completion. There is no summary of what already ran, it just stops and nothing else is printed. Other errors are thrown before and tests continued, so not sure what is going on - it occurred for me locally a couple times too but is not consistent.

@spalger can you help?

See:
https://internal-ci.elastic.co/view/Stack%20Tests/job/elastic+estf-cloud-kibana+master+multijob-selenium/1/consoleText

Tests are being run with command: node scripts/functional_test_runner with --debug option, not sure if there is more logging we can obtain, if it occurs again.

[elasticmachine]

Pinging @elastic/kibana-operations

[liza-mae]

This problem was opened around the time of 6.6 release back in Jan, then it went away and I have not seen again until now when testing 7.3. Looks like we have more debug messages available, so far I have been able to reproduce this after several attempts.

│ debg TestSubjects.exists(dashboardPanelHeading-APie)
           │ debg Find.existsByDisplayedByCssSelector('[data-test-subj~="dashboardPanelHeading-APie"]') with timeout=120000
           │ERROR UNHANDLED PROMISE REJECTION
           │ERROR Error: retry.try timeout: TimeoutError: Waiting for element to be located By(css selector, .euiTableRow)
           │      Wait timed out after 122234ms
           │          at /home/liza/TESTING/k73/node_modules/selenium-webdriver/lib/webdriver.js:834:17
           │          at process._tickCallback (internal/process/next_tick.js:68:7)
           │          at lastError (/home/liza/TESTING/k73/test/common/services/retry/retry_for_success.ts:28:9)
           │          at onFailure (/home/liza/TESTING/k73/test/common/services/retry/retry_for_success.ts:68:13)

[elasticmachine]

Pinging @elastic/kibana-test-triage

[tylersmalley]

How often is this happening?

[liza-mae]

Hard to say, some releases it would happen more often than others. It is not cloud specific though, I was able to reproduce it running node functional_tests_server and in another terminal running node functional_test_runner.

[liza-mae]

This started happening again 7.8, 7.9 and master test runs.

05:56:27          └-> can navigate to app
05:56:27            └-> "before each" hook: global before each
05:56:40 Unhandled Promise rejection detected:
05:56:40 
05:56:40 { NoSuchElementError: no such element: Unable to locate element: {"method":"css selector","selector":"[data-test-subj=collapsibleNav] > button"}
05:56:40   (Session info: headless chrome=81.0.4044.122)
05:56:40     at Object.throwDecodedError (/var/lib/jenkins/workspace/elastic+estf-cloud-kibana-tests/JOB/xpackGrp1/TASK/saas_run_kibana_tests/node/linux-immutable/ci/cloud/common/build/kibana/node_modules/selenium-webdriver/lib/error.js:550:15)
05:56:40     at parseHttpResponse (/var/lib/jenkins/workspace/elastic+estf-cloud-kibana-tests/JOB/xpackGrp1/TASK/saas_run_kibana_tests/node/linux-immutable/ci/cloud/common/build/kibana/node_modules/selenium-webdriver/lib/http.js:565:13)
05:56:40     at Executor.execute (/var/lib/jenkins/workspace/elastic+estf-cloud-kibana-tests/JOB/xpackGrp1/TASK/saas_run_kibana_tests/node/linux-immutable/ci/cloud/common/build/kibana/node_modules/selenium-webdriver/lib/http.js:491:26)
05:56:40     at process._tickCallback (internal/process/next_tick.js:68:7)
05:56:40   name: 'NoSuchElementError',
05:56:40   remoteStacktrace: '#0 0x55aa770385b9 <unknown>\n' }
05:56:40 
05:56:40 Terminating process...

[liza-mae]

I consider this high priority since it is preventing the test suites from completing.

[spalger]

@liza-mae the Unhandled Promise rejection detected failure is something else, a test must be triggering this error by not having an await in the right spot and is caused by some effort we've put into hardening our node processes.

I've traced all the uses of appsMenu.closeCollapsibleNav() where this error seems to be coming from, and they all seem to be using the promises correctly, but I've also merged a few PRs in the last month or so that add missing awaits (including #66079 and #65566) so it's possible this is fixed in the latest version of the branches. Either way, I think this is a different issue from the spontaneous and unexplained FTR shutdowns this issue was tracking. Please ping me on a new issue with some details the specific commit you're seeing this error happen on and I'd be happy to track down where the issue is and get it fixed.

Additionally, it doesn't seem like we're seeing the same spontaneous and unexplained shutdowns anymore, so I'm going to close this issue.