[FTR] Switch to new browser headless mode #153828

pheyos · 2023-03-28T08:11:18Z

Summary

This PR updates the way how we start the headless browser for testing.

The current way of starting in headless mode is eventually going away and the new headless mode offers more capabilities and stability, see https://www.selenium.dev/blog/2023/headless-is-going-away/ and https://developer.chrome.com/articles/new-headless/.

Test adjustments

All the adjusted discover, dashboard, maps and infra tests showed the same pattern during failure investigation that's around the fact that the new headless mode is closer to the regular / non-headless mode:

Tests passed with the old headless mode
Tests failed in regular / non-headless mode the same way they failed in new headless mode
The failure reasons were mostly around slightly different font rendering and slightly different browser sizes

pheyos · 2023-03-30T15:25:02Z

Checking stability of the adjusted discover, dashboard and maps tests in a flaky test runner job ... no failure in 50 runs each ✔️

pheyos · 2023-03-30T15:28:13Z

@elasticmachine merge upstream

pheyos · 2023-03-31T07:33:26Z

@elasticmachine merge upstream

pheyos · 2023-03-31T12:58:12Z

@elasticmachine merge upstream

pheyos · 2023-04-03T09:23:39Z

@elasticmachine merge upstream

pheyos · 2023-04-03T11:09:42Z

Recent failures were unrelated to this change (API integration tests don't use the headless browser).
This will hopefully be resolved soon, still opening this PR for reviews.

jughosta

Data Discovery changes LGTM 👍

wayneseymour · 2023-04-03T12:12:14Z

test/functional/services/remote/webdriver.ts

@@ -99,9 +99,12 @@ function initChromiumOptions(browserType: Browsers, acceptInsecureCerts: boolean
  }

  if (headlessBrowser === '1') {
+    // Using the new headless mode (instead of `options.headless()`)
+    // See: https://www.selenium.dev/blog/2023/headless-is-going-away/


Smart adding the link in the comment! 🎉

nreese

Could you provide some details into why these test changes are required? Did the screen size change for functional tests or something?

pheyos · 2023-04-03T15:53:03Z

@nreese Indeed, the size of the browser content area changed with the new headless mode. Reading the article that describes how the new headless mode shares code with the headful version and also seeing how running in headful mode gives the same browser content dimensions (same as new headless, different from old headless), makes me suspect the following (although I haven't found a clear statement about it):

When setting the browser window size during tests,

the old headless mode (a separate, alternate browser implementation) sets the content area to that size
the new headless mode (same as the regular headful mode) sets the browser window to that size, so the actual content area is smaller (window decorations, menu bar, ...)

I hope that helps. Let me know if you want me to add more details.

nreese

kibana-gis changes LGTM
code review

x-pack/test/functional/apps/maps/group1/documents_source/search_hits.js

x-pack/test/functional/apps/maps/group2/es_geo_grid_source.js

dmlemeshko · 2023-04-04T10:11:53Z

Code LGTM, Tested locally on Mac OS:

Chrome head: all changed tests passed
Chrome headless: maps related tests are failing, but I think we had the Mac OS issue with WebGL for a while (correct?)
Firefox head: few tests are failing due to changes

pheyos · 2023-04-04T11:57:24Z

An infra test started failing on new headless mode after a recent test change, again due to the slightly different browser size. Adjusted the browser size for this suite and started a flaky test runner job for the modified infra suite to check test stability... no failure in 50 runs ✔️

kibana-ci · 2023-04-04T12:43:47Z

💚 Build Succeeded

Buildkite Build
Commit: 8ef6986

Metrics [docs]

Unknown metric groups

ESLint disabled line counts

id	before	after	diff
`securitySolution`	432	435	+3

Total ESLint disabled count

id	before	after	diff
`securitySolution`	512	515	+3

History

💔 Build #117609 failed d36972e
💔 Build #117579 failed f1f6954
💚 Build #117411 succeeded b1ea744
💔 Build #117291 failed 019f799
💛 Build #117070 was flaky 56fea27

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @pheyos

jennypavlova

Infra UI changes look good, thanks 🙂

kibanamachine · 2023-04-04T13:58:37Z

💔 All backports failed

Status	Branch	Result
❌	8.7	Backport failed because of merge conflicts

Manual backport

To create the backport manually run:

node scripts/backport --pr 153828

Questions ?

Please refer to the Backport tool documentation

[FTR] Switch to new browser headless mode

df60f84

pheyos self-assigned this Mar 28, 2023

pheyos added 6 commits March 29, 2023 10:46

Merge branch 'main' into new_headless_mode

9cd0657

Fix discover tests

a96c811

Fix dashboard tests

fdd14ca

Merge branch 'main' into new_headless_mode

5af7a82

Fix maps tests

861311c

Merge branch 'main' into new_headless_mode

f7c5076

Merge branch 'main' into new_headless_mode

6f4bcf7

Merge branch 'main' into new_headless_mode

99c6730

Merge branch 'main' into new_headless_mode

56fea27

Merge branch 'main' into new_headless_mode

019f799

pheyos added v8.8.0 release_note:skip Skip the PR/issue when compiling release notes backport:prev-minor Backport to (8.x) the previous minor version (i.e. one version back from main) test_ui_functional labels Apr 3, 2023

pheyos marked this pull request as ready for review April 3, 2023 11:11

pheyos requested review from a team as code owners April 3, 2023 11:11

pheyos requested a review from dmlemeshko April 3, 2023 11:11

jughosta approved these changes Apr 3, 2023

View reviewed changes

wayneseymour approved these changes Apr 3, 2023

View reviewed changes

nreese reviewed Apr 3, 2023

View reviewed changes

Merge branch 'main' into new_headless_mode

b1ea744

nreese approved these changes Apr 4, 2023

View reviewed changes

pheyos added 2 commits April 4, 2023 09:03

Merge branch 'main' into new_headless_mode

f1f6954

Merge branch 'main' into new_headless_mode

d36972e

dmlemeshko reviewed Apr 4, 2023

View reviewed changes

x-pack/test/functional/apps/maps/group1/documents_source/search_hits.js Show resolved Hide resolved

dmlemeshko reviewed Apr 4, 2023

View reviewed changes

x-pack/test/functional/apps/maps/group2/es_geo_grid_source.js Show resolved Hide resolved

pheyos added 2 commits April 4, 2023 13:01

Merge branch 'main' into new_headless_mode

9a56503

Fix infra tests

8ef6986

pheyos requested a review from a team as a code owner April 4, 2023 11:54

jennypavlova approved these changes Apr 4, 2023

View reviewed changes

This comment was marked as duplicate.

Sign in to view

pheyos merged commit 98df0c2 into elastic:main Apr 4, 2023

pheyos deleted the new_headless_mode branch April 4, 2023 13:54

pheyos removed the backport:prev-minor Backport to (8.x) the previous minor version (i.e. one version back from main) label Apr 4, 2023

kibanamachine added the backport:skip This commit does not require backporting label Apr 4, 2023

nickpeihl mentioned this pull request Apr 11, 2023

[Dashboard] Unskip functional test related to webdriver #154767

Merged

pheyos mentioned this pull request Apr 28, 2023

[ML] Functional tests - stabilize and re-enable export jobs tests #152334

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[FTR] Switch to new browser headless mode #153828

[FTR] Switch to new browser headless mode #153828

pheyos commented Mar 28, 2023 •

edited by kibanamachine

Loading

pheyos commented Mar 30, 2023

pheyos commented Mar 30, 2023

pheyos commented Mar 31, 2023

pheyos commented Mar 31, 2023

pheyos commented Apr 3, 2023

pheyos commented Apr 3, 2023

jughosta left a comment

wayneseymour Apr 3, 2023

nreese left a comment

pheyos commented Apr 3, 2023 •

edited

Loading

nreese left a comment

dmlemeshko commented Apr 4, 2023

pheyos commented Apr 4, 2023 •

edited

Loading

kibana-ci commented Apr 4, 2023

ESLint disabled line counts

Total ESLint disabled count

jennypavlova left a comment

This comment was marked as duplicate.

kibanamachine commented Apr 4, 2023

[FTR] Switch to new browser headless mode #153828

[FTR] Switch to new browser headless mode #153828

Conversation

pheyos commented Mar 28, 2023 • edited by kibanamachine Loading

Summary

Test adjustments

pheyos commented Mar 30, 2023

pheyos commented Mar 30, 2023

pheyos commented Mar 31, 2023

pheyos commented Mar 31, 2023

pheyos commented Apr 3, 2023

pheyos commented Apr 3, 2023

jughosta left a comment

Choose a reason for hiding this comment

wayneseymour Apr 3, 2023

Choose a reason for hiding this comment

nreese left a comment

Choose a reason for hiding this comment

pheyos commented Apr 3, 2023 • edited Loading

nreese left a comment

Choose a reason for hiding this comment

dmlemeshko commented Apr 4, 2023

pheyos commented Apr 4, 2023 • edited Loading

kibana-ci commented Apr 4, 2023

💚 Build Succeeded

Metrics [docs]

ESLint disabled line counts

Total ESLint disabled count

History

jennypavlova left a comment

Choose a reason for hiding this comment

This comment was marked as duplicate.

kibanamachine commented Apr 4, 2023

💔 All backports failed

Manual backport

Questions ?

pheyos commented Mar 28, 2023 •

edited by kibanamachine

Loading

pheyos commented Apr 3, 2023 •

edited

Loading

pheyos commented Apr 4, 2023 •

edited

Loading