[rb] Fully support Chrome 120+ old headless mode #13271

neilvcarvalho · 2023-12-08T20:44:42Z

Motivation and Context

When Chrome's new headless mode was announced, it was mentioned that they intended to remove the old headless mode from the Chrome binary. Some people still want to use the old headless mode, as it's more lightweight, performant, and has fewer dependencies - at the expense of having fewer features.

Chrome 120+ has a standalone chrome-headless-shell binary to keep the old headless mode working for those who wish to use it. If your Ruby code sets up the Chrome driver with --headless (which currently defaults to old) or --headless=old explicitly, you will lose capabilities such as setting permissions, fetching the logs, and using the CDP API.

This change is explained on the Chrome for Developers blog: Download old Headless Chrome as chrome-headless-shell

That happens because the new binary has a browser name of chrome-headless-shell, instead of chrome. That makes it be recognized as something other than Chrome, not setting its capabilities.

Description

This commit introduces chrome-headless-shell as an alternative Chrome browser name, fixing this issue.

Types of changes

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to change)

Checklist

I have read the contributing document.
My change requires a change to the documentation.
I have updated the documentation accordingly.
I have added tests to cover my changes.
All new and existing tests passed.

Fixes #13112

CLAassistant · 2023-12-08T20:44:48Z

All committers have signed the CLA.

titusfortner

Thanks for this PR.

Let's leave the test files alone, those get toggled off of the values we pass in, not what comes back from the browser. (also we just aren't going to be running tests with the old headless).

rb/CHANGES

rb/lib/selenium/webdriver/remote/bridge.rb

When Chrome's new headless mode was announced, it was mentioned that they intended to remove the old headless mode from the Chrome binary. Some people still want to use the old headless mode, as it's more lightweight, performant, and has fewer dependencies - at the expense of having fewer features. Chrome 120+ has a standalone `chrome-headless-shell` binary to keep the old headless mode working for those who wish to use it. If your Ruby code sets up the Chrome driver with `--headless` (which currently defaults to `old`) or `--headless=old` explicitly, you will lose capabilities such as setting permissions, fetching the logs and using the CDP API. That happens because the new binary has a browser name of `chrome-headless-shell`, instead of `chrome`. That makes it be recognized as something other than Chrome, not setting its capabilities. This commit introduces `chrome-headless-shell` as an alternative Chrome browser name, fixing this issue. Fixes SeleniumHQ#13112

neilvcarvalho · 2023-12-08T23:23:56Z

Let's leave the test files alone, those get toggled off of the values we pass in, not what comes back from the browser. (also we just aren't going to be running tests with the old headless).

@titusfortner I removed the specs changes, amending on the same commit. Thank you for the explanation!

ryanb · 2023-12-12T22:37:28Z

I'm running into this issue and having trouble with the new headless. Do you know when the next release will be that contains this fix?

titusfortner · 2023-12-13T14:41:20Z

The current plan is to release about as often as Google does https://chromiumdash.appspot.com/schedule
The recommendation is to use --headless=new

ryanb · 2023-12-13T18:37:12Z

Thanks. I'm having trouble migrating to --headless=new so for now I'll stick to Chrome v119 until this PR is released.

For those who may be facing the same issue, you can set the version like this.

Selenium::WebDriver::Chrome::Options.new(
  args: ["headless=old"],
  browser_version: "119",
),

`./modules/my_page/spec/features/my/my_spent_time_widget_with_a_negative_time_zone_spec.rb` is failing with error `undefined method `execute_cdp' for #<Selenium::WebDriver::Chrome::Driver:0x..fe768862b9701efbc browser=:"chrome-headless-shell">`. The `--headless` flag uses the old headless mode of Chrome. Since version 120, this version's name is "chrome-headless-shell". This name is not properly recognized in selenium-webdriver 4.16.0 and some extensions are not loaded, like the one allowing to use execute_cdp. This will be fixed in next selenium-webdriver with this PR: SeleniumHQ/selenium#13271 In the meantime, using the new headless mode with `--headless=new` fixes the issue.

jeppester · 2023-12-18T11:04:53Z

The current plan is to release about as often as Google does https://chromiumdash.appspot.com/schedule The recommendation is to use --headless=new

I think a lot of projects will need to implement this temporary workaround in order to get their CI to pass.
Many of hours will be spent investigating the problem and eventually implementing the workaround, and then after the next release, the workaround will become obsolete code in those projects.

@titusfortner
Is it completely out of question to make a bugfix release containing this fix?

titusfortner · 2023-12-18T18:10:37Z

Using --headless=new is not a temporary workaround. It is the recommendation going forward. And maybe on a patch release, I've got other things on my plate right now, so haven't looked at what's feasible.

ryanb · 2023-12-18T19:41:48Z

@titusfortner a couple questions that might help those migrating to the new headless.

Should we file a bug if we find an inconsistency between old and new headless?
If old and new headless has intentional differences, are those differences documented somewhere?

Thanks for your time.

titusfortner · 2023-12-19T22:56:31Z

Should we file a bug if we find an inconsistency between old and new headless?

Probably. It depends on if it was actually a bug in the old headless. :)
Old Headless was essentially a stripped down fake browser.
New Headless is an actual headless implementation of Real Chrome.

ryanb · 2023-12-20T00:20:10Z

@titusfortner thanks for the reply.

Probably. It depends on if it was actually a bug in the old headless. :)

I guess a better check is if it works in non headless chrome but not in new headless chrome. In my case drag & drop isn't working in new headless but it does work in old headless and non headless Chrome. I'll look into it further and file a bug once I've isolated the issue.

titusfortner · 2023-12-20T14:19:48Z

Oh that's interesting. Drag and Drop with HTML 5 draggable / droppable elements hasn't actually ever worked with the drivers as far as I know. (perpetual complaints as you may have guessed).

runephilosof-abtion · 2023-12-21T15:46:39Z

Using --headless=new is not a temporary workaround. It is the recommendation going forward.

It is a temporary workaround in projects that use selenium, because it is only necessary until this commit is released.

Just see the linked pull requests
Shopify Shopify/maintenance_tasks#936

Instead of waiting for a new release of selenium-webdriver that includes the fix to continue using the old headless, or configuring Selenium to use an older version of Chrome, this moves to the new headless Chrome.

Openproject opf/openproject#14426

This will be fixed in next selenium-webdriver with this PR: #13271

In the meantime, using the new headless mode with --headless=new fixes the issue.

They are working around the problem that this pull request fixes. It seems they would have preferred to just update to a selenium-webdriver version that included this fix.

If it is the recommendation going forward to add --headless=new, then that should be more prominent in the documentation. The only place it is mentioned is https://www.selenium.dev/documentation/webdriver/browsers/edge/#arguments

There are probably also a lot of frameworks that need to catch up and add --headless=new to their templates, so it would be a nice Christmas gift for a lot of people that they would not have to use a lot of time to figure out that they need to add --headless=new.

If you release a patch version, everyone can upgrade to --headless=new when they are ready, or when it becomes the default.

titusfortner · 2023-12-21T16:29:38Z

It is a temporary workaround in projects that use selenium, because it is only necessary until this commit is released.

We're arguing semantics. Encouraging people to use the recommended settings shouldn't include the baggage of the term "workaround," even if it entails "extra work," but I think we each get the other's point here.

Google will soon(?) be switching --headless to use --headless=new instead of --headless=old as it is now. I wish they had done this at the same time as making a backwards incompatible name change, or at least given us a heads up about it, but they didn't.

There is code in trunk that for /reasons/ cannot be released, yet, so doing a patch release is no longer an option. The next minor release will be for all languages when Chrome 121 comes out in January.

Looks like there was some change in headless chrome where some features were removed, including execute_cdp, which we rely on for the tests which exercise the clipboard. There's an old mode which is faster but doesn't have all the features of a real browser, and a new mode which does. SeleniumHQ/selenium#13112 SeleniumHQ/selenium#13271 Maybe a better approach would be to keep the old mode for fast tests which don't need these features and use new mode for tests which do.

3.40 will use the new headless chrome, but it hasn't been merged yet. As far as I can gather there was some change in Chrome where they split headless into two versions: old and new. The old one is used by default, but is missing some features were removed, including execute_cdp, which we rely on for the tests which exercise the clipboard This PR explains it pretty well: SeleniumHQ/selenium#13271 What makes this confusing is that the headless browser is provided by the selenium-webdrivers gem, but driver is specified by Capybara: teamcapybara/capybara@e2f9f6c The result was that we got failures on some environments saying that execute_cdp is not defined - this is presumably due to the Chrome version being updated - independent of the codebase. Note that unfortunately the "new" headless browser is slower, but I think it's probably worth it for keeping the config simple and using a more realistic browser.

neilvcarvalho · 2024-01-24T21:53:36Z

For folks who were following this and want to use the old headless mode: this was released this week 🎉

Looks like there was some change in headless chrome where some features were removed, including execute_cdp, which we rely on for the tests which exercise the clipboard. There's an old mode which is faster but doesn't have all the features of a real browser, and a new mode which does. SeleniumHQ/selenium#13112 SeleniumHQ/selenium#13271 Maybe a better approach would be to keep the old mode for fast tests which don't need these features and use new mode for tests which do.

neilvcarvalho mentioned this pull request Dec 8, 2023

[🐛 Bug]: Ruby: undefined method `logs' for #<Selenium::WebDriver::Chrome::Driver> with Chrome Beta (120) + headless mode #13112

Closed

neilvcarvalho force-pushed the nc-chrome-120 branch from fa480b3 to 58e58d2 Compare December 8, 2023 21:20

titusfortner requested changes Dec 8, 2023

View reviewed changes

rb/CHANGES Outdated Show resolved Hide resolved

rb/lib/selenium/webdriver/remote/bridge.rb Show resolved Hide resolved

neilvcarvalho force-pushed the nc-chrome-120 branch from 58e58d2 to 4be5859 Compare December 8, 2023 23:22

titusfortner approved these changes Dec 9, 2023

View reviewed changes

titusfortner merged commit e586190 into SeleniumHQ:trunk Dec 11, 2023
25 of 30 checks passed

neilvcarvalho deleted the nc-chrome-120 branch December 11, 2023 17:34

valfirst mentioned this pull request Dec 12, 2023

[java] Recognise chrome-headless-shell as Chrome browser #13292

Merged

8 tasks

cbliard mentioned this pull request Dec 15, 2023

Use --headless=new for Chrome in Capybara opf/openproject#14426

Merged

etiennebarrie mentioned this pull request Dec 15, 2023

Fix Chrome headless Shopify/maintenance_tasks#936

Merged

jeppester mentioned this pull request Dec 18, 2023

Temporarily Work around issue with selenium abtion/rails-template#1942

Merged

neilvcarvalho mentioned this pull request Jan 24, 2024

Use the new headless Chrome on the pre-registered driver teamcapybara/capybara#2726

Merged

mjankowski mentioned this pull request Feb 1, 2024

Configure selenium to use Chrome version 120 mastodon/mastodon#29038

Merged

komagata mentioned this pull request Feb 2, 2024

chromedriverを更新すると、テスト実行時にNoMethodError: undefined method 'execute_cdp'が発生する fjordllc/bootcamp#7168

Closed

sudoudaisuke mentioned this pull request Feb 19, 2024

[🚀 Feature]: Add chrome-headless-shell to the browser options of selenium-manager. #13600

Closed

dgmstuart mentioned this pull request Feb 25, 2024

Fix: use new selenium headless chrome dgmstuart/swing-out-london#320

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[rb] Fully support Chrome 120+ old headless mode #13271

[rb] Fully support Chrome 120+ old headless mode #13271

neilvcarvalho commented Dec 8, 2023 •

edited

Loading

CLAassistant commented Dec 8, 2023 •

edited

Loading

titusfortner left a comment

neilvcarvalho commented Dec 8, 2023

ryanb commented Dec 12, 2023

titusfortner commented Dec 13, 2023

ryanb commented Dec 13, 2023

jeppester commented Dec 18, 2023 •

edited

Loading

titusfortner commented Dec 18, 2023

ryanb commented Dec 18, 2023 •

edited

Loading

titusfortner commented Dec 19, 2023

ryanb commented Dec 20, 2023

titusfortner commented Dec 20, 2023

runephilosof-abtion commented Dec 21, 2023

titusfortner commented Dec 21, 2023 •

edited

Loading

neilvcarvalho commented Jan 24, 2024

[rb] Fully support Chrome 120+ old headless mode #13271

[rb] Fully support Chrome 120+ old headless mode #13271

Conversation

neilvcarvalho commented Dec 8, 2023 • edited Loading

Motivation and Context

Description

Types of changes

Checklist

CLAassistant commented Dec 8, 2023 • edited Loading

titusfortner left a comment

Choose a reason for hiding this comment

neilvcarvalho commented Dec 8, 2023

ryanb commented Dec 12, 2023

titusfortner commented Dec 13, 2023

ryanb commented Dec 13, 2023

jeppester commented Dec 18, 2023 • edited Loading

titusfortner commented Dec 18, 2023

ryanb commented Dec 18, 2023 • edited Loading

titusfortner commented Dec 19, 2023

ryanb commented Dec 20, 2023

titusfortner commented Dec 20, 2023

runephilosof-abtion commented Dec 21, 2023

titusfortner commented Dec 21, 2023 • edited Loading

neilvcarvalho commented Jan 24, 2024

neilvcarvalho commented Dec 8, 2023 •

edited

Loading

CLAassistant commented Dec 8, 2023 •

edited

Loading

jeppester commented Dec 18, 2023 •

edited

Loading

ryanb commented Dec 18, 2023 •

edited

Loading

titusfortner commented Dec 21, 2023 •

edited

Loading