Chrome headless hangs - fix to work with new behaviour of newer Chrome versions #890
Comments
|
Adding on, running the same flow in visible Chrome mode runs at normal speed instead of hanging waiting v long for reply. This is likely due to some change in behaviour of headless Chrome in newer Chrome releases, as it previously works. |
|
on debugging the websocket connection, keeps getting below errors non-stop - |
|
running below code to test Chrome headless works - https://medium.com/@lagenar/using-headless-chrome-via-the-websockets-interface-5f498fb67e0f import json
import time
import subprocess
import requests
from websocket import create_connection
def start_browser(browser_path, debugging_port):
options = ['--headless', ' --disable-gpu',
'--remote-debugging-port={}'.format(debugging_port)]
browser_proc = subprocess.Popen([browser_path] + options)
wait_seconds = 10.0
sleep_step = 0.25
while wait_seconds > 0:
try:
url = 'http://127.0.0.1:{}/json'.format(debugging_port)
resp = requests.get(url).json()
ws_url = resp[0]['webSocketDebuggerUrl']
return browser_proc, create_connection(ws_url)
except requests.exceptions.ConnectionError:
time.sleep(sleep_step)
wait_seconds -= sleep_step
raise Exception('Unable to connect to chrome')
request_id = 0
def run_command(conn, method, **kwargs):
global request_id
request_id += 1
command = {'method': method,
'id': request_id,
'params': kwargs}
conn.send(json.dumps(command))
while True:
msg = json.loads(conn.recv())
if msg.get('id') == request_id:
return msg
gnews_url = 'https://news.google.com/news/?ned=us&hl=en'
chrome_path = '/Applications/Google Chrome.app/Contents/MacOS/Google Chrome'
browser, conn = start_browser(chrome_path, 9222)
run_command(conn, 'Page.navigate', url=gnews_url)
time.sleep(5) # let it load
js = """
var sel = 'h3 > a';
var headings = document.querySelectorAll(sel);
headings = [].slice.call(headings).map((link)=>{return link.innerText});
JSON.stringify(headings);
"""
result = run_command(conn, 'Runtime.evaluate', expression=js)
headings = json.loads(result['result']['result']['value'])
for heading in headings:
print(heading)
browser.terminate() |
|
following simple TagUI script base on above, works in normal mode, but throws the same error using headless mode - |
|
consider the following simple script the same simple calls can lead to persistent timeouts after a couple of times |
|
doing a wait to throttle the requests will see the same timeout messages, but still seeing response after some time - |
|
Some clues found. Error happens in headless mode when the user profile directory is a relative path - However, above in tagui/src/tagui works in normal visible mode. And changing the relative path to absolute path makes it work for headless mode. Ie some difference in behaviour for newer versions of Chrome in headless mode. More references on using full path name -
|
|
Adding on below what I shared with Chrome Remote Interface (another project using DevTools Protocol) maintainer -
|
|
above commit fixes headless Chrome to work on macOS and Linux. to check status for Windows and see if fix required |
kensoh
changed the title
kensoh
changed the title
|
Above commits fix headless Chrome to work on Windows. So headless now working for all OSes. Users can download the latest copy of TagUI from here and unzip to overwrite your existing installation (please drag the folders under tagui\src to overwrite your existing installation) - https://github.com/kelaberetiv/TagUI/archive/master.zip In the next release, this fix will be part of the packaged zip files. |
|
Closing issue since the latest packaged release TagUI v6.14 is out. Release notes - https://github.com/kelaberetiv/TagUI/releases/tag/v6.14.0 |
see simple flow
run output log
tagui chrome log
The text was updated successfully, but these errors were encountered: