[nodriver] Using Nodriver in already running event loops

[lunden23]

Hi there, for some reason Nodriver should be initiated in the following way according to the documentation:

if __name__ == "__main__":
    # since asyncio.run never worked (for me)
    # i use
    uc.loop().run_until_complete(main())

However, this precludes using it in an already running event loop. Due to the complex structure of my app where Nodriver would serve a part as a scraper in background processes managed by Simple Asynchronous Queue (SAQ) I am using it in already running asynchronous loops. I can create a new thread to run Nodriver (theoretically) but that would be synchronous and I imagine it would cause issues on its own. I am therefore wondering if it is possible to run and close Nodriver from an already running loop without seeing the following generic exception raised every time when closing Nodriver:

Exception ignored in: <function BaseSubprocessTransport.__del__ at 0x7fee26a2ccc0>
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/asyncio/base_subprocess.py", line 126, in __del__
  File "/usr/local/lib/python3.12/asyncio/base_subprocess.py", line 104, in close
  File "/usr/local/lib/python3.12/asyncio/unix_events.py", line 568, in close
  File "/usr/local/lib/python3.12/asyncio/unix_events.py", line 592, in _close
  File "/usr/local/lib/python3.12/asyncio/base_events.py", line 795, in call_soon
  File "/usr/local/lib/python3.12/asyncio/base_events.py", line 541, in _check_closed
RuntimeError: Event loop is closed

I have tried many various methods already, but always encounter this exception when Nodriver is closed. Are there any ways to bypass this shortcoming?

[ultrafunkamsterdam]

You can run it in whatever loop you want . To prevent the error you should close the websocket connections manually. Or, create task and run_until_complete like UC loop does but using your own loop.

You could also use contextlib suppress to , well, suppress the exception

[lunden23]

Thank you for your input @ultrafunkamsterdam. I obviously bit off a bit more than I can chew when transitioning to Nodriver, but it is super performant so have to use it. Based on your input I devised this method to close the browser without seeing the exception:

        try:
            # Close all tabs
            for t in browser.tabs:
                await t.close()

            # Attempt to close the websocket connection manually
            if browser.connection:
                await browser.connection.aclose()

            # Close the browser
            browser.stop()
        except Exception as e:
            print(f"Error during browser closure: {e}")
        finally:
            # Ensure the browser process is terminated
            if browser._process:
                browser._process.terminate()
                await browser._process.wait()

Per your suggestion I tried to close the websocket connections manually, but they are usually already closed by Nodriver itself. So the code that actually results in no async exception being raised is the last part in the finally clause wherein the browser._process is manually terminated. That however causes Nodriver to catch an exception in the nodriver.core.connection package in listener_loop method, where it prints:

2024-08-08 16:01:46,263 - uc.connection - DEBUG - connection listener exception while reading websocket:
no close frame received or sent

Since it doesn't re-raise an exception and handles it internally I am content with this solution. I also played around a lot with the event loops and run_until_complete and create_task but they always causes the same exception to be raised. I assume I am not knowledgeable enough to make that solution work. Any suggestion to handle this more gracefully would still be appreciated. Thank you.