Repeated asyncio errors
#576
Comments
|
We should always have a Pinger facade. So either we're not connected any longer or we're attempting to create a Pinger without first building the facades. There are couple of locations that I think need investigation:
|
jujubot
added a commit
that referenced
this issue
Nov 19, 2021
#580 ### Description A bunch of background noise is happening in the concurrent asyncio events resulting in repeated error logs to build up, despite that nothing in the foreground is failing. These are often: 1. Actual exceptions that are ignored in the main task/thread. 2. Error logs bleeding from other libraries (potentially when trying to handle the ignored exceptions in (1) (b/c `asyncio` manages their events as well). 3. Tasks that are not gathered properly at teardown. This PR attempts to streamline all the tasks and events that are happening on our `connection.py` and make sure we're: * not producing any background exceptions, * not producing any unnecessary/useless/excess error logs, and * tearing down properly. Hopefully will fix #576 Jira card [#138](https://warthogs.atlassian.net/browse/JUJU-138) ### QA Steps In the CI test runs (especially in the ones with failed tests), we should only see the pass info and exception stacktraces that are relevant to the test logic (i.e. there shouldn't be any asyncio related errors in there). ### Notes & Discussion * This first commit ensures the `Pinger` facade is available when the `_pinger_task` is created, by moving the `_pinger_task` creation after `_build_facade` calls. * Also disables the `_pinger_task` completely for `debug_log` connections, because debug_log connections are not doing any RPCs, they directly connect through the WebSocketClientProtocol, so we're not getting any facades from the apiserver (like we do in the case of `_connect_with_login` etc) * A relevant bug here is https://bugs.python.org/issue36709 : which is about `asyncio.sslproto.SSLProtocol` raising an error whenever the event_loop is closed during an alive websocket connection. So in our `client/connection.py` we avoid keeping the connection alive (by removing `ping_interval=None`). * The websocket object (_ws) is tightly coupled with the asyncio-managed tasks (e.g. Pinger) as well as the Monitor, so interacting with it directly (e.g. doing conn._ws.close() instead of conn.close()) pushes things out of sync/control. So it shouldn't be used directly (e.g. avoid `conn.ws.close()`, instead use `conn.close()`) * Functions such as `model.get_controller` returns a newly constructed Controller object that is already connected with the model's own connection parameters. This puts the responsibility of closing out the connection (and destroying the spawned asyncio tasks etc) to the caller. If not done, everyone (asyncio, websockets etc) yells at pylibjuju about lingering tasks and open connections. So we introduce a `ConnectedController` object that when it's used with the `async with` form automatically disconnects after done. * asyncio loop event_handler is only called on Task exceptions when the Task object is cleared from the memory. But the garbabe collector doesn't clear the Task if we keep a reference for it (e.g. putting it in a neverending Task ---like Pinger) until the very end. So installing our own handler makes sure that the exceptions are retrieved and properly handled and logged regardless of when the Task is being destroyed.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I've been using
pylibjujuthrough thepytest-operatorfor testing, and I'm seeing this error a lot:and
These occur consistently on every run. More examples can be found here: https://github.com/jnsgruk/charm-kubernetes-dashboard/runs/4069754517?check_suite_focus=true#step:4:256
The text was updated successfully, but these errors were encountered: