OSError: [WinError 64] The specified network name is no longer available

[sentry-for-tribler]

Sentry Issue: main branch

OSError: [WinError 64] The specified network name is no longer available
  File "asyncio\windows_events.py", line 571, in accept_coro
    await future
  File "asyncio\windows_events.py", line 817, in _poll
    value = callback(transferred, key, ov)
  File "asyncio\windows_events.py", line 560, in finish_accept
    ov.getresult()

[drew2a]

It seems like this is the most frequent bug for the 7.13.1 release:

https://sentry.tribler.org/organizations/tribler/issues/2516

EVENTS
85

USERS
36

[qstokkink]

After some offline discussion and given the fact that these errors happen deep in cpython, we decided that it may be best to simply add this to the ignored errors:

tribler/src/tribler/core/components/reporter/exception_handler.py

Lines 19 to 33 in e6d2c58

	IGNORED_ERRORS_BY_CLASS: Tuple[Type[Exception], ...] = (
	ConnectionResetError, # Connection forcibly closed by the remote host.
	gaierror, # all gaierror is ignored
	)
	IGNORED_ERRORS_BY_CODE: Set[Tuple[Type[Exception], int]] = {
	(OSError, 113), # No route to host is non-critical since Tribler can still function when a request fails.
	# Socket block: this sometimes occurs on Windows and is non-critical.
	(BlockingIOError, 10035 if sys.platform == 'win32' else errno.EWOULDBLOCK),
	(OSError, 51), # Could not send data: network is unreachable.
	(ConnectionAbortedError, 10053), # An established connection was aborted by the software in your host machine.
	(OSError, 10022), # Failed to get address info.
	(OSError, 16), # Socket error: Device or resource busy.
	(OSError, 0)
	}

[drew2a]

Ignored exception means the exception will not be sent to us, but Tribler will crash. Perhaps we should catch the error and try to continue work if this error is not critical.

[qstokkink]

@drew2a I got the wrong list then. Do we also have a list of exceptions that are ignored while Tribler continues to work? We had one in the past and I thought this was its refactored form.

[drew2a]

Do we also have a list of exceptions that are ignored while Tribler continues to work? We had one in the past and I thought this was its refactored form.

Perhaps I didn't express myself completely accurately because there are two mechanisms by which Tribler can crash due to an error in the core.

The first is an error in the core that allows it to continue operating, but this error is sent to the GUI, and the GUI then terminates the Tribler operation (in this case, you are right about the ignored errors).

The second mechanism is an error in the core that makes it unable to continue operating, and then it doesn't matter whether the exception is in the ignored list or not. I was referring to the second mechanism. For it, one might try to catch the exception earlier so that it doesn't reach the exception_handler and is resolved at the level of the function/module where it occurred.

Sorry for any possible misunderstanding.

[xoriole]

I'm able to often reproduce this issue with a relatively large database and quite some downloads. Looking at Sentry as well, it is affecting several users so I believe putting the error on ignored list will not be a good solution.

Looking at the stack trace, it is clear that it is not related to API port but rather SocksServer/circuit relay port.

This makes me believe it could be linked to #7805.

[xoriole]

The fix #7889 did not solve this issue and there are still reports of this issue on 7.13.2

Sentry report here

[xoriole]

Investigating this issue further, found this issue which seems relevant: python/cpython#93821

[kozlovsky]

@xoriole, great finding! The described patch should fix the problem. We can monkey-patch asyncio code with these changes.

[xoriole]

Here is a client and a server code to reproduce this issue:

# socks_clients.py
import asyncio
import threading
from unittest.mock import Mock

from tribler.core.components.socks_servers.socks5.client import Socks5Client


async def async_create_socks_client():
    for j in range(100):
        client = Socks5Client(('127.0.0.1', 8001), Mock())
        await client.connect_tcp(('127.0.0.1', 8088))
        client.write(b' ')


def create_socks_client():
    asyncio.set_event_loop(asyncio.new_event_loop())
    asyncio.run(async_create_socks_client())


if __name__ == "__main__":
    for i in range(100):
        thread = threading.Thread(target=create_socks_client, args=(), daemon=True)
        thread.start()

# socks_server.py
import asyncio

from tribler.core.components.socks_servers.socks5.server import Socks5Server


async def main():
    socks_server = Socks5Server(port=8001)
    await socks_server.start()
    print(f"socks port: {socks_server.port}")

    async with socks_server.server:
        await socks_server.server.serve_forever()


asyncio.run(main(), debug=True)

[xoriole]

Closed by #7926