Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ElectrumX becoming unresponsive #92

Open
ZacharyACoon opened this issue Jan 11, 2021 · 7 comments
Open

ElectrumX becoming unresponsive #92

ZacharyACoon opened this issue Jan 11, 2021 · 7 comments

Comments

@ZacharyACoon
Copy link

ZacharyACoon commented Jan 11, 2021 •
edited by SomberNight
Loading

Versions

python=3.7.1, 3.9.1
electrumx=1.16.0
aiohttp=3.7.0, 4.0

Context

After running ElectrumX for a few hours, when under very high load (2k+ sessions), sometimes it "locks up".
That is, the main thread blocks and stops making progress, it uses 100% CPU,
and it stops accepting new sessions.
The LocalRPC interface stops responding too; all requests there time out.
It never recovers on its own; the server (python process) must be restarted.

Symptoms

ElectrumX Nonresponsive for tcp, ssl, or rpc services
ElectrumX 100% cpu

Stack Trace

pyrasite-trace.txt

main thread traceback 
--- Stack for thread 140338657445696 ---
  File "/electrumx.git/./electrumx_server", line 43, in <module>
    main()
  File "/electrumx.git/./electrumx_server", line 35, in main
    asyncio.run(controller.run())
  File "/usr/local/lib/python3.9/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.9/asyncio/base_events.py", line 629, in run_until_complete
    self.run_forever()
  File "/usr/local/lib/python3.9/asyncio/base_events.py", line 596, in run_forever
    self._run_once()
  File "/usr/local/lib/python3.9/asyncio/base_events.py", line 1890, in _run_once
    handle._run()
  File "/usr/local/lib/python3.9/asyncio/events.py", line 80, in _run
    self._context.run(self._callback, *self._args)
  File "/usr/local/lib/python3.9/site-packages/aiorpcx/session.py", line 496, in _throttled_request
    await self._send_message(message)
  File "/usr/local/lib/python3.9/site-packages/aiorpcx/session.py", line 153, in _send_message
    await self.transport.write(message)
  File "/usr/local/lib/python3.9/site-packages/aiorpcx/rawsocket.py", line 118, in write
    self._asyncio_transport.write(framed_message)
  File "/usr/local/lib/python3.9/asyncio/sslproto.py", line 386, in write
    self._ssl_protocol._write_appdata(data)
  File "/usr/local/lib/python3.9/asyncio/sslproto.py", line 598, in _write_appdata
    self._process_write_backlog()
  File "/usr/local/lib/python3.9/asyncio/sslproto.py", line 674, in _process_write_backlog
    ssldata, offset = self._sslpipe.feed_appdata(data, offset)
  File "/usr/local/lib/python3.9/asyncio/sslproto.py", line 261, in feed_appdata
    offset += self._sslobj.write(view[offset:])
  File "/usr/local/lib/python3.9/ssl.py", line 897, in write
    return self._sslobj.write(data)

Known Workarounds

  1. set up nginx and let it terminate SSL
    • If one places nginx in front of ElectrumX, and lets nginx handle SSL termination, the problem should not occur.
    • In such a setup, nginx is reporting these errors: 
      2021/01/11 02:28:30 [crit] 21#21: *437620 SSL_shutdown() failed (SSL: error:14094123:SSL routines:ssl3_read_bytes:application data after close notify) while proxying connection, client: REDACTED, server: 0.0.0.0:50002, upstream: "127.0.0.1:50213", bytes from/to client:81/205, bytes from/to upstream:205/0
  2. Patch CPython.
  3. Use uvloop
@ZacharyACoon
Copy link
Author

Note, this seems to be occurring with other coins as well.

This was referenced Jan 26, 2021
Closed
Closed
@Emzy
Copy link
Contributor

Emzy commented Apr 8, 2021

I don't get this problem on my old setup (electumx running for weeks):
Python 3.7.5
aiorpcX version: 0.18.3
aiohttp Version: 3.5.4

$ git rev-parse HEAD
76dca02

@SomberNight
Copy link
Member

SomberNight commented May 3, 2021 •
edited
Loading

I have this traceback for the main thread, on python 3.8.5 (from ubuntu 20.04; with openssl=1.1.1f-1ubuntu2.3).
It is similar to the one in the OP.

(gdb) py-bt
Traceback (most recent call first):
  File "/usr/lib/python3.8/asyncio/sslproto.py", line 535, in feed_appdata

  File "/usr/lib/python3.8/asyncio/sslproto.py", line 675, in _process_write_backlog
    ssldata, offset = self._sslpipe.feed_appdata(data, offset)
  File "/usr/lib/python3.8/asyncio/sslproto.py", line 599, in _write_appdata
    self._process_write_backlog()
  File "/usr/lib/python3.8/asyncio/sslproto.py", line 387, in write
    self._ssl_protocol._write_appdata(data)
  File "/usr/local/lib/python3.8/dist-packages/aiorpcx/rawsocket.py", line 118, in write
    self._asyncio_transport.write(framed_message)
  File "/usr/local/lib/python3.8/dist-packages/aiorpcx/session.py", line 153, in _send_message
    await self.transport.write(message)
  File "/usr/local/lib/python3.8/dist-packages/aiorpcx/session.py", line 496, in _throttled_request
    await self._send_message(message)
  <built-in method run of Context object at remote 0x7f09e30eb6c0>
  File "/usr/lib/python3.8/asyncio/events.py", line 81, in _run
    self._context.run(self._callback, *self._args)
  File "/usr/lib/python3.8/asyncio/base_events.py", line 2627, in _run_once
--Type <RET> for more, q to quit, c to continue without paging--
  File "/usr/lib/python3.8/asyncio/base_events.py", line 826, in run_forever
    None, getaddr_func, host, port, family, type, proto, flags)
  File "/usr/lib/python3.8/asyncio/base_events.py", line 603, in run_until_complete
    self.run_forever()
  File "/usr/lib/python3.8/asyncio/runners.py", line 299, in run
  File "/usr/local/bin/electrumx_server", line 35, in main
    asyncio.run(controller.run())
  File "/usr/local/bin/electrumx_server", line 43, in <module>
    main()
(gdb) py-list
 530            except (SystemExit, KeyboardInterrupt):
 531                raise
 532            except BaseException as e:
 533                self._fatal_error(e, 'SSL error in data received')
 534                return
>535
 536            for chunk in ssldata:
 537                self._transport.write(chunk)
 538
 539            for chunk in appdata:
 540                if chunk:
(gdb) bt 
(gdb) bt
#0  0x0000000000569b51 in _PyEval_EvalFrameDefault (f=<optimized out>,
    throwflag=<optimized out>) at ../Python/ceval.c:2487
#1  0x000000000056822a in PyEval_EvalFrameEx (throwflag=0,
    f=Frame 0x1c08960, for file /usr/lib/python3.8/asyncio/sslproto.py, line 535, in feed_appdata (self=<_SSLPipe(_context=<SSLContext at remote 0x7f0a30dc00c0>, _server_side=True, _server_hostname=None, _state='WRAPPED', _incoming=<_ssl.MemoryBIO at remote 0x7f09cc0925b0>, _outgoing=<_ssl.MemoryBIO at remote 0x7f09b919c430>, _sslobj=<SSLObject(_sslobj=<_ssl._SSLSocket at remote 0x7f09cc8c1630>) at remote 0x7f09db428910>, _need_ssldata=False, _handshake_cb=<method at remote 0x7f09c0177a00>, _shutdown_cb=None) at remote 0x7f09db43f160>, data=b'{"jsonrpc": "2.0", "result": null, "id": 18}\n', offset=0, ssldata=[b'\x17\x03\x03\x00\x13<\x8fh\x93\xd3\xd7M\xb9\xbd\xb6\xad\x08\x1bZ\x857\x0c\xe4\xac'], view=<memoryview at remote 0x7f09d295be80>, exc_errno=5)) at ../Python/ceval.c:741
#2  _PyEval_EvalCodeWithName (_co=<optimized out>, globals=<optimized out>,
    locals=<optimized out>, args=<optimized out>, argcount=<optimized out>,
    kwnames=<optimized out>, kwargs=0xea652b8, kwcount=<optimized out>,
    kwstep=1, defs=0x7f0a34816088, defcount=1, kwdefs=0x0, closure=0x0,
    name='feed_appdata', qualname='_SSLPipe.feed_appdata')
    at ../Python/ceval.c:4298
#3  0x00000000005f6033 in _PyFunction_Vectorcall (func=<optimized out>,
    stack=0xea652a0, nargsf=<optimized out>, kwnames=<optimized out>)
    at ../Objects/call.c:435
#4  0x000000000056a136 in _PyObject_Vectorcall (kwnames=0x0,
--Type <RET> for more, q to quit, c to continue without paging--
    nargsf=<optimized out>, args=0xea652a0,
    callable=<function at remote 0x7f0a34828940>)
    at ../Include/cpython/abstract.h:127
#5  call_function (kwnames=0x0, oparg=<optimized out>,
    pp_stack=<synthetic pointer>, tstate=0x143e650) at ../Python/ceval.c:4963
#6  _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>)
    at ../Python/ceval.c:3486
#7  0x00000000005f5e56 in PyEval_EvalFrameEx (throwflag=0,
    f=Frame 0xea650f0, for file /usr/lib/python3.8/asyncio/sslproto.py, line 675, in _process_write_backlog (self=<SSLProtocol(_server_side=True, _server_hostname=None, _sslcontext=<SSLContext at remote 0x7f0a30dc00c0>, _extra={'sslcontext': <...>, 'peercert': None, 'cipher': ('TLS_AES_256_GCM_SHA384', 'TLSv1.3', 256), 'compression': None, 'ssl_object': <SSLObject(_sslobj=<_ssl._SSLSocket at remote 0x7f09cc8c1630>) at remote 0x7f09db428910>}, _write_backlog=<collections.deque at remote 0x7f09cc8c1820>, _write_buffer_size=45, _waiter=None, _loop=<_UnixSelectorEventLoop(_timer_cancelled_count=128, _closed=False, _stopping=False, _ready=<collections.deque at remote 0x7f0a33e29940>, _scheduled=[<TimerHandle at remote 0x7f09cc78c6d0>, <TimerHandle at remote 0x7f09b88b8430>, <TimerHandle at remote 0x7f09c0fee3c0>, <TimerHandle at remote 0x7f09b88b8cf0>, <TimerHandle at remote 0x7f09b58403c0>, <TimerHandle at remote 0x7f09b88b8890>, <TimerHandle at remote 0x7f09cc86eba0>, <TimerHandle at remote 0x7f09c4b17580>, <TimerHand...(truncated)) at ../Python/ceval.c:741
#8  function_code_fastcall (globals=<optimized out>, nargs=<optimized out>,
--Type <RET> for more, q to quit, c to continue without paging--
    args=<optimized out>, co=<optimized out>) at ../Objects/call.c:283
#9  _PyFunction_Vectorcall (func=<optimized out>, stack=<optimized out>,
    nargsf=<optimized out>, kwnames=<optimized out>) at ../Objects/call.c:410
#10 0x000000000056a136 in _PyObject_Vectorcall (kwnames=0x0,
    nargsf=<optimized out>, args=0x2225e0e0,
    callable=<function at remote 0x7f0a34826b80>)
    at ../Include/cpython/abstract.h:127
#11 call_function (kwnames=0x0, oparg=<optimized out>,
    pp_stack=<synthetic pointer>, tstate=0x143e650) at ../Python/ceval.c:4963
#12 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>)
    at ../Python/ceval.c:3486
#13 0x00000000005f5e56 in PyEval_EvalFrameEx (throwflag=0,
    f=Frame 0x2225df60, for file /usr/lib/python3.8/asyncio/sslproto.py, line 599, in _write_appdata (self=<SSLProtocol(_server_side=True, _server_hostname=None, _sslcontext=<SSLContext at remote 0x7f0a30dc00c0>, _extra={'sslcontext': <...>, 'peercert': None, 'cipher': ('TLS_AES_256_GCM_SHA384', 'TLSv1.3', 256), 'compression': None, 'ssl_object': <SSLObject(_sslobj=<_ssl._SSLSocket at remote 0x7f09cc8c1630>) at remote 0x7f09db428910>}, _write_backlog=<collections.deque at remote 0x7f09cc8c1820>, _write_buffer_size=45, _waiter=None, _loop=<_UnixSelectorEventLoop(_timer_cancelled_count=128, _closed=False, _stopping=False, _ready=<collections.deque at remote 0x7f0a33e29940>, _scheduled=[<TimerHandle at remote 0x7f09cc78c6d0>, <TimerHandle at remote 0x7f09b88b8430>, <TimerHandle at remote 0x7f09c0fee3c0>, <TimerHandle at remote 0x7f09b88b8cf0>, <TimerHandle at remote 0x7f--Type <RET> for more, q to quit, c to continue without paging--
09b58403c0>, <TimerHandle at remote 0x7f09b88b8890>, <TimerHandle at remote 0x7f09cc86eba0>, <TimerHandle at remote 0x7f09c4b17580>, <TimerHandle at r...(truncated)) at ../Python/ceval.c:741
#14 function_code_fastcall (globals=<optimized out>, nargs=<optimized out>,
    args=<optimized out>, co=<optimized out>) at ../Objects/call.c:283
#15 _PyFunction_Vectorcall (func=<optimized out>, stack=<optimized out>,
    nargsf=<optimized out>, kwnames=<optimized out>) at ../Objects/call.c:410
#16 0x000000000056a136 in _PyObject_Vectorcall (kwnames=0x0,
    nargsf=<optimized out>, args=0x276b5a10,
    callable=<function at remote 0x7f0a34826940>)
    at ../Include/cpython/abstract.h:127
#17 call_function (kwnames=0x0, oparg=<optimized out>,
    pp_stack=<synthetic pointer>, tstate=0x143e650) at ../Python/ceval.c:4963
#18 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>)
    at ../Python/ceval.c:3486
#19 0x00000000005f5e56 in PyEval_EvalFrameEx (throwflag=0,
    f=Frame 0x276b5890, for file /usr/lib/python3.8/asyncio/sslproto.py, line 387, in write (self=<_SSLProtocolTransport(_ssl_protocol=<SSLProtocol(_server_side=True, _server_hostname=None, _sslcontext=<SSLContext at remote 0x7f0a30dc00c0>, _extra={'sslcontext': <...>, 'peercert': None, 'cipher': ('TLS_AES_256_GCM_SHA384', 'TLSv1.3', 256), 'compression': None, 'ssl_object': <SSLObject(_sslobj=<_ssl._SSLSocket at remote 0x7f09cc8c1630>) at remote 0x7f09db428910>}, _write_backlog=<collections.deque at remote 0x7f09cc8c1820>, _write_buffer_size=45, _waiter=--Type <RET> for more, q to quit, c to continue without paging--
None, _loop=<_UnixSelectorEventLoop(_timer_cancelled_count=128, _closed=False, _stopping=False, _ready=<collections.deque at remote 0x7f0a33e29940>, _scheduled=[<TimerHandle at remote 0x7f09cc78c6d0>, <TimerHandle at remote 0x7f09b88b8430>, <TimerHandle at remote 0x7f09c0fee3c0>, <TimerHandle at remote 0x7f09b88b8cf0>, <TimerHandle at remote 0x7f09b58403c0>, <TimerHandle at remote 0x7f09b88b8890>, <TimerHandle at remote 0x7f09cc86eba0>, <TimerHandle at remote 0x7f09...(truncated)) at ../Python/ceval.c:741
#20 function_code_fastcall (globals=<optimized out>, nargs=<optimized out>,
    args=<optimized out>, co=<optimized out>) at ../Objects/call.c:283
#21 _PyFunction_Vectorcall (func=<optimized out>, stack=<optimized out>,
    nargsf=<optimized out>, kwnames=<optimized out>) at ../Objects/call.c:410
#22 0x000000000056a136 in _PyObject_Vectorcall (kwnames=0x0,
    nargsf=<optimized out>, args=0x7f09e8565568,
    callable=<function at remote 0x7f0a34826160>)
    at ../Include/cpython/abstract.h:127
#23 call_function (kwnames=0x0, oparg=<optimized out>,
    pp_stack=<synthetic pointer>, tstate=0x143e650) at ../Python/ceval.c:4963
#24 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>)
    at ../Python/ceval.c:3486
#25 0x00000000005002e8 in PyEval_EvalFrameEx (throwflag=<optimized out>,
    f=Frame 0x7f09e85653e0, for file /usr/local/lib/python3.8/dist-packages/aiorpcx/rawsocket.py, line 118, in write (self=<RSTransport(session_factory=<functools.partial at remote 0x7f0a30ba9860>, loop=<_UnixSelectorEventLoop(_timer_cancel--Type <RET> for more, q to quit, c to continue without paging--
led_count=128, _closed=False, _stopping=False, _ready=<collections.deque at remote 0x7f0a33e29940>, _scheduled=[<TimerHandle at remote 0x7f09cc78c6d0>, <TimerHandle at remote 0x7f09b88b8430>, <TimerHandle at remote 0x7f09c0fee3c0>, <TimerHandle at remote 0x7f09b88b8cf0>, <TimerHandle at remote 0x7f09b58403c0>, <TimerHandle at remote 0x7f09b88b8890>, <TimerHandle at remote 0x7f09cc86eba0>, <TimerHandle at remote 0x7f09c4b17580>, <TimerHandle at remote 0x7f09c0f4b970>, <TimerHandle at remote 0x7f09cd4d7200>, <TimerHandle at remote 0x7f09bcd904a0>, <TimerHandle at remote 0x7f0a31f9c970>, <TimerHandle at remote 0x7f09bf8655f0>, <TimerHandle at remote 0x7f0a32488c80>, <TimerHandle at remote 0x7f09cd6bf040>, <TimerHandle at remote 0x7f09c0ef0820>, <TimerHandle at remote 0x7f09cd14b430>...(truncated)) at ../Python/ceval.c:741
#26 gen_send_ex (gen=0x7f09b7ee5740, arg=<optimized out>, exc=<optimized out>,
    closing=<optimized out>) at ../Objects/genobject.c:222
#27 0x000000000056b95e in _PyGen_Send (arg=None, gen=0x7f09b7ee5740)
    at ../Objects/genobject.c:292
#28 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>)
    at ../Python/ceval.c:2053
#29 0x00000000005002e8 in PyEval_EvalFrameEx (throwflag=<optimized out>,
    f=Frame 0x7f09c56b3220, for file /usr/local/lib/python3.8/dist-packages/aiorpcx/session.py, line 153, in _send_message (self=<ElectrumX(transport=<RSTransport(session_factory=<functools.partial at remote 0x7f0a30ba9860>, loop=<_UnixSelectorEventLoop(_timer_cancelled_count=128, _closed=False, _stopping=False, _ready=<collections.deque at remote 0x7f0a33e29940>, _scheduled=[<TimerHandle at remot--Type <RET> for more, q to quit, c to continue without paging--
e 0x7f09cc78c6d0>, <TimerHandle at remote 0x7f09b88b8430>, <TimerHandle at remote 0x7f09c0fee3c0>, <TimerHandle at remote 0x7f09b88b8cf0>, <TimerHandle at remote 0x7f09b58403c0>, <TimerHandle at remote 0x7f09b88b8890>, <TimerHandle at remote 0x7f09cc86eba0>, <TimerHandle at remote 0x7f09c4b17580>, <TimerHandle at remote 0x7f09c0f4b970>, <TimerHandle at remote 0x7f09cd4d7200>, <TimerHandle at remote 0x7f09bcd904a0>, <TimerHandle at remote 0x7f0a31f9c970>, <TimerHandle at remote 0x7f09bf8655f0>, <TimerHandle at remote 0x7f0a32488c80>, <TimerHandle at remote 0x7f09cd6bf040>, <TimerHandle at remote 0x7f09c0ef0820>, <TimerHandl...(truncated)) at ../Python/ceval.c:741
#30 gen_send_ex (gen=0x7f09b7ee5c40, arg=<optimized out>, exc=<optimized out>,
    closing=<optimized out>) at ../Objects/genobject.c:222
#31 0x000000000056b95e in _PyGen_Send (arg=None, gen=0x7f09b7ee5c40)
    at ../Objects/genobject.c:292
#32 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>)
    at ../Python/ceval.c:2053
#33 0x00000000005002e8 in PyEval_EvalFrameEx (throwflag=<optimized out>,
    f=Frame 0x284097d0, for file /usr/local/lib/python3.8/dist-packages/aiorpcx/session.py, line 496, in _throttled_request (self=<ElectrumX(transport=<RSTransport(session_factory=<functools.partial at remote 0x7f0a30ba9860>, loop=<_UnixSelectorEventLoop(_timer_cancelled_count=128, _closed=False, _stopping=False, _ready=<collections.deque at remote 0x7f0a33e29940>, _scheduled=[<TimerHandle at remote 0x7f09cc78c6d0>, <TimerHandle at remote 0x7f09b88b8430>, <TimerHandle at remote 0x7f09c0fee3c0>, <TimerHandle at remote 0x7f09b88b8cf0>, <TimerHandle at remo--Type <RET> for more, q to quit, c to continue without paging--
te 0x7f09b58403c0>, <TimerHandle at remote 0x7f09b88b8890>, <TimerHandle at remote 0x7f09cc86eba0>, <TimerHandle at remote 0x7f09c4b17580>, <TimerHandle at remote 0x7f09c0f4b970>, <TimerHandle at remote 0x7f09cd4d7200>, <TimerHandle at remote 0x7f09bcd904a0>, <TimerHandle at remote 0x7f0a31f9c970>, <TimerHandle at remote 0x7f09bf8655f0>, <TimerHandle at remote 0x7f0a32488c80>, <TimerHandle at remote 0x7f09cd6bf040>, <TimerHandle at remote 0x7f09c0ef0820>, <TimerHand...(truncated)) at ../Python/ceval.c:741
#34 gen_send_ex (gen=0x7f09b4c6e540, arg=<optimized out>, exc=<optimized out>,
    closing=<optimized out>) at ../Objects/genobject.c:222
#35 0x00007f0a34eaaef9 in task_step_impl (exc=0x0, task=0x7f09baf4a0e0)
    at ./Modules/_asynciomodule.c:2641
#36 task_step (task=0x7f09baf4a0e0, exc=<optimized out>)
    at ./Modules/_asynciomodule.c:2934
#37 0x00007f0a34eac083 in task_wakeup (o=<optimized out>, task=0x7f09baf4a0e0)
    at ./Modules/_asynciomodule.c:2971
#38 TaskWakeupMethWrapper_call (o=0x7f09da6b2fa0, args=<optimized out>,
    kwds=<optimized out>) at ./Modules/_asynciomodule.c:1791
#39 0x00000000005f3446 in _PyObject_MakeTpCall (
    callable=<TaskWakeupMethWrapper at remote 0x7f09da6b2fa0>,
    args=<optimized out>, nargs=<optimized out>, keywords=<optimized out>)
    at ../Include/internal/pycore_pyerrors.h:13
#40 0x00000000005fdc33 in context_run (self=0x7f09e30eb6c0,
    args=0x7f09c1f45cd8, nargs=2, kwnames=0x0) at ../Python/context.c:634
--Type <RET> for more, q to quit, c to continue without paging--
#41 0x00000000005c3b97 in cfunction_vectorcall_FASTCALL_KEYWORDS (
    func=<built-in method run of Context object at remote 0x7f09e30eb6c0>,
    args=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>)
    at ../Objects/methodobject.c:437
#42 0x00000000005f257d in PyVectorcall_Call (
    callable=<built-in method run of Context object at remote 0x7f09e30eb6c0>,
    tuple=<optimized out>, kwargs=0x0) at ../Objects/call.c:199
#43 0x000000000056fcb6 in do_call_core (kwdict=0x0,
    callargs=(<TaskWakeupMethWrapper at remote 0x7f09da6b2fa0>, <_asyncio.Future at remote 0x7f09b7ee57c0>),
    func=<built-in method run of Context object at remote 0x7f09e30eb6c0>,
    tstate=<optimized out>) at ../Python/ceval.c:4983
#44 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>)
    at ../Python/ceval.c:3559
#45 0x00000000005f5e56 in PyEval_EvalFrameEx (throwflag=0,
    f=Frame 0x7f0a185f6c20, for file /usr/lib/python3.8/asyncio/events.py, line 81, in _run (self=<Handle at remote 0x7f09ccfd15e0>)) at ../Python/ceval.c:741
#46 function_code_fastcall (globals=<optimized out>, nargs=<optimized out>,
    args=<optimized out>, co=<optimized out>) at ../Objects/call.c:283
#47 _PyFunction_Vectorcall (func=<optimized out>, stack=<optimized out>,
    nargsf=<optimized out>, kwnames=<optimized out>) at ../Objects/call.c:410
#48 0x000000000056a136 in _PyObject_Vectorcall (kwnames=0x0,
    nargsf=<optimized out>, args=0x7f09ed010f68,
--Type <RET> for more, q to quit, c to continue without paging--
    callable=<function at remote 0x7f0a3486dc10>)
    at ../Include/cpython/abstract.h:127
#49 call_function (kwnames=0x0, oparg=<optimized out>,
    pp_stack=<synthetic pointer>, tstate=0x143e650) at ../Python/ceval.c:4963
#50 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>)
    at ../Python/ceval.c:3486
#51 0x00000000005f5e56 in PyEval_EvalFrameEx (throwflag=0,
    f=Frame 0x7f09ed010d90, for file /usr/lib/python3.8/asyncio/base_events.py, line 2627, in _run_once (self=<_UnixSelectorEventLoop(_timer_cancelled_count=128, _closed=False, _stopping=False, _ready=<collections.deque at remote 0x7f0a33e29940>, _scheduled=[<TimerHandle at remote 0x7f09cc78c6d0>, <TimerHandle at remote 0x7f09b88b8430>, <TimerHandle at remote 0x7f09c0fee3c0>, <TimerHandle at remote 0x7f09b88b8cf0>, <TimerHandle at remote 0x7f09b58403c0>, <TimerHandle at remote 0x7f09b88b8890>, <TimerHandle at remote 0x7f09cc86eba0>, <TimerHandle at remote 0x7f09c4b17580>, <TimerHandle at remote 0x7f09c0f4b970>, <TimerHandle at remote 0x7f09cd4d7200>, <TimerHandle at remote 0x7f09bcd904a0>, <TimerHandle at remote 0x7f0a31f9c970>, <TimerHandle at remote 0x7f09bf8655f0>, <TimerHandle at remote 0x7f0a32488c80>, <TimerHandle at remote 0x7f09cd6bf040>, <TimerHandle at remote 0x7f09c0ef0820>, <TimerHandle at remote 0x7f09cd14b430>, <TimerHandle at remote 0x7f09ccd99040>, <TimerHandle at remote 0x7f09ccd99c10>, <TimerHandl...(truncated)) at ../Python/ceval.c:741
#52 function_code_fastcall (globals=<optimized out>, nargs=<optimized out>,
    args=<optimized out>, co=<optimized out>) at ../Objects/call.c:283
--Type <RET> for more, q to quit, c to continue without paging--
#53 _PyFunction_Vectorcall (func=<optimized out>, stack=<optimized out>,
    nargsf=<optimized out>, kwnames=<optimized out>) at ../Objects/call.c:410
#54 0x000000000056a136 in _PyObject_Vectorcall (kwnames=0x0,
    nargsf=<optimized out>, args=0x7f0a33d5dd00,
    callable=<function at remote 0x7f0a34780e50>)
    at ../Include/cpython/abstract.h:127
#55 call_function (kwnames=0x0, oparg=<optimized out>,
    pp_stack=<synthetic pointer>, tstate=0x143e650) at ../Python/ceval.c:4963
#56 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>)
    at ../Python/ceval.c:3486
#57 0x00000000005f5e56 in PyEval_EvalFrameEx (throwflag=0,
    f=Frame 0x7f0a33d5db80, for file /usr/lib/python3.8/asyncio/base_events.py, line 826, in run_forever (self=<_UnixSelectorEventLoop(_timer_cancelled_count=128, _closed=False, _stopping=False, _ready=<collections.deque at remote 0x7f0a33e29940>, _scheduled=[<TimerHandle at remote 0x7f09cc78c6d0>, <TimerHandle at remote 0x7f09b88b8430>, <TimerHandle at remote 0x7f09c0fee3c0>, <TimerHandle at remote 0x7f09b88b8cf0>, <TimerHandle at remote 0x7f09b58403c0>, <TimerHandle at remote 0x7f09b88b8890>, <TimerHandle at remote 0x7f09cc86eba0>, <TimerHandle at remote 0x7f09c4b17580>, <TimerHandle at remote 0x7f09c0f4b970>, <TimerHandle at remote 0x7f09cd4d7200>, <TimerHandle at remote 0x7f09bcd904a0>, <TimerHandle at remote 0x7f0a31f9c970>, <TimerHandle at remote 0x7f09bf8655f0>, <TimerHandle at remote 0x7f0a32488c80>, <TimerHandle at remote 0x7f09cd6bf040>, <TimerHandle at remote 0x7f09c0ef0820>, <TimerHandle at remote 0x7f09cd14b430>, <TimerHandle at remo--Type <RET> for more, q to quit, c to continue without paging--
te 0x7f09ccd99040>, <TimerHandle at remote 0x7f09ccd99c10>, <TimerHand...(truncated)) at ../Python/ceval.c:741
#58 function_code_fastcall (globals=<optimized out>, nargs=<optimized out>,
    args=<optimized out>, co=<optimized out>) at ../Objects/call.c:283
#59 _PyFunction_Vectorcall (func=<optimized out>, stack=<optimized out>,
    nargsf=<optimized out>, kwnames=<optimized out>) at ../Objects/call.c:410
#60 0x000000000056a136 in _PyObject_Vectorcall (kwnames=0x0,
    nargsf=<optimized out>, args=0x7f0a33d5d948,
    callable=<function at remote 0x7f0a3477f310>)
    at ../Include/cpython/abstract.h:127
#61 call_function (kwnames=0x0, oparg=<optimized out>,
    pp_stack=<synthetic pointer>, tstate=0x143e650) at ../Python/ceval.c:4963
#62 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>)
    at ../Python/ceval.c:3486
#63 0x00000000005f5e56 in PyEval_EvalFrameEx (throwflag=0,
    f=Frame 0x7f0a33d5d7c0, for file /usr/lib/python3.8/asyncio/base_events.py, line 603, in run_until_complete (self=<_UnixSelectorEventLoop(_timer_cancelled_count=128, _closed=False, _stopping=False, _ready=<collections.deque at remote 0x7f0a33e29940>, _scheduled=[<TimerHandle at remote 0x7f09cc78c6d0>, <TimerHandle at remote 0x7f09b88b8430>, <TimerHandle at remote 0x7f09c0fee3c0>, <TimerHandle at remote 0x7f09b88b8cf0>, <TimerHandle at remote 0x7f09b58403c0>, <TimerHandle at remote 0x7f09b88b8890>, <TimerHandle at remote 0x7f09cc86eba0>, <TimerHandle at remote 0x7f09c4b17580>, <TimerHandle at remote 0x7f09c0f4b970>, <TimerHandle --Type <RET> for more, q to quit, c to continue without paging--
at remote 0x7f09cd4d7200>, <TimerHandle at remote 0x7f09bcd904a0>, <TimerHandle at remote 0x7f0a31f9c970>, <TimerHandle at remote 0x7f09bf8655f0>, <TimerHandle at remote 0x7f0a32488c80>, <TimerHandle at remote 0x7f09cd6bf040>, <TimerHandle at remote 0x7f09c0ef0820>, <TimerHandle at remote 0x7f09cd14b430>, <TimerHandle at remote 0x7f09ccd99040>, <TimerHandle at remote 0x7f09ccd99c10>, <Ti...(truncated)) at ../Python/ceval.c:741
#64 function_code_fastcall (globals=<optimized out>, nargs=<optimized out>,
    args=<optimized out>, co=<optimized out>) at ../Objects/call.c:283
#65 _PyFunction_Vectorcall (func=<optimized out>, stack=<optimized out>,
    nargsf=<optimized out>, kwnames=<optimized out>) at ../Objects/call.c:410
#66 0x000000000056a136 in _PyObject_Vectorcall (kwnames=0x0,
    nargsf=<optimized out>, args=0x18dc468,
    callable=<function at remote 0x7f0a3477f3a0>)
    at ../Include/cpython/abstract.h:127
#67 call_function (kwnames=0x0, oparg=<optimized out>,
    pp_stack=<synthetic pointer>, tstate=0x143e650) at ../Python/ceval.c:4963
#68 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>)
    at ../Python/ceval.c:3486
#69 0x000000000056822a in PyEval_EvalFrameEx (throwflag=0,
    f=Frame 0x18dc2e0, for file /usr/lib/python3.8/asyncio/runners.py, line 299, in run (main=<coroutine at remote 0x7f0a33e31d40>, debug=False, loop=<_UnixSelectorEventLoop(_timer_cancelled_count=128, _closed=False, _stopping=False, _ready=<collections.deque at remote 0x7f0a33e29940>, _scheduled=[<TimerHandle at remot--Type <RET> for more, q to quit, c to continue without paging--
e 0x7f09cc78c6d0>, <TimerHandle at remote 0x7f09b88b8430>, <TimerHandle at remote 0x7f09c0fee3c0>, <TimerHandle at remote 0x7f09b88b8cf0>, <TimerHandle at remote 0x7f09b58403c0>, <TimerHandle at remote 0x7f09b88b8890>, <TimerHandle at remote 0x7f09cc86eba0>, <TimerHandle at remote 0x7f09c4b17580>, <TimerHandle at remote 0x7f09c0f4b970>, <TimerHandle at remote 0x7f09cd4d7200>, <TimerHandle at remote 0x7f09bcd904a0>, <TimerHandle at remote 0x7f0a31f9c970>, <TimerHandle at remote 0x7f09bf8655f0>, <TimerHandle at remote 0x7f0a32488c80>, <TimerHandle at remote 0x7f09cd6bf040>, <TimerHandle at remote 0x7f09c0ef0820>, <TimerHandle at remote 0x7f09cd14b430>, <TimerHandle at remote 0x7f09ccd99040>, <TimerHandl...(truncated)) at ../Python/ceval.c:741
#70 _PyEval_EvalCodeWithName (_co=<optimized out>, globals=<optimized out>,
    locals=<optimized out>, args=<optimized out>, argcount=<optimized out>,
    kwnames=<optimized out>, kwargs=0x7f0a34e907b8, kwcount=<optimized out>,
    kwstep=1, defs=0x0, defcount=0, kwdefs={'debug': False}, closure=0x0,
    name='run', qualname='run') at ../Python/ceval.c:4298
#71 0x00000000005f6033 in _PyFunction_Vectorcall (func=<optimized out>,
    stack=0x7f0a34e907b0, nargsf=<optimized out>, kwnames=<optimized out>)
    at ../Objects/call.c:435
#72 0x000000000056ef97 in _PyObject_Vectorcall (kwnames=0x0,
    nargsf=<optimized out>, args=0x7f0a34e907b0,
    callable=<function at remote 0x7f0a34e715e0>)
    at ../Include/cpython/abstract.h:127
#73 call_function (kwnames=0x0, oparg=<optimized out>,
--Type <RET> for more, q to quit, c to continue without paging--
    pp_stack=<synthetic pointer>, tstate=0x143e650) at ../Python/ceval.c:4963
#74 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>)
    at ../Python/ceval.c:3469
#75 0x00000000005f5e56 in PyEval_EvalFrameEx (throwflag=0,
    f=Frame 0x7f0a34e90610, for file /usr/local/bin/electrumx_server, line 35, in main (log_fmt='%(levelname)s:%(name)s:%(message)s', handler=<StreamHandler(filters=[], _name=None, level=0, formatter=<CompactFormatter(_style=<PercentStyle(_fmt='%(levelname)s:%(name)s:%(message)s') at remote 0x7f0a34f13790>, _fmt='%(levelname)s:%(name)s:%(message)s', datefmt=None) at remote 0x7f0a34e1b040>, lock=<_thread.RLock at remote 0x7f0a34f13a20>, stream=<_io.TextIOWrapper at remote 0x7f0a34f9ce10>) at remote 0x7f0a34ec1fa0>, logger=<Logger(filters=[], name='electrumx', level=20, parent=<RootLogger(filters=[], name='root', level=30, parent=None, propagate=True, handlers=[], disabled=False, _cache={}) at remote 0x7f0a34d33130>, propagate=False, handlers=[<...>], disabled=False, _cache={}, manager=<Manager(root=<...>, disable=0, emittedNoHandlerWarning=False, loggerDict={'concurrent.futures': <Logger(filters=[], name='concurrent.futures', level=0, parent=<...>, propagate=True, handlers=[], disabled=False, _cache={}, manager=<......(truncated)) at ../Python/ceval.c:741
#76 function_code_fastcall (globals=<optimized out>, nargs=<optimized out>,
    args=<optimized out>, co=<optimized out>) at ../Objects/call.c:283
#77 _PyFunction_Vectorcall (func=<optimized out>, stack=<optimized out>,
    nargsf=<optimized out>, kwnames=<optimized out>) at ../Objects/call.c:410
#78 0x0000000000569f5e in _PyObject_Vectorcall (kwnames=0x0,
--Type <RET> for more, q to quit, c to continue without paging--
    nargsf=<optimized out>, args=0x7f0a34f027b0,
    callable=<function at remote 0x7f0a34e5df70>)
    at ../Include/cpython/abstract.h:127
#79 call_function (kwnames=0x0, oparg=<optimized out>,
    pp_stack=<synthetic pointer>, tstate=0x143e650) at ../Python/ceval.c:4963
#80 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>)
    at ../Python/ceval.c:3500
#81 0x000000000056822a in PyEval_EvalFrameEx (throwflag=0,
    f=Frame 0x7f0a34f02640, for file /usr/local/bin/electrumx_server, line 43, in <module> ()) at ../Python/ceval.c:741
#82 _PyEval_EvalCodeWithName (_co=<optimized out>, globals=<optimized out>,
    locals=<optimized out>, args=<optimized out>, argcount=<optimized out>,
    kwnames=<optimized out>, kwargs=0x0, kwcount=<optimized out>, kwstep=2,
    defs=0x0, defcount=0, kwdefs=0x0, closure=0x0, name=0x0, qualname=0x0)
    at ../Python/ceval.c:4298
#83 0x000000000068c1e7 in PyEval_EvalCodeEx (closure=0x0, kwdefs=0x0,
    defcount=0, defs=0x0, kwcount=0, kws=0x0, argcount=0, args=0x0,
    locals=<optimized out>, globals=<optimized out>, _co=<optimized out>)
    at ../Python/ceval.c:4327
#84 PyEval_EvalCode (co=<optimized out>, globals=<optimized out>,
    locals=<optimized out>) at ../Python/ceval.c:718
#85 0x000000000067d5a1 in run_eval_code_obj (co=0x7f0a34f3ef50,
    globals={'__name__': '__main__', '__doc__': 'Script to kick off the server.'--Type <RET> for more, q to quit, c to continue without paging--
, '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='/usr/local/bin/electrumx_server') at remote 0x7f0a34f50f70>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7f0a34f8e0e0>, '__file__': '/usr/local/bin/electrumx_server', '__cached__': None, 'asyncio': <module at remote 0x7f0a34e5fd60>, 'logging': <module at remote 0x7f0a34dda180>, 'sys': <module at remote 0x7f0a34f82e00>, 'Controller': <type at remote 0x18f5070>, 'Env': <type at remote 0x190f1e0>, 'CompactFormatter': <type at remote 0x15f1c20>, 'make_logger': <function at remote 0x7f0a34683b80>, 'main': <function at remote 0x7f0a34e5df70>},
    locals={'__name__': '__main__', '__doc__': 'Script to kick off the server.', '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='/usr/local/bin/electrumx_server') at remote 0x7f0a34f50f70>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7f0a34f8e0e0>, '__file__': '/usr/local/bin/electrumx_server', '__cached__': None, 'asyncio': <module at remote 0x7f0a34e5fd60>, 'logging': <module at remote 0x7f0a34dda180>, 'sys': <module at remote 0x7f0a34f82e00>, 'Controller': <type at remote 0x18f5070>, 'Env': <type at remote 0x190f1e0>, 'CompactFormatter': <type at remote 0x15f1c20>, 'make_logger': <function at remote 0x7f0a34683b80>, 'main': <function at remote 0x7f0a34e5df70>}) at ../Python/pythonrun.c:1125
#86 0x000000000067d61f in run_mod (mod=<optimized out>,
    filename=<optimized out>,
    globals={'__name__': '__main__', '__doc__': 'Script to kick off the server.', '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='/u--Type <RET> for more, q to quit, c to continue without paging--
sr/local/bin/electrumx_server') at remote 0x7f0a34f50f70>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7f0a34f8e0e0>, '__file__': '/usr/local/bin/electrumx_server', '__cached__': None, 'asyncio': <module at remote 0x7f0a34e5fd60>, 'logging': <module at remote 0x7f0a34dda180>, 'sys': <module at remote 0x7f0a34f82e00>, 'Controller': <type at remote 0x18f5070>, 'Env': <type at remote 0x190f1e0>, 'CompactFormatter': <type at remote 0x15f1c20>, 'make_logger': <function at remote 0x7f0a34683b80>, 'main': <function at remote 0x7f0a34e5df70>},
    locals={'__name__': '__main__', '__doc__': 'Script to kick off the server.', '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='/usr/local/bin/electrumx_server') at remote 0x7f0a34f50f70>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7f0a34f8e0e0>, '__file__': '/usr/local/bin/electrumx_server', '__cached__': None, 'asyncio': <module at remote 0x7f0a34e5fd60>, 'logging': <module at remote 0x7f0a34dda180>, 'sys': <module at remote 0x7f0a34f82e00>, 'Controller': <type at remote 0x18f5070>, 'Env': <type at remote 0x190f1e0>, 'CompactFormatter': <type at remote 0x15f1c20>, 'make_logger': <function at remote 0x7f0a34683b80>, 'main': <function at remote 0x7f0a34e5df70>}, flags=<optimized out>, arena=<optimized out>)
    at ../Python/pythonrun.c:1147
#87 0x000000000067d6db in PyRun_FileExFlags (fp=0x143c340,
    filename_str=<optimized out>, start=<optimized out>,
    globals={'__name__': '__main__', '__doc__': 'Script to kick off the server.', '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='/u--Type <RET> for more, q to quit, c to continue without paging--
sr/local/bin/electrumx_server') at remote 0x7f0a34f50f70>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7f0a34f8e0e0>, '__file__': '/usr/local/bin/electrumx_server', '__cached__': None, 'asyncio': <module at remote 0x7f0a34e5fd60>, 'logging': <module at remote 0x7f0a34dda180>, 'sys': <module at remote 0x7f0a34f82e00>, 'Controller': <type at remote 0x18f5070>, 'Env': <type at remote 0x190f1e0>, 'CompactFormatter': <type at remote 0x15f1c20>, 'make_logger': <function at remote 0x7f0a34683b80>, 'main': <function at remote 0x7f0a34e5df70>},
    locals={'__name__': '__main__', '__doc__': 'Script to kick off the server.', '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='/usr/local/bin/electrumx_server') at remote 0x7f0a34f50f70>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7f0a34f8e0e0>, '__file__': '/usr/local/bin/electrumx_server', '__cached__': None, 'asyncio': <module at remote 0x7f0a34e5fd60>, 'logging': <module at remote 0x7f0a34dda180>, 'sys': <module at remote 0x7f0a34f82e00>, 'Controller': <type at remote 0x18f5070>, 'Env': <type at remote 0x190f1e0>, 'CompactFormatter': <type at remote 0x15f1c20>, 'make_logger': <function at remote 0x7f0a34683b80>, 'main': <function at remote 0x7f0a34e5df70>}, closeit=1, flags=0x7ffcb682af78) at ../Python/pythonrun.c:1063
#88 0x000000000067da6e in PyRun_SimpleFileExFlags (fp=0x143c340,
    filename=<optimized out>, closeit=1, flags=0x7ffcb682af78)
    at ../Python/pythonrun.c:428
#89 0x00000000006b6132 in pymain_run_file (cf=0x7ffcb682af78, config=0x143d990)
    at ../Modules/main.c:381
--Type <RET> for more, q to quit, c to continue without paging--
#90 pymain_run_python (exitcode=0x7ffcb682af70) at ../Modules/main.c:606
#91 Py_RunMain () at ../Modules/main.c:685
#92 0x00000000006b64bd in Py_BytesMain (argc=<optimized out>,
    argv=<optimized out>) at ../Modules/main.c:739
#93 0x00007f0a3522b0b3 in __libc_start_main (main=0x4eec80 <main>, argc=2,
    argv=0x7ffcb682b158, init=<optimized out>, fini=<optimized out>,
    rtld_fini=<optimized out>, stack_end=0x7ffcb682b148)
    at ../csu/libc-start.c:308
#94 0x00000000005f927e in _start () at ../Objects/bytesobject.c:2560
(gdb) py-locals
self = <_SSLPipe(_context=<SSLContext at remote 0x7f0a30dc00c0>, _server_side=True, _server_hostname=None, _state='WRAPPED', _incoming=<_ssl.MemoryBIO at remote 0x7f09cc0925b0>, _outgoing=<_ssl.MemoryBIO at remote 0x7f09b919c430>, _sslobj=<SSLObject(_sslobj=<_ssl._SSLSocket at remote 0x7f09cc8c1630>) at remote 0x7f09db428910>, _need_ssldata=False, _handshake_cb=<method at remote 0x7f09c0177a00>, _shutdown_cb=None) at remote 0x7f09db43f160>
data = b'{"jsonrpc": "2.0", "result": null, "id": 18}\n'
offset = 0
ssldata = [b'\x17\x03\x03\x00\x13<\x8fh\x93\xd3\xd7M\xb9\xbd\xb6\xad\x08\x1bZ\x857\x0c\xe4\xac']
view = <memoryview at remote 0x7f09d295be80>
exc_errno = 5

The server is stuck in a busy loop.
The deepest common stack frame is at
https://github.com/python/cpython/blob/v3.8.5/Lib/asyncio/sslproto.py#L675

Some more gdb after some `cont`s and SIGINTs 
(gdb) cont
Continuing.
^C
Thread 1 "electrumx_serve" received signal SIGINT, Interrupt.
memory_getbuf (self=0x7f09b69a3880, view=0x7ffcb6829200, flags=0)
    at ../Objects/memoryobject.c:1441
1441    ../Objects/memoryobject.c: No such file or directory.
(gdb) py-bt
Traceback (most recent call first):
  File "/usr/lib/python3.8/ssl.py", line 897, in write
    return self._sslobj.write(data)
  File "/usr/lib/python3.8/asyncio/sslproto.py", line 262, in feed_appdata
    offset += self._sslobj.write(view[offset:])
  File "/usr/lib/python3.8/asyncio/sslproto.py", line 675, in _process_write_backlog
    ssldata, offset = self._sslpipe.feed_appdata(data, offset)
  File "/usr/lib/python3.8/asyncio/sslproto.py", line 599, in _write_appdata
    self._process_write_backlog()
  File "/usr/lib/python3.8/asyncio/sslproto.py", line 387, in write
    self._ssl_protocol._write_appdata(data)
  File "/usr/local/lib/python3.8/dist-packages/aiorpcx/rawsocket.py", line 118, in write
    self._asyncio_transport.write(framed_message)
  File "/usr/local/lib/python3.8/dist-packages/aiorpcx/session.py", line 153, in _send_message
    await self.transport.write(message)
  File "/usr/local/lib/python3.8/dist-packages/aiorpcx/session.py", line 496, in _throttled_request
    await self._send_message(message)
  <built-in method run of Context object at remote 0x7f09e30eb6c0>
  File "/usr/lib/python3.8/asyncio/events.py", line 81, in _run
--Type <RET> for more, q to quit, c to continue without paging--
    self._context.run(self._callback, *self._args)
  File "/usr/lib/python3.8/asyncio/base_events.py", line 2627, in _run_once
  File "/usr/lib/python3.8/asyncio/base_events.py", line 826, in run_forever
    None, getaddr_func, host, port, family, type, proto, flags)
  File "/usr/lib/python3.8/asyncio/base_events.py", line 603, in run_until_complete
    self.run_forever()
  File "/usr/lib/python3.8/asyncio/runners.py", line 299, in run
  File "/usr/local/bin/electrumx_server", line 35, in main
    asyncio.run(controller.run())
  File "/usr/local/bin/electrumx_server", line 43, in <module>
    main()
(gdb)
(gdb) cont
Continuing.
^C
(gdb) py-bt
Traceback (most recent call first):
  <built-in method len of module object at remote 0x7f0a34f8e0e0>
  File "/usr/lib/python3.8/asyncio/sslproto.py", line 535, in feed_appdata

  File "/usr/lib/python3.8/asyncio/sslproto.py", line 675, in _process_write_backlog
    ssldata, offset = self._sslpipe.feed_appdata(data, offset)
  File "/usr/lib/python3.8/asyncio/sslproto.py", line 599, in _write_appdata
    self._process_write_backlog()
  File "/usr/lib/python3.8/asyncio/sslproto.py", line 387, in write
    self._ssl_protocol._write_appdata(data)
  File "/usr/local/lib/python3.8/dist-packages/aiorpcx/rawsocket.py", line 118, in write
    self._asyncio_transport.write(framed_message)
  File "/usr/local/lib/python3.8/dist-packages/aiorpcx/session.py", line 153, in _send_message
    await self.transport.write(message)
  File "/usr/local/lib/python3.8/dist-packages/aiorpcx/session.py", line 496, in _throttled_request
    await self._send_message(message)
  <built-in method run of Context object at remote 0x7f09e30eb6c0>
  File "/usr/lib/python3.8/asyncio/events.py", line 81, in _run
    self._context.run(self._callback, *self._args)
--Type <RET> for more, q to quit, c to continue without paging--
  File "/usr/lib/python3.8/asyncio/base_events.py", line 2627, in _run_once
  File "/usr/lib/python3.8/asyncio/base_events.py", line 826, in run_forever
    None, getaddr_func, host, port, family, type, proto, flags)
  File "/usr/lib/python3.8/asyncio/base_events.py", line 603, in run_until_complete
    self.run_forever()
  File "/usr/lib/python3.8/asyncio/runners.py", line 299, in run
  File "/usr/local/bin/electrumx_server", line 35, in main
    asyncio.run(controller.run())
  File "/usr/local/bin/electrumx_server", line 43, in <module>
    main()
    main()
(gdb) cont
Continuing.
^C
Thread 1 "electrumx_serve" received signal SIGINT, Interrupt.
PySlice_New (start=0, stop=None, step=0x0) at ../Objects/sliceobject.c:123
123     ../Objects/sliceobject.c: No such file or directory.
(gdb)
Continuing.
^C
Thread 1 "electrumx_serve" received signal SIGINT, Interrupt.
0x00000000005d4390 in long_richcompare (
    self=<SSLErrorNumber(_value_=5, _name_='SSL_ERROR_SYSCALL', __objclass__=<EnumMeta(_generate_next_value_=<function at remote 0x7f0a34e119d0>, __doc__='An enumeration.', __module__='ssl', _member_names_=['SSL_ERROR_SSL', 'SSL_ERROR_WANT_READ', 'SSL_ERROR_WANT_WRITE', 'SSL_ERROR_WANT_X509_LOOKUP', 'SSL_ERROR_SYSCALL', 'SSL_ERROR_ZERO_RETURN', 'SSL_ERROR_WANT_CONNECT', 'SSL_ERROR_EOF', 'SSL_ERROR_INVALID_ERROR_CODE'], _member_map_={'SSL_ERROR_SSL': <SSLErrorNumber(_value_=1, _name_='SSL_ERROR_SSL', __objclass__=<...>) at remote 0x7f0a3487fe00>, 'SSL_ERROR_WANT_READ': <SSLErrorNumber(_value_=2, _name_='SSL_ERROR_WANT_READ', __objclass__=<...>) at remote 0x7f0a3487fe80>, 'SSL_ERROR_WANT_WRITE': <SSLErrorNumber(_value_=3, _name_='SSL_ERROR_WANT_WRITE', __objclass__=<...>) at remote 0x7f0a3487ff00>, 'SSL_ERROR_WANT_X509_LOOKUP': <SSLErrorNumber(_value_=4, _name_='SSL_ERROR_WANT_X509_LOOKUP', __objclass__=<...>) at remote 0x7f0a3487ff80>, 'SSL_ERROR_SYSCALL': <...>, 'SSL_ERROR_ZERO_RETURN': <SSLErrorNumber(_value_=6, _nam...(truncated), other=5, op=2) at ../Objects/longobject.c:3083
3083    ../Objects/longobject.c: No such file or directory.
(gdb) py-bt
Traceback (most recent call first):
  File "/usr/lib/python3.8/asyncio/sslproto.py", line 526, in feed_appdata
    return
  File "/usr/lib/python3.8/asyncio/sslproto.py", line 675, in _process_write_backlog
    ssldata, offset = self._sslpipe.feed_appdata(data, offset)
  File "/usr/lib/python3.8/asyncio/sslproto.py", line 599, in _write_appdata
    self._process_write_backlog()
  File "/usr/lib/python3.8/asyncio/sslproto.py", line 387, in write
    self._ssl_protocol._write_appdata(data)
  File "/usr/local/lib/python3.8/dist-packages/aiorpcx/rawsocket.py", line 118, in write
    self._asyncio_transport.write(framed_message)
  File "/usr/local/lib/python3.8/dist-packages/aiorpcx/session.py", line 153, in _send_message
    await self.transport.write(message)
  File "/usr/local/lib/python3.8/dist-packages/aiorpcx/session.py", line 496, in _throttled_request
    await self._send_message(message)
  <built-in method run of Context object at remote 0x7f09e30eb6c0>
  File "/usr/lib/python3.8/asyncio/events.py", line 81, in _run
    self._context.run(self._callback, *self._args)
  File "/usr/lib/python3.8/asyncio/base_events.py", line 2627, in _run_once
--Type <RET> for more, q to quit, c to continue without paging--
(gdb)
(gdb) cont
Continuing.
^C
Thread 1 "electrumx_serve" received signal SIGINT, Interrupt.
0x00000000004f7fa0 in builtin_len (module=<optimized out>, obj=<optimized out>)
    at ../Objects/longobject.c:61
61      ../Objects/longobject.c: No such file or directory.
(gdb) py-bt
Traceback (most recent call first):
  <built-in method len of module object at remote 0x7f0a34f8e0e0>
  File "/usr/lib/python3.8/asyncio/sslproto.py", line 535, in feed_appdata

  File "/usr/lib/python3.8/asyncio/sslproto.py", line 675, in _process_write_backlog
    ssldata, offset = self._sslpipe.feed_appdata(data, offset)
  File "/usr/lib/python3.8/asyncio/sslproto.py", line 599, in _write_appdata
    self._process_write_backlog()
  File "/usr/lib/python3.8/asyncio/sslproto.py", line 387, in write
    self._ssl_protocol._write_appdata(data)
  File "/usr/local/lib/python3.8/dist-packages/aiorpcx/rawsocket.py", line 118, in write
    self._asyncio_transport.write(framed_message)
  File "/usr/local/lib/python3.8/dist-packages/aiorpcx/session.py", line 153, in _send_message
    await self.transport.write(message)
  File "/usr/local/lib/python3.8/dist-packages/aiorpcx/session.py", line 496, in _throttled_request
    await self._send_message(message)
  <built-in method run of Context object at remote 0x7f09e30eb6c0>
  File "/usr/lib/python3.8/asyncio/events.py", line 81, in _run
    self._context.run(self._callback, *self._args)
--Type <RET> for more, q to quit, c to continue without paging--
(gdb) cont
Continuing.
^C
Thread 1 "electrumx_serve" received signal SIGINT, Interrupt.
PyType_IsSubtype (b=0x907020 <_PyNone_Type>, a=0x90a860 <PyUnicode_Type>)
    at ../Objects/typeobject.c:1370
1370    ../Objects/typeobject.c: No such file or directory.
(gdb) py-bt
Traceback (most recent call first):
  File "/usr/lib/python3.8/asyncio/sslproto.py", line 268, in feed_appdata
    if exc.reason == 'PROTOCOL_IS_SHUTDOWN':
  File "/usr/lib/python3.8/asyncio/sslproto.py", line 675, in _process_write_backlog
    ssldata, offset = self._sslpipe.feed_appdata(data, offset)
  File "/usr/lib/python3.8/asyncio/sslproto.py", line 599, in _write_appdata
    self._process_write_backlog()
  File "/usr/lib/python3.8/asyncio/sslproto.py", line 387, in write
    self._ssl_protocol._write_appdata(data)
  File "/usr/local/lib/python3.8/dist-packages/aiorpcx/rawsocket.py", line 118, in write
    self._asyncio_transport.write(framed_message)
  File "/usr/local/lib/python3.8/dist-packages/aiorpcx/session.py", line 153, in _send_message
    await self.transport.write(message)
  File "/usr/local/lib/python3.8/dist-packages/aiorpcx/session.py", line 496, in _throttled_request
    await self._send_message(message)
  <built-in method run of Context object at remote 0x7f09e30eb6c0>
  File "/usr/lib/python3.8/asyncio/events.py", line 81, in _run
    self._context.run(self._callback, *self._args)
  File "/usr/lib/python3.8/asyncio/base_events.py", line 2627, in _run_once
--Type <RET> for more, q to quit, c to continue without paging--
(gdb) cont
Continuing.
^C
Thread 1 "electrumx_serve" received signal SIGINT, Interrupt.
__GI___errno_location () at errno-loc.c:25
25      errno-loc.c: No such file or directory.
(gdb) py-bt
Traceback (most recent call first):
  File "/usr/lib/python3.8/ssl.py", line 897, in write
    return self._sslobj.write(data)
  File "/usr/lib/python3.8/asyncio/sslproto.py", line 262, in feed_appdata
    offset += self._sslobj.write(view[offset:])
  File "/usr/lib/python3.8/asyncio/sslproto.py", line 675, in _process_write_backlog
    ssldata, offset = self._sslpipe.feed_appdata(data, offset)
  File "/usr/lib/python3.8/asyncio/sslproto.py", line 599, in _write_appdata
    self._process_write_backlog()
  File "/usr/lib/python3.8/asyncio/sslproto.py", line 387, in write
    self._ssl_protocol._write_appdata(data)
  File "/usr/local/lib/python3.8/dist-packages/aiorpcx/rawsocket.py", line 118, in write
    self._asyncio_transport.write(framed_message)
  File "/usr/local/lib/python3.8/dist-packages/aiorpcx/session.py", line 153, in _send_message
    await self.transport.write(message)
  File "/usr/local/lib/python3.8/dist-packages/aiorpcx/session.py", line 496, in _throttled_request
    await self._send_message(message)
  <built-in method run of Context object at remote 0x7f09e30eb6c0>
  File "/usr/lib/python3.8/asyncio/events.py", line 81, in _run

Note that around line https://github.com/python/cpython/blob/v3.8.5/Lib/asyncio/sslproto.py#L262,
exc_errno = 5 corresponds to

>>> ssl.SSL_ERROR_SYSCALL
<SSLErrorNumber.SSL_ERROR_SYSCALL: 5>

re ssl.SSL_ERROR_SYSCALL, see:

@SomberNight
Copy link
Member

Note that around line https://github.com/python/cpython/blob/v3.8.5/Lib/asyncio/sslproto.py#L262,
exc_errno = 5 corresponds to

>>> ssl.SSL_ERROR_SYSCALL
<SSLErrorNumber.SSL_ERROR_SYSCALL: 5>

I have been running with a patched python for the last two days, and did not yet get a hang.
The server looks to be functioning well.
The only change I made is not suppressing these ssl.SSL_ERROR_SYSCALL errors.
SomberNight/cpython@8296a26

@SomberNight
Copy link
Member

SomberNight commented May 4, 2021 •
edited
Loading

I've opened upstream python bug report: https://bugs.python.org/issue44036

@SomberNight
Copy link
Member

SomberNight commented May 8, 2021 •
edited
Loading

Note: Another workaround is to set EVENT_LOOP_POLICY=uvloop (which needs uvloop as an additional dependency)
https://electrumx-spesmilo.readthedocs.io/en/latest/environment.html#envvar-EVENT_LOOP_POLICY

I haven't tested this but it should work as the problematic code is in the SSL wrappers in the stdlib asyncio code, which is not used with uvloop.

@lukechilds
Copy link
Contributor

I've been hitting this for a while too, I get CPU pegged to 100% with the server unresponsive after a month or so of running and I have to kill the process. I've just changed bitcoin.lukechilds.co over to use EVENT_LOOP_POLICY=uvloop to see if that resolves it.

For anyone else who ends up here and uses my Docker images, I've published an alternate version (lukechilds/electrumx:v1.16.0-uvloop) which you can use as a drop in replacement.

lukechilds added a commit to lukechilds/docker-electrumx that referenced this issue May 16, 2021
@lukechilds
Use uvloop
550590c 
Resolves spesmilo/electrumx#92
@justinvforvendetta justinvforvendetta mentioned this issue Feb 22, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@Emzy @lukechilds @SomberNight @ZacharyACoon