This repository has been archived by the owner on Nov 6, 2020. It is now read-only.
Light client sometimes deadlocks from incoming RPC requests #8918
Labels
F2-bug 🐞
The client fails to follow expected behavior.
M4-core ⛓
Core client code / Rust.
P5-sometimesoon 🌲
Issue is worth doing soon.
Milestone
Comments
tomaka
added
F2-bug 🐞
|
Could be fixed by #8977 ? |
|
I could reproduce and trace. Any RPC such as getBalance don't get answered.
Note that the last one, the problem seems to be the peer discovery.. oscillating between 1 and 0. |
|
@niklasad1 this is correct |
andresilva
pushed a commit
that referenced
this issue
Sep 4, 2018
This PR is fixing deadlock for #8918 It avoids some recursive calls on light_sync by making state check optional for Informant. The current behavior is to display the information when informant checks if block is major version. This change a bit the informant behavior, but not on most cases. To remember where and how this kind of deadlock are likely to happen (not seen with Parkinglot deadlock detection because it uses std condvar), I am adding a description of the deadlock. Also, for the reviewers there may be better solution than modifying the informant. ### Thread1 - ethcore/sync/light_sync/mod.rs A call to the light handler through any Io (having a loop of rpc query running on like client makes the dead lock way more likely). At the end of those calls we systematically call `maintain_sync` method. Here maintain_sync locks `state` (it is the deadlock cause), with a write purpose `maintain_sync` -> `begin_search` with the state locked open `begin_search` -> lightcliennt `flush_queue` method - ethcore/light/src/client/mod.rs `flush_queue` -> `flush` on queue (HeaderQueue aka VerificationQueue of headers) - ethcore/src/verification/queue/mod.rs Condition there is some unverified or verifying content `flush` wait on a condvar until the queue is empty. The only way to unlock condvar is that worker is empty and unlock it (so thread 2 is Verification worker). ### Thread2 A verification worker at the end of a verify loop (new block). - ethcore/src/verification/queue/mod.rs thread loops on `verify` method. End of loop condition is_ready -> Import the block immediately calls `set_sync` on QueueSignal which send a BlockVerified ClientIoMessage in inner channel (IoChannel of ClientIoMessage) using `send_sync` - util/io/src/service_mio.rs IoChannel `send_sync` method calls all handlers with `message` method; one of the handlers is ImportBlocks IoHandler (with a single inner Client service field) - ethcore/light/src/client/service.rs `message` trigger inner method `import_verified` - core/light/src/client/mod.rs `import_verified` at the very end notify the listeners of a new_headers, one of the listeners is Informant `listener` method - parity/informant.rs `newHeaders` run up to call to `is_major_importing` on its target (again clinet) - ethcore/sync/src/light_sync/mod.rs Here `is_major_importing` tries to get state lock (read purpose only) but cannot because of previous state lock, thus deadlock
|
Closed by #9385 |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Sometimes, when making an RPC request, the light client doesn't answer and becomes unresponsive through the JSON-RPC interface.
The logs don't show anything out of the ordinary, except a lot of:
However this seems to happen even when the node was running correctly.
If I had an hypothesis, it would be that we disconnect from the peer we're making a request from. But I'm really not sure that this is the problem.
cc @Tbaut @amaurymartiny @rphmeier
The text was updated successfully, but these errors were encountered: