Road to eth2 multiclients in DAppNode ⧫

[dapplion]

DAppNode supports diversification of clients in the eth2 (POS consensus layer) for the good of the network. However we are not able to support any other client but Prysm since it's the only one that features a UI. Note that the value proposition of DAppNode is to reduce the technical complexity of running nodes and other DApps, which means not having to interact with a CLI.

We are exploring different paths to integrate a UI into the usage flow of other clients

A: Add a UI to the client

Like Prysm did, embed a UI in the validator client of that implementation with a supporting backend API to perform keymanagment. The cost of this route is very high and isn't likely to be completed anytime soon.

• Prysm: ✔️ Already has a UI + backend
• Lighthouse: ⏳ UI in the works, unknown ETA. Backend already implemented
• Teku: ❌ Not actively pursuing this
• Nimbus: ❌ Interested in developing one potentially
• Lodestar: ❌ Not actively pursuing this

B: Add only UI backend to the client

Given that there's already a functional UI developed, clients could implement just the keymanager UI backend to their signer binaries (either validator client or remote signer) and achieve the same result. This option should be far cheaper to develop, since it does not involve any front-end work: just exposing existing functionality through an HTTP API.

This PR Add local validator manager API #151 ethereum/beacon-APIs#151 wants to signal the minimal routes to develop to client teams. Note that no client supports any version of this proposed simple keymanager API, so it's unclear how soon this option would materialize:

• Prysm: ⏳ Interested in following this standard once there's bandwidth
• Lighthouse: ⏳ Might follow the API, however it conflicts with the current implemented backend
• Teku: ⁉️ No signal
• Nimbus: ⏳ Might be interested in implementing the backed
• Lodestar: ❌ Not actively pursuing this

C: Use a remote signer with a UI attached

Connect a validator client of some implementation to a unique remote signer that has a UI for key managment. This solution would allow a safe way to switching between clients. However remote signer support is not universal

• Prysm: ⏳ Pending implementation, high priority item
• Lighthouse: ⏳ developed and merged to unstable, to be released soon [Merged by Bors] - Web3Signer support for VC sigp/lighthouse#2522
• Teku: ✔️ Supports
• Nimbus: ⏳ Not yet, requested Support remote signer EIP-3030 status-im/nimbus-eth2#2948
• Lodestar: ⏳ To be implemented soon (in pipeline)

EIP-3030: BLS Remote Signer HTTP API https://eips.ethereum.org/EIPS/eip-3030 wants to standardize the remote signer API, such that any validator client can connect to it. However the current situation is that remote signer clients are not standardized, so interop might not be possible yet. It's unclear if all clients are gonna support Altair in time for their remote signers.

Regarding remote signers, this implementation by Teku is actively used in production and uses a slashing protection db 👍 https://github.com/ConsenSys/web3signer/. However, it requires a CLI for managing keys and slashing protection data, so again missing a UI here. Note it also requires a PostgreSQL instance.

D: Use Prysm validator client with other beacon nodes

The Prysm validator client is the only eth2 keymanager implementation with UI support. Given that the beacon node does most of the work and poses the highest risk of consensus failure, users running a Prysm validator client + a non-Prysm beacon node would still contribute significantly to the network client diversity.

The eth2 API for validator client -> beacon node interactions is standardized in https://github.com/ethereum/eth2.0-APIs but Prysm has just started to follow it. We should test the matrix of compatibility between clients to study how feasible this option is:

• Prysm vc + Lighthouse: ❌ Prysm won't support eth2 standard API in the VC till the merge
• Prysm vc + Teku: ❌ Prysm won't support eth2 standard API in the VC till the merge
• Prysm vc + Nimbus: ❌ Prysm won't support eth2 standard API in the VC till the merge
• Prysm vc + Lodestar: ❌ Prysm won't support eth2 standard API in the VC till the merge

[dapplion]

To explore D: Use Prysm validator client with other beacon nodes:

1. Run one beacon node (Lighthouse or Teku or Nimbus) in a sufficiently powerful machine (4-8GB or RAM). You may use docker-compose with their distributed builds.
2. Run one Prysm validator with their v2 release (currently in beta).
3. Connect the Prysm validator to the (1) node API.
4. Monitor logs and performance to ensure all is good.

[tropicar]

In this probe, we have tried to connect a Prysm validator ( version v2.0.0-rc) to a Lighthouse Beacon Node (version 1.5.2). We tested this on the Pyrmont Network.
I followed the guide to run the Lighthouse beacon node as lighthouse docs indicate https://lighthouse-book.sigmaprime.io/docker.html , only it's necessary to modify the network flag to Pyrmont.

As a prysm validator, I used the pyrmont package we have in DAppNode (https://github.com/dappnode/DAppNodePackage-prysm-pyrmont). To connect this validator to the lighthouse beacon-node we have to add the flag :5052 as BEACON_RPC_PROVIDER.

The result of this test has not been good.

• We did not find any logs in the validator that indicates something goes well or bad. In beaconchain.in we checked that the validator was not working properly.
• We used the debug mode of the validator, but the logs do not show any information about the validator or the status of the connection with the beacon node.
• We asked about how to deal with this situation in the Prysm Discord channel(https://discord.com/channels/476244492043812875/603588441741066241/889798572852277300).
  The answer of Radek indicates that the prysm validator uses its own API to connect to the beacon node, not the official so what we are trying is not possible yet. They say that although the plan is to switch to the official APIs, this is not a PRIORITY at the moment.

[rauljordan]

In this probe, we have tried to connect a Prysm validator ( version v2.0.0-rc) to a Lighthouse Beacon Node (version 1.5.2). We tested this on the Pyrmont Network.
I followed the guide to run the Lighthouse beacon node as lighthouse docs indicate https://lighthouse-book.sigmaprime.io/docker.html , only it's necessary to modify the network flag to Pyrmont.

As a prysm validator, I used the pyrmont package we have in DAppNode (https://github.com/dappnode/DAppNodePackage-prysm-pyrmont). To connect this validator to the lighthouse beacon-node we have to add the flag :5052 as BEACON_RPC_PROVIDER.

The result of this test has not been good.

• We did not find any logs in the validator that indicates something goes well or bad. In beaconchain.in we checked that the validator was not working properly.
• We used the debug mode of the validator, but the logs do not show any information about the validator or the status of the connection with the beacon node.
• We asked about how to deal with this situation in the Prysm Discord channel(https://discord.com/channels/476244492043812875/603588441741066241/889798572852277300).
  The answer of Radek indicates that the prysm validator uses its own API to connect to the beacon node, not the official so what we are trying is not possible yet. They say that although the plan is to switch to the official APIs, this is not a PRIORITY at the moment.

@tropicar this is correct: there is no current test for trying a LH validator with a Prysm beacon, so there are possibly tons of bugs that have not been uncovered and this is not in our short or mid-term plans, unfortunately. We eventually plan to use the official APIs as canonical but that will probably happen until after the merge given priorities.

[dapplion]

UPDATE: Thanks to Jeff <3 for spreading the word about initiative C (Use a remote signer with a UI attached) https://twitter.com/technocrypto/status/1453128929418661892

I've started a doc to be expanded with more technical details about the project -> https://hackmd.io/@dapplion/web3signer

[joaquim-verges]

I'm willing to lend a hand on this, opened an issue on the web3signer repo to discuss about option C: Consensys/web3signer#439 - there's a couple of open questions, would love some eyes on it!

[Pol-Lanski]

Update on the situation:

From @dapplion 's original post, there are 2 solutions that are developing nicely:
B: Add only UI backend to the client
and
C: Use a remote signer with a UI attached

I'm going to slightly rename them since the scope has shifted and is now better defined.

B) Enable all validator clients to connect to a UI

DAppLion's initial API proposal has been reviewed and discussed by all teams and it now has its own repo: https://github.com/ethereum/keymanager-APIs

This would allow us to have a generic UI that connects to any client that implements the standard.

😀 At least Teku and Lightnouse are implementing it, which means that they could be offered as options in DAppNode.
😔 No generic UI yet

C) Use remote signer with a UI attached

😀 Thanks to @joaquim-verges , the standard is being implemented in the web3signer as per: Consensys/web3signer#439
😔 No generic UI yet

Work on DAppNode's side

Regardless of which option, B or C, becomes available first, there's 3 things that need to happen in parallel.

1️⃣ Users in DAppNode are currently using Prysm. To migrate to any other solution, they will need to extract: 1) Validator keys, 2) Slashing protection data from the current Prysm client. This is being addressed by @pablomendezroyo here: dappnode/DNP_DAPPMANAGER#916

2️⃣ When more validator packages become available, there is a chance that if the user does not follow a particular sequence of steps, two validators might be running and due to some bug, two validate with the same key. We must mitigate this by all means possible. This is being addressed by @pablomendezroyo here: dappnode/DNP_DAPPMANAGER#913 and here dappnode/DNP_DAPPMANAGER#914

3️⃣ Whether it's for a validator client that does not have a UI (all of them except for Prysm) or for the web3signer, a UI that utilizes the keymanagerAPI needs to be done. We, DAppNode, would prefer NOT to touch this and are looking for someone that can develop it.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions!

[alexpeterson91]

Resolved!