ethclient/simulated: implement new sim backend #28202

MariusVanDerWijden · 2023-09-26T10:05:43Z

This is a rewrite of the 'simulated backend', an implementation of the ethclient interfaces
which is backed by a simulated blockchain. It was getting annoying to maintain the old
version of the simulated backend feature because there was a lot of code duplication with
the main client.

The new version is built using parts that we already have: an in-memory geth node instance
running in developer mode provides the chain, while the Go API is provided by ethclient.
A backwards-compatibility wrapper is provided, but the simulated backend has also moved to
a more sensible import path: github.com/ethereum/go-ethereum/ethclient/simulated

closes #28573
closes #21457
closes #28431
closes #28144

accounts/abi/bind/backends/newsim.go

MariusVanDerWijden · 2023-11-24T10:47:17Z

One thing I'm not quite sure about is that I actually slightly changed the interface of the constructor from
NewSimulatedBackend() (*SimulatedBackend) to NewSimulatedBackend() (*SimulatedBackend, error) since way more stuff can go wrong. I could just return nil and expect the caller to make sure that the backend is initialized correctly, but I don't know, would be good to discuss a bit

MariusVanDerWijden · 2023-11-27T12:22:47Z

This PR actually changes the way AdjustTime works. Previously the simulated backend would adjust the time of the pending block (as long as it was empty). Now AdjustTime will create a new block. This behavior change is okay in my opinion since the old way required callers to have an empty block anyway when calling adjust time.

holiman

I don't have much experience, in general, with using the simulated backend, so can't really review the "ux" of these changes.

accounts/abi/bind/backends/simulated.go

fjl · 2023-12-04T15:09:42Z

I think we should create more interfaces in the top-level package to cover these methods. We should probably also review the ethclient API for anything we missed that isn't covered by tests.

fjl · 2023-12-07T09:32:41Z

So my proposal is:

we move the simulated backend implementation to a new package ethclient/simulated, and rename it to something like Backend. So users of the package would see it as simulated.Backend.

In package accounts/abi/bind, we create a wrapper struct and mark it deprecated.

For the new type, one thing I would like to figure out is whether we should separate the SimChainManagement methods from the simulated backend somehow. At the moment, the simulated backend object implements all of the ethclient methods and also Commit, Fork, etc.

I think it might be better to have these separated, i.e. the simulated backend object should have Commit, Fork, Close etc. and also a method Client to retrieve another object that implements the ethclient interfaces. The advantage would be, if we introduce ethclient v2 later, we could add another method to return the v2 as well.

MariusVanDerWijden · 2023-12-07T11:08:01Z

I moved the Sim implementation to its new package now and added a stump & deprecation notice

ARR4N · 2023-12-08T09:34:52Z

One thing I'm not quite sure about is that I actually slightly changed the interface of the constructor from NewSimulatedBackend() (*SimulatedBackend) to NewSimulatedBackend() (*SimulatedBackend, error) since way more stuff can go wrong. I could just return nil and expect the caller to make sure that the backend is initialized correctly, but I don't know, would be good to discuss a bit

I think you should just return the errors if they occur. The return of an error in a function signature signals to the user that they need to check it, whereas the lack of one suggests that failure is impossible. Expecting users to know to check the backend initialisation is more error prone and, IMO, will frustrate users who experience unreported errors that they need to search for.

fjl · 2023-12-08T12:07:21Z

The constructor really cannot fail since no I/O will occur. And it is way more convenient to avoid the error in tests.

eth/catalyst/simulated_beacon.go

ethclient/simulated/backend.go

eth/catalyst/simulated_beacon.go

accounts/abi/bind/util_test.go

ethclient/simulated/backend.go

lightclient · 2024-01-08T13:36:14Z

eth/catalyst/simulated_beacon.go

@@ -117,6 +126,10 @@ func (c *SimulatedBeacon) setFeeRecipient(feeRecipient common.Address) {
 func (c *SimulatedBeacon) Start() error {
 	if c.period == 0 {
 		go c.loopOnDemand()
+	} else if c.period == math.MaxUint64 {


I think it would be better to supplant the "loopOnDemand" logic with your commit format. Right now, we have two special periods for SimulatedBeacon that do kind of the same thing: mine blocks arbitrarily.

Since dev mode is really the only thing that consumes this, you could wrap SimulatedBeacon in another object which retains the original behavior for period == 0, it just does so by first sending the txs / wxs and then calls Commit().

This would mean all transactions would need to be cached in this other object, and we would only pass them to the txpool on Commit. The issue with that would be that all the tx verification logic would need to be duplicated

Why would you need to cache them? This is what I had in mind (that wrapper already existed, it's called "api"):

diff --git a/eth/catalyst/simulated_beacon.go b/eth/catalyst/simulated_beacon.go index 291f91874..7012a53e1 100644 --- a/eth/catalyst/simulated_beacon.go +++ b/eth/catalyst/simulated_beacon.go @@ -19,14 +19,12 @@ package catalyst import ( "crypto/rand" "errors" - "math" "math/big" "sync" "time" "github.com/ethereum/go-ethereum/beacon/engine" "github.com/ethereum/go-ethereum/common" - "github.com/ethereum/go-ethereum/core" "github.com/ethereum/go-ethereum/core/types" "github.com/ethereum/go-ethereum/eth" "github.com/ethereum/go-ethereum/log" @@ -125,9 +123,7 @@ func (c *SimulatedBeacon) setFeeRecipient(feeRecipient common.Address) { // Start invokes the SimulatedBeacon life-cycle function in a goroutine. func (c *SimulatedBeacon) Start() error { if c.period == 0 { - go c.loopOnDemand() - } else if c.period == math.MaxUint64 { - // if period is set to MaxUint, do not mine at all + // if period is set to 0, do not mine at all // this is used in the simulated backend where blocks // are explicitly mined via Commit, AdjustTime and Fork } else { @@ -204,32 +200,6 @@ func (c *SimulatedBeacon) sealBlock(withdrawals []*types.Withdrawal, timestamp u return nil } -// loopOnDemand runs the block production loop for "on-demand" configuration (period = 0) -func (c *SimulatedBeacon) loopOnDemand() { - var ( - newTxs = make(chan core.NewTxsEvent) - sub = c.eth.TxPool().SubscribeTransactions(newTxs, true) - ) - defer sub.Unsubscribe() - - for { - select { - case <-c.shutdownCh: - return - case w := <-c.withdrawals.pending: - withdrawals := append(c.withdrawals.gatherPending(9), w) - if err := c.sealBlock(withdrawals, uint64(time.Now().Unix())); err != nil { - log.Warn("Error performing sealing work", "err", err) - } - case <-newTxs: - withdrawals := c.withdrawals.gatherPending(10) - if err := c.sealBlock(withdrawals, uint64(time.Now().Unix())); err != nil { - log.Warn("Error performing sealing work", "err", err) - } - } - } -} - // loop runs the block production loop for non-zero period configuration func (c *SimulatedBeacon) loop() { timer := time.NewTimer(0) @@ -318,10 +288,12 @@ func (c *SimulatedBeacon) AdjustTime(adjustment time.Duration) error { } func RegisterSimulatedBeaconAPIs(stack *node.Node, sim *SimulatedBeacon) { + api := &api{sim} + go api.loop() stack.RegisterAPIs([]rpc.API{ { Namespace: "dev", - Service: &api{sim}, + Service: api, Version: "1.0", }, }) diff --git a/eth/catalyst/simulated_beacon_api.go b/eth/catalyst/simulated_beacon_api.go index 93670257f..73d0a5921 100644 --- a/eth/catalyst/simulated_beacon_api.go +++ b/eth/catalyst/simulated_beacon_api.go @@ -18,19 +18,44 @@ package catalyst import ( "context" + "time" "github.com/ethereum/go-ethereum/common" + "github.com/ethereum/go-ethereum/core" "github.com/ethereum/go-ethereum/core/types" + "github.com/ethereum/go-ethereum/log" ) type api struct { - simBeacon *SimulatedBeacon + sim *SimulatedBeacon +} + +func (a *api) loop() { + var ( + newTxs = make(chan core.NewTxsEvent) + sub = a.sim.eth.TxPool().SubscribeTransactions(newTxs, true) + ) + defer sub.Unsubscribe() + + for { + select { + case <-a.sim.shutdownCh: + return + case w := <-a.sim.withdrawals.pending: + withdrawals := append(a.sim.withdrawals.gatherPending(9), w) + if err := a.sim.sealBlock(withdrawals, uint64(time.Now().Unix())); err != nil { + log.Warn("Error performing sealing work", "err", err) + } + case <-newTxs: + a.sim.Commit() + } + } } func (a *api) AddWithdrawal(ctx context.Context, withdrawal *types.Withdrawal) error { - return a.simBeacon.withdrawals.add(withdrawal) + return a.sim.withdrawals.add(withdrawal) } func (a *api) SetFeeRecipient(ctx context.Context, feeRecipient common.Address) { - a.simBeacon.setFeeRecipient(feeRecipient) + a.sim.setFeeRecipient(feeRecipient) }

Basically just moved "loopOnDemand" into the api struct so it only runs when in dev mode. Now if period is 0, you have to manually call commit (as desired when using MaxUint before).

I see, that makes sense to me

applied your patch (and some more fixes) pls check

Look correct to me, but i haven't run in the on-demand devmode with the update yet. I noticed there are tests for devmode, but only if the period is non-zero.

eth/catalyst/simulated_beacon.go

ethclient/simulated/backend.go

interfaces.go

moves the loopOnDemand outside making it part of the api

It's not a 'reader' since the result is computed by the oracle.

This is a rewrite of the 'simulated backend', an implementation of the ethclient interfaces which is backed by a simulated blockchain. It was getting annoying to maintain the old version of the simulated backend feature because there was a lot of code duplication with the main client. The new version is built using parts that we already have: an in-memory geth node instance running in developer mode provides the chain, while the Go API is provided by ethclient. A backwards-compatibility wrapper is provided, but the simulated backend has also moved to a more sensible import path: github.com/ethereum/go-ethereum/ethclient/simulated --------- Co-authored-by: Felix Lange <[email protected]> Co-authored-by: Gary Rong <[email protected]>

MariusVanDerWijden added the status:triage label Nov 21, 2023

rjl493456442 reviewed Nov 21, 2023

View reviewed changes

accounts/abi/bind/backends/newsim.go Outdated Show resolved Hide resolved

MariusVanDerWijden removed the status:triage label Nov 22, 2023

MariusVanDerWijden force-pushed the simulated-backend-v2 branch from 6619ec3 to 87b6067 Compare November 24, 2023 10:42

MariusVanDerWijden marked this pull request as ready for review November 27, 2023 11:26

MariusVanDerWijden requested a review from gballet as a code owner November 27, 2023 11:26

holiman reviewed Dec 1, 2023

View reviewed changes

fjl reviewed Dec 1, 2023

View reviewed changes

accounts/abi/bind/backends/simulated.go Outdated Show resolved Hide resolved

MariusVanDerWijden force-pushed the simulated-backend-v2 branch from 4fcb939 to 23c6b79 Compare December 5, 2023 10:22

This was referenced Dec 5, 2023

accounts/abi: simulator implements interface PendingStateReader #28438

Closed

accounts/abi: simulate detail insufficient fund for EstimateGas #28162

Closed

accounts/abi/bind/backend: SimulatedBackend options to always honour vm.Config #25072

Closed

rjl493456442 reviewed Dec 8, 2023

View reviewed changes

eth/catalyst/simulated_beacon.go Show resolved Hide resolved

s1na reviewed Dec 12, 2023

View reviewed changes

ethclient/simulated/backend.go Outdated Show resolved Hide resolved

s1na reviewed Dec 12, 2023

View reviewed changes

ethclient/simulated/backend.go Show resolved Hide resolved

rjl493456442 reviewed Dec 19, 2023

View reviewed changes

eth/catalyst/simulated_beacon.go Outdated Show resolved Hide resolved

MariusVanDerWijden requested a review from karalabe as a code owner December 19, 2023 12:13

rjl493456442 approved these changes Dec 19, 2023

View reviewed changes

fjl reviewed Dec 19, 2023

View reviewed changes

accounts/abi/bind/util_test.go Outdated Show resolved Hide resolved

MariusVanDerWijden added 4 commits January 2, 2024 08:18

eth/catalyst: accounts/abi/bind/backend: implement new sim backend

f0b5799

accounts/abi/bind/backends: fix bugs, add basic test

8887559

accounts/abi/bind/backends: new sim backend, ported tests, fixed tests

d50932f

accounts/abi/bind/backends: revert to old interface for constructor

790ce4a

fjl and others added 9 commits January 2, 2024 08:19

ethclient/simulated: move interface assertion to tests

8eb927b

all: fix typo and comments

2a12a97

ethclient/simulated: unexport NewWithNode

45953a9

eth/catalyst: add comment

d71bfff

eth/catalyst: rework rollback procedure

e648fb5

core/txpool/legacypool: remove all txs if called with negative gasTip

c557960

eth/catalyst: properly flush txs

7ab3038

accounts/abi: use deprecated, ignore in lint

9c754a0

accounts/abi: don't use the deprecated, fix tests

12c05b9

MariusVanDerWijden force-pushed the simulated-backend-v2 branch from 99b81fd to 12c05b9 Compare January 2, 2024 07:19

rjl493456442 added this to the 1.13.8 milestone Jan 2, 2024

fjl reviewed Jan 2, 2024

View reviewed changes

ethclient/simulated/backend.go Outdated Show resolved Hide resolved

MariusVanDerWijden force-pushed the simulated-backend-v2 branch from 6bc68a9 to 12c05b9 Compare January 2, 2024 13:07

lightclient reviewed Jan 8, 2024

View reviewed changes

MariusVanDerWijden added 2 commits January 8, 2024 15:05

ethclient/simulated: added comments

a327647

eth/catalyst: apply lightclients suggestions

6b29940

moves the loopOnDemand outside making it part of the api

fjl changed the title ~~eth/catalyst: accounts/abi/bind/backend: implement new sim backend~~ ethclient/simulated: implement new sim backend Jan 9, 2024

ethclient/simulated: split gasTipCap and feeHistory interfaces

f016bff

MariusVanDerWijden modified the milestones: 1.13.8, 1.13.9 Jan 10, 2024

fjl added 2 commits January 10, 2024 12:40

ethereum: rename to GasPricer1559

f895169

It's not a 'reader' since the result is computed by the oracle.

accounts/abi/bind: embed more interfaces instead of redeclaring methods

d3b0135

fjl merged commit 2d08c99 into ethereum:master Jan 10, 2024
1 of 3 checks passed

BrewTestBot mentioned this pull request Jan 10, 2024

ethereum 1.13.9 Homebrew/homebrew-core#159552

Merged

dnhn mentioned this pull request Jan 11, 2024

ethereum 1.13.10 Homebrew/homebrew-core#159681

Merged

ian-shim mentioned this pull request Mar 7, 2024

Upgrade eigensdk-go and go-ethereum Layr-Labs/eigenda#323

Merged

5 tasks

This was referenced Sep 13, 2024

Simulated backend Rollback() function should set gas tip back to the parameter passed by config #30429

Closed

Simulated backend Fork & blockchain SetHead don't emit removed logs event #30448

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ethclient/simulated: implement new sim backend #28202

ethclient/simulated: implement new sim backend #28202

MariusVanDerWijden commented Sep 26, 2023 •

edited by fjl

Loading

MariusVanDerWijden commented Nov 24, 2023

MariusVanDerWijden commented Nov 27, 2023

holiman left a comment •

edited

Loading

fjl commented Dec 4, 2023

fjl commented Dec 7, 2023 •

edited

Loading

MariusVanDerWijden commented Dec 7, 2023

ARR4N commented Dec 8, 2023 •

edited

Loading

fjl commented Dec 8, 2023

lightclient Jan 8, 2024

MariusVanDerWijden Jan 8, 2024

lightclient Jan 8, 2024

MariusVanDerWijden Jan 8, 2024

MariusVanDerWijden Jan 8, 2024

lightclient Jan 8, 2024

ethclient/simulated: implement new sim backend #28202

ethclient/simulated: implement new sim backend #28202

Conversation

MariusVanDerWijden commented Sep 26, 2023 • edited by fjl Loading

MariusVanDerWijden commented Nov 24, 2023

MariusVanDerWijden commented Nov 27, 2023

holiman left a comment • edited Loading

Choose a reason for hiding this comment

fjl commented Dec 4, 2023

fjl commented Dec 7, 2023 • edited Loading

MariusVanDerWijden commented Dec 7, 2023

ARR4N commented Dec 8, 2023 • edited Loading

fjl commented Dec 8, 2023

lightclient Jan 8, 2024

Choose a reason for hiding this comment

MariusVanDerWijden Jan 8, 2024

Choose a reason for hiding this comment

lightclient Jan 8, 2024

Choose a reason for hiding this comment

MariusVanDerWijden Jan 8, 2024

Choose a reason for hiding this comment

MariusVanDerWijden Jan 8, 2024

Choose a reason for hiding this comment

lightclient Jan 8, 2024

Choose a reason for hiding this comment

MariusVanDerWijden commented Sep 26, 2023 •

edited by fjl

Loading

holiman left a comment •

edited

Loading

fjl commented Dec 7, 2023 •

edited

Loading

ARR4N commented Dec 8, 2023 •

edited

Loading