Deprecating the pending state #495

s1na · 2023-11-23T11:37:23Z

This issue is to keep track of the conversation started at the JSON-RPC session at devconnect.

Most state-related methods which take in a blocktag allow users to specify "pending". The semantic difference to "latest" is that the EL keeps track of a pending state by applying all of the pending transactions on top of the current state.

We're questioning the practical use of this tag as it depends heavily on the local view of the txpool. The mempool is fluid, there is no guarantee that the transactions there will be mined. Moreover different nodes within an RPC provider infrastructure have different views on the mempool and some notable ones do not serve pending state.

The path to deprecating "pending" we are proposing is to create an empty pending block. This means the block context fields such as number and timestamp will reflect the "next" block, but state is from that of "latest". This also means that there will be no guarantees that the client can deliver pending logs. The only exception would be eth_getTransactionCount which should return the latest nonce for an account, considering also txes in the mempool.

This will alleviate some load from nodes to build the pending block and maintain its state while keeping most of the current use-cases.

Looking into the future: by adding block context overrides to relevant methods such as eth_call, eth_estimateGas and eth_createAccessList we remove the need for creating an empty pending block. Users can simulate on top of "latest" but pass in overrides for the next block. Furthermore if clients implement eth_multicall, it will be possible to simulate the pending state by passing higher weighted pending txes to multicall.

The text was updated successfully, but these errors were encountered:

protolambda · 2023-11-23T21:00:51Z

In the OP-Stack we've been aliasing the pending RPC label to latest, to:

Save the cost of computing a pending-block (low block-time of 2 seconds makes recomputing a 30M gas block quite expensive).
Serve sensible data, when the tx-pool is private.

This has been working well enough for L2 users (although maybe better than L1 due to the lower block time). I support the aliasing of it to latest as deprecation strategy on L1.

And we still support eth_getTransactionCount, since it's attached to the mem-pool, not the actual pending state, so is not as costly to maintain. And since it does not reveal the full transaction data, generally not regarded as privacy issue, and a good UX trade-off.

So far we've left computation of pending state still as opt-in feature for individual node users, which then allows them to inspect the block-building of their local mempool (which just contains a copy of their own transactions, that the node passed on privately to the sequencer endpoint), which they can tune the gas-limit of as well (again to save unnecessary re-computation cost). But from user-issues it appears that nobody is actually using the opt-in feature.

With the removal of Clique from EL clients, removing the pending flag unblocks the cleanup of a lot of legacy pre-engine-API block-building, so I'm very much in favor from the perspective of reducing code complexity.

s1na · 2023-11-24T09:23:31Z

Adding a note that this means it will not be possible to retrieve pending logs anymore. I personally think if pending state/logs are required they can be computed using eth_multicall over the mempool set.

Link to geth implementation: ethereum/go-ethereum#28440

lmittmann · 2023-11-24T09:27:44Z

The pending header is very useful for simulating transactions for the pending block when constructing the VM's block context.

s1na · 2023-11-24T09:50:53Z

The pending header is very useful for simulating transactions for the pending block when constructing the VM's block context.

Interesting. The fields that might be useful are: timestamp, number, baseFeePerGas and gasLimit. They are all computable off-band based on previous header but I see the point that they are consensus logic and better handled within the node.

If my assumption is right I think a path forward is to return a pending header with 0 transactions. It will fill the fields above but every other field will be trash.

niklasnatter · 2023-11-24T10:55:16Z

The pending header is very useful for simulating transactions for the pending block when constructing the VM's block context.

One relevant example for this is using the eth_createAccessList with the pending block tag. A lot of contracts that use checkpoints (eg. Uniswap v3 for oracle observations) do a binary search that uses the current block number. Falling back to latest will lead to wrong entries in the access list in this case.

A more flexible way to cover these cases could be supporting blockOverrides (like already implemented for eth_call in go-ethereum: ethereum/go-ethereum#26414) for the relevant RPC methods.

s1na · 2023-11-24T11:51:37Z

Gotcha. I will update the description above to clearly specify: "we will use a pending block with zero transactions" when "pending" is passed. This should cover most cases.

That said, I agree we should push blockOverrides to be implemented across the board by the time we fully remove "pending".

ryanschneider · 2023-11-28T20:29:04Z

And we still support eth_getTransactionCount

I would advocate against this since I think it gives users a false sense of "correctness", instead I think we should be encouraging users/wallets to do their own "off chain/out of pool" nonce management. In fact I'd be in favor of a new 2718 tx type that supports a different nonce/replay protection scheme.

Given the discussions around deprecating pending (see #28623 or ethereum/execution-apis#495), we can move away from using the pending block internally, and use latest instead

Given the discussions around deprecating pending (see ethereum#28623 or ethereum/execution-apis#495), we can move away from using the pending block internally, and use latest instead

* core, trie, rpc: speed up tests (#28461) * rpc: make subscription test faster reduces time for TestClientSubscriptionChannelClose from 25 sec to < 1 sec. * trie: cache trie nodes for faster sanity check This reduces the time spent on TestIncompleteSyncHash from ~25s to ~16s. * core/forkid: speed up validation test This takes the validation test from > 5s to sub 1 sec * core/state: improve snapshot test run brings the time for TestSnapshotRandom from 13s down to 6s * accounts/keystore: improve keyfile test This removes some unnecessary waits and reduces the runtime of TestUpdatedKeyfileContents from 5 to 3 seconds * trie: remove resolver * trie: only check ~5% of all trie nodes * ethdb/pebble: don't double-close iterator inside pebbleIterator (#28566) Adds 'released' flag to pebbleIterator to avoid double closing cockroachdb/pebble.Iterator as it is an invalid operation. Fixes #28565 * eth/filters: reuse error msg for invalid block range (#28479) * core/types: make 'v' optional for DynamicFeeTx and BlobTx (#28564) This fixes an issue where transactions would not be accepted when they have only 'yParity' and not 'v'. * rpc: improve performance of subscription notification encoding (#28328) It turns out that encoding json.RawMessage is slow because package json basically parses the message again to ensure it is valid. We can avoid the slowdown by encoding the entire RPC notification once, which yields a 30% speedup. * cmd/utils: validate pre-existing genesis in --dev mode (#28468) geth --dev can be used with an existing data directory and genesis block. Since dev mode only works with PoS, we need to verify that the merge has happened. Co-authored-by: Felix Lange <[email protected]> * cmd/geth: add support for --dev flag in dumpgenesis (#28463) Co-authored-by: Felix Lange <[email protected]> Co-authored-by: lightclient <[email protected]> * les/vflux: run tests in parallel (#28524) * cmd/{geth,utils}: add cmd to export preimages in snap enumeration order (#28256) Adds a subcommand: `geth snapshot export-preimages`, to export preimages of every hash found during a snapshot enumeration: that is, it exports _only the active state_, and not _all_ preimages that have been used but are no longer part of the state. This tool is needed for the verkle transition, in order to distribute the preimages needed for the conversion. Since only the 'active' preimages are exported, the output is shrunk from ~70GB to ~4GB. The order of the output is the order used by the snapshot enumeration, which avoids database thrashing. However, it also means that storage-slot preimages are not deduplicated. * cmd/geth: fix build error (#28585) * cmd/devp2p/internal/ethtest: undo debug-hack (#28588) cmd/devp2p/internal/ethtest: remove a debug-hack flaw which prevented certain tests from running * params: update discV5 bootnodes (#28562) update discV5 bootnodes from https://github.com/eth-clients/eth2-networks/blob/master/shared/mainnet/bootstrap_nodes.txt * cmd, les, tests: remove light client code (#28586) * cmd, les, tests: remove light client code This commit removes the light client (LES) code. Since the merge the light client has been broken and it is hard to maintain it alongside the normal client. We decided it would be best to remove it for now and maybe rework and reintroduce it in the future. * cmd, eth: remove some more mentions of light mode * cmd: re-add flags and mark as deprecated * cmd: warn the user about deprecated flags * eth: better error message * eth, internal/ethapi: drop some weird indirection (#28597) * trie: fix random test generator early terminate (#28590) This change fixes a minor bug in the `randTest.Generate` function, which caused the `quick.Check` to be a no-op. * eth/gasestimator, internal/ethapi: move gas estimator out of rpc (#28600) * go.mod: update uint256 to v1.2.4 (#28612) * eth/catalyst, eth/downloader: expose more sync information (#28584) This change exposes more information from sync module internally * light: remove package light(#28614) This changes removes the package 'light', which is currently unused. * cmd/evm, core/state: fix post-exec dump of state (statetests, blockchaintests) (#28504) There were several problems related to dumping state. - If a preimage was missing, even if we had set the `OnlyWithAddresses` to `false`, to export them anyway, the way the mapping was constructed (using `common.Address` as key) made the entries get lost anyway. Concerns both state- and blockchain tests. - Blockchain test execution was not configured to store preimages. This changes makes it so that the block test executor takes a callback, just like the state test executor already does. This callback can be used to examine the post-execution state, e.g. to aid debugging of test failures. * ethereum: remove TODO comment about subscription (#28609) * eth/tracers/js: fix type inconsistencies (#28488) This change fixes two type-inconsistencies in the JS tracer: - In most places we return byte arrays as a `Uint8Array` to the tracer. However it seems we missed doing the conversion for `ctx` fields which are passed to the tracer during `result`. They are passed as simple arrays. I think Uint8Arrays are more suitable and we should change this inconsistency. Note: this will be a breaking-change. But I believe the effect is small. If we look at our tracers we see that these fields (`ctx.from`, `ctx.to`, etc.) are used in 2 ways. Passed to `toHex` which takes both array or buffer. Or the length was measured which is the same for both types. - The `slice` taking in `int, int` params versus `memory.slice` taking `int64, int64` params. I suggest changing `slice` types to `int64`. This should have no effect almost in any case. * crypto/secp256k1: fix 32-bit tests when CGO_ENABLED=0 (#28602) * consensus: verify the nonexistence of shanghai- and cancun-specific header fields (#28605) * eth/gasestimator: allow slight estimation error in favor of less iterations (#28618) * eth/gasestimator: early exit for plain transfer and error allowance * core, eth/gasestimator: hard guess at a possible required gas * internal/ethapi: update estimation tests with the error ratio * eth/gasestimator: I hate you linter * graphql: fix gas estimation test --------- Co-authored-by: Oren <[email protected]> * all: replace log15 with slog (#28187) This PR replaces Geth's logger package (a fork of [log15](https://github.com/inconshreveable/log15)) with an implementation using slog, a logging library included as part of the Go standard library as of Go1.21. Main changes are as follows: * removes any log handlers that were unused in the Geth codebase. * Json, logfmt, and terminal formatters are now slog handlers. * Verbosity level constants are changed to match slog constant values. Internal translation is done to make this opaque to the user and backwards compatible with existing `--verbosity` and `--vmodule` options. * `--log.backtraceat` and `--log.debug` are removed. The external-facing API is largely the same as the existing Geth logger. Logger method signatures remain unchanged. A small semantic difference is that a `Handler` can only be set once per `Logger` and not changed dynamically. This just means that a new logger must be instantiated every time the handler of the root logger is changed. ---- For users of the `go-ethereum/log` module. If you were using this module for your own project, you will need to change the initialization. If you previously did ```golang log.Root().SetHandler(log.LvlFilterHandler(log.LvlInfo, log.StreamHandler(os.Stderr, log.TerminalFormat(true)))) ``` You now instead need to do ```golang log.SetDefault(log.NewLogger(log.NewTerminalHandlerWithLevel(os.Stderr, log.LevelInfo, true))) ``` See more about reasoning here: ethereum/go-ethereum#28558 (comment) * core/state: make stateobject.create selfcontain (#28459) * trie/triedb/hashdb: take lock around access to dirties cache (#28542) Add read locking of db lock around access to dirties cache in hashdb.Database to prevent data race versus hashdb.Database.dereference which can modify the dirities map by deleting an item. Fixes #28541 --------- Co-authored-by: Gary Rong <[email protected]> * accounts/abi/bind: fix typo (#28630) * slog: faster and less memory-consumption (#28621) These changes improves the performance of the non-coloured terminal formatting, _quite a lot_. ``` name old time/op new time/op delta TerminalHandler-8 10.2µs ±15% 5.4µs ± 9% -47.02% (p=0.008 n=5+5) name old alloc/op new alloc/op delta TerminalHandler-8 2.17kB ± 0% 0.40kB ± 0% -81.46% (p=0.008 n=5+5) name old allocs/op new allocs/op delta TerminalHandler-8 33.0 ± 0% 5.0 ± 0% -84.85% (p=0.008 n=5+5) ``` I tried to _somewhat_ organize the commits, but the it might still be a bit chaotic. Some core insights: - The function `terminalHandler.Handl` uses a mutex, and writes all output immediately to 'upstream'. Thus, it can reuse a scratch-buffer every time. - This buffer can be propagated internally, making all the internal formatters either write directly to it, - OR, make use of the `tmp := buf.AvailableBuffer()` in some cases, where a byte buffer "extra capacity" can be temporarily used. - The `slog` package uses `Attr` by value. It makes sense to minimize operating on them, since iterating / collecting into a new slice, iterating again etc causes copy-on-heap. Better to operate on them only once. - If we want to do padding, it's better to copy from a constant `space`-buffer than to invoke `bytes.Repeat` every single time. * eth/tracers: tx-level state in debug_traceCall (#28460) * cmd/evm: fix Env struct json tag (#28635) * accounts/abi/bind: fixed typos (#28634) * Update auth.go * Update backend.go * Update bind.go * Update bind_test.go * eth/fetcher: fix invalid tracking of received at time for block (#28637) eth/fetcher: fix invalid tracking of received at time * accounts: run tests in parallel (#28544) * eth/tracers/logger: make structlog/json-log stack hex again (#28628) * common/hexutil: define hex wrappers for uint256.Int * eth/tracers/logger: make structlog/json-log stack hex again * common/hexutil: goimports * log: remove lazy, remove unused interfaces, unexport methods (#28622) This change - Removes interface `log.Format`, - Removes method `log.FormatFunc`, - unexports `TerminalHandler.TerminalFormat` formatting methods (renamed to `TerminalHandler.format`) - removes the notion of `log.Lazy` values The lazy handler was useful in the old log package, since it could defer the evaluation of costly attributes until later in the log pipeline: thus, if the logging was done at 'Trace', we could skip evaluation if logging only was set to 'Info'. With the move to slog, this way of deferring evaluation is no longer needed, since slog introduced 'Enabled': the caller can thus do the evaluate-or-not decision at the callsite, which is much more straight-forward than dealing with lazy reflect-based evaluation. Also, lazy evaluation would not work with 'native' slog, as in, these two statements would be evaluated differently: ```golang log.Info("foo", "my lazy", lazyObj) slog.Info("foo", "my lazy", lazyObj) ``` * .github: use github actions to run 32-bit linux tests (#28549) use github actions to run 32-bit linux tests * fix blob fee cap too low error * update go mod * ethdb/pebble: remove a dependency (#28627) The dependency was not really used anyway, so we can get rid of it. Co-authored-by: Felix Lange <[email protected]> * tests/fuzzers/bls12381: deactivate BLS fuzzer when CGO_ENABLED=0 (#28653) tests/fuzzers/bls12381: deactivate fuzzer when CGO_ENABLED=0 * build: upgrade -dlgo version to Go 1.21.5 (#28648) * rpc: fix ns/µs mismatch in metrics (#28649) The rpc/duration/all meter was in nanoseconds, the individual meter in microseconds. This PR changes it so both of them use nanoseconds. * cmd/evm: fix dump after state-test exec (#28650) The dump after state-test didn't work, the problem was an error, "Already committed", which was silently ignored. This change re-initialises the state, so the dumping works again. * beacon/light: add CommitteeChain (#27766) This change implements CommitteeChain which is a key component of the beacon light client. It is a passive data structure that can validate, hold and update a chain of beacon light sync committees and updates, starting from a checkpoint that proves the starting committee through a beacon block hash, header and corresponding state. Once synced to the current sync period, CommitteeChain can also validate signed beacon headers. * cmd/utils, eth: disallow invalid snap sync / snapshot flag combos (#28657) * eth: prevent startup in snap mode without snapshots * cmd/utils: try to fix bad flag combos wrt snap sync and snapshot generation * trie: remove inconsistent trie nodes during sync in path mode (#28595) This fixes a database corruption issue that could occur during state healing. When sync is aborted while certain modifications were already committed, and a reorg occurs, the database would contain incorrect trie nodes stored by path. These nodes need to detected/deleted in order to obtain a complete and fully correct state after state healing. --------- Co-authored-by: Felix Lange <[email protected]> * cmd/utils: fix HTTPHost, WSHost flag priority (#28669) Co-authored-by: Felix Lange <[email protected]> * eth/protocols/eth: fix typos in comments (#28652) * core/txpool : small cleanup refactors (#28654) * eth/fetcher, eth/gasestimator: fix typos in comments (#28675) * all: fix typos in comments (#28662) Co-authored-by: Felix Lange <[email protected]> * miner: eliminate the dead loop possibility for `newWorkLoop` and `mainLoop` (#28677) discard the intervalAdjust message if the channel is full * all: fix typos in comments (#28682) chore(core,eth):fix a couple of typos * p2p/discover: add liveness check in collectTableNodes (#28686) * p2p/discover: add liveness check in collectTableNodes * p2p/discover: fix test * p2p/discover: rename to appendLiveNodes * p2p/discover: add dedup logic back * p2p/discover: simplify * p2p/discover: fix issue found by test * internal/flags: add missing flag types for auto-env-var generation (#28692) Certain flags, such as `--rpc.txfeecap` currently do not have an env-var auto-generated for them. This change adds three missing cli flag types to the auto env-var helper function to fix this. * cmd/evm: default to mirror mainnet forks enabled (#28691) cmd/evm: default to using dev chain config (all mainnet HFs activated at block/timestamp 0 * cmd/evm, cmd/clef, cmd/bootnode: fix / unify logging (#28696) This change fixes a problem with our non-core binaries: evm, clef, bootnode. First of all, they failed to convert from legacy loglevels 1 to 5, to the new slog loglevels -4 to 4. Secondly, the logging was actually setup in the init phase, and then overridden in the main. This is not needed for evm, since it used the same flag name as the main geth verbosity. Better to let the flags/internal handle the logging init. * cmd/evm: t8n support custom tracers (#28557) This change implements ability for the `evm t8n` tool to use custom tracers; either 'native' golang tracers or javascript tracers. * params: release go-ethereum v1.13.6 stable * params: begin v1.13.7 release cycle * internal/ethapi: ethSendTransaction check baseFee (#27834) If the EIP-1559 is activated, reject 0-priced transactions in the rpc level * core/rawdb: implement size reporting for live items in freezer_table (#28525) This is the fix to issue #27483. A new hiddenBytes() is introduced to calculate the byte size of hidden items in the freezer table. When reporting the size of the freezer table, size of the hidden items will be subtracted from the total size. --------- Co-authored-by: Yifan <Yifan Wang> Co-authored-by: Gary Rong <[email protected]> * build: make linter emit output (#28704) * .travis: set lower GOGC value (#28705) As documented on https://golangci-lint.run/usage/performance/ , a lower GOGC value causes less peak mem consumption when running the linter. Exceeding 3Gb is a common cause for build failures, according to https://docs.travis-ci.com/user/common-build-problems/#my-build-script-is-killed-without-any-error * ci: disable lint on travis (#28706) * build(deps): bump golang.org/x/crypto from 0.15.0 to 0.17.0 (#28702) Bumps [golang.org/x/crypto](https://github.com/golang/crypto) from 0.15.0 to 0.17.0. - [Commits](golang/crypto@v0.15.0...v0.17.0) --- updated-dependencies: - dependency-name: golang.org/x/crypto dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * params: go-ethereum v1.13.7 stable * params: begin go-ethereum v1.13.8 release cycle * internal/build: fix crash in MustRunCommandWithOutput (#28709) * accounts: properly close managed wallets when closing manager (#28710) * build: upgrade to golangci-lint v1.55.2 (#28712) This is primarily to make lint work again on macOS 14. The older version of golangci-lint kept crashing. Also included is a fix for a goroutine leak in the recently-introduced function MustRunCommandWithOutput. * cmd/devp2p: update eth/snap protocol test suites for PoS (#28340) Here we update the eth and snap protocol test suites with a new test chain, created by the hivechain tool. The new test chain uses proof-of-stake. As such, tests using PoW block propagation in the eth protocol are removed. The test suite now connects to the node under test using the engine API in order to make it accept transactions. The snap protocol test suite has been rewritten to output test descriptions and log requests more verbosely. --------- Co-authored-by: Felix Lange <[email protected]> * core, cmd, trie: fix the condition of pathdb initialization (#28718) Original problem was caused by #28595, where we made it so that as soon as we start to sync, the root of the disk layer is deleted. That is not wrong per se, but another part of the code uses the "presence of the root" as an init-check for the pathdb. And, since the init-check now failed, the code tried to re-initialize it which failed since a sync was already ongoing. The total impact being: after a state-sync has begun, if the node for some reason is is shut down, it will refuse to start up again, with the error message: `Fatal: Failed to register the Ethereum service: waiting for sync.`. This change also modifies how `geth removedb` works, so that the user is prompted for two things: `state data` and `ancient chain`. The former includes both the chaindb aswell as any state history stored in ancients. --------- Co-authored-by: Martin HS <[email protected]> * core/rawdb: improve state scheme checking (#28724) This pull request improves the condition to check if path state scheme is in use. Originally, root node presence was used as the indicator if path scheme is used or not. However due to fact that root node will be deleted during the initial snap sync, this condition is no longer useful. If PersistentStateID is present, it shows that we've already configured for path scheme. * params: go-ethereum v1.13.8 stable * params: begin v1.13.9 release cycle * core/state: logic equivalence for GetCodeHash (#28733) * tests: add currentExcessBlobGas to state tests (#28735) * accounts,signer: fix typos in comments (#28730) * build: add support for ubuntu 23.10 (mantic minotaur) (#28728) * log: avoid setting default slog logger in init (#28747) slog.SetDefault has undesirable side effects. It also sets the default logger destination, for example. So we should not call it by default in init. * cmd/evm: fix link in README.md (#28755) * core/vm: update comments to match eip number (#28743) * cmd/evm: Fix blob-gas-used on invalid transactions in t8n (#28734) cmd/evm: fixes the blob gas calculation if a transaction is invalid * internal/flags: update copyright year to 2024 (#28760) Co-authored-by: Felix Lange <[email protected]> * ethclient: simplify error handling in TransactionReceipt (#28748) Co-authored-by: Martin HS <[email protected]> Co-authored-by: Felix Lange <[email protected]> * eth/downloader, eth/filters: use defer to call Unsubscribe (#28762) * log: emit error level string as "error", not "eror" (#28774) * eth/filters: fix early Unsubscribe of log events (#28769) * cmd/devp2p/internal/ethtest: fix typos in comments (#28772) * params, core/forkid: schedule cancun fork on goerli (#28719) This PR schedules the cancun fork for the goerli testnet as discussed on ACD. Spec: ethereum/execution-specs#860 We schedule: goerli at 1705473120 * cmd/geth: make it possible to autopilot removedb (#28725) When managing geth, it is sometimes desirable to do a partial wipe; deleting state but retaining freezer data. A partial wipe can be somewhat tricky to accomplish. This change implements the ability to perform partial wipe by making it possible to run geth removedb non-interactive, using command line options instead. * accounts/abi: fix bigInt topic encoding (#28764) * cmd/geth: update log test data (#28780) update logger test data * ethclient/simulated: implement new sim backend (#28202) This is a rewrite of the 'simulated backend', an implementation of the ethclient interfaces which is backed by a simulated blockchain. It was getting annoying to maintain the old version of the simulated backend feature because there was a lot of code duplication with the main client. The new version is built using parts that we already have: an in-memory geth node instance running in developer mode provides the chain, while the Go API is provided by ethclient. A backwards-compatibility wrapper is provided, but the simulated backend has also moved to a more sensible import path: github.com/ethereum/go-ethereum/ethclient/simulated --------- Co-authored-by: Felix Lange <[email protected]> Co-authored-by: Gary Rong <[email protected]> * params: go-ethereum v1.13.9 stable * params: begin v1.13.10 release cycle * version: release v1.13.10 to fix bad tag * params: begin v1.13.11 release cycle * docs: fix badge in README (#28796) * Fix broken badge in README.md Replaced broken Github link with IPFS link for long-term storage. * update go badge Co-authored-by: lightclient <[email protected]> --------- Co-authored-by: lightclient <[email protected]> * eth: minor change of config-accessor (#28782) eth: refactor `GetVM` * cmd: fix typos (#28798) * build: fix typo in comment (#28800) * accounts, ethclient: minor tweaks on the new simulated backend (#28799) * accounts, ethclient: minor tweaks on the new simulated backend * ethclient/simulated: add an initial batch of gas options * accounts, ethclient: remove mandatory gasLimit constructor param * accounts, ethclient: minor option naming tweaks * cmd/geth, internal/debug: get rid of by-default log config (#28801) * cmd/devp2p/internal/ethtest: skip large tx test on github build (#28794) This test was failling consistently on the github 32-bit build probably due to slow IO. Skipping it for that green check. * p2p/dnsdisc: use strings.Cut over strings.IndexByte (#28787) * internal/ethapi: avoid using pending for defaults (#28784) Given the discussions around deprecating pending (see #28623 or ethereum/execution-apis#495), we can move away from using the pending block internally, and use latest instead * core/state: unexport GetOrNewStateObject (#28804) * cmd/rlpdump: add -pos flag, displaying byte positions (#28785) * tests: update reference tests (#28778) Updates the reference tests to the latest version * ethclient: add tests for TransactionInBlock (#28283) Co-authored-by: Felix Lange <[email protected]> * eth: fix potential hang in waitSnapExtension (#28744) This should fix a rare hang in waitSnapExtension during shutdown. * core/txpool/blobpool: fix typos * acounts/usbwallet: fix typo (#28815) acounts:fix typo * tests: more verbosity if block decoding fails (#28814) * tracer: use proper base fee in tests (#28775) In the tracing tests, the base fee was generally set to nil. This commit changes this to pass the proper base instead, and fixes the few tests which become broken by the change. * miner: fix typo in payload_building_test.go (#28825) * internal/ethapi: handle blobs in API methods (#28786) EIP-4844 adds a new transaction type for blobs. Users can submit such transactions via `eth_sendRawTransaction`. In this PR we refrain from adding support to `eth_sendTransaction` and in fact it will fail if the user passes in a blob hash. However since the chain can handle such transactions it makes sense to allow simulating them. E.g. an L2 operator should be able to simulate submitting a rollup blob and updating the L2 state. Most methods that take in a transaction object should recognize blobs. The change boils down to adding `blobVersionedHashes` and `maxFeePerBlobGas` to `TransactionArgs`. In summary: - `eth_sendTransaction`: will fail for blob txes - `eth_signTransaction`: will fail for blob txes The methods that sign txes does not, as of this PR, add support the for new EIP-4844 transaction types. Resuming the summary: - `eth_sendRawTransaction`: can send blob txes - `eth_fillTransaction`: will fill in a blob tx. Note: here we simply fill in normal transaction fields + possibly `maxFeePerBlobGas` when blobs are present. One can imagine a more elaborate set-up where users can submit blobs themselves and we fill in proofs and commitments and such. Left for future PRs if desired. - `eth_call`: can simulate blob messages - `eth_estimateGas`: blobs have no effect here. They have a separate unit of gas which is not tunable in the transaction. * eth/filters: reset filter.begin in BenchmarkFilters (#28830) * set head in sbundle pool * crypto/kzg4844: add helpers for versioned blob hashes (#28827) The code to compute a versioned hash was duplicated a couple times, and also had a small issue: if we ever change params.BlobTxHashVersion, it will most likely also cause changes to the actual hash computation. So it's a bit useless to have this constant in params. * ethclient: apply accessList field in toCallArg (#28832) Co-authored-by: Felix Lange <[email protected]> * params, core/forkid: enable cancun on sepolia and holesky (#28834) This change enables Cancun - Sepolia at 1706655072 (Jan 31st, 2024) - Holesky at 1707305664 (Feb 7th, 2024) Specification: ethereum/execution-specs#860 * core, core/rawdb, eth/sync: no tx indexing during snap sync (#28703) This change simplifies the logic for indexing transactions and enhances the UX when transaction is not found by returning more information to users. Transaction indexing is now considered as a part of the initial sync, and `eth.syncing` will thus be `true` if transaction indexing is not yet finished. API consumers can use the syncing status to determine if the node is ready to serve users. * docs: remove reference to being official (#28858) * go.{mod,sum}: upgrade go-ole to support arm64 (#28859) go.{mod,sum}: upgrade go-ole * core: fix genesis setup in benchReadChain (#28856) * all: use uint256 in state (#28598) This change makes use of uin256 to represent balance in state. It touches primarily upon statedb, stateobject and state processing, trying to avoid changes in transaction pools, core types, rpc and tracers. * build: upgrade -dlgo version to Go 1.21.6 (#28836) * core/state/snapshot: use AddHash/ContainHash instead of Hasher interface (#28849) This change switches from using the `Hasher` interface to add/query the bloomfilter to implementing it as methods. This significantly reduces the allocations for Search and Rebloom. * core/vm: fix misleading comment (#28860) fix misleading comment * eth/catalyst: add timestamp checks to fcu and new payload and improve param checks (#28230) This PR introduces a few changes with respect to payload verification in fcu and new payload requests: * First of all, it undoes the `verifyPayloadAttributes(..)` simplification I attempted in #27872. * Adds timestamp validation to fcu payload attributes [as required](https://github.com/ethereum/execution-apis/blob/main/src/engine/cancun.md#specification-1) (section 2) by the Engine API spec. * For the new payload methods, I also update the verification of the executable data. For `newPayloadV2`, it does not currently ensure that cancun values are `nil`. Which could make it possible to submit cancun payloads through it. * On `newPayloadV3` the same types of checks are added. All shanghai and cancun related fields in the executable data must be non-nil, with the addition that the timestamp is _only_ with cancun. * Finally it updates a newly failing catalyst test to call the correct fcu and new payload methods depending on the fork. * core/txpool, eth/catalyst: fix racy simulator due to txpool background reset (#28837) This PR fixes an issues in the new simulated backend. The root cause is the fact that the transaction pool has an internal reset operation that runs on a background thread. When a new transaction is added to the pool via the RPC, the transaction is added to a non-executable queue and will be moved to its final location on a background thread. If the machine is overloaded (or simply due to timing issues), it can happen that the simulated backend will try to produce the next block, whilst the pool has not yet marked the newly added transaction executable. This will cause the block to not contain the transaction. This is an issue because we want determinism from the simulator: add a tx, mine a block. It should be in there. The PR fixes it by adding a Sync function to the txpool, which waits for the current reset operation (if any) to finish, and then runs an entire round of reset on top. The new round is needed because resets are only triggered by new head events, so newly added transactions will not trigger the outer resets that we can wait on. The transaction pool would eventually internally do a reset even on transaction addition, but there's no easy way to wait on that and there's no meaningful reason to bubble that across everything. A clean outer reset will at worse be a small noop goroutine. * core: move tx indexer to its own file (#28857) This change moves all the transaction indexing functions to a separate txindexer.go file and defines a txIndexer structure as a refactoring. * eth/catalyst: prefix payload id with version (#28246) GetPayloadVX should only return payloads which match its version. GetPayloadV2 is a special snowflake that supports v1 and v2 payloads. This change uses a a version-specific prefix within in the payload id, basically a namespace for the version number. * ethclient: fix flaky test (#28864) Fix flaky test due to incomplete transaction indexing * params: go-ethereum v1.13.11 stable * params: begin v.1.13.12 release cycle * merge v1.13.11 * internal/flags: fix typo (#28876) * core/types: fix and test handling of faulty nil-returning signer (#28879) This adds an error if the signer returns a nil value for one of the signature value fields. * README.md: fix travis badge (#28889) The hyperlink in the README file that directs to the Travis CI build was broken. This commit updates the link to point to the corrent build page. * eth/catalyst: allow payload attributes v1 in fcu v2 (#28882) At some point, `ForkchoiceUpdatedV2` stopped working for `PayloadAttributesV1` while `paris` was active. This was causing a few failures in hive. This PR fixes that, and also adds a gate in `ForkchoiceUpdatedV1` to disallow `PayloadAttributesV3`. * docs/postmortems: fix outdated link (#28893) * core: reset tx lookup cache if necessary (#28865) This pull request resets the txlookup cache if chain reorg happens, preventing them from remaining reachable. It addresses failures in the hive tests. * build: fix problem with windows line-endings in CI download (#28900) fixes #28890 * eth/downloader: fix skeleton cleanup (#28581) * eth/downloader: fix skeleton cleanup * eth/downloader: short circuit if nothing to delete * eth/downloader: polish the logic in cleanup * eth/downloader: address comments * deps: update memsize (#28916) * core/txpool/blobpool: post-crash cleanup and addition/removal metrics (#28914) * core/txpool/blobpool: clean up resurrected junk after a crash * core/txpool/blobpool: track transaction insertions and rejections * core/txpool/blobpool: linnnnnnnt * core/txpool: don't inject lazy resolved transactions into the container (#28917) * core/txpool: don't inject lazy resolved transactions into the container * core/txpool: minor typo fixes * add bundle support * core/types: fix typo (#28922) * fix tests * add blob fields * fix logging * p2p: fix accidental termination of portMappingLoop (#28911) * internal/flags: fix --miner.gasprice default listing (#28932) * all: fix typos in comments (#28881) * Makefile: add help target to display available targets (#28845) Co-authored-by: Martin HS <[email protected]> Co-authored-by: Felix Lange <[email protected]> * core: cache transaction indexing tail in memory (#28908) * eth, miner: fix enforcing the minimum miner tip (#28933) * eth, miner: fix enforcing the minimum miner tip * ethclient/simulated: fix failing test due the min tip change * accounts/abi/bind: fix simulater gas tip issue * remove private txs from blobs * core/state, core/vm: minor uint256 related perf improvements (#28944) * cmd,internal/era: implement `export-history` subcommand (#26621) * all: implement era format, add history importer/export * internal/era/e2store: refactor e2store to provide ReadAt interface * internal/era/e2store: export HeaderSize * internal/era: refactor era to use ReadAt interface * internal/era: elevate anonymous func to named * cmd/utils: don't store entire era file in-memory during import / export * internal/era: better abstraction between era and e2store * cmd/era: properly close era files * cmd/era: don't let defers stack * cmd/geth: add description for import-history * cmd/utils: better bytes buffer * internal/era: error if accumulator has more records than max allowed * internal/era: better doc comment * internal/era/e2store: rm superfluous reader, rm superfluous testcases, add fuzzer * internal/era: avoid some repetition * internal/era: simplify clauses * internal/era: unexport things * internal/era,cmd/utils,cmd/era: change to iterator interface for reading era entries * cmd/utils: better defer handling in history test * internal/era,cmd: add number method to era iterator to get the current block number * internal/era/e2store: avoid double allocation during write * internal/era,cmd/utils: fix lint issues * internal/era: add ReaderAt func so entry value can be read lazily Co-authored-by: lightclient <[email protected]> Co-authored-by: Martin Holst Swende <[email protected]> * internal/era: improve iterator interface * internal/era: fix rlp decode of header and correctly read total difficulty * cmd/era: fix rebase errors * cmd/era: clearer comments * cmd,internal: fix comment typos --------- Co-authored-by: Martin Holst Swende <[email protected]> * core,params: add holesky to default genesis function (#28903) * fix tests * node, rpc: add configurable HTTP request limit (#28948) Adds a configurable HTTP request limit, and bumps the engine default * all: fix docstring names (#28923) * fix wrong comment * reviewers input * Update log/handler_glog.go --------- Co-authored-by: Martin HS <[email protected]> * ethclient/simulated: fix typo (#28952) (ethclient/simulated):fix typo * eth/gasprice: fix percentile validation in eth_feeHistory (#28954) * cmd/devp2p, eth: drop support for eth/67 (#28956) * params, core/forkid: add mainnet timestamp for Cancun (#28958) * params: add cancun timestamp for mainnet * core/forkid: add test for mainnet cancun forkid * core/forkid: update todo tests for cancun * internal/ethapi: add support for blobs in eth_fillTransaction (#28839) This change adds support for blob-transaction in certain API-endpoints, e.g. eth_fillTransaction. A follow-up PR will add support for signing such transactions. * internal/era: update block index format to be based on record offset (#28959) As mentioned in #26621, the block index format for era1 is not in line with the regular era block index. This change modifies the index so all relative offsets are based against the beginning of the block index record. * params: go-ethereum v1.13.12 stable * params: begin v1.13.13 release cycle * build: remove ubuntu 'lunar' build (#28962) * fix: update outdated link to trezor docs (#28966) fix: update link to trezor * internal/ethapi: fix gas estimation bug in eth_fillTransaction for blob tx (#28929) * fix nil pointer resolving tx * fix memory leak in prefetcher * core/txpool/legacypool: use uint256.Int instead of big.Int (#28606) This change makes the legacy transaction pool use of `uint256.Int` instead of `big.Int`. The changes are made primarily only on the internal functions of legacypool. --------- Co-authored-by: Martin Holst Swende <[email protected]> * internal/ethapi, signer/core: fix documentation-links (#28979) fix: management api links * all: remove the dependency from trie to triedb (#28824) This change removes the dependency from trie package to triedb package. * cmd/utils: fix merge-breakage in test (#28985) * tests: fix goroutine leak related to state snapshot generation (#28974) --------- Co-authored-by: Felix Lange <[email protected]> * ethereum, ethclient: add blob transaction fields in CallMsg (#28989) Co-authored-by: Felix Lange <[email protected]> * core/txpool/blobpool: rename variables in comments (#28981) Co-authored-by: Felix Lange <[email protected]> * cmd/devp2p: fix modulo in makeBlobTxs (#28970) * eth/catalyst,beacon/engine: implement GetClientVersionV1 (#28915) * tests: update execution spec tests + split statetest exec (#28993) * eth/catalyst: add getClientVersion to capabilities (#28994) * cmd/evm: fix typo in test script (#28995) * cmd/devp2p/internal/ethtest: some fixes for the eth test suite (#28996) Improving two things here: On hive, where we look at these tests, the Go code comment above the test is not visible. When there is a failure, it's not obvious what the test is actually expecting. I have converted the comments in to printed log messages to explain the test more. Second, I noticed that besu is failing some tests because it happens to request a header when we want it to send transactions. Trying the minimal fix here to serve the headers. Co-authored-by: lightclient <[email protected]> * core/txpool/legacypool: remove a redundant heap.Init (#28910) Co-authored-by: Martin HS <[email protected]> Co-authored-by: Felix Lange <[email protected]> * core/txpool/blobpool: update the blob db with corruption handling (#29001) Updates billy to a more recent version which is more robust in the face of corrupt data (e.g. after a hard crash) * core: move genesis alloc types to core/types (#29003) We want to use these types in public user-facing APIs, so they shouldn't be in core. Co-authored-by: Felix Lange <[email protected]> * core/txpool, eth, miner: pre-filter dynamic fees during pending tx retrieval (#29005) * core/txpool, eth, miner: pre-filter dynamic fees during pending tx retrieval * miner: fix typo * core/txpool: handle init-error in blobpool without panicing --------- Co-authored-by: Martin Holst Swende <[email protected]> * ethstats: prevent panic if head block is not available (#29020) This pull request fixes a flaw in ethstats which can lead to node crash A panic could happens when the local blockchain is reorging which causes the original head block not to be reachable (since number->hash canonical mapping is deleted). In order to prevent the panic, the block nilness is now checked in ethstats. * core: using math.MaxUint64 instead of 0xffffffffffffffff (#29022) * core/txpool, miner: speed up blob pool pending retrievals (#29008) * core/txpool, miner: speed up blob pool pending retrievals * miner: fix test merge issue * eth: same same * core/txpool/blobpool: speed up blobtx creation in benchmark a bit * core/txpool/blobpool: fix linter --------- Co-authored-by: Martin Holst Swende <[email protected]> * core/vm, params: ensure order of forks, prevent overflow (#29023) This PR fixes an overflow which can could happen if inconsistent blockchain rules were configured. Additionally, it tries to prevent such inconsistencies from occurring by making sure that merge cannot be enabled unless previous fork(s) are also enabled. * core/txpool, eth, miner: retrieve plain and blob txs separately (#29026) * core/txpool, eth, miner: retrieve plain and blob txs separately * core/txpool: fix typo, no farming * miner: farm all the typos Co-authored-by: Martin HS <[email protected]> --------- Co-authored-by: Martin HS <[email protected]> * core/txpool: fix typo (#29031) * core,eth,internal: fix typo (#29024) * params: add cancun upgrade banner (#29042) params: add cancun banner Signed-off-by: tmelhao <[email protected]> Co-authored-by: tmelhao <[email protected]> * core/txpool: fix typo (#29036) * fix typos * address comments * internal/ethapi: fix defaults for blob fields (#29037) Co-authored-by: Martin HS <[email protected]> * params: release go-ethereum v1.13.13 stable * params: begin v1.13.14 release cycle * internal/ethapi: use overriden baseFee for gasPrice (#29051) eth_call and debug_traceCall allow users to override various block fields, among them base fee. However the overriden base fee was not considered for computing the effective gas price of that message, and instead base fee of the base block was used. This has been fixed in this commit. * internal/ethapi:fix zero rpc gas cap in eth_createAccessList (#28846) This PR enhances eth_createAccessList RPC call to support scenarios where the node is launched with an unlimited gas cap (--rpc.gascap 0). The eth_createAccessList RPC call returns failure if user doesn't explicitly set a gas limit. * eth/catalyst: fix wrong error message of payloadV2 after cancun (#29049) * eth/catalyst: the same error format Signed-off-by: tmelhao <[email protected]> * eth/catalyst: wrong error message for payloadV2 post-cancun Signed-off-by: tmelhao <[email protected]> * eth/catalyst: parentBeaconBlockRoot -> parentBlockBeaconRoot Signed-off-by: tmelhao <[email protected]> * apply commit review Signed-off-by: tmelhao <[email protected]> --------- Signed-off-by: tmelhao <[email protected]> Co-authored-by: tmelhao <[email protected]> * fix tests * core/txpool: reject blob txs with blob fee cap below the minimum (#29081) * make blobpool reject blob transactions with fee below the minimum * core/txpool: some minot nitpick polishes and unified error formats * core/txpool: do less big.Int constructions with the min blob cap --------- Co-authored-by: Péter Szilágyi <[email protected]> * p2p, log, rpc: use errors.New to replace fmt.Errorf with no parameters (#29074) * eth/catalyst: enable some commented-out testcases (#29073) * internal/ethapi: pass blob hashes to gas estimation (#29085) * cmd/clef: add spaces in README.md table (#29077) Add space after links in so they are clickable in vscode. * eth/catalyst: remove variable in tx conversion loop (#29076) * core/txpool/blobpool: reduce default database cap for rollout (#29090) xcore/txpool/blobpool: reduce default database cap for rollout * core/txpool: no need to log loud rotate if no local txs (#29083) * core/txpool: no need to run rotate if no local txs Signed-off-by: jsvisa <[email protected]> * Revert "core/txpool: no need to run rotate if no local txs" This reverts commit 17fab173883168c586d57ca9c05dfcbd9e7831b4. Signed-off-by: jsvisa <[email protected]> * use Debug if todo is empty Signed-off-by: jsvisa <[email protected]> --------- Signed-off-by: jsvisa <[email protected]> * eth/tracers: Fix callTracer logs on onlyTopCall == true (#29068) * core/txpool: elevate the 'already reserved' error into a constant (#29095) declare the 'already reserved' error in errors.go * params: release Geth v1.13.14 * fix statedb.Finalize() * minimize diff from upstream (#147) * add back private tx logic * add excess blob gas to mev sim bundle (#148) * remove panic from worker.go --------- Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: tmelhao <[email protected]> Signed-off-by: jsvisa <[email protected]> Co-authored-by: Marius van der Wijden <[email protected]> Co-authored-by: Maciej Kulawik <[email protected]> Co-authored-by: ucwong <[email protected]> Co-authored-by: Mario Vega <[email protected]> Co-authored-by: Delweng <[email protected]> Co-authored-by: jwasinger <[email protected]> Co-authored-by: Felix Lange <[email protected]> Co-authored-by: lightclient <[email protected]> Co-authored-by: Håvard Anda Estensen <[email protected]> Co-authored-by: Guillaume Ballet <[email protected]> Co-authored-by: Martin Holst Swende <[email protected]> Co-authored-by: Mikel Cortes <[email protected]> Co-authored-by: Péter Szilágyi <[email protected]> Co-authored-by: Ng Wei Han <[email protected]> Co-authored-by: rjl493456442 <[email protected]> Co-authored-by: lightclient <[email protected]> Co-authored-by: Shivam Sandbhor <[email protected]> Co-authored-by: Sina Mahmoodi <[email protected]> Co-authored-by: Jakub Freebit <[email protected]> Co-authored-by: Oren <[email protected]> Co-authored-by: BorkBorked <[email protected]> Co-authored-by: ddl <[email protected]> Co-authored-by: Manav Darji <[email protected]> Co-authored-by: Marius Kjærstad <[email protected]> Co-authored-by: Felföldi Zsolt <[email protected]> Co-authored-by: Ford <[email protected]> Co-authored-by: Ursulafe <[email protected]> Co-authored-by: Elias Rad <[email protected]> Co-authored-by: FletcherMan <[email protected]> Co-authored-by: alex <[email protected]> Co-authored-by: Sebastian Stammler <[email protected]> Co-authored-by: wangyifan <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: cygaar <[email protected]> Co-authored-by: Taeguk Kwon <[email protected]> Co-authored-by: ddl <[email protected]> Co-authored-by: Darioush Jalali <[email protected]> Co-authored-by: Rossen Krastev <[email protected]> Co-authored-by: vuittont60 <[email protected]> Co-authored-by: drstevenbrule <[email protected]> Co-authored-by: HAOYUatHZ <[email protected]> Co-authored-by: hyunchel <[email protected]> Co-authored-by: 牛晓婕 <[email protected]> Co-authored-by: Alfie John <[email protected]> Co-authored-by: Paul Lange <[email protected]> Co-authored-by: Thabokani <[email protected]> Co-authored-by: colin <[email protected]> Co-authored-by: trocher <[email protected]> Co-authored-by: protolambda <[email protected]> Co-authored-by: KeienWang <[email protected]> Co-authored-by: zoereco <[email protected]> Co-authored-by: Chris Ziogas <[email protected]> Co-authored-by: Dimitris Apostolou <[email protected]> Co-authored-by: Halimao <[email protected]> Co-authored-by: lmittmann <[email protected]> Co-authored-by: Peter Straus <[email protected]> Co-authored-by: maskpp <[email protected]> Co-authored-by: Lindlof <[email protected]> Co-authored-by: bk <[email protected]> Co-authored-by: cui <[email protected]> Co-authored-by: buddho <[email protected]> Co-authored-by: Haotian <[email protected]> Co-authored-by: tmelhao <[email protected]> Co-authored-by: ArtificialPB <[email protected]> Co-authored-by: Roberto Bayardo <[email protected]> Co-authored-by: Qt <[email protected]> Co-authored-by: Justin Dhillon <[email protected]> Co-authored-by: Andrei Silviu Dragnea <[email protected]> Co-authored-by: Vitaly Drogan <[email protected]>

* params: begin v1.13.9 release cycle * core/state: logic equivalence for GetCodeHash (#28733) * tests: add currentExcessBlobGas to state tests (#28735) * accounts,signer: fix typos in comments (#28730) * build: add support for ubuntu 23.10 (mantic minotaur) (#28728) * log: avoid setting default slog logger in init (#28747) slog.SetDefault has undesirable side effects. It also sets the default logger destination, for example. So we should not call it by default in init. * cmd/evm: fix link in README.md (#28755) * core/vm: update comments to match eip number (#28743) * cmd/evm: Fix blob-gas-used on invalid transactions in t8n (#28734) cmd/evm: fixes the blob gas calculation if a transaction is invalid * internal/flags: update copyright year to 2024 (#28760) Co-authored-by: Felix Lange <[email protected]> * ethclient: simplify error handling in TransactionReceipt (#28748) Co-authored-by: Martin HS <[email protected]> Co-authored-by: Felix Lange <[email protected]> * eth/downloader, eth/filters: use defer to call Unsubscribe (#28762) * log: emit error level string as "error", not "eror" (#28774) * eth/filters: fix early Unsubscribe of log events (#28769) * cmd/devp2p/internal/ethtest: fix typos in comments (#28772) * params, core/forkid: schedule cancun fork on goerli (#28719) This PR schedules the cancun fork for the goerli testnet as discussed on ACD. Spec: ethereum/execution-specs#860 We schedule: goerli at 1705473120 * cmd/geth: make it possible to autopilot removedb (#28725) When managing geth, it is sometimes desirable to do a partial wipe; deleting state but retaining freezer data. A partial wipe can be somewhat tricky to accomplish. This change implements the ability to perform partial wipe by making it possible to run geth removedb non-interactive, using command line options instead. * accounts/abi: fix bigInt topic encoding (#28764) * cmd/geth: update log test data (#28780) update logger test data * ethclient/simulated: implement new sim backend (#28202) This is a rewrite of the 'simulated backend', an implementation of the ethclient interfaces which is backed by a simulated blockchain. It was getting annoying to maintain the old version of the simulated backend feature because there was a lot of code duplication with the main client. The new version is built using parts that we already have: an in-memory geth node instance running in developer mode provides the chain, while the Go API is provided by ethclient. A backwards-compatibility wrapper is provided, but the simulated backend has also moved to a more sensible import path: github.com/ethereum/go-ethereum/ethclient/simulated --------- Co-authored-by: Felix Lange <[email protected]> Co-authored-by: Gary Rong <[email protected]> * params: go-ethereum v1.13.9 stable * params: begin v1.13.10 release cycle * version: release v1.13.10 to fix bad tag * params: begin v1.13.11 release cycle * docs: fix badge in README (#28796) * Fix broken badge in README.md Replaced broken Github link with IPFS link for long-term storage. * update go badge Co-authored-by: lightclient <[email protected]> --------- Co-authored-by: lightclient <[email protected]> * eth: minor change of config-accessor (#28782) eth: refactor `GetVM` * cmd: fix typos (#28798) * build: fix typo in comment (#28800) * accounts, ethclient: minor tweaks on the new simulated backend (#28799) * accounts, ethclient: minor tweaks on the new simulated backend * ethclient/simulated: add an initial batch of gas options * accounts, ethclient: remove mandatory gasLimit constructor param * accounts, ethclient: minor option naming tweaks * cmd/geth, internal/debug: get rid of by-default log config (#28801) * cmd/devp2p/internal/ethtest: skip large tx test on github build (#28794) This test was failling consistently on the github 32-bit build probably due to slow IO. Skipping it for that green check. * p2p/dnsdisc: use strings.Cut over strings.IndexByte (#28787) * internal/ethapi: avoid using pending for defaults (#28784) Given the discussions around deprecating pending (see #28623 or ethereum/execution-apis#495), we can move away from using the pending block internally, and use latest instead * core/state: unexport GetOrNewStateObject (#28804) * cmd/rlpdump: add -pos flag, displaying byte positions (#28785) * tests: update reference tests (#28778) Updates the reference tests to the latest version * ethclient: add tests for TransactionInBlock (#28283) Co-authored-by: Felix Lange <[email protected]> * eth: fix potential hang in waitSnapExtension (#28744) This should fix a rare hang in waitSnapExtension during shutdown. * core/txpool/blobpool: fix typos * acounts/usbwallet: fix typo (#28815) acounts:fix typo * tests: more verbosity if block decoding fails (#28814) * tracer: use proper base fee in tests (#28775) In the tracing tests, the base fee was generally set to nil. This commit changes this to pass the proper base instead, and fixes the few tests which become broken by the change. * miner: fix typo in payload_building_test.go (#28825) * internal/ethapi: handle blobs in API methods (#28786) EIP-4844 adds a new transaction type for blobs. Users can submit such transactions via `eth_sendRawTransaction`. In this PR we refrain from adding support to `eth_sendTransaction` and in fact it will fail if the user passes in a blob hash. However since the chain can handle such transactions it makes sense to allow simulating them. E.g. an L2 operator should be able to simulate submitting a rollup blob and updating the L2 state. Most methods that take in a transaction object should recognize blobs. The change boils down to adding `blobVersionedHashes` and `maxFeePerBlobGas` to `TransactionArgs`. In summary: - `eth_sendTransaction`: will fail for blob txes - `eth_signTransaction`: will fail for blob txes The methods that sign txes does not, as of this PR, add support the for new EIP-4844 transaction types. Resuming the summary: - `eth_sendRawTransaction`: can send blob txes - `eth_fillTransaction`: will fill in a blob tx. Note: here we simply fill in normal transaction fields + possibly `maxFeePerBlobGas` when blobs are present. One can imagine a more elaborate set-up where users can submit blobs themselves and we fill in proofs and commitments and such. Left for future PRs if desired. - `eth_call`: can simulate blob messages - `eth_estimateGas`: blobs have no effect here. They have a separate unit of gas which is not tunable in the transaction. * eth/filters: reset filter.begin in BenchmarkFilters (#28830) * crypto/kzg4844: add helpers for versioned blob hashes (#28827) The code to compute a versioned hash was duplicated a couple times, and also had a small issue: if we ever change params.BlobTxHashVersion, it will most likely also cause changes to the actual hash computation. So it's a bit useless to have this constant in params. * ethclient: apply accessList field in toCallArg (#28832) Co-authored-by: Felix Lange <[email protected]> * params, core/forkid: enable cancun on sepolia and holesky (#28834) This change enables Cancun - Sepolia at 1706655072 (Jan 31st, 2024) - Holesky at 1707305664 (Feb 7th, 2024) Specification: ethereum/execution-specs#860 * core, core/rawdb, eth/sync: no tx indexing during snap sync (#28703) This change simplifies the logic for indexing transactions and enhances the UX when transaction is not found by returning more information to users. Transaction indexing is now considered as a part of the initial sync, and `eth.syncing` will thus be `true` if transaction indexing is not yet finished. API consumers can use the syncing status to determine if the node is ready to serve users. * docs: remove reference to being official (#28858) * go.{mod,sum}: upgrade go-ole to support arm64 (#28859) go.{mod,sum}: upgrade go-ole * core: fix genesis setup in benchReadChain (#28856) * all: use uint256 in state (#28598) This change makes use of uin256 to represent balance in state. It touches primarily upon statedb, stateobject and state processing, trying to avoid changes in transaction pools, core types, rpc and tracers. * build: upgrade -dlgo version to Go 1.21.6 (#28836) * core/state/snapshot: use AddHash/ContainHash instead of Hasher interface (#28849) This change switches from using the `Hasher` interface to add/query the bloomfilter to implementing it as methods. This significantly reduces the allocations for Search and Rebloom. * core/vm: fix misleading comment (#28860) fix misleading comment * eth/catalyst: add timestamp checks to fcu and new payload and improve param checks (#28230) This PR introduces a few changes with respect to payload verification in fcu and new payload requests: * First of all, it undoes the `verifyPayloadAttributes(..)` simplification I attempted in #27872. * Adds timestamp validation to fcu payload attributes [as required](https://github.com/ethereum/execution-apis/blob/main/src/engine/cancun.md#specification-1) (section 2) by the Engine API spec. * For the new payload methods, I also update the verification of the executable data. For `newPayloadV2`, it does not currently ensure that cancun values are `nil`. Which could make it possible to submit cancun payloads through it. * On `newPayloadV3` the same types of checks are added. All shanghai and cancun related fields in the executable data must be non-nil, with the addition that the timestamp is _only_ with cancun. * Finally it updates a newly failing catalyst test to call the correct fcu and new payload methods depending on the fork. * core/txpool, eth/catalyst: fix racy simulator due to txpool background reset (#28837) This PR fixes an issues in the new simulated backend. The root cause is the fact that the transaction pool has an internal reset operation that runs on a background thread. When a new transaction is added to the pool via the RPC, the transaction is added to a non-executable queue and will be moved to its final location on a background thread. If the machine is overloaded (or simply due to timing issues), it can happen that the simulated backend will try to produce the next block, whilst the pool has not yet marked the newly added transaction executable. This will cause the block to not contain the transaction. This is an issue because we want determinism from the simulator: add a tx, mine a block. It should be in there. The PR fixes it by adding a Sync function to the txpool, which waits for the current reset operation (if any) to finish, and then runs an entire round of reset on top. The new round is needed because resets are only triggered by new head events, so newly added transactions will not trigger the outer resets that we can wait on. The transaction pool would eventually internally do a reset even on transaction addition, but there's no easy way to wait on that and there's no meaningful reason to bubble that across everything. A clean outer reset will at worse be a small noop goroutine. * core: move tx indexer to its own file (#28857) This change moves all the transaction indexing functions to a separate txindexer.go file and defines a txIndexer structure as a refactoring. * eth/catalyst: prefix payload id with version (#28246) GetPayloadVX should only return payloads which match its version. GetPayloadV2 is a special snowflake that supports v1 and v2 payloads. This change uses a a version-specific prefix within in the payload id, basically a namespace for the version number. * ethclient: fix flaky test (#28864) Fix flaky test due to incomplete transaction indexing * params: go-ethereum v1.13.11 stable * go get github.com/ethereum-optimism/superchain-registry/superchain@52d3dbd1605dd43f419e838584abd0ec163d462b * params: begin v1.13.9 release cycle * core/state: logic equivalence for GetCodeHash (#28733) * tests: add currentExcessBlobGas to state tests (#28735) * accounts,signer: fix typos in comments (#28730) * build: add support for ubuntu 23.10 (mantic minotaur) (#28728) * log: avoid setting default slog logger in init (#28747) slog.SetDefault has undesirable side effects. It also sets the default logger destination, for example. So we should not call it by default in init. * cmd/evm: fix link in README.md (#28755) * core/vm: update comments to match eip number (#28743) * cmd/evm: Fix blob-gas-used on invalid transactions in t8n (#28734) cmd/evm: fixes the blob gas calculation if a transaction is invalid * internal/flags: update copyright year to 2024 (#28760) Co-authored-by: Felix Lange <[email protected]> * ethclient: simplify error handling in TransactionReceipt (#28748) Co-authored-by: Martin HS <[email protected]> Co-authored-by: Felix Lange <[email protected]> * eth/downloader, eth/filters: use defer to call Unsubscribe (#28762) * log: emit error level string as "error", not "eror" (#28774) * eth/filters: fix early Unsubscribe of log events (#28769) * cmd/devp2p/internal/ethtest: fix typos in comments (#28772) * params, core/forkid: schedule cancun fork on goerli (#28719) This PR schedules the cancun fork for the goerli testnet as discussed on ACD. Spec: ethereum/execution-specs#860 We schedule: goerli at 1705473120 * cmd/geth: make it possible to autopilot removedb (#28725) When managing geth, it is sometimes desirable to do a partial wipe; deleting state but retaining freezer data. A partial wipe can be somewhat tricky to accomplish. This change implements the ability to perform partial wipe by making it possible to run geth removedb non-interactive, using command line options instead. * accounts/abi: fix bigInt topic encoding (#28764) * cmd/geth: update log test data (#28780) update logger test data * ethclient/simulated: implement new sim backend (#28202) This is a rewrite of the 'simulated backend', an implementation of the ethclient interfaces which is backed by a simulated blockchain. It was getting annoying to maintain the old version of the simulated backend feature because there was a lot of code duplication with the main client. The new version is built using parts that we already have: an in-memory geth node instance running in developer mode provides the chain, while the Go API is provided by ethclient. A backwards-compatibility wrapper is provided, but the simulated backend has also moved to a more sensible import path: github.com/ethereum/go-ethereum/ethclient/simulated --------- Co-authored-by: Felix Lange <[email protected]> Co-authored-by: Gary Rong <[email protected]> * params: go-ethereum v1.13.9 stable * add ability to create simulated backend from ethconfig update fork.yaml * Revert "Merge upstream geth v1.13.9 into op-geth" * remove duplicate file reference which breaks forkdiff (#285) * fix JSON logger to properly handle debug & lower log level * core/vm, params: ensure order of forks, prevent overflow (#29023) This PR fixes an overflow which can could happen if inconsistent blockchain rules were configured. Additionally, it tries to prevent such inconsistencies from occurring by making sure that merge cannot be enabled unless previous fork(s) are also enabled. --------- Co-authored-by: Martin Holst Swende <[email protected]> Co-authored-by: cygaar <[email protected]> Co-authored-by: Mario Vega <[email protected]> Co-authored-by: Taeguk Kwon <[email protected]> Co-authored-by: Felix Lange <[email protected]> Co-authored-by: ddl <[email protected]> Co-authored-by: Darioush Jalali <[email protected]> Co-authored-by: Marius Kjærstad <[email protected]> Co-authored-by: Rossen Krastev <[email protected]> Co-authored-by: ucwong <[email protected]> Co-authored-by: jwasinger <[email protected]> Co-authored-by: vuittont60 <[email protected]> Co-authored-by: Marius van der Wijden <[email protected]> Co-authored-by: rjl493456442 <[email protected]> Co-authored-by: Péter Szilágyi <[email protected]> Co-authored-by: drstevenbrule <[email protected]> Co-authored-by: lightclient <[email protected]> Co-authored-by: HAOYUatHZ <[email protected]> Co-authored-by: Sina Mahmoodi <[email protected]> Co-authored-by: hyunchel <[email protected]> Co-authored-by: 牛晓婕 <[email protected]> Co-authored-by: Alfie John <[email protected]> Co-authored-by: alex <[email protected]> Co-authored-by: Paul Lange <[email protected]> Co-authored-by: Thabokani <[email protected]> Co-authored-by: colin <[email protected]> Co-authored-by: Guillaume Ballet <[email protected]> Co-authored-by: trocher <[email protected]> Co-authored-by: geoknee <[email protected]> Co-authored-by: Sebastian Stammler <[email protected]> Co-authored-by: Roberto Bayardo <[email protected]> Co-authored-by: Joshua Gutow <[email protected]> Co-authored-by: Joshua Gutow <[email protected]> Co-authored-by: protolambda <[email protected]>

Given the discussions around deprecating pending (see #28623 or ethereum/execution-apis#495), we can move away from using the pending block internally, and use latest instead

s1na mentioned this issue Jan 10, 2024

internal/ethapi: avoid using pending for defaults ethereum/go-ethereum#28784

Merged

egonspace mentioned this issue Jun 13, 2024

An eth_estimate request uses stale state in a mining node. wemixarchive/go-wemix#109

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Deprecating the pending state #495

Deprecating the pending state #495

s1na commented Nov 23, 2023 •

edited

Loading

protolambda commented Nov 23, 2023

s1na commented Nov 24, 2023

lmittmann commented Nov 24, 2023

s1na commented Nov 24, 2023

niklasnatter commented Nov 24, 2023

s1na commented Nov 24, 2023

ryanschneider commented Nov 28, 2023

Deprecating the pending state #495

Deprecating the pending state #495

Comments

s1na commented Nov 23, 2023 • edited Loading

protolambda commented Nov 23, 2023

s1na commented Nov 24, 2023

lmittmann commented Nov 24, 2023

s1na commented Nov 24, 2023

niklasnatter commented Nov 24, 2023

s1na commented Nov 24, 2023

ryanschneider commented Nov 28, 2023

s1na commented Nov 23, 2023 •

edited

Loading