eth_gasPrice ERROR Could not retrieve block

[siladu]

While retesting #6965, which involved running a QBFT node and throwing https://github.com/MariusVanDerWijden/tx-fuzz at it. I received a new error:

2024-07-01 16:08:56.279+10:00 | vert.x-worker-thread-8 | ERROR | BaseJsonRpcProcessor | Error processing method: eth_gasPrice null
java.lang.IllegalStateException: Could not retrieve block #10
	at org.hyperledger.besu.ethereum.api.query.BlockchainQueries.lambda$gasPrice$25(BlockchainQueries.java:994)
	at java.base/java.util.Optional.orElseThrow(Optional.java:403)
	at org.hyperledger.besu.ethereum.api.query.BlockchainQueries.lambda$gasPrice$26(BlockchainQueries.java:993)
	at java.base/java.util.stream.LongPipeline$1$1.accept(LongPipeline.java:177)
	at java.base/java.util.stream.Streams$RangeLongSpliterator.forEachRemaining(Streams.java:232)
	at java.base/java.util.Spliterator$OfLong.forEachRemaining(Spliterator.java:777)
	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
	at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:575)
	at java.base/java.util.stream.AbstractPipeline.evaluateToArrayNode(AbstractPipeline.java:260)
	at java.base/java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:616)
	at org.hyperledger.besu.ethereum.api.query.BlockchainQueries.gasPrice(BlockchainQueries.java:999)
	at org.hyperledger.besu.ethereum.api.jsonrpc.internal.methods.EthGasPrice.calculateGasPrice(EthGasPrice.java:49)
	at org.hyperledger.besu.ethereum.api.jsonrpc.internal.methods.EthGasPrice.response(EthGasPrice.java:45)
	at org.hyperledger.besu.ethereum.api.jsonrpc.execution.BaseJsonRpcProcessor.process(BaseJsonRpcProcessor.java:44)
	at org.hyperledger.besu.ethereum.api.jsonrpc.execution.TracedJsonRpcProcessor.process(TracedJsonRpcProcessor.java:54)
	at org.hyperledger.besu.ethereum.api.jsonrpc.execution.TimedJsonRpcProcessor.process(TimedJsonRpcProcessor.java:45)
	at org.hyperledger.besu.ethereum.api.jsonrpc.execution.JsonRpcExecutor.execute(JsonRpcExecutor.java:92)
	at org.hyperledger.besu.ethereum.api.handlers.AbstractJsonRpcExecutor.executeRequest(AbstractJsonRpcExecutor.java:87)
	at org.hyperledger.besu.ethereum.api.handlers.JsonRpcObjectExecutor.execute(JsonRpcObjectExecutor.java:57)
	at org.hyperledger.besu.ethereum.api.handlers.JsonRpcExecutorHandler.lambda$handler$0(JsonRpcExecutorHandler.java:57)
	at java.base/java.util.Optional.ifPresentOrElse(Optional.java:196)
	at org.hyperledger.besu.ethereum.api.handlers.JsonRpcExecutorHandler.lambda$handler$2(JsonRpcExecutorHandler.java:54)
	at io.vertx.ext.web.impl.BlockingHandlerDecorator.lambda$handle$0(BlockingHandlerDecorator.java:48)
	at io.vertx.core.impl.ContextImpl.lambda$executeBlocking$1(ContextImpl.java:191)
	at io.vertx.core.impl.ContextInternal.dispatch(ContextInternal.java:279)
	at io.vertx.core.impl.ContextImpl.lambda$internalExecuteBlocking$2(ContextImpl.java:210)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:1583)

This PR had main pulled in recently. Previous test was based on May 21st main, that area of code had changed around May 2nd, in 8df6bca#diff-2ef41086175ba1eea041526cfd1d92a80688eeeb8e4e1f17566b2f3d65ed7cae

Still trying to recreate it consistently.

[siladu]

I can reliably recreate this - seems to happen the first time the tx-fuzz runs on a new network. Why this is the case is unknown, would need to study the pattern of requests it makes.

Having tested older versions of Besu, I am confident 8df6bca introduced this error.

24.5.4 - works
24.6.0 - breaks
8df6bca - Make eth_gasPrice aware of the base fee market - breaks
bd12986 (the commit before "Make eth_gasPrice aware...") - works.

Attaching a terminal log showing the various errors/lack of errors for each version
7287.log

---

The impact isn't fully clear as only QBFT has been tested. It sometimes kills the tx-fuzz tool...

➜  tx-fuzz git:(master) ./cmd/livefuzzer/livefuzzer spam
No seed provided, creating one
Spamming 3758 transactions per account on 10 accounts with seed: 0x-5f2e05649bf1dd5
panic: invalid chain id for signer

goroutine 83 [running]:
main.SendBaikalTransactions(0x1400010e180, 0x20?, 0x40?, {0x102c24d95, 0x2a}, 0xeae, 0x0?)
	/Users/simon.dudley/IdeaProjects/tx-fuzz/cmd/livefuzzer/main.go:193 +0x438
main.SpamTransactions.func1({0x102c2e13f?, 0x0?}, {0x102c24d95, 0x2a}, 0x0?)
	/Users/simon.dudley/IdeaProjects/tx-fuzz/cmd/livefuzzer/main.go:163 +0xa4
created by main.SpamTransactions
	/Users/simon.dudley/IdeaProjects/tx-fuzz/cmd/livefuzzer/main.go:160 +0x3a0

Restarting the tool allows more transactions to be submitted though so impact seems limited.

---

To recreate (this might not be limited to QBFT or tx-fuzz though):

1. Setup a fresh single-node QBFT network (I used attached genesis-with-transition.json which includes the necessary account for tx-fuzz to work)
2. besu --config-file besu1.conf (config file attached, you need to tweak data path, key path and genesis file path)
3. Run tx-fuzz against it
   a. run go build .
   b. ./cmd/livefuzzer/livefuzzer spam

7287-config.tar.gz

[non-fungible-nelson]

@matthew1001 flagging this for you

[matthew1001]

Could this be similar to the issues fixed in #6143 and #6345?

Those stemmed from issues around chainHead being updated before blockchain storage had been committed. DefaultBlockchain.appendBlockHelper(...) updates the chainHead before committing the stored block to the DB. This means that any non-synchronized method on DefaultBlockchain (of which DefaultBlockchain.getBlockHashByNumber(...) is one) chain head could be new but storage could be yet to update.

If you look at BlockchainQueries.gasPrice(...) it calls blockchain.getChainHeadHeader() which gets the new chain head via an unsynchronized call to DefaultBlockchain.getChainHeadHash(...). It then calls blockchain.getBlockByNumber(...) which ends up at the un-synchronized blockchainStorage.getBlockHash(...). There's a timing window therefore where chain head has been updated but storage hasn't been committed.