callTracer execution timeout for blocks 0x228088C, 0x226AECC

[qn-srikanth]

Describe the bug
debug_traceBlockByNumber returns execution timeout for blocks :
36178060 = 0x228088C
36089548 = 0x226AECC

To Reproduce
Steps to reproduce the behavior:
Running harmony archive version *v4.3.12*-gf8777e0c

---

curl -s -H "Content-Type: application/json" http://localhost:9501 -d '{"jsonrpc":"2.0","method":"debug_traceBlockByNumber","params":["0x228088C", {"tracer": "callTracer", "timeout": "1h"}],"id":1}'

{"jsonrpc":"2.0","id":1,"result":[{"error":"execution timeout"}]}

---

curl -s -H "Content-Type: application/json" http://localhost:9501 -d '{"jsonrpc":"2.0","method":"debug_traceBlockByNumber","params":["0x226AECC", {"tracer": "callTracer", "timeout": "1h"}],"id":1}'

{"jsonrpc":"2.0","id":1,"result":[{"error":"execution timeout"}]}

---

Expected behavior
Debug trace output

Environment (please complete the following information):

• OS: [Linux]

Additional context

• Adding "timeout": "1h" doesn't help and returns an empty response.

curl -s -H "Content-Type: application/json" http://localhost:9501 -d '{"jsonrpc":"2.0","method":"debug_traceBlockByNumber","params":["0x228088c", {"tracer": "callTracer", "timeout": "1h"}],"id":1}'

curl -s -H "Content-Type: application/json" http://localhost:9501 -d '{"jsonrpc":"2.0","method":"debug_traceBlockByNumber","params":["0x226AECC", {"tracer": "callTracer", "timeout": "1h"}],"id":1}'

• Can reproduce via public RPC as well.

curl -s -H "Content-Type: application/json" https://api.s0.t.hmny.io/ -d '{"jsonrpc":"2.0","method":"debug_traceBlockByNumber","params":["0x228088C", {"tracer": "callTracer"}],"id":1}'

504 Gateway Time-out

• Closely related to #4282
  

[qk-santi]

Is this due to reaching the internal EVM timeout when processing each call?
Would we be able to have a similar flag to Geth's --rpc.evmtimeout available to override that default timeout?

[mchinaloy]

FWIW I've seen similar behaviour on other Geth based clients where we've been able to fix the problem by applying the following fix:

jacquesvcritien/metis-l2geth@c78d647

To essentially set a bunch of timeouts where CLI arguments/ENV vars were not having the desired effect i.e.

cfg.HTTPTimeouts.WriteTimeout
cfg.HTTPTimeouts.ReadTimeout 
cfg.HTTPTimeouts.IdleTimeout

[MaxMustermann2]

I'm strangely unable to (always) reproduce the error on a local node; I get the desired result on one and a blank value on another node with possibly a timeout but no OOMs in either case.

However, the timeout on the public RPC appears to be imposed by Nginx or the load balancer (504 Gateway Timeout) and not the node.

[a26nine]

@MaxMustermann2, would the above PR fix this issue?

I found another one:
curl -H "Content-Type: application/json" -X POST --data '{"jsonrpc":"2.0","method":"debug_traceBlockByHash","params":["0xd0587aff2780b84860c007808edaa3e76a6f12f77515e150fe5997d6eeaf794f", {"tracer": "callTracer", "timeout": "60s"}],"id":1}' localhost:9501

curl: (52) Empty reply from server

[MaxMustermann2]

@a26nine

The issue is two fold. One is the timeout and the other is an OOM. If you get an empty reply, I believe it is the time out causing trouble and not an out of memory error. If you'd like a binary made from this PR to test, let me know and I'd be happy to share. You can also clone the PR and make one yourself.

[a26nine]

@MaxMustermann2 We will wait for the PR to be included in the official release. If we can expedite the process, that would be great!