-
Notifications
You must be signed in to change notification settings - Fork 44
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support ethereum-spec-evm statetest
#131
Conversation
Oh wow that's awesome! Can you add it to the docker/Dockerfile too? |
I don't know docker that well 😅 Our installation is basically: pip install git+https://github.com/ethereum/execution-specs |
Alright, I'll fix, no worries! |
I'll let you know when |
Oh right! Please also see this readme: https://github.com/holiman/goevmlab/tree/master/evms/testdata If you could provide the eels output for those testcases, that would be the next step. I will also play with this today though |
How do I actually run the thing? I tried on a debian image, using an
Note, there was one error during install:
This is more or less the steps I followed
|
You have to use pip. Calling If you want to split the clone and the install into two steps, it should look something like: git clone https://github.com/ethereum/execution-specs.git --branch statetests --depth 1
pip install ./execution-specs |
If I do that, then I get
|
Generated the testdata and took a stab at the Dockerfile. While we don't entirely barf on the test inputs, I very much cannot promise the outputs are correct 🤣 |
Sweet!
But actually, eels does show the
So, what happens here is that This appears to be the only failure surfaced by the reference outputs, which is very good! Unfortunately, it's not something that can be fixed on the goevmlab-side, at least not without quite a lot of work (namely: go through all other vms and zero out the opcode on errors) [1]: When geth encounters end of code, it executes a STOP. Other clients do not output anything for that. Hence, goevmlab strips all |
I was able to simplify the output-processing a bit, and now all tests are enabled on the "reference outputs". Once the inclusion of the op on error is fixed, the next step will be to actually run the fuzzer |
I'm actually able to run the fuzzer for a while -- here's geth vs eels. After a while though, it hits the error described above
|
Found another flaw. Doing
That seems to be a problem with When in "skip trace" mode, (a.k.a speedMode), we still read from So the |
Using the
It's extremely slow, averaging |
Some general observations:
|
After
After
After
So, I think I know what the problem here is. For averaging, I goevmlab doesn't use a real average, rather a sliding window average.
So it is So that's a red herring that can be ignored (in this context) |
Hm. Do you collect traces interally before emitting them? I'd recommend against doing so. The reason we use |
Awesome! I'm running this on my machine now, I'll let it run geth vs eels for a while then merge. Thanks for this! |
With all engines enabled:
✔️ |
Would be nice to remove be able to remove |
Wow, the
|
New docker image pushed
All 8 clients! |
This pull request adds support for EELS'
ethereum-spec-evm
.Requires an extremely new checkout (at least ethereum/execution-specs@c4f5b8a).