ethash: implement Progpow

[andresilva]

This is a proof-of-concept implementation of progpow, this is still work in progress and I don't intend this to be merged yet.

• Progpow light verification
• Pass existing test vectors (https://github.com/chfast/ethash/blob/master/test/unittests/progpow_test_vectors.hpp)
• Benchmark and optimize
• Update to latest spec
• Integrate with ethash engine

References:

• EIP 1057 (https://eips.ethereum.org/EIPS/eip-1057)
• reference implementation (https://github.com/ifdefelse/ProgPOW)
• go-ethereum implementation (Progpow integration ethereum/go-ethereum#17731)
• aleth's ethash implementation (https://github.com/chfast/ethash)

[andresilva]

Benchmarks of baseline performance:

test benchmarks::bench_hashimoto_light       ... bench:     981,214 ns/iter (+/- 32,031)
test benchmarks::bench_progpow_light         ... bench:  16,766,912 ns/iter (+/- 1,429,102)
test benchmarks::bench_progpow_optimal_light ... bench:   5,055,690 ns/iter (+/- 2,379,231)

bench_progpow_optimal_light - doesn't take into account cdag creation, this is realistic since the cdag can be cached during it's epoch (50 blocks).

Will have a look at what can be optimized now that it's passing all test vectors.

[holiman]

doesn't take into account cdag creation, this is realistic since the cdag can be cached during it's epoch (50 blocks).

Isn't the cDag epoch the same as dag epoch, 30k blocks?

[andresilva]

@holiman Yes, you're right, the only thing that changes every 50 blocks is the program generated from the cDag, but the cDag itself has the same epoch as the ethash dag.

[andresilva]

I think I've addressed all grumbles. The only unsafe code that is kept is for transmuting from [u32; 8] to [u8; 32], I think doing the byte shuffling is unnecessary but can remove it if you think otherwise. There is probably a lot of low hanging fruit for optimization here. Completely unrolling the hash function kecak_f800 like we do for keccak_f1600 would probably speed it up a bit.

[andresilva]

There's a test with a bunch of test vectors (shared among implementations) that might take a little while to run. We might want to disable this test on the CI.

[cheme]

This looks good to me, I agree some tests takes to long for CI (even for my pc) and should be disabled somehow before merging.

[niklasad1]

Does it make sense to Box CDag in the Algorithm enum?
It is quite big (16384 bytes), thus it will make the enum Algorithm as big.

EDIT:
Have seen any regression on Hashimoto after this change?

[andresilva]

Good idea. No I didn't see any regression in the benchmarks with all the changes I've done so far.

[andresilva]

Boxed the cdag inside algorithm and adapted an existing cargo feature to skip the slow progpow test vectors in CI.

[andresilva]

Actually since CI is running the tests with --release the test vectors don't take that long to run (20s on my laptop vs like 4 minutes for a debug build), so I'm going to keep them.

[niklasad1]

I'm in favor of merging this, good work Andre 👍

optimization(s) can be fixed in follow-up PRs

[andresilva]

I removed the verification that triggered the compile error, it wasn't working in the first place the reason it wasn't being triggered is because the feature flags were using different names. As far as I understand since json_tests is a part of ethcore it will be compiled in non-test builds (e.g. evmbin directly uses json_tests functionality).

[5chdn]

🎉