go/storage/mkvs/urkel: Figure out why the remove tests are so slow

[Yawning]

The CI time consumed by the LevelDB/BadgerDB urkel removal tests is gigantic.

There is a stopgap fix for this in #2018 that reduces the number of k/v pairs used from 1000 to 500 that shaves 5 mins off the typical CI time, but the root cause should be determined, and this should be solved properly if possible.

[kostko]

After a very quick glance the biggest increase comes from write log merging, #1975. Will take a look when I come back.

The other time comes from fsync after every commit which should definitely be disabled for tests.

[kostko]

It seems most of the time is spent fetchng, decoding, re-encoding and storing the write logs during merges (this was introduced in #1975).

While in reality the merge log chains will be much smaller than in this test case (e.g., max 2 steps instead of 500 or 1000 like in this test) and this should be enforced to prevent DoS (see #1914, "limit the depth of forked roots"), there are still things that need to actually be fixed to ensure correctness (while looking into this I figured the current implementation can actually be exploited to make the node lose finalized write logs).

[kostko]

After the fixes in #2051 the complete LevelDB test suite takes ~1 sec and Badger around ~2 sec. The difference with CI (and make test-unit) is the race detector which adds a couple of seconds more.