Perfomance Degradation: Heavy Disk Usage

[hleb-albau]

Starting new chain, generating about 500 tx each 40kb with overall 100k messages. After some point performance dropped to 1 tx/s. We observing heavy usage of disk - writes from 5 MiB/s to 30-40 MiB/s for periods up to 20 seconds. LevedDb size is about few mb. So, general chain workflow become :

for 30sec block generation:
1) propose,commit block few secs
2) up to 20sec disk heavy usage
3) only few seconds tendermint accepts new mempool txes
4) back to point 1

Starting new chain, gives next disk usage:

Here max value is 40MiB/sec. As you can see, most of time, chain spend on db staff.

Same disk usage stays even if we produce empty blocks for a long period of time. Stop sending txes, just create empty blocks.

ref #2131
ref #1835

[ebuchman]

As far as we know, this is due to a problem in the underlying goleveldb library. You should be able to solve it by compiling Tendermint to use the real C version of LevelDB. See https://tendermint.com/docs/introduction/install.html#compile-with-cleveldb-support

[hleb-albau]

@ebuchman, thanks for answer.
In our case, compiling with cleveldb and adjusting multistore usage fixed problem.
For each 30sec-block we previously rewrite about 5mb, that caused a lot of compactions.
Sorry for interruption. Thanks for tendermint and cosmos.

[ebuchman]

Please, no need to apologize. We appreciate the initial report and confirmation of the fix. Let us know if you encounter any other issues! Thanks.

[gcohler]

@ebuchman When I follow the instructions for installing cleveldb, I get a panic that it is not supported. Has this support been eliminated?

panic: Unknown db_backend cleveldb, expected either fsdb or leveldb or goleveldb or memdb

Thanks much.

[ebuchman]

No it has not. How are you installing it? make install_c should do it. Do you have gcc installed?