perf: explore use of direct IO for the RocksDB WAL

[petermattis]

RocksDB 5.1.2 added support for using direct IO (O_DIRECT) for writes. This might be a faster option than using fsync/fdatasync after writing the WAL.

[tbg]

cc @irfansharif - not sure, but maybe this option is something that would apply to the Raft log RocksDB instance?

[irfansharif]

it does, have this in my notes somewhere. on my radar for when tuning RocksDB for raft log workload.

[petermattis]

In more recent RocksDB releases, the option became use_direct_io_for_flush_and_compaction and only applies to reads and writes performed by background flushes and compactions. So direct IO could not be used as a replacement for syncing the WAL.

Enabling use_direct_io_for_flush_and_compaction on a write-only workload showed a 5-10% performance decrease. Disk utilization jump from ~50% to ~75% and disk read bandwidth jumped from 0 to 60-80 MB/sec. Avoiding the OS buffer cache doesn't seem to be a good idea at this time.

[leventov]

https://github.com/facebook/rocksdb/wiki/Direct-IO says:

use_direct_io_for_flush_and_compaction and use_direct_reads will only be applied to SST file I/O but not WAL I/O or MANIFEST I/O because the I/O pattern of these files are not suitable for direct I/O

I wonder why WAL I/O pattern is considered not suitable for direct I/O?

---

Update: Sorry, didn't realise this issue is not in the facebook/rocksdb repo. Asked the same question here: https://groups.google.com/d/msg/rocksdb/hYT9_xSZmyw/maYMZj8CBQAJ