Fake PR just for review

[innerr]

Signed-off-by: Liu Cong [email protected]

What problem does this PR solve?

Improve TiKV performance on non-NVMe disk.

Problem Summary:

• The 'sync' performance is varied on different types of disk, it could from several hundred to 50K times per sec, we could use fio -fdatasync or pg_test_sync to profile the 'sync' performance of a disk
• Typically, a non-NVMe disk could perform 2K~5K sync per sec
• When TiKV runs under proper stress (lots of write, disk latency is good), the 'sync' frequency could reach 20K as we discovered on grafana panel

So, the 'sync' performance will be one of the bottlenecks of TiKV performance, especially under some types of disk.
This PR provide a config entry to control the raftstore's 'sync' frequency.

What is changed and how it works?

When peers receive data, the 'write' operation goes as usual, but the 'sync' will be hold for a while.
(not holding 'write' operation is the key, IO will be smoother comparing to simple batching, which will hold both 'write' and 'sync')

Before 'sync' is called, the related raft follower's messages of AppendResp will be blocked(held and cached),
when follower perform 'sync', the raft log will be considered well persisted, then the messages will be released and delivered to leader.

In current implementation of raft-rs, the committed index of a raft group will advance immediately, even though the log is not persisted yet, because currently it assume that the log will(should, must) be persisted before next round of handling.
This assumption is not fit for our purpose, so we changed raft-rs for a bit:

• Added a new method raft_group.on_synced(index) to deliver 'sync' event to raft group, the committed index will not advance until on_synced is called.
• Enhanced raft_group.ready_since(index) to raft_group.ready_from_range(index_low, index_high), to fetch ready only belong to synced logs.
• PR: Fake PR for code review raft-rs#1
• Detail proposal: Proposal: support fully control fsync frequency in raft etcd-io/etcd#12257

Related changes

Tests

• Integration test
• Manual test (add detailed scripts or steps below)

Side effects

• These changes could delay the committed logs' applying, and cause greater latency, so we provide a config entry to enable/disable it when running TiKV on different types of disk.

Release note

• Control sync frequency using config entry: raftstore.delay-sync-ns, performance on some types of disk could be better with this.