Region is unavailable while scale up tiflash

[lilinghai]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

2. What did you expect to see? (Required)

3. What did you see instead (Required)

4. What is your TiFlash version? (Required)

master

[CalvinNeo]

#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007fd767a86c9f in std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&) () from /tiup/deploy/tiflash-9000/bin/tiflash/libc++.so.1
#2  0x00007fd76715d13b in rocksdb::WriteThread::AwaitState(rocksdb::WriteThread::Writer*, unsigned char, rocksdb::WriteThread::AdaptationContext*) ()
   from /tiup/deploy/tiflash-9000/bin/tiflash/[libtiflash_proxy.so](http://libtiflash_proxy.so/)
#3  0x00007fd7671fd2d0 in rocksdb::DBImpl::PebbleWriteImpl(rocksdb::WriteOptions const&, rocksdb::WriteBatch*, rocksdb::WriteCallback*, unsigned long*, unsigned long, unsigned long*) ()
   from /tiup/deploy/tiflash-9000/bin/tiflash/[libtiflash_proxy.so](http://libtiflash_proxy.so/)
#4  0x00007fd7671fbaef in rocksdb::DBImpl::WriteImpl(rocksdb::WriteOptions const&, rocksdb::WriteBatch*, rocksdb::WriteCallback*, unsigned long*, unsigned long, bool, unsigned long*, unsigned long, rocksdb::PreReleaseCallback*) () from /tiup/deploy/tiflash-9000/bin/tiflash/[libtiflash_proxy.so](http://libtiflash_proxy.so/)
#5  0x00007fd7671fb6c8 in rocksdb::DBImpl::Write(rocksdb::WriteOptions const&, rocksdb::WriteBatch*) () from /tiup/deploy/tiflash-9000/bin/tiflash/[libtiflash_proxy.so](http://libtiflash_proxy.so/)
#6  0x00007fd766f78a67 in crocksdb_write () from /tiup/deploy/tiflash-9000/bin/tiflash/[libtiflash_proxy.so](http://libtiflash_proxy.so/)
#7  0x00007fd76501fec2 in _$LT$engine_tiflash..write_batch..RocksWriteBatch$u20$as$u20$engine_traits..write_batch..WriteBatch$GT$::write_opt::h9b624632afa1536d ()
   from /tiup/deploy/tiflash-9000/bin/tiflash/[libtiflash_proxy.so](http://libtiflash_proxy.so/)
#8  0x00007fd765a3be31 in raftstore::store::fsm::apply::ApplyContext$LT$EK$GT$::write_to_db::hc9ec1383c60b7c8c () from /tiup/deploy/tiflash-9000/bin/tiflash/[libtiflash_proxy.so](http://libtiflash_proxy.so/)
#9  0x00007fd765a3b60c in raftstore::store::fsm::apply::ApplyContext$LT$EK$GT$::commit_opt::h28f0d1dcd36c2064 () from /tiup/deploy/tiflash-9000/bin/tiflash/[libtiflash_proxy.so](http://libtiflash_proxy.so/)
#10 0x00007fd765a3b50a in raftstore::store::fsm::apply::ApplyContext$LT$EK$GT$::commit::h26550b755a952c76 () from /tiup/deploy/tiflash-9000/bin/tiflash/[libtiflash_proxy.so](http://libtiflash_proxy.so/)
#11 0x00007fd765a432a7 in raftstore::store::fsm::apply::ApplyDelegate$LT$EK$GT$::process_raft_cmd::hd795c1ccd8b465b7 () from /tiup/deploy/tiflash-9000/bin/tiflash/[libtiflash_proxy.so](http://libtiflash_proxy.so/)
#12 0x00007fd765a41137 in raftstore::store::fsm::apply::ApplyDelegate$LT$EK$GT$::handle_raft_committed_entries::ha5616e5f0541ac90 () from /tiup/deploy/tiflash-9000/bin/tiflash/[libtiflash_proxy.so](http://libtiflash_proxy.so/)
#13 0x00007fd765a5f301 in raftstore::store::fsm::apply::ApplyFsm$LT$EK$GT$::handle_apply::h04935313931cff3f () from /tiup/deploy/tiflash-9000/bin/tiflash/[libtiflash_proxy.so](http://libtiflash_proxy.so/)
#14 0x00007fd765a62031 in raftstore::store::fsm::apply::ApplyFsm$LT$EK$GT$::handle_tasks::hff18fa04fb65880c () from /tiup/deploy/tiflash-9000/bin/tiflash/[libtiflash_proxy.so](http://libtiflash_proxy.so/)
#15 0x00007fd765a649f0 in _$LT$raftstore..store..fsm..apply..ApplyPoller$LT$EK$GT$$u20$as$u20$batch_system..batch..PollHandler$LT$raftstore..store..fsm..apply..ApplyFsm$LT$EK$GT$$C$raftstore..store..fsm..apply..ControlFsm$GT$$GT$::handle_normal::hc32cc0d4ecb3132f () from /tiup/deploy/tiflash-9000/bin/tiflash/[libtiflash_proxy.so](http://libtiflash_proxy.so/)
#16 0x00007fd7655643f2 in batch_system::batch::Poller$LT$N$C$C$C$Handler$GT$::poll::hdb369f813ddd22e5 () from /tiup/deploy/tiflash-9000/bin/tiflash/[libtiflash_proxy.so](http://libtiflash_proxy.so/)
#17 0x00007fd765dce618 in std::sys_common::backtrace::__rust_begin_short_backtrace::h182f537039797994 () from /tiup/deploy/tiflash-9000/bin/tiflash/[libtiflash_proxy.so](http://libtiflash_proxy.so/)
#18 0x00007fd765456071 in core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::h9b437b20434f6586 () from /tiup/deploy/tiflash-9000/bin/tiflash/[libtiflash_proxy.so](http://libtiflash_proxy.so/)
#19 0x00007fd766556aea in std::sys::unix::thread::Thread::new::thread_start::hd39c5f08bdcda277 () from /tiup/deploy/tiflash-9000/bin/tiflash/[libtiflash_proxy.so](http://libtiflash_proxy.so/)
#20 0x00007fd763ef9ea5 in start_thread (arg=0x7fd58fa2a700) at pthread_create.c:307
#21 0x00007fd7632feb0d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

We found "enable-pipelined-write":false on rocksdb. This may cause bug if we have an empty write batch.