mdtest quits halfway through

[ilixiaocui]

Describe the bug (描述bug)
sudo mdtest -C -F -L -z 4 -b 10 -I 10000 -d /mnt/fakes3_2/100_million_small_files/mdtest3

client error log

metaserver error log

To Reproduce (复现方法)

Expected behavior (期望行为)

Versions (各种版本)
OS:
Compiler:
branch: master with no diskcache and with fakeS3
commit id:

Additional context/screenshots (更多上下文/截图)

[Wine93]

According to the client logs we found that some inode and dentry which should belong to the same partition but sended request to different partitions:

CreateInode :  inodeId: 686037139456 (copysetId: 161 partitionId: 40931) -> SUCCESS
CreateDentry:  inodeId: 686037139456 parentInodeId: 1142863953920 name: mdtest_tree.138 type: TYPE_DIRECTORY channel: 10.166.16.60:7802 (copysetId: 267 partitionId: 68158) -> SUCCESS
GetDentry   :  parentInodeId: 686037139456 name: "mdtest_tree.1381" (copysetId: 165 partitionId: 40930) -> NOT_FOUND
CreateInode :  inodeId: 340929806336 (copysetId: 84 partitionId: 20358) -> SUCCESS
CreateDentry:  inodeId: 340929806336 parentInodeId: 686037139456 name: mdtest_tree.1381 type: TYPE_DIRECTORY Channel: 10.166.16.61:7801 (copysetId: 165, partitionId: 40930) -> NOT_FOUND

then we list all partitions and found that two different partitions with the same inode range, so we doubt that something went wrong while creating the partition:

$ curvefs_tool list-partition -fsId=1

...
partitionInfoList {
  fsId: 1
  poolId: 1
  copysetId: 165
  partitionId: 40930
  start: 686037139456
  end: 686053916671
  txId: 0
  status: READWRITE
  inodeNum: 0
  dentryNum: 1
}
partitionInfoList {
  fsId: 1
  poolId: 1
  copysetId: 161
  partitionId: 40931
  start: 686037139456
  end: 686053916671
  txId: 0
  status: READWRITE
  inodeNum: 1
  dentryNum: 0
}
...

[SeanHai]

The process of create partition have no lock to limit sequential execution at mds. So when create partition occurs on multiple mount points may create the partitions with same range.

curve/curvefs/src/mds/topology/topology_manager.cpp

Line 628 in d0e9dfb

void TopologyManager::CreatePartitions(const CreatePartitionRequest *request,