Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix dead lock when stop storage while balancing #4016

Merged
merged 5 commits into from
Mar 17, 2022
Merged

fix dead lock when stop storage while balancing #4016

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
b6bb08f
fix dead lock when stop storage while balancing
liwenhui-soul Feb 16, 2022
7f58f28
use SemiFuture in sendSnapshot to avoid deadlock
liwenhui-soul Feb 17, 2022
e639ad8
Merge branch 'master' into deadlock
critical27 Mar 15, 2022
9af3f51
Merge branch 'master' into deadlock
critical27 Mar 16, 2022
8365c1d
Merge branch 'master' into deadlock
liwenhui-soul Mar 16, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 7 additions & 5 deletions src/kvstore/raftex/RaftexService.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -126,14 +126,16 @@ void RaftexService::stop() {

// stop service
LOG(INFO) << "Stopping the raftex service on port " << serverPort_;
std::unordered_map<std::pair<GraphSpaceID, PartitionID>, std::shared_ptr<RaftPart>> parts;
{
folly::RWSpinLock::WriteHolder wh(partsLock_);
for (auto& p : parts_) {
p.second->stop();
}
parts_.clear();
LOG(INFO) << "All partitions have stopped";
// partsLock_ should not be hold when waiting for parts stop, so swap them out first
parts.swap(parts_);
}
for (auto& p : parts) {
p.second->stop();
}
LOG(INFO) << "All partitions have stopped";
server_->stop();
}

Expand Down
5 changes: 4 additions & 1 deletion src/kvstore/raftex/SnapshotManager.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,10 @@ SnapshotManager::SnapshotManager() {
folly::Future<StatusOr<std::pair<LogID, TermID>>> SnapshotManager::sendSnapshot(
std::shared_ptr<RaftPart> part, const HostAddr& dst) {
folly::Promise<StatusOr<std::pair<LogID, TermID>>> p;
auto fut = p.getFuture();
// if use getFuture(), the future's executor is InlineExecutor, and if the promise setValue first,
// the future's callback will be called directly in thenValue in the same thread, the Host::lock_
// would be locked twice in one thread, this will cause deadlock
auto fut = p.getSemiFuture().via(executor_.get());
executor_->add([this, p = std::move(p), part, dst]() mutable {
auto spaceId = part->spaceId_;
auto partId = part->partId_;
Expand Down