Skip to content
This repository has been archived by the owner on Jun 23, 2022. It is now read-only.

perf: optimizing write latency using independent IO queues replace of libaio #633

Merged
merged 74 commits into from
Oct 19, 2020
Merged
Changes from 1 commit
Commits
Show all changes
74 commits
Select commit Hold shift + click to select a range
02c608d
refactor: implement thread-pool-based asynchrounous IO
neverchanje Jul 17, 2020
24cedc7
merge update
foreverneverer Sep 8, 2020
4b31613
add tracer
foreverneverer Sep 9, 2020
18ad5e9
update tracer
foreverneverer Sep 9, 2020
caa7535
update tracer
foreverneverer Sep 9, 2020
bb14fe4
update tracer
foreverneverer Sep 9, 2020
b9776e5
update tracer
foreverneverer Sep 9, 2020
147b595
update tracer
foreverneverer Sep 9, 2020
fd7cf38
update tracer
foreverneverer Sep 10, 2020
298942b
merge
foreverneverer Sep 10, 2020
1bc4dd7
add comment
foreverneverer Sep 10, 2020
31c30f9
add comment
foreverneverer Sep 10, 2020
9db31ba
Merge branch 'fix-tracer' into newaio-with-tracer
foreverneverer Sep 10, 2020
32fcef1
add comment
foreverneverer Sep 10, 2020
c1f33f1
add comment
foreverneverer Sep 11, 2020
5656d64
add comment
foreverneverer Sep 11, 2020
fb55339
add comment
foreverneverer Sep 11, 2020
676bd85
add comment
foreverneverer Sep 11, 2020
aabab6a
add comment
foreverneverer Sep 11, 2020
6f6d221
add comment
foreverneverer Sep 11, 2020
7d32244
add comment
foreverneverer Sep 11, 2020
9e7d3fc
add perfcounter
foreverneverer Sep 11, 2020
1718381
add perfcounter
foreverneverer Sep 11, 2020
eec5dcc
add perfcounter
foreverneverer Sep 11, 2020
6297287
add max
foreverneverer Sep 12, 2020
e384c78
update
foreverneverer Sep 12, 2020
c1fe66c
add debug info
foreverneverer Sep 14, 2020
b804b72
add debug info
foreverneverer Sep 14, 2020
c88a1be
add debug info
foreverneverer Sep 14, 2020
026301f
add debug info
foreverneverer Sep 14, 2020
db66a2c
add debug info
foreverneverer Sep 14, 2020
eb30ebf
split all aio task
foreverneverer Sep 14, 2020
03934e6
dequeue one
foreverneverer Sep 16, 2020
ee78661
init
foreverneverer Sep 23, 2020
72dcbe6
init
foreverneverer Sep 23, 2020
b388192
init
foreverneverer Sep 23, 2020
72ae740
init
foreverneverer Sep 23, 2020
a417ff4
init
foreverneverer Sep 23, 2020
542238e
init
foreverneverer Sep 23, 2020
5abfec5
init
foreverneverer Sep 23, 2020
6e37a6d
init but not compile
foreverneverer Sep 23, 2020
1b2568d
ci ok
foreverneverer Sep 23, 2020
8b937dc
ci ok
foreverneverer Sep 23, 2020
24a960a
add_tracer
foreverneverer Sep 24, 2020
5b07031
refactor io queue
foreverneverer Sep 27, 2020
30e5460
refactor io queue
foreverneverer Sep 27, 2020
b5bd3f8
refactor io queue
foreverneverer Sep 27, 2020
a0cbabe
refactor io queue
foreverneverer Sep 27, 2020
65b68c1
refactor io queue
foreverneverer Sep 27, 2020
572d7da
merge master
foreverneverer Sep 27, 2020
1adcb92
delete tracer
foreverneverer Sep 27, 2020
27b58a1
delete tracer
foreverneverer Sep 27, 2020
cc9456b
delete tracer
foreverneverer Sep 27, 2020
da40281
delete tracer
foreverneverer Sep 27, 2020
b0a1f3e
Merge branch 'master' into newaio-with-tracer
foreverneverer Sep 28, 2020
9800ce8
Merge branch 'newaio-with-tracer' of github.com:Shuo-Jia/rdsn into ne…
foreverneverer Sep 28, 2020
dde5c0a
delete tracer
foreverneverer Sep 28, 2020
c936764
fix compile error
foreverneverer Sep 28, 2020
0e7aabc
fix compile error
foreverneverer Sep 28, 2020
1805766
fix compile error
foreverneverer Sep 29, 2020
1c5570a
fix call back exec in replication pool
foreverneverer Oct 10, 2020
2d8e420
move plog pool into rep_long
foreverneverer Oct 10, 2020
0d818d6
merge task queue
foreverneverer Oct 12, 2020
366f18f
merge task queue
foreverneverer Oct 12, 2020
f6d3fcf
merge task queue
foreverneverer Oct 12, 2020
5ec2054
merge task queue
foreverneverer Oct 12, 2020
acb4363
merge task queue
foreverneverer Oct 12, 2020
a677af4
merge task queue
foreverneverer Oct 12, 2020
4caad55
Merge branch 'master' into newaio-with-tracer
foreverneverer Oct 12, 2020
e7977ac
Merge branch 'master' into newaio-with-tracer
hycdong Oct 13, 2020
e46d657
Merge branch 'master' into newaio-with-tracer
levy5307 Oct 15, 2020
d0bc147
review
foreverneverer Oct 19, 2020
1e18d05
Merge branch 'newaio-with-tracer' of github.com:Shuo-Jia/rdsn into ne…
foreverneverer Oct 19, 2020
5cf7373
review
foreverneverer Oct 19, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
add comment
foreverneverer committed Sep 10, 2020
commit 32fcef18d9f2d6b7f1c6a9d41e1279c6e6134a2d
5 changes: 5 additions & 0 deletions src/replica/mutation_log.cpp
Original file line number Diff line number Diff line change
@@ -57,6 +57,7 @@ ::dsn::task_ptr mutation_log_shared::append(mutation_ptr &mu,
// init pending buffer
if (nullptr == _pending_write) {
_pending_write = std::make_shared<log_appender>(mark_new_offset(0, true).second);
ADD_CUSTOM_POINT(mu->tracer, "create_new_log");
}
_pending_write->append_mutation(mu, cb);

@@ -65,6 +66,7 @@ ::dsn::task_ptr mutation_log_shared::append(mutation_ptr &mu,

// start to write if possible
if (!_is_writing.load(std::memory_order_acquire)) {
ADD_CUSTOM_POINT(mu->tracer, "is_write_false");
write_pending_mutations(true);
if (pending_size) {
*pending_size = 0;
@@ -168,6 +170,9 @@ void mutation_log_shared::commit_pending_mutations(log_file_ptr &lf,
// because the following callbacks may run before "block" released, which may cause
// the next init_prepare() not starting the write.
_is_writing.store(false, std::memory_order_relaxed);
for (auto &mu : pending->mutations()) {
ADD_CUSTOM_POINT(mu->tracer, "is_writing_false");
}

// notify the callbacks
// ATTENTION: callback may be called before this code block executed done.