feat: support to force send non-idempotent write when doing duplication #1908

ninsmiracle · 2024-02-18T12:20:11Z

What problem does this PR solve?

#1907

What is changed and how does it work?

Change the judgment logic in the writing process, and configure it to allow non-idempotent operations in the master cluster.
Modify the logic of the specified mutation committer in prepare_list
Add configuration items and add them to the replica side to modify them through the server-config of admin-cli.
Add the parsing logic after the backup cluster receives the packaged RPC of duplication.
Add the logic for the master cluster to obtain relevant non-idempotent RPC from the plog
Add the metric index of dup non-idempotent write QPS

Tests

Unit test
Manual test (add detailed scripts or steps below)

when user want to force dup non-idempotent mutations. Type following command in admin-cli:

server-config replica force_send_no_idempotent_when_duplication set true

ninsmiracle · 2024-02-21T03:02:13Z

This PR's base function is in line with expectations , however I was requested by business department to add more matrix for surveillance system about this feature. I will update this PR in recently days.

ninsmiracle · 2024-02-27T02:58:33Z

Now , when user config force_send_no_idempotent_when_duplication to true . We can dup non-idempotent write to backup cluster , and we can verify the data consistency by check the number rows of pegasus app.
app in master-cluster

app in backup-cluster

And because the retries for non-idempotent writes will make influence of the consistency between two clusters. So we should show the metric like following picture:

show in backup cluster:

src/common/duplication_common.cpp

acelyc111 · 2024-03-01T06:44:49Z

src/common/duplication_common.h

@@ -31,6 +31,7 @@
 #include "utils/fmt_utils.h"

 DSN_DECLARE_uint32(duplicate_log_batch_bytes);
+DSN_DECLARE_bool(force_send_no_idempotent_when_duplication);


Is it necessary to add it here?

I think it's necessary . Cause once I remove this line , compilation can not be passed.

The DSN_DECLARE_xxx macros are recommended to move to cpp files where actually use it, don't make the header files to be too large.

The same to DSN_DECLARE_uint32(duplicate_log_batch_bytes);

src/server/pegasus_write_service.cpp

src/server/pegasus_mutation_duplicator.cpp

acelyc111 · 2024-03-01T07:05:34Z

src/server/pegasus_write_service.cpp

+
+            if (request.task_code == dsn::apps::RPC_RRDB_RRDB_INCR) {
+                incr_rpc rpc(write);
+                resp.__set_error(_impl->incr(ctx.decree, rpc.request(), rpc.response()));


Add some comments to describe what's you aim here.

Co-authored-by: Yingchun Lai <[email protected]>

src/common/duplication_common.cpp

src/server/pegasus_mutation_duplicator.cpp

Co-authored-by: Yingchun Lai <[email protected]>

…cubator-pegasus into support_no_idempotent_dup

src/common/duplication_common.cpp

acelyc111 · 2024-03-14T02:45:42Z

src/common/duplication_common.h

@@ -31,6 +31,7 @@
 #include "utils/fmt_utils.h"

 DSN_DECLARE_uint32(duplicate_log_batch_bytes);
+DSN_DECLARE_bool(duplication_unsafe_allow_non_idempotent);


Move these declarations to the places where actually use them, don't put them in header files.

acelyc111 · 2024-03-14T02:51:45Z

src/server/pegasus_mutation_duplicator.h

@@ -89,8 +94,14 @@ class pegasus_mutation_duplicator : public dsn::replication::mutation_duplicator

    size_t _total_shipped_size{0};

+    const std::set<dsn::task_code> _non_idempotent_code = {


Suggested change

const std::set<dsn::task_code> _non_idempotent_code = {

const std::set<dsn::task_code> _non_idempotent_codes = {

acelyc111 · 2024-03-14T02:52:34Z

src/server/pegasus_mutation_duplicator.h

@@ -89,8 +94,14 @@ class pegasus_mutation_duplicator : public dsn::replication::mutation_duplicator

    size_t _total_shipped_size{0};

+    const std::set<dsn::task_code> _non_idempotent_code = {


Try to reuse it in some other places, e.g. src/server/pegasus_write_service.cpp

acelyc111 · 2024-03-14T02:54:54Z

src/server/pegasus_write_service.cpp

+            // receive non-idempotent request from master cluster via duplication when
+            // FLAG_duplication_unsafe_allow_non_idempotent set as true.
+            // This metric greater than zero means that there is already the possibility of
+            // inconsistency between clusters


Move these comments to the dup_unsafe_received_non_idempotent_duplicate_request metric description, then users know what does it mean from http API, rather than finding out the comments in code.

src/server/pegasus_mutation_duplicator.cpp

acelyc111 · 2024-03-14T03:01:57Z

src/server/pegasus_mutation_duplicator.cpp

@@ -189,6 +211,9 @@ void pegasus_mutation_duplicator::on_duplicate_reply(uint64_t hash,
            // retry this rpc
            _inflights[hash].push_front(rpc);
            _env.schedule([hash, cb, this]() { send(hash, cb); }, 1_s);


Will the non-idempotent request be retried twice here?

Yes,it will. So I record specific raw_key into log.

Co-authored-by: Yingchun Lai <[email protected]>

… to define

Co-authored-by: Yingchun Lai <[email protected]>

ninsmiracle added 2 commits February 18, 2024 19:27

support_no_idempotent_dup

e60dc0a

format and IWYU

31a4271

github-actions bot added the cpp label Feb 18, 2024

ninsmiracle changed the title ~~Support no idempotent dupFeature: support force no idempotent wirte when doing duplication~~ Feature: support force no idempotent wirte when doing duplication Feb 18, 2024

ninsmiracle changed the title ~~Feature: support force no idempotent wirte when doing duplication~~ Feature: support force send non-idempotent wirte when doing duplication Feb 18, 2024

empiredan changed the title ~~Feature: support force send non-idempotent wirte when doing duplication~~ feat: support to force send non-idempotent write when doing duplication Feb 20, 2024

add two matric and type key if dup non-idempotent write retried

61b446f

ninsmiracle force-pushed the support_no_idempotent_dup branch from 31a4271 to 61b446f Compare February 26, 2024 12:38

ninsmiracle added 5 commits February 26, 2024 20:45

deal with conflict

0d1152d

pull

0db1a56

small fix

cef197a

PASS IWYU

a4f510a

format code

538090b

acelyc111 reviewed Mar 1, 2024

View reviewed changes

ninsmiracle and others added 6 commits March 4, 2024 14:07

Update src/common/duplication_common.cpp

c5e39ca

Co-authored-by: Yingchun Lai <[email protected]>

Merge branch 'apache:master' into support_no_idempotent_dup

11c013f

fix by github comment

b521752

format code

0e19a01

pass IWYU

e827c78

pass IWYU2

bb56598

acelyc111 reviewed Mar 13, 2024

View reviewed changes

src/common/duplication_common.cpp Outdated Show resolved Hide resolved

src/server/pegasus_mutation_duplicator.cpp Outdated Show resolved Hide resolved

ninsmiracle and others added 6 commits March 13, 2024 19:47

Update src/server/pegasus_mutation_duplicator.cpp

3d0694f

Co-authored-by: Yingchun Lai <[email protected]>

fix by comment and format

7833e8c

Merge branch 'support_no_idempotent_dup' of github.com:ninsmiracle/in…

95953ac

…cubator-pegasus into support_no_idempotent_dup

make comment in code more clearly

d8e26b1

make function name better

3831f7a

fix type_force_send_non_idempotent_if_need

01fa7af

acelyc111 reviewed Mar 14, 2024

View reviewed changes

ninsmiracle and others added 8 commits March 14, 2024 11:09

Update src/common/duplication_common.cpp

c400ef7

Co-authored-by: Yingchun Lai <[email protected]>

move DSN_DECLARE in duplication common

8dc378f

move dup_unsafe_received_non_idempotent_duplicate_request discription…

fcaabfe

… to define

pass IWYU

cc853e7

Update src/server/pegasus_mutation_duplicator.cpp

5d35724

Co-authored-by: Yingchun Lai <[email protected]>

Update src/server/pegasus_mutation_duplicator.cpp

76faaa7

Co-authored-by: Yingchun Lai <[email protected]>

Update src/server/pegasus_mutation_duplicator.cpp

2f90f2e

Co-authored-by: Yingchun Lai <[email protected]>

Merge branch 'master' into support_no_idempotent_dup

884aabe

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: support to force send non-idempotent write when doing duplication #1908

feat: support to force send non-idempotent write when doing duplication #1908

ninsmiracle commented Feb 18, 2024

ninsmiracle commented Feb 21, 2024

ninsmiracle commented Feb 27, 2024

acelyc111 Mar 1, 2024

ninsmiracle Mar 4, 2024

acelyc111 Mar 13, 2024

acelyc111 Mar 1, 2024

acelyc111 Mar 14, 2024

acelyc111 Mar 14, 2024

acelyc111 Mar 14, 2024

acelyc111 Mar 14, 2024

acelyc111 Mar 14, 2024

ninsmiracle Mar 14, 2024

		@@ -89,8 +94,14 @@ class pegasus_mutation_duplicator : public dsn::replication::mutation_duplicator

		size_t _total_shipped_size{0};

		const std::set<dsn::task_code> _non_idempotent_code = {

feat: support to force send non-idempotent write when doing duplication #1908

Are you sure you want to change the base?

feat: support to force send non-idempotent write when doing duplication #1908

Conversation

ninsmiracle commented Feb 18, 2024

What problem does this PR solve?

What is changed and how does it work?

Tests

ninsmiracle commented Feb 21, 2024

ninsmiracle commented Feb 27, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment