membuffer: fix memory leak in red-black tree

[you06]

close #1375, ref pingcap/tidb#56837

The issue #1375 figures out that the RBT membuffer cannot reuse free nodes which causes a memory leak. This pr uses marking the node as deleted to replace the actual deletion.

Pros:

• Because the memory arena cannot handle the memory fragmentation well, we avoid the memory leak by this mark-delete strategy.

Cons:

• In certain cases, the mark-delete tree may be taller than the actual deletion, which can slightly impact the performance.

[you06]

@cfzjywxk I test this PR (with RBT enabled) is ok to pass TiDB's CI, PTAL.

[cfzjywxk]

@you06
Do we have a benchmark test with tidb to verify the memory leak issue is resolved?

[crazycs520]

REST LGTM

[you06]

Do we have a benchmark test with tidb to verify the memory leak issue is resolved?

This test can reproduce the memory leak issue in TiDB side, will add it to TiDB.

func TestInsertRollbackMemoryLeak(t *testing.T) {
	store := testkit.CreateMockStore(t)
	tk := testkit.NewTestKit(t, store)
	tk.MustExec("use test")
	tk.MustExec("create table t(a varchar(255) primary key)")
	key1 := strings.Repeat("a", 255)
	key2 := strings.Repeat("b", 255)
	tk.MustExec(`set global tidb_mem_oom_action='cancel'`)
	tk.MustExec("set session tidb_mem_quota_query=10240")
	tk.MustExec("begin")
	tk.MustExec("insert into t values(?)", key2)
	for i := 0; i < 100; i++ {
		err := tk.ExecToErr("insert into t values(?), (?)", key1, key2)
		require.Error(t, err)
		if strings.Contains(err.Error(), "Duplicate") {
			continue
		}
		require.NoError(t, err)
	}
	tk.MustExec("commit")
}

[cfzjywxk]

LGTM

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: cfzjywxk, zyguan

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [cfzjywxk,zyguan]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ti-chi-bot]

[LGTM Timeline notifier]

Timeline:

• 2024-11-13 06:07:49.885070903 +0000 UTC m=+422832.075939892: ☑️ agreed by zyguan.
• 2024-11-14 14:37:56.801349824 +0000 UTC m=+539838.992218823: ☑️ agreed by cfzjywxk.