flaky test TestMppExecution

[hawkingrei]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

2. What did you expect to see? (Required)

3. What did you see instead (Required)

     testkit.go:174: 
        	Error Trace:	testkit/testkit.go:174
        	            				testkit/testkit.go:151
        	            				executor/test/tiflashtest/tiflash_test.go:277
        	Error:      	Received unexpected error:
        	            	The number of tasks is not right, expect 3 tasks but actually there are 4 tasks
        	            	github.com/pingcap/tidb/executor/internal/mpp.(*localMppCoordinator).Execute
        	            		executor/internal/mpp/local_mpp_coordinator.go:720
        	            	github.com/pingcap/tidb/executor.(*MPPGather).Open
        	            		executor/mpp_gather.go:115
        	            	github.com/pingcap/tidb/executor/internal/exec.(*BaseExecutor).Open
        	            		executor/internal/exec/executor.go:160
        	            	github.com/pingcap/tidb/executor.(*HashAggExec).Open
        	            		executor/aggregate.go:306
        	            	github.com/pingcap/tidb/executor.(*ExecStmt).openExecutor
        	            		executor/adapter.go:1216
        	            	github.com/pingcap/tidb/executor.(*ExecStmt).Exec
        	            		executor/adapter.go:544
        	            	github.com/pingcap/tidb/session.runStmt
        	            		session/session.go:2419
        	            	github.com/pingcap/tidb/session.(*session).ExecuteStmt
        	            		session/session.go:2269
        	            	github.com/pingcap/tidb/testkit.(*TestKit).ExecWithContext
        	            		testkit/testkit.go:342
        	            	github.com/pingcap/tidb/testkit.(*TestKit).MustQueryWithContext
        	            		testkit/testkit.go:173
        	            	github.com/pingcap/tidb/testkit.(*TestKit).MustQuery
        	            		testkit/testkit.go:151
        	            	executor/test/tiflashtest/tiflashtest_test.TestMppExecution
        	            		executor/test/tiflashtest/tiflash_test.go:277
        	            	testing.tRunner
        	            		GOROOT/src/testing/testing.go:1595
        	            	runtime.goexit
        	            		src/runtime/asm_amd64.s:1650
        	Test:       	TestMppExecution
        	Messages:   	sql:select avg(t.a) from t join t t1 on t.a = t1.a, args:[] 

4. What is your TiDB version? (Required)

[gonnafaraway]

@hawkingrei can't reproduce it, did you change anything in test?
locally on my machine this test passes correctly without errors.

[hawkingrei]

@gonnafaraway In fact, TiDB CI has a critical problem. it cannot control the cpu usage when you run the UT. so if you run this test cases with high cpu usage, you will meet the flaky test.

[hawkingrei]

https://tiprow.hawkingrei.com/view/gs/pingcapprow/pr-logs/pull/pingcap_tidb/46478/tiprow_fast_test/1696729879719972864

[gonnafaraway]

i understand problem, seems it's not a bug, you can also execute this test on machine without free RAM and it could fall too :)