The integration test is not stable

[JaySon-Huang]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

2. What did you expect to see? (Required)

3. What did you see instead (Required)

Sometimes we run into error like this, https://ci.pingcap.net/blue/organizations/jenkins/tiflash-ghpr-integration-tests/detail/tiflash-ghpr-integration-tests/14731/pipeline

[2023-10-18T04:08:59.291Z] ++ cat ./log/tiflash/server.log
[2023-10-18T04:08:59.291Z] ++ grep 'Ready for connections'
[2023-10-18T04:08:59.291Z] + [[ -n [2023/10/18 12:08:53.086 +08:00] [INFO] [Server.cpp:1660] ["Ready for connections."] [thread_id=1] ]]
[2023-10-18T04:08:59.291Z] + local failed=false
[2023-10-18T04:08:59.291Z] + break
[2023-10-18T04:08:59.291Z] + '[' false == true ']'
[2023-10-18T04:08:59.291Z] + echo '   available'
[2023-10-18T04:08:59.291Z]    available
[2023-10-18T04:08:59.291Z] + docker-compose -f cluster_disable_new_collation.yaml -f tiflash-dt.yaml exec -T tiflash0 bash -c 'cd /tests ; ./run-test.sh tidb-ci/disable_new_collation_fullstack'
[2023-10-18T04:09:02.555Z] tidb-ci/disable_new_collation_fullstack/alter_default_value.test: Running
[2023-10-18T04:09:02.555Z]   File: tidb-ci/disable_new_collation_fullstack/alter_default_value.test
[2023-10-18T04:09:02.555Z]   Error line: 17
[2023-10-18T04:09:02.555Z]   Error: alter table test.t set tiflash replica 1
[2023-10-18T04:09:02.555Z]   Result:
[2023-10-18T04:09:02.555Z]     ERROR 1105 (HY000) at line 1: the tiflash replica count: 1 should be less than the total tiflash server count: 0
[2023-10-18T04:09:02.555Z]   Expected:
[2023-10-18T04:09:02.555Z]     <nothing>

sometimes, https://ci.pingcap.net/blue/organizations/jenkins/tiflash-ghpr-integration-tests/detail/tiflash-ghpr-integration-tests/14718/pipeline/291

[2023-10-17T10:10:29.342Z] fullstack-test2/ddl/alter_partition_by.test: Running
[2023-10-17T10:10:39.284Z]   File: fullstack-test2/ddl/alter_partition_by.test
[2023-10-17T10:10:39.284Z]   Error line: 127
[2023-10-17T10:10:39.284Z]   Error: select tidb_database,tidb_name from system.tables where tidb_database = 'test' and tidb_name = 't2' and is_tombstone = 0
[2023-10-17T10:10:39.284Z]   Result:
[2023-10-17T10:10:39.284Z]     <nothing>
[2023-10-17T10:10:39.284Z]   Expected:
[2023-10-17T10:10:39.284Z]     ┌─tidb_database─┬─tidb_name─┐
[2023-10-17T10:10:39.284Z]     │ test          │ t2        │
[2023-10-17T10:10:39.284Z]     └───────────────┴───────────┘

4. What is your TiFlash version? (Required)

master

[JaySon-Huang]

Error

[2023-11-08T08:13:23.708Z] fullstack-test2/ddl/alter_partition_by.test: Running
[2023-11-08T08:13:32.460Z]   File: fullstack-test2/ddl/alter_partition_by.test
[2023-11-08T08:13:32.460Z]   Error line: 171
[2023-11-08T08:13:32.460Z]   Error: select tidb_database,tidb_name from system.tables where tidb_database = 'test' and tidb_name = 't2' and is_tombstone = 0
[2023-11-08T08:13:32.460Z]   Result:
[2023-11-08T08:13:32.460Z]     <nothing>
[2023-11-08T08:13:32.460Z]   Expected:
[2023-11-08T08:13:32.460Z]     ┌─tidb_database─┬─tidb_name─┐
[2023-11-08T08:13:32.460Z]     │ test          │ t2        │
[2023-11-08T08:13:32.460Z]     └───────────────┴───────────┘

failure logging: https://ci.pingcap.net/blue/organizations/jenkins/tiflash-ghpr-integration-tests/detail/tiflash-ghpr-integration-tests/15005/artifacts

tiflash/tests/fullstack-test2/ddl/alter_partition_by.test

Lines 147 to 174 in 1bdd6ad

	mysql> select /*+ READ_FROM_STORAGE(TIFLASH[t2]) */ * from test.t2 order by a;
	+---------+---------+----------+
	| a | b | c |
	+---------+---------+----------+
	| 1 | 1 | -1 |
	| 2 | 2 | -2 |
	| 3 | 3 | -3 |
	| 4 | 4 | -4 |
	| 500001 | 500001 | -500001 |
	| 500002 | 500002 | -500002 |
	| 500003 | 500003 | -500003 |
	| 500004 | 500004 | -500004 |
	| 1000001 | 1000001 | -1000001 |
	| 1000002 | 1000002 | -1000002 |
	| 1000003 | 1000003 | -1000003 |
	| 1000004 | 1000004 | -1000004 |
	| 1500001 | 1500001 | -1500001 |
	| 1500002 | 1500002 | -1500002 |
	| 1500003 | 1500003 | -1500003 |
	| 1500004 | 1500004 | -1500004 |
	+---------+---------+----------+
	mysql> drop table test.t;
	>> select tidb_database,tidb_name from system.tables where tidb_database = 'test' and tidb_name = 't2' and is_tombstone = 0
	┌─tidb_database─┬─tidb_name─┐
	│ test │ t2 │
	└───────────────┴───────────┘

From the logging, we can see that the table is created after the select tidb_database,tidb_name from system.tables where tidb_database = 'test' and tidb_name = 't2' and is_tombstone = 0 returned

It should be cause by the hint does not take effects and the query is not handled by TiFlash