Data inconsistency when tpcc workload and consistency check

[lilinghai]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

tpcc workload and consistency check. the correct result is 30000000

mysql> begin;
Query OK, 0 rows affected (0.05 sec)

mysql> set tidb_isolation_read_engines='tiflash';
Query OK, 0 rows affected (0.05 sec)

mysql> select count(*) from customer;
+----------+
| count(*) |
+----------+
| 29999991 |
+----------+
1 row in set (0.11 sec)

mysql> select count(*) from customer;
+----------+
| count(*) |
+----------+
| 29999989 |
+----------+
1 row in set (0.10 sec)

mysql> explain analyze select count(*) from customer;
+------------------------------+-------------+----------+--------------+----------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------------------------------+---------+------+
| id                           | estRows     | actRows  | task         | access object  | execution info                                                                                                                                                                                                                                                                                                                                                                                        | operator info                              | memory  | disk |
+------------------------------+-------------+----------+--------------+----------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------------------------------+---------+------+
| HashAgg_27                   | 1.00        | 1        | root         |                | time:574.1ms, loops:2, partial_worker:{wall_time:573.997579ms, concurrency:5, task_num:1, tot_wait:2.869544929s, tot_exec:18.431µs, tot_time:2.869595816s, max:573.970407ms, p95:573.970407ms}, final_worker:{wall_time:574.030217ms, concurrency:5, task_num:1, tot_wait:2.869907181s, tot_exec:36.645µs, tot_time:2.86994861s, max:574.003103ms, p95:574.003103ms}                                  | funcs:count(Column#39)->Column#23          | 9.86 KB | N/A  |
| └─TableReader_29             | 1.00        | 3        | root         |                | time:573.9ms, loops:2, cop_task: {num: 6, max: 0s, min: 0s, avg: 0s, p95: 0s, copr_cache_hit_ratio: 0.00}                                                                                                                                                                                                                                                                                             | data:ExchangeSender_28                     | N/A     | N/A  |
|   └─ExchangeSender_28        | 1.00        | 3        | mpp[tiflash] |                | tiflash_task:{proc max:572.5ms, min:40.5ms, avg: 230.7ms, p80:572.5ms, p95:572.5ms, iters:3, tasks:3, threads:24}                                                                                                                                                                                                                                                                                     | ExchangeType: PassThrough                  | N/A     | N/A  |
|     └─HashAgg_9              | 1.00        | 3        | mpp[tiflash] |                | tiflash_task:{proc max:572.5ms, min:40.5ms, avg: 230.7ms, p80:572.5ms, p95:572.5ms, iters:3, tasks:3, threads:24}                                                                                                                                                                                                                                                                                     | funcs:count(tpcc.customer.c_id)->Column#39 | N/A     | N/A  |
|       └─TableFullScan_26     | 30000000.00 | 29999992 | mpp[tiflash] | table:customer | tiflash_task:{proc max:572.5ms, min:40.5ms, avg: 230ms, p80:572.5ms, p95:572.5ms, iters:477, tasks:3, threads:25}, tiflash_scan:{dtfile:{total_scanned_packs:3547, total_skipped_packs:413, total_scanned_rows:28737689, total_skipped_rows:3324291, total_rs_index_load_time: 0ms, total_read_time: 240ms}, total_create_snapshot_time: 0ms, total_local_region_num: 12, total_remote_region_num: 1} | keep order:false                           | N/A     | N/A  |
+------------------------------+-------------+----------+--------------+----------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------------------------------+---------+------+
5 rows in set (0.69 sec)

mysql> commit;
Query OK, 0 rows affected (0.04 sec)

2. What did you expect to see? (Required)

3. What did you see instead (Required)

4. What is your TiFlash version? (Required)

master

[lilinghai]

mysql> explain analyze  select /*+ USE_INDEX(customer) */ count(*) from customer;
+----------------------------+-------------+----------+-----------+----------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------------------+-----------+------+
| id                         | estRows     | actRows  | task      | access object  | execution info                                                                                                                                                                                                                                                                                                                                                                                 | operator info                     | memory    | disk |
+----------------------------+-------------+----------+-----------+----------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------------------+-----------+------+
| HashAgg_11                 | 1.00        | 1        | root      |                | time:1.45s, loops:2, partial_worker:{wall_time:1.448681141s, concurrency:5, task_num:1, tot_wait:7.242872371s, tot_exec:39.812µs, tot_time:7.242923666s, max:1.448589242s, p95:1.448589242s}, final_worker:{wall_time:0s, concurrency:5, task_num:1, tot_wait:7.242998351s, tot_exec:33.873µs, tot_time:7.243036622s, max:1.448616962s, p95:1.448616962s}                                      | funcs:count(Column#24)->Column#23 | 17.1 KB   | N/A  |
| └─TableReader_12           | 1.00        | 210      | root      |                | time:1.45s, loops:2, cop_task: {num: 210, max: 1.23s, min: 407.4µs, avg: 100.2ms, p95: 166.2ms, max_proc_keys: 1885717, p95_proc_keys: 145074, tot_proc: 20.5s, tot_wait: 313ms, rpc_num: 213, rpc_time: 21s, copr_cache_hit_ratio: 0.10, distsql_concurrency: 15}, backoff{regionMiss: 2ms}                                                                                                   | data:HashAgg_5                    | 396 Bytes | N/A  |
|   └─HashAgg_5              | 1.00        | 210      | cop[tikv] |                | tikv_task:{proc max:1.23s, min:37ms, avg: 112.9ms, p80:131ms, p95:194ms, iters:29416, tasks:210}, scan_detail: {total_process_keys: 26444185, total_process_keys_size: 713992995, total_keys: 26452080, get_snapshot_time: 40.8ms, rocksdb: {delete_skipped_count: 769, key_skipped_count: 26452647, block: {cache_hit_count: 15757, read_count: 616, read_byte: 2.26 MB, read_time: 4.14ms}}} | funcs:count(1)->Column#24         | N/A       | N/A  |
|     └─TableFullScan_10     | 30000000.00 | 30000000 | cop[tikv] | table:customer | tikv_task:{proc max:1.23s, min:36ms, avg: 112.8ms, p80:131ms, p95:194ms, iters:29416, tasks:210}                                                                                                                                                                                                                                                                                               | keep order:false                  | N/A       | N/A  |
+----------------------------+-------------+----------+-----------+----------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------------------+-----------+------+
4 rows in set (1.50 sec)

[hehechen]

No problem in 0b1ffce

[hehechen]

introduced by #6538, investigating.