v5.4.0 list partition can't work after restarting tidb

[wshwsh12]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

New v5.4.0 cluster with new collation, and run the following sql:

set @@session.tidb_enable_list_partition = ON;
set @@tidb_partition_prune_mode = 'dynamic';
drop table if exists github_events;
CREATE TABLE `github_events` (
`id` bigint(20) DEFAULT NULL,
`type` varchar(255) COLLATE utf8mb4_unicode_ci DEFAULT NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci
PARTITION BY LIST COLUMNS(`type`)
(PARTITION `push_event` VALUES IN ("PushEvent"),
PARTITION `create_event` VALUES IN ("CreateEvent"),
PARTITION `pull_request_event` VALUES IN ("PullRequestEvent"),
PARTITION `watch_event` VALUES IN ("WatchEvent")
);

tidb> desc select id,type from github_events  where  type = 'WatchEvent';
+-------------------------+----------+-----------+-----------------------+-------------------------------------------+
| id                      | estRows  | task      | access object         | operator info                             |
+-------------------------+----------+-----------+-----------------------+-------------------------------------------+
| TableReader_7           | 10.00    | root      | partition:watch_event | data:Selection_6                          |
| └─Selection_6           | 10.00    | cop[tikv] |                       | eq(test.github_events.type, "WatchEvent") |
|   └─TableFullScan_5     | 10000.00 | cop[tikv] | table:github_events   | keep order:false, stats:pseudo            |
+-------------------------+----------+-----------+-----------------------+-------------------------------------------+
3 rows in set (0.00 sec)

Then restart tidb

tidb> desc select id,type from github_events  where  type = 'WatchEvent';
+-------------+---------+------+---------------+---------------+
| id          | estRows | task | access object | operator info |
+-------------+---------+------+---------------+---------------+
| TableDual_6 | 0.00    | root |               | rows:0        |
+-------------+---------+------+---------------+---------------+
1 row in set (0.00 sec)

set @@session.tidb_enable_list_partition = ON;
set @@tidb_partition_prune_mode = 'dynamic';

tidb>  desc select id,type from github_events  where  type = 'WatchEvent';
+-------------------------+----------+-----------+---------------------+-------------------------------------------+
| id                      | estRows  | task      | access object       | operator info                             |
+-------------------------+----------+-----------+---------------------+-------------------------------------------+
| TableReader_7           | 10.00    | root      | partition:dual      | data:Selection_6                          |
| └─Selection_6           | 10.00    | cop[tikv] |                     | eq(test.github_events.type, "WatchEvent") |
|   └─TableFullScan_5     | 10000.00 | cop[tikv] | table:github_events | keep order:false, stats:pseudo            |
+-------------------------+----------+-----------+---------------------+-------------------------------------------+
3 rows in set (0.00 sec)

And then upgrade to master(3738dec):
it works.. But I don't think it is a fix. There may be some potential problems.

2. What did you expect to see? (Required)

All veresion can works. Or some explicit bug fix.

3. What did you see instead (Required)

4. What is your TiDB version? (Required)

v5.4, master

[wshwsh12]

@Defined2014 @wjhuang2016 PTAL

[zimulala]

@mjonss PTAL

[wjhuang2016]

The problem is that, ForListPruning builds the hashMap in TableFromMeta, which is initialized before setting the new_collation.
Since that we need to rebuild the hashMap after setting the new_collation.

[zimulala]

This also affects the master. This problem can also occur in the following scenario:

1. Set the new collation configuration item to false and run the SQL statement above (create list partition table).
2. Restart the TiDB and execute the select statement.

The above scenario is analyzed in detail:

In step 1

1. In the bootstrap phase, new collation is true
2. After bootstrap ends, new collation is false.

In step 2

1. In the bootstrap phase, new collation is true. ForListPruning builds the hashMap(generate the partition hash key) in TableFromMeta
2. After bootstrap ends, new collation is false. The generated partition hash key could not be found in the hashMap.