- Author: Yunyan Hong
This RFC enables tables to support both setting placement rules and setting tiflash replicas.
Currently, there is a conflict between setting tiflash replicas and setting placement rules. For example, if a table has a tiflash replica, it is not supported to set placement rules for this table. Similarly, if a table has set some placement rules, it is not allowed to set tiflash replica for this table. The conflict between tiflash replicas and placement rules has a negative impact on user experience. Thus, this RFC will resolve this conflict, to support setting tiflash replicas and placement rules on tables at the same time.
The conflict between tiflash replicas and placement rules setting is because setting tiflash replicas and placement rules will both generate placement configuration in PD.
- In the configuration generated by placement rules,
group_index = 40, group_override = true
. - In the configuration generated by tiflash replicas,
group_index = 0
,group_override = false
.
According to the rules in PD, if group_override
is true , the configuration will override the configuration of other groups whose group_index
is smaller; While if group_override
is false, other configurations will not be overridden. Therefore, placement rules will overwrite the setting of tiflash replicas. To avoid this problem, the coexistence of placement rules and tiflash replicas was prohibited in the original development.
Because placement rules and tiflash replica do not affect each other in their own rules and work independently, we can give a largest value to group_index for the configuration generated by tiflash replicas. We set RuleIndexTiFlash = 120
, which is larger than all other placement configurations.
In this way, other placement rules can not override the settings of tiflash replicas. Besides, due to group_override = false
in tiflash replicas' configuration, tiflash replicas configuration will not override other placement rules. This allows the tables to support both placement rules and tiflash replicas settings.
Therefore, before setting tiflash replicas(POST /pd/api/v1/config/rule
), we call POST /pd/api/v1/config/rule-group
to modify the group_index value of the tiflash group as RuleIndexTiFlash.
For the already existing placement configuration of the tiflash replica setting in the old version, we will call POST /pd/api/v1/config/rule-group
in the background task of PollTiFlashRoutine
to update the group index of the placement configuration generated by the existing tiflash replicas. In this way, we can ensure normal settings for subsequent placement rules.
- Add setting group_index in the api
POST /pd/api/v1/config/rule
called by the tiflash replica settings, so that the group_index isRuleIndexTiFlash
when the tiflash replica is set.- This practice changes the semantics of
/pd/api/v1/config/rule
, which is inappropriate.
- This practice changes the semantics of
- tiflash uses the
POST /pd/api/v1/config/placement-rule
API to replace the original API.- This API will require setting the full number of rules in the group. All the settings of tiflash are in a group, which will lead to a large package of http requests when there are many tiflash tables.
- Compared with the original code, this change is relatively large, and the introduction of concurrency ddl may cause concurrent coverage problems.