lightning: support table-level local-backend #35148
Labels
component/lightning
This issue is related to Lightning of TiDB.
type/enhancement
The issue or PR belongs to an enhancement.
Enhancement
We want to use tidb-lightning to import data to a cluster that is running online services. To reduce the impact on online services, we need to reduce the impact of local-backend to the table-level.
We have to do several enhancements:
First, tidb-lightning shouldn't disable the whole PD scheduler during importing. Instead, lightning should only disable the scheduler on the region that is being imported.tikv/pd#4695 supported TTL for region label. We can use the region label to disable the scheduler on a specific key range. With the help of TTL, we don't need to worry that the scheduler won't be recovered when tidb-lightning is crashed. For region label, please see tikv/pd#3839 for more details.
Second, tidb-lightning need to provide a mechanism to limit the write speed to each TiKV store. We can add a new config
tikv-importer.store-write-bwlimit
and limit the write speed to TiKV nodes.The text was updated successfully, but these errors were encountered: