Incompatible table router in DM and Lightning

[buchuitoudegou]

What did you do?

1. Migrate data using DM, where loader unit is now replaced by lightning by default.
2. Use regular expression in table router setting

What did you expect to see?

migration succeeds. Designated tables are migrated to target table.

1. source table in upstream: test_lightning_topsql.test
   config:

---
name: test_topsql
task-mode: full

target-database:
  host: "127.0.0.1"
  port: 4000
  user: "root"

mysql-instances:
  - source-id: "mysql1"
    block-allow-list: "instance"
    loader-config-name: "global"
    route-rules: ["rule1"]

block-allow-list:
  instance:
    do-dbs: ["test_lightning_topsql"]

loaders:
  global:
    pool-size: 16
    import-mode: "sql"
routes:
  rule1:
    schema-pattern: "~.*"
    table-pattern: "~test.*"
    target-schema: "dtest"
    target-table: "dtest_table"

What did you see instead?

In downstream database, the source table is routed to test_lightning_topsql.test instead of dtest.dtest_table. According to the source code, the router config is directly transmitted to the lightning unit, where the lightning router cannot recognize regular expression. This issue probably attributes to the PR #4358.

Versions of the cluster

DM version (run dmctl -V or dm-worker -V or dm-master -V):

(paste DM version here, and you must ensure versions of dmctl, DM-worker and DM-master are same)

Upstream MySQL/MariaDB server version:

(paste upstream MySQL/MariaDB server version here)

Downstream TiDB cluster version (execute SELECT tidb_version(); in a MySQL client):

(paste TiDB cluster version here)

How did you deploy DM: tiup or manually?

(leave TiUP or manually here)

Other interesting information (system version, hardware config, etc):

>
>

current status of DM cluster (execute query-status <task-name> in dmctl)

(paste current status of DM cluster here)

[buchuitoudegou]

It seems intractable and unreasonable to introduce the dm's router to tidb's repo. What about adding a new package called regexpr-router to the tidb-tools repo?

/cc @lance6716 @niubell

[lance6716]

It seems intractable and unreasonable to introduce the dm's router to tidb's repo. What about adding a new package called regexpr-router to the tidb-tools repo?

/cc @lance6716 @niubell

Yes. I remember we already recommended to do so.