*: refine mysql.tidb_mdl_view #48728

wjhuang2016 · 2023-11-20T12:23:00Z

What problem does this PR solve?

Issue Number: close #50149, close #46933, close #47743

Problem Summary:

What is changed and how it works?

Use mysql.tidb_mdl_info and mysql.ddl_job so that we don't query all the history DDLs

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)

mysql> SELECT tidb_mdl_info.job_id, JSON_EXTRACT(cast(cast(job_meta as char) as json), "$.schema_name"), JSON_EXTRACT(cast(cast(job_meta as char) as json), "$.table_name"), JSON_EXTRACT(cast(cast(job_meta as char) as json), "$.query"), session_id, cluster_tidb_trx.start_time, tidb_decode_sql_digests(all_sql_digests, 4096) AS SQL_DIGESTS FROM mysql.tidb_ddl_job, mysql.tidb_mdl_info, information_schema.cluster_tidb_trx WHERE tidb_ddl_job.job_id=tidb_mdl_info.job_id AND CONCAT(',', tidb_mdl_info.table_ids, ',') REGEXP CONCAT(',', REPLACE(cluster_tidb_trx.related_table_ids, ',', '|'), ',') != 0;
+--------+---------------------------------------------------------------------+--------------------------------------------------------------------+---------------------------------------------------------------+------------+----------------------------+----------------------------------+
| job_id | JSON_EXTRACT(cast(cast(job_meta as char) as json), "$.schema_name") | JSON_EXTRACT(cast(cast(job_meta as char) as json), "$.table_name") | JSON_EXTRACT(cast(cast(job_meta as char) as json), "$.query") | session_id | start_time                 | SQL_DIGESTS                      |
+--------+---------------------------------------------------------------------+--------------------------------------------------------------------+---------------------------------------------------------------+------------+----------------------------+----------------------------------+
|   1336 | test                                                              | t                                                                | alter table t add index idx(id)                             |    2097160 | 2024-01-08 15:37:26.450000 | ["begin","select `id` from `t`"] |
+--------+---------------------------------------------------------------------+--------------------------------------------------------------------+---------------------------------------------------------------+------------+----------------------------+----------------------------------+
1 row in set (0.00 sec)

No need to test
- I checked and no code files have been changed.

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

Signed-off-by: wjhuang2016 <[email protected]>

tangenta · 2023-11-20T12:35:16Z

pkg/sessionctx/variable/session.go

@@ -2084,6 +2084,7 @@ func NewSessionVars(hctx HookContext) *SessionVars {
 	if EnableRowLevelChecksum.Load() {
 		vars.EnableRowLevelChecksum = true
 	}
+	vars.systems[MaxAllowedPacket] = strconv.Itoa(int(DefMaxAllowedPacket))


Why change this?

So that we can use concat in UT

codecov · 2023-11-20T12:37:01Z

Codecov Report

Merging #48728 (cd956d8) into master (f19d876) will decrease coverage by 11.7139%.
Report is 24 commits behind head on master.
The diff coverage is 82.6087%.

Additional details and impacted files

@@                Coverage Diff                @@
##             master     #48728         +/-   ##
=================================================
- Coverage   79.3015%   67.5877%   -11.7139%     
=================================================
  Files          2447       2568        +121     
  Lines        673721     846719     +172998     
=================================================
+ Hits         534271     572278      +38007     
- Misses       118049     250789     +132740     
- Partials      21401      23652       +2251

Flag	Coverage Δ
integration	`37.2688% <56.5217%> (?)`
unit	`79.5812% <60.8695%> (+0.2797%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
dumpling	`73.6130% <ø> (ø)`
parser	`∅ <ø> (∅)`
br	`71.7976% <ø> (+3.3231%)`	⬆️

ti-chi-bot · 2023-11-23T05:38:14Z

PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Signed-off-by: wjhuang2016 <[email protected]>

wjhuang2016 · 2024-01-08T14:33:42Z

/retest

hawkingrei · 2024-01-08T15:01:05Z

/test tiprow_fast_test

ti-chi-bot · 2024-01-08T15:01:09Z

@hawkingrei: The specified target(s) for /test were not found.
The following commands are available to trigger required jobs:

/test build
/test check-dev
/test check-dev2
/test mysql-test
/test pull-integration-ddl-test
/test pull-lightning-integration-test
/test pull-mysql-client-test
/test unit-test

The following commands are available to trigger optional jobs:

/test canary-notify-when-compatibility-sections-changed
/test pingcap/tidb/canary_ghpr_unit_test
/test pull-br-integration-test
/test pull-common-test
/test pull-e2e-test
/test pull-integration-common-test
/test pull-integration-copr-test
/test pull-integration-jdbc-test
/test pull-integration-mysql-test
/test pull-integration-nodejs-test
/test pull-sqllogic-test
/test pull-tiflash-test

Use /test all to run the following jobs that were automatically triggered:

pingcap/tidb/ghpr_build
pingcap/tidb/ghpr_check
pingcap/tidb/ghpr_check2
pingcap/tidb/ghpr_mysql_test
pingcap/tidb/ghpr_unit_test
pingcap/tidb/pull_integration_ddl_test
pingcap/tidb/pull_mysql_client_test

In response to this:

/test tiprow_fast_test

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Signed-off-by: wjhuang2016 <[email protected]>

wjhuang2016 · 2024-01-08T15:47:16Z

/retest

D3Hunter · 2024-01-09T04:10:48Z

pkg/infoschema/test/clustertablestest/cluster_tables_test.go

-			tk.MustExec(c.createTable)
+			for _, cr := range c.createTable {
+				if strings.Contains(c.name, "err") {
+					_, _ = tk.Exec(cr)


check it does return err?

Not all the SQLs return an error, I don't think it's necessary to check the error since it is just a preparation.

D3Hunter · 2024-01-09T04:13:34Z

pkg/session/session.go

+		// But we need to init max_allowed_packet to use concat function during bootstrap or upgrade.
+		err := vars.SetSystemVar(variable.MaxAllowedPacket, strconv.FormatUint(variable.DefMaxAllowedPacket, 10))
+		if err != nil {
+			logutil.BgLogger().Error("set system variable max_allowed_packet error", zap.Error(err))


should we return err if it's required?

It's not always required, only when we need to create or replace this view. Do you think we still need to return an error?

ti-chi-bot · 2024-01-09T05:18:08Z

[LGTM Timeline notifier]

Timeline:

2024-01-08 08:04:03.277253315 +0000 UTC m=+257632.861506999: ☑️ agreed by tangenta.
2024-01-09 05:18:07.490750927 +0000 UTC m=+334077.075004600: ☑️ agreed by D3Hunter.

zimulala · 2024-01-09T07:49:27Z

pkg/session/bootstrap.go

-		WHERE (ddl_jobs.state != 'synced' and ddl_jobs.state != 'cancelled')
-			AND Find_in_set(ddl_jobs.table_id, cluster_tidb_trx.related_table_ids)
-			AND cluster_tidb_trx.session_id = cluster_processlist.id
+		FROM mysql.tidb_ddl_job,


TiDB versions less than 6.2 do not have this table. Does this have compatibility issues?

After testing, the old owner doesn't check if the table exists or not. So there it no compatibility issues for now.

Got it. Maybe we can create an issue.

wjhuang2016 · 2024-01-09T07:55:04Z

/hold

wjhuang2016 · 2024-01-09T09:40:12Z

/unhold

yudongusa

LGTM for now but version rollback needs to be considered for the cases like this. @Benjamin2037

ti-chi-bot · 2024-01-09T16:32:40Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: D3Hunter, tangenta, yudongusa, zimulala

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [D3Hunter,tangenta,zimulala]
~~pkg/infoschema/OWNERS~~ [D3Hunter,tangenta,zimulala]
~~pkg/session/OWNERS~~ [yudongusa]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

wjhuang2016 · 2024-01-09T17:59:10Z

/retest

ti-chi-bot · 2024-01-09T18:20:36Z

In response to a cherrypick label: new pull request created to branch release-7.5: #50236.

Signed-off-by: ti-chi-bot <[email protected]>

done

df07f0d

Signed-off-by: wjhuang2016 <[email protected]>

ti-chi-bot bot added do-not-merge/needs-linked-issue do-not-merge/needs-tests-checked release-note-none Denotes a PR that doesn't merit a release note. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Nov 20, 2023

tangenta reviewed Nov 20, 2023

View reviewed changes

ti-chi-bot bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Nov 23, 2023

wjhuang2016 added 2 commits January 8, 2024 15:53

refine

624c93c

Signed-off-by: wjhuang2016 <[email protected]>

merge master

f6d77d6

Signed-off-by: wjhuang2016 <[email protected]>

ti-chi-bot bot added do-not-merge/needs-triage-completed and removed do-not-merge/needs-linked-issue do-not-merge/needs-triage-completed do-not-merge/needs-tests-checked labels Jan 8, 2024

tangenta approved these changes Jan 8, 2024

View reviewed changes

ti-chi-bot bot added the needs-1-more-lgtm Indicates a PR needs 1 more LGTM. label Jan 8, 2024

wjhuang2016 added 3 commits January 8, 2024 16:11

fix bazel

7b2f530

Signed-off-by: wjhuang2016 <[email protected]>

fix

6caac1c

Signed-off-by: wjhuang2016 <[email protected]>

fix

b649ad6

Signed-off-by: wjhuang2016 <[email protected]>

wjhuang2016 removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jan 8, 2024

wjhuang2016 added 2 commits January 8, 2024 20:26

try

4aa07d7

Signed-off-by: wjhuang2016 <[email protected]>

fix

d94c369

Signed-off-by: wjhuang2016 <[email protected]>

ti-chi-bot bot added do-not-merge/needs-triage-completed needs-cherry-pick-release-7.5 Should cherry pick this PR to release-7.5 branch. and removed do-not-merge/needs-triage-completed labels Jan 8, 2024

add ut

7a1d3f8

Signed-off-by: wjhuang2016 <[email protected]>

fix

cd956d8

Signed-off-by: wjhuang2016 <[email protected]>

D3Hunter reviewed Jan 9, 2024

View reviewed changes

D3Hunter approved these changes Jan 9, 2024

View reviewed changes

ti-chi-bot bot added lgtm and removed needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Jan 9, 2024

zimulala reviewed Jan 9, 2024

View reviewed changes

ti-chi-bot bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jan 9, 2024

ti-chi-bot bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jan 9, 2024

zimulala approved these changes Jan 9, 2024

View reviewed changes

yudongusa approved these changes Jan 9, 2024

View reviewed changes

ti-chi-bot bot added the approved label Jan 9, 2024

ti-chi-bot bot merged commit 35c0e8a into pingcap:master Jan 9, 2024
22 checks passed

ti-chi-bot pushed a commit to ti-chi-bot/tidb that referenced this pull request Jan 9, 2024

This is an automated cherry-pick of pingcap#48728

8d5bfd0

Signed-off-by: ti-chi-bot <[email protected]>

ti-chi-bot mentioned this pull request Jan 9, 2024

*: refine mysql.tidb_mdl_view (#48728) #50236

Closed

13 tasks

This was referenced Jan 10, 2024

TestUpgradeVersionForResumeJob execution timeout #50292

Closed

session: make TestUpgradeVersionForResumeJob faster #50299

Merged

tangenta mentioned this pull request May 15, 2024

Root user lack rights to use tidb_mdl_view #53292

Closed

Jayjlchen mentioned this pull request May 24, 2024

DDL cancellation stalled by DML: Expecting lock indication in mysql.tidb_mdl_view #53539

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

*: refine mysql.tidb_mdl_view #48728

*: refine mysql.tidb_mdl_view #48728

wjhuang2016 commented Nov 20, 2023 •

edited

Loading

tangenta Nov 20, 2023

wjhuang2016 Jan 8, 2024

codecov bot commented Nov 20, 2023 •

edited

Loading

ti-chi-bot bot commented Nov 23, 2023

wjhuang2016 commented Jan 8, 2024

hawkingrei commented Jan 8, 2024

ti-chi-bot bot commented Jan 8, 2024

wjhuang2016 commented Jan 8, 2024

D3Hunter Jan 9, 2024

wjhuang2016 Jan 9, 2024

D3Hunter Jan 9, 2024

wjhuang2016 Jan 9, 2024

ti-chi-bot bot commented Jan 9, 2024

zimulala Jan 9, 2024

wjhuang2016 Jan 9, 2024

zimulala Jan 9, 2024

wjhuang2016 commented Jan 9, 2024

wjhuang2016 commented Jan 9, 2024

yudongusa left a comment

ti-chi-bot bot commented Jan 9, 2024

wjhuang2016 commented Jan 9, 2024

ti-chi-bot commented Jan 9, 2024

*: refine mysql.tidb_mdl_view #48728

*: refine mysql.tidb_mdl_view #48728

Conversation

wjhuang2016 commented Nov 20, 2023 • edited Loading

What problem does this PR solve?

What is changed and how it works?

Check List

Release note

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Nov 20, 2023 • edited Loading

Codecov Report

ti-chi-bot bot commented Nov 23, 2023

wjhuang2016 commented Jan 8, 2024

hawkingrei commented Jan 8, 2024

ti-chi-bot bot commented Jan 8, 2024

wjhuang2016 commented Jan 8, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ti-chi-bot bot commented Jan 9, 2024

[LGTM Timeline notifier]

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wjhuang2016 commented Jan 9, 2024

wjhuang2016 commented Jan 9, 2024

yudongusa left a comment

Choose a reason for hiding this comment

ti-chi-bot bot commented Jan 9, 2024

wjhuang2016 commented Jan 9, 2024

ti-chi-bot commented Jan 9, 2024

wjhuang2016 commented Nov 20, 2023 •

edited

Loading

codecov bot commented Nov 20, 2023 •

edited

Loading