Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ttl: add info schema cache and ttl table status table cache for ttl scheduler #39467

Merged
merged 1 commit into from
Dec 5, 2022

Conversation

YangKeao
Copy link
Member

@YangKeao YangKeao commented Nov 29, 2022

What problem does this PR solve?

Issue Number: close #39466

Problem Summary:

To implement a ttl job scheduler, we'll need a local map for info schema and ttl table status, to map from a physical table id to related schema information and the information stored in the ttl table status

What is changed and how it works?

It does the following changes:

  1. Refine the code structure. The sql generator and cache are split and placed in different directories. If we are using sql generator and cache from the manager, both placing them in one level and "reverse (about the folder hierarchy) dependency" is not suggested.
  2. Move codes related with session into ttl/session, and add an interface of session pool into it (though there are some existing session pool, with the same definition here). I don't know whether it appropriate to add yet another session pool interface (but others are in extensions, br... pkg).
  3. Add two struct: InfoSchemaCache and the TTLTableStatusCache. They are the local mirror of the info schema and mysql.tidb_ttl_table_status system table.

This PR relies on #39315. Please merge it before this one.

Check List

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)
  • No code

@YangKeao YangKeao requested a review from a team as a code owner November 29, 2022 13:36
@ti-chi-bot
Copy link
Member

ti-chi-bot commented Nov 29, 2022

[REVIEW NOTIFICATION]

This pull request has been approved by:

  • hawkingrei
  • lcwangchao

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

@ti-chi-bot ti-chi-bot added do-not-merge/needs-triage-completed do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Nov 29, 2022
@YangKeao YangKeao added release-note-none Denotes a PR that doesn't merit a release note. and removed size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Nov 29, 2022
@ti-chi-bot ti-chi-bot removed the do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. label Nov 29, 2022
@YangKeao
Copy link
Member Author

/run-check-issue-triage-complete

@ti-chi-bot ti-chi-bot added the size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. label Nov 29, 2022
@YangKeao YangKeao force-pushed the add-ttl-job-schedule-1 branch 5 times, most recently from 798836f to 41852c7 Compare November 30, 2022 04:56
@YangKeao
Copy link
Member Author

/run-build

@YangKeao
Copy link
Member Author

/run-unit-test

@YangKeao
Copy link
Member Author

/run-build

@YangKeao
Copy link
Member Author

/run-check_dev_2

@YangKeao YangKeao force-pushed the add-ttl-job-schedule-1 branch from 6a41335 to e8a389e Compare November 30, 2022 05:38
ttl/cache/base.go Show resolved Hide resolved
ttl/cache/table.go Outdated Show resolved Hide resolved
ttl/cache/infoschema.go Show resolved Hide resolved
ttl/cache/ttlstatus_test.go Show resolved Hide resolved
ttl/cache/ttlstatus.go Outdated Show resolved Hide resolved
@YangKeao YangKeao force-pushed the add-ttl-job-schedule-1 branch from e8a389e to 42bb993 Compare December 1, 2022 09:20
@ti-chi-bot ti-chi-bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Dec 1, 2022
@YangKeao YangKeao force-pushed the add-ttl-job-schedule-1 branch from 42bb993 to 53d8b33 Compare December 1, 2022 09:29
@ti-chi-bot ti-chi-bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Dec 1, 2022
@ti-chi-bot ti-chi-bot added needs-cherry-pick-release-6.5 Should cherry pick this PR to release-6.5 branch. and removed needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. labels Dec 5, 2022
@YangKeao
Copy link
Member Author

YangKeao commented Dec 5, 2022

/run-check_dev_2

@YangKeao YangKeao force-pushed the add-ttl-job-schedule-1 branch from 4d5e89c to 7c3c7d6 Compare December 5, 2022 07:45
@YangKeao YangKeao requested review from lcwangchao and hawkingrei and removed request for a team December 5, 2022 07:46
ttl/session/session.go Outdated Show resolved Hide resolved
@YangKeao YangKeao force-pushed the add-ttl-job-schedule-1 branch 2 times, most recently from 9044725 to 04cd9b8 Compare December 5, 2022 07:52
@ti-chi-bot ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Dec 5, 2022
@YangKeao YangKeao requested a review from xhebox December 5, 2022 07:57
ttl/cache/infoschema.go Outdated Show resolved Hide resolved
ttl/cache/table.go Outdated Show resolved Hide resolved
@YangKeao YangKeao force-pushed the add-ttl-job-schedule-1 branch from 04cd9b8 to bcec10c Compare December 5, 2022 09:03
@YangKeao YangKeao force-pushed the add-ttl-job-schedule-1 branch from bcec10c to 1d4c2b1 Compare December 5, 2022 09:06
@YangKeao YangKeao requested a review from hawkingrei December 5, 2022 09:24
@ti-chi-bot ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Dec 5, 2022
@hawkingrei
Copy link
Member

/merge

@ti-chi-bot
Copy link
Member

This pull request has been accepted and is ready to merge.

Commit hash: 1d4c2b1

@ti-chi-bot ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Dec 5, 2022
@ti-chi-bot ti-chi-bot merged commit b770bc6 into pingcap:master Dec 5, 2022
@ti-chi-bot
Copy link
Member

In response to a cherrypick label: new pull request created: #39628.

@sre-bot
Copy link
Contributor

sre-bot commented Dec 5, 2022

TiDB MergeCI notify

🔴 Bad News! [1] CI still failing after this pr merged.
These failed integration tests don't seem to be introduced by the current PR.

CI Name Result Duration Compare with Parent commit
idc-jenkins-ci/integration-cdc-test 🔴 failed 1, success 39, total 40 19 min Existing failure
idc-jenkins-ci-tidb/common-test 🟢 all 11 tests passed 12 min Existing passed
idc-jenkins-ci-tidb/integration-common-test 🟢 all 17 tests passed 12 min Existing passed
idc-jenkins-ci-tidb/tics-test 🟢 all 1 tests passed 6 min 27 sec Existing passed
idc-jenkins-ci-tidb/sqllogic-test-2 🟢 all 28 tests passed 5 min 32 sec Existing passed
idc-jenkins-ci-tidb/integration-ddl-test 🟢 all 6 tests passed 5 min 2 sec Existing passed
idc-jenkins-ci-tidb/sqllogic-test-1 🟢 all 26 tests passed 4 min 19 sec Existing passed
idc-jenkins-ci-tidb/mybatis-test 🟢 all 1 tests passed 4 min 8 sec Existing passed
idc-jenkins-ci-tidb/integration-compatibility-test 🟢 all 1 tests passed 3 min 35 sec Existing passed
idc-jenkins-ci-tidb/plugin-test 🟢 build success, plugin test success 4min Existing passed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
affects-6.5 This bug affects the 6.5.x(LTS) versions. needs-cherry-pick-release-6.5 Should cherry pick this PR to release-6.5 branch. release-note-none Denotes a PR that doesn't merit a release note. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. status/can-merge Indicates a PR has been approved by a committer. status/LGT2 Indicates that a PR has LGTM 2.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add info schema cache and ttl table status cache
5 participants