-
Notifications
You must be signed in to change notification settings - Fork 5.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
lightning: finer granularity for tidb backend finished bytes #39318
Conversation
Signed-off-by: lance6716 <[email protected]>
[REVIEW NOTIFICATION] This pull request has been approved by:
To complete the pull request process, please ask the reviewers in the list to review by filling The full list of commands accepted by this bot can be found here. Reviewer can indicate their review by submitting an approval review. |
Signed-off-by: lance6716 <[email protected]>
Signed-off-by: lance6716 <[email protected]>
Signed-off-by: lance6716 <[email protected]>
Signed-off-by: lance6716 <[email protected]>
Signed-off-by: lance6716 <[email protected]>
Signed-off-by: lance6716 <[email protected]>
/run-integration-br-tests |
Signed-off-by: lance6716 <[email protected]>
/run-integration-br-tests |
@@ -406,6 +406,11 @@ func (tr *TableRestore) restoreEngine( | |||
if err != nil { | |||
return closedEngine, errors.Trace(err) | |||
} | |||
if rc.status != nil && rc.status.backend == config.BackendTiDB { | |||
for _, chunk := range cp.Chunks { | |||
rc.status.FinishedFileSize.Add(chunk.Chunk.EndOffset) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
should it be chunk.Chunk.EndOffset-chunk.Key.Offset
, in case of strict format where chunk.Key.Offset
is not 0
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm not familiar with strict format, can you give some reference to the behaviour?
I think you want to keep a property that the value of FinishedFileSize.Add
should be same no matter it's processed without failure or recovered from checkpoint?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
OK I guess the strict mode will split a file into many chunks, the split position is written in Key.Offset
@@ -475,6 +480,9 @@ func (tr *TableRestore) restoreEngine( | |||
|
|||
// Restore table data | |||
for chunkIndex, chunk := range cp.Chunks { | |||
if rc.status != nil && rc.status.backend == config.BackendTiDB { | |||
rc.status.FinishedFileSize.Add(chunk.Chunk.Offset) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
should it be chunk.Chunk.Offset - chunk.Key.Offset
to count previous imported data
@@ -475,6 +480,9 @@ func (tr *TableRestore) restoreEngine( | |||
|
|||
// Restore table data | |||
for chunkIndex, chunk := range cp.Chunks { | |||
if rc.status != nil && rc.status.backend == config.BackendTiDB { | |||
rc.status.FinishedFileSize.Add(chunk.Chunk.Offset) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
seems that not restore completedly here?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
here I want to recover FinishedFileSize from checkpoint, the restore process of current run is not started
Signed-off-by: lance6716 <[email protected]>
/run-integration-br-tests |
/merge |
This pull request has been accepted and is ready to merge. Commit hash: 6250609
|
TiDB MergeCI notify✅ Well Done! New fixed [2] after this pr merged.
|
Signed-off-by: lance6716 [email protected]
What problem does this PR solve?
Issue Number: close #xxx
Problem Summary:
I still use the old Lightning.Status to get the finished bytes and total bytes. For TiDB backend, finished bytes will be counted after write to TiDB, and also will recover from checkpoints
What is changed and how it works?
Check List
Tests
Side effects
Documentation
Release note
Please refer to Release Notes Language Style Guide to write a quality release note.