import into: log checksum for each KV group #50871

lance6716 · 2024-02-01T04:14:58Z

What problem does this PR solve?

Issue Number: ref #50832

Problem Summary:

What changed and how does it work?

prints log for the checksum of each KV group
move the calculation of checksum from delivering goroutine to encoding goroutine, to avoid checksum is lost after encoding before delivering
unify the two encoders for file source and query source by using a readFn

Check List

Tests

Unit test
Integration test
Manual test: pass 40T test
No need to test

pass existing tests

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

tiprow · 2024-02-01T04:15:38Z

Hi @lance6716. Thanks for your PR.

PRs from untrusted users cannot be marked as trusted with /ok-to-test in this repo meaning untrusted PR authors can never trigger tests themselves. Collaborators can still trigger tests on the PR using /test all.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Signed-off-by: lance6716 <[email protected]>

codecov · 2024-02-01T06:45:25Z

Codecov Report

Merging #50871 (1510ec1) into master (464a126) will decrease coverage by 0.0688%.
The diff coverage is 89.7755%.

Additional details and impacted files

@@               Coverage Diff                @@
##             master     #50871        +/-   ##
================================================
- Coverage   72.3101%   72.2414%   -0.0688%     
================================================
  Files          1467       1468         +1     
  Lines        361663     437922     +76259     
================================================
+ Hits         261519     316361     +54842     
- Misses        80851     101491     +20640     
- Partials      19293      20070       +777

Flag	Coverage Δ
integration	`27.9971% <87.7805%> (?)`
unit	`70.2789% <75.5611%> (-1.8902%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
dumpling	`54.3668% <ø> (-1.9461%)`	⬇️
parser	`∅ <ø> (∅)`
br	`51.3057% <91.9191%> (+4.9857%)`	⬆️

Signed-off-by: lance6716 <[email protected]>

lance6716 · 2024-02-01T09:12:47Z

/retest

tiprow · 2024-02-01T09:13:08Z

@lance6716: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

lance6716 · 2024-02-01T10:00:26Z

/retest

tiprow · 2024-02-01T10:00:46Z

@lance6716: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

…group

D3Hunter · 2024-02-02T06:31:10Z

pkg/executor/importer/chunk_process.go

+	task.Info("data group checksum", zap.Object("checksum", p.enc.dataChecksum))
+	for indexID, c := range p.enc.indexChecksum {
+		p.chunkInfo.Checksum.Add(c)
+		task.Info("index group checksum", zap.Int64("indexID", indexID), zap.Object("checksum", c))


it's too many to have each set of logs for each chunk, can we have each set for each subtask, and only print it at post-process step, else we have to sum them up manually

Signed-off-by: lance6716 <[email protected]>

D3Hunter

will review later

br/pkg/lightning/verification/checksum.go

D3Hunter

rest lgtm

br/pkg/lightning/verification/checksum.go

pkg/disttask/importinto/proto.go

br/pkg/lightning/verification/checksum.go

pkg/disttask/importinto/subtask_executor.go

pkg/disttask/importinto/task_executor.go

pkg/executor/importer/chunk_process.go

Signed-off-by: lance6716 <[email protected]>

lance6716 · 2024-02-18T02:22:23Z

/retest

tiprow · 2024-02-18T02:22:43Z

@lance6716: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Benjamin2037

LGTM

ti-chi-bot · 2024-02-18T07:41:43Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Benjamin2037, D3Hunter

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~br/pkg/lightning/OWNERS~~ [Benjamin2037,D3Hunter]
~~pkg/disttask/OWNERS~~ [Benjamin2037,D3Hunter]
~~pkg/executor/importer/OWNERS~~ [D3Hunter]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

ti-chi-bot · 2024-02-18T07:41:47Z

[LGTM Timeline notifier]

Timeline:

2024-02-05 07:34:59.709180524 +0000 UTC m=+169425.275950399: ☑️ agreed by D3Hunter.
2024-02-18 07:41:46.456144539 +0000 UTC m=+170195.203767650: ☑️ agreed by Benjamin2037.

tiprow · 2024-02-18T07:44:07Z

@lance6716: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
tiprow_fast_test	`0fee9cf`	link	true	`/test tiprow_fast_test`

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

…group

Signed-off-by: lance6716 <[email protected]>

lance6716 · 2024-02-18T08:34:56Z

/retest

tiprow · 2024-02-18T08:35:20Z

@lance6716: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

lance6716 added 3 commits January 31, 2024 17:34

--wip-- [skip ci]

c808feb

--wip-- [skip ci]

f4b38dd

--wip-- [skip ci]

5775c48

lance6716 marked this pull request as draft February 1, 2024 04:15

finish?

adf72ba

Signed-off-by: lance6716 <[email protected]>

ti-chi-bot bot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Feb 1, 2024

lance6716 marked this pull request as ready for review February 1, 2024 06:32

lance6716 added 4 commits February 1, 2024 15:08

fix bazel

48a1965

Signed-off-by: lance6716 <[email protected]>

fix UT

add9d93

Signed-off-by: lance6716 <[email protected]>

fix panic

d668aa5

Signed-off-by: lance6716 <[email protected]>

fix mock

3e90383

Signed-off-by: lance6716 <[email protected]>

ti-chi-bot bot removed do-not-merge/needs-linked-issue do-not-merge/needs-tests-checked labels Feb 1, 2024

lance6716 changed the title ~~[WIP]Separate index group~~ import into: log checksum for each KV group Feb 1, 2024

ti-chi-bot bot removed do-not-merge/invalid-title do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. labels Feb 1, 2024

Merge branch 'master' of github.com:pingcap/tidb into separate-index-…

056ee97

…group

D3Hunter reviewed Feb 2, 2024

View reviewed changes

lance6716 added 5 commits February 4, 2024 10:38

address comment

162c70c

Signed-off-by: lance6716 <[email protected]>

fix bazel

d7db720

Signed-off-by: lance6716 <[email protected]>

fix UT

06e2150

Signed-off-by: lance6716 <[email protected]>

fix UT

9322272

Signed-off-by: lance6716 <[email protected]>

fix UT

827ae5a

Signed-off-by: lance6716 <[email protected]>

D3Hunter reviewed Feb 5, 2024

View reviewed changes

br/pkg/lightning/verification/checksum.go Outdated Show resolved Hide resolved

D3Hunter reviewed Feb 5, 2024

View reviewed changes

address comment

0fee9cf

Signed-off-by: lance6716 <[email protected]>

D3Hunter approved these changes Feb 5, 2024

View reviewed changes

ti-chi-bot bot added approved needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Feb 5, 2024

Benjamin2037 approved these changes Feb 18, 2024

View reviewed changes

ti-chi-bot bot added lgtm and removed needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Feb 18, 2024

lance6716 added 2 commits February 18, 2024 15:55

Merge branch 'master' of github.com:pingcap/tidb into separate-index-…

e240f62

…group

fix lint

1510ec1

Signed-off-by: lance6716 <[email protected]>

ti-chi-bot bot merged commit 65d57c0 into pingcap:master Feb 18, 2024
20 of 21 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

import into: log checksum for each KV group #50871

import into: log checksum for each KV group #50871

lance6716 commented Feb 1, 2024 •

edited

Loading

tiprow bot commented Feb 1, 2024

codecov bot commented Feb 1, 2024 •

edited

Loading

lance6716 commented Feb 1, 2024

tiprow bot commented Feb 1, 2024

lance6716 commented Feb 1, 2024

tiprow bot commented Feb 1, 2024

D3Hunter Feb 2, 2024 •

edited

Loading

D3Hunter left a comment

D3Hunter left a comment

lance6716 commented Feb 18, 2024

tiprow bot commented Feb 18, 2024

Benjamin2037 left a comment

ti-chi-bot bot commented Feb 18, 2024

ti-chi-bot bot commented Feb 18, 2024

tiprow bot commented Feb 18, 2024

lance6716 commented Feb 18, 2024

tiprow bot commented Feb 18, 2024

import into: log checksum for each KV group #50871

import into: log checksum for each KV group #50871

Conversation

lance6716 commented Feb 1, 2024 • edited Loading

What problem does this PR solve?

What changed and how does it work?

Check List

Release note

tiprow bot commented Feb 1, 2024

codecov bot commented Feb 1, 2024 • edited Loading

Codecov Report

lance6716 commented Feb 1, 2024

tiprow bot commented Feb 1, 2024

lance6716 commented Feb 1, 2024

tiprow bot commented Feb 1, 2024

D3Hunter Feb 2, 2024 • edited Loading

Choose a reason for hiding this comment

D3Hunter left a comment

Choose a reason for hiding this comment

D3Hunter left a comment

Choose a reason for hiding this comment

lance6716 commented Feb 18, 2024

tiprow bot commented Feb 18, 2024

Benjamin2037 left a comment

Choose a reason for hiding this comment

ti-chi-bot bot commented Feb 18, 2024

ti-chi-bot bot commented Feb 18, 2024

[LGTM Timeline notifier]

tiprow bot commented Feb 18, 2024

lance6716 commented Feb 18, 2024

tiprow bot commented Feb 18, 2024

lance6716 commented Feb 1, 2024 •

edited

Loading

codecov bot commented Feb 1, 2024 •

edited

Loading

D3Hunter Feb 2, 2024 •

edited

Loading