importinto: calculate writer mem size using step resource & add a option for test #51486

D3Hunter · 2024-03-04T11:19:14Z

What problem does this PR solve?

Issue Number: ref #49008

Problem Summary:

What changed and how does it work?

calculate writer mem size using step resource & add a test option to force merge-step
add some logs

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
before this pr, import 1t data

2024-03-05 18:21:09 [task_executor.go:309] ["execute task step completed"] [task-id=4] [task-type=ImportInto] [step=encode&sort] [mem-limit-percent=0.99] [server-mem-limit=88%] [resource="[CPU=16, Mem=27.83GiB]"] [takeTime=1h34m2.774450553s] []
2024-03-05 19:04:51 [task_executor.go:309] ["execute task step completed"] [task-id=4] [task-type=ImportInto] [step=write&ingest] [mem-limit-percent=0.99] [server-mem-limit=88%] [resource="[CPU=16, Mem=27.83GiB]"] [takeTime=42m49.945995706s] []
2024-03-05 19:13:38 [task_executor.go:309] ["execute task step completed"] [task-id=4] [task-type=ImportInto] [step=post-process] [mem-limit-percent=0.99] [server-mem-limit=88%] [resource="[CPU=16, Mem=27.83GiB]"] [takeTime=8m46.067248935s] []

after, the time to encode and ingest is similar, the checksum time increase, but it's not related to this pr change

2024-03-05 21:41:52 [task_executor.go:309] ["execute task step completed"] [task-id=1] [task-type=ImportInto] [step=encode&sort] [mem-limit-percent=0.99] [server-mem-limit=88%] [resource="[CPU=16, Mem=27.83GiB]"] [takeTime=1h33m19.647405717s] []
2024-03-05 22:25:21 [task_executor.go:309] ["execute task step completed"] [task-id=1] [task-type=ImportInto] [step=write&ingest] [mem-limit-percent=0.99] [server-mem-limit=88%] [resource="[CPU=16, Mem=27.83GiB]"] [takeTime=42m5.874066197s] []
2024-03-05 22:53:11 [task_executor.go:309] ["execute task step completed"] [task-id=1] [task-type=ImportInto] [step=post-process] [mem-limit-percent=0.99] [server-mem-limit=88%] [resource="[CPU=16, Mem=27.83GiB]"] [takeTime=27m49.204570776s] []

with this pr, run import 1t with merge-step, success

[task_executor.go:309] ["execute task step completed"] [task-id=1] [task-type=ImportInto] [step=merge-sort] [mem-limit-percent=0.99] [server-mem-limit=88%] [resource="[CPU=16, Mem=27.83GiB]"] [takeTime=31m43.024898826s] []

No need to test
- I checked and no code files have been changed.

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

tiprow · 2024-03-04T11:19:29Z

Hi @D3Hunter. Thanks for your PR.

PRs from untrusted users cannot be marked as trusted with /ok-to-test in this repo meaning untrusted PR authors can never trigger tests themselves. Collaborators can still trigger tests on the PR using /test all.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

codecov · 2024-03-04T11:38:46Z

Codecov Report

Merging #51486 (825d84c) into master (4f4f911) will increase coverage by 2.7126%.
Report is 34 commits behind head on master.
The diff coverage is 100.0000%.

Additional details and impacted files

@@               Coverage Diff                @@
##             master     #51486        +/-   ##
================================================
+ Coverage   70.7359%   73.4486%   +2.7126%     
================================================
  Files          1462       1467         +5     
  Lines        435442     443717      +8275     
================================================
+ Hits         308014     325904     +17890     
+ Misses       108116      97806     -10310     
- Partials      19312      20007       +695

Flag	Coverage Δ
integration	`49.5718% <100.0000%> (?)`
unit	`71.1529% <57.4074%> (+0.6146%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
dumpling	`53.9957% <ø> (ø)`
parser	`∅ <ø> (∅)`
br	`51.6902% <ø> (+5.8681%)`	⬆️

D3Hunter · 2024-03-06T02:39:12Z

@lance6716

br/pkg/lightning/backend/external/merge.go

D3Hunter · 2024-03-06T08:56:54Z

update result of merge-sort
@lance6716

D3Hunter · 2024-03-06T10:38:41Z

@lance6716

lance6716 · 2024-03-06T11:22:08Z

pkg/disttask/importinto/planner.go

@@ -482,7 +482,7 @@ func getSortedKVMetasForIngest(planCtx planner.PlanCtx) (map[string]*external.So
 	for kvGroup, kvMeta := range kvMetasOfEncodeStep {
 		// only part of kv files are merge sorted. we need to merge kv metas that
 		// are not merged into the kvMetasOfMergeSort.
-		if skipMergeSort(kvGroup, kvMeta.MultipleFilesStats) {
+		if !p.Plan.ForceMergeStep && skipMergeSort(kvGroup, kvMeta.MultipleFilesStats) {


maybe we can remove the failpoint, ForceMergeStep seems cover all calling

i deleted it once, but find that some test only want to skip for some kv group

ti-chi-bot · 2024-03-07T06:12:10Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: lance6716, tangenta

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [lance6716,tangenta]
~~br/pkg/lightning/OWNERS~~ [lance6716]
~~pkg/disttask/OWNERS~~ [lance6716,tangenta]
~~pkg/executor/importer/OWNERS~~ [lance6716]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

ti-chi-bot · 2024-03-07T06:12:14Z

[LGTM Timeline notifier]

Timeline:

2024-03-06 11:27:53.86026721 +0000 UTC m=+252900.882513600: ☑️ agreed by lance6716.
2024-03-07 06:12:12.842519957 +0000 UTC m=+320359.864766344: ☑️ agreed by tangenta.

hawkingrei · 2024-03-07T07:21:58Z

/retest

D3Hunter · 2024-03-07T09:53:19Z

/retest

tiprow · 2024-03-07T09:53:41Z

@D3Hunter: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

D3Hunter added 4 commits March 1, 2024 11:53

add log

7e3c27a

use step resource

9767aa3

add option to force merge-sort

097dd4e

Merge remote-tracking branch 'upstream/master' into use-rc-allocator

8105ce5

ti-chi-bot bot added release-note-none Denotes a PR that doesn't merit a release note. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Mar 4, 2024

D3Hunter changed the title ~~importinto: calculate writer mem size using step resource & add a test option~~ importinto: calculate writer mem size using step resource & add a option for test Mar 4, 2024

Merge remote-tracking branch 'upstream/master' into use-rc-allocator

2f4457b

D3Hunter mentioned this pull request Mar 5, 2024

resource control inside dist framework #49008

Open

43 tasks

lance6716 reviewed Mar 6, 2024

View reviewed changes

br/pkg/lightning/backend/external/merge.go Show resolved Hide resolved

fix comments

825d84c

lance6716 approved these changes Mar 6, 2024

View reviewed changes

ti-chi-bot bot added approved needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Mar 6, 2024

tangenta approved these changes Mar 7, 2024

View reviewed changes

ti-chi-bot bot added lgtm and removed needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Mar 7, 2024

ti-chi-bot bot merged commit 911bcf1 into pingcap:master Mar 7, 2024
31 of 35 checks passed

D3Hunter deleted the use-rc-allocator branch March 7, 2024 15:46

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

importinto: calculate writer mem size using step resource & add a option for test #51486

importinto: calculate writer mem size using step resource & add a option for test #51486

D3Hunter commented Mar 4, 2024 •

edited

Loading

tiprow bot commented Mar 4, 2024

codecov bot commented Mar 4, 2024 •

edited

Loading

D3Hunter commented Mar 6, 2024

D3Hunter commented Mar 6, 2024

D3Hunter commented Mar 6, 2024

lance6716 Mar 6, 2024

D3Hunter Mar 6, 2024

ti-chi-bot bot commented Mar 7, 2024

ti-chi-bot bot commented Mar 7, 2024

hawkingrei commented Mar 7, 2024

D3Hunter commented Mar 7, 2024

tiprow bot commented Mar 7, 2024

importinto: calculate writer mem size using step resource & add a option for test #51486

importinto: calculate writer mem size using step resource & add a option for test #51486

Conversation

D3Hunter commented Mar 4, 2024 • edited Loading

What problem does this PR solve?

What changed and how does it work?

Check List

Release note

tiprow bot commented Mar 4, 2024

codecov bot commented Mar 4, 2024 • edited Loading

Codecov Report

D3Hunter commented Mar 6, 2024

D3Hunter commented Mar 6, 2024

D3Hunter commented Mar 6, 2024

lance6716 Mar 6, 2024

Choose a reason for hiding this comment

D3Hunter Mar 6, 2024

Choose a reason for hiding this comment

ti-chi-bot bot commented Mar 7, 2024

ti-chi-bot bot commented Mar 7, 2024

[LGTM Timeline notifier]

hawkingrei commented Mar 7, 2024

D3Hunter commented Mar 7, 2024

tiprow bot commented Mar 7, 2024

D3Hunter commented Mar 4, 2024 •

edited

Loading

codecov bot commented Mar 4, 2024 •

edited

Loading