Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

executor: LOAD DATA INFILE support asterisk matching #42050

Merged
merged 9 commits into from
Mar 13, 2023
Merged

executor: LOAD DATA INFILE support asterisk matching #42050

merged 9 commits into from
Mar 13, 2023

Conversation

lance6716
Copy link
Contributor

What problem does this PR solve?

Issue Number: ref #40499

Problem Summary:

as title

What is changed and how it works?

Check List

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)
  • No code

Side effects

  • Performance regression: Consumes more CPU
  • Performance regression: Consumes more Memory
  • Breaking backward compatibility

Documentation

  • Affects user behaviors
  • Contains syntax changes
  • Contains variable changes
  • Contains experimental features
  • Changes MySQL compatibility

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

@ti-chi-bot
Copy link
Member

ti-chi-bot commented Mar 9, 2023 •
edited
Loading

[REVIEW NOTIFICATION]

This pull request has been approved by:

  • D3Hunter
  • gozssky

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

@ti-chi-bot ti-chi-bot added release-note-none Denotes a PR that doesn't merit a release note. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Mar 9, 2023
@lance6716
revert unwanted change
fdb111b 
Signed-off-by: lance6716 <[email protected]>
@lance6716
revert more
0f8e835 
Signed-off-by: lance6716 <[email protected]>
@lance6716
finish
5867dbe 
Signed-off-by: lance6716 <[email protected]>
@lance6716
Copy link
Contributor Author

/cc @Leavrth @D3Hunter @GMHDBJD

@lance6716
Copy link
Contributor Author

/cc @gozssky

D3Hunter
D3Hunter reviewed Mar 9, 2023
View reviewed changes
executor/load_data.go Outdated
Comment on lines 223 to 224
loggedError = true
logutil.Logger(ctx).Warn("failed to build pattern for file matching",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we can move this in param checking part

executor/load_data.go

filesize := int64(-1)
reader, err2 := opener(ctx)
if err2 == nil {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

odd to continue load when open failed, we don't have retry for this now, maybe just return err

executor/load_data.go Outdated
return
}

TrySendTask:
pos, err2 := seeker.Seek(0, io.SeekCurrent)
if err2 != nil && !loggedError {
pos, err = seeker.Seek(0, io.SeekCurrent)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
pos, err = seeker.Seek(0, io.SeekCurrent)
currScannedSize, err = seeker.Seek(0, io.SeekCurrent)

executor/load_data.go Outdated
@@ -997,7 +1093,8 @@ func (e *LoadDataWorker) ReadRows(ctx context.Context, parser mydump.Parser) err
}
}

for e.IgnoreLines > 0 {
ignoreLineCnt := e.IgnoreLines
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should save this ignored line count, else we skip N lines for every read-row

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should be saved for each file

D3Hunter
D3Hunter reviewed Mar 9, 2023
View reviewed changes
executor/loadremotetest/multi_file_test.go

s.tk.MustExec("TRUNCATE TABLE multi_load.t;")
sql = fmt.Sprintf(`LOAD DATA INFILE 'gs://test-multi-load/db.tbl.*.tsv?endpoint=%s'
INTO TABLE multi_load.t IGNORE 1 LINES;`, gcsEndpoint)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please add a case which calls readRows twice to make sure ignoreNLines works

@lance6716
address comment
ce0a410 
Signed-off-by: lance6716 <[email protected]>
@lance6716
Copy link
Contributor Author

/hold

@ti-chi-bot ti-chi-bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Mar 10, 2023
@lance6716
add and fix UT
773bef5 
Signed-off-by: lance6716 <[email protected]>
@lance6716 lance6716 changed the title executor: LOAD DATA support INFILE glob matching executor: LOAD DATA support INFILE asterisk matching Mar 10, 2023
@ti-chi-bot ti-chi-bot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Mar 10, 2023
@lance6716
Copy link
Contributor Author

/unhold

@ti-chi-bot ti-chi-bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Mar 10, 2023
@lance6716
add a util function
9c0a541 
Signed-off-by: lance6716 <[email protected]>
@lance6716 lance6716 changed the title executor: LOAD DATA support INFILE asterisk matching executor: LOAD DATA INFILE support asterisk matching Mar 13, 2023
@lance6716
Copy link
Contributor Author

ptal @D3Hunter @gozssky @GMHDBJD

@ti-chi-bot ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Mar 13, 2023
sleepymole
sleepymole reviewed Mar 13, 2023
View reviewed changes
Copy link
Contributor

@sleepymole sleepymole left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we have any document to describe the asterisk matching?

executor/load_data.go Show resolved Hide resolved
@ti-chi-bot ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Mar 13, 2023
@lance6716
Copy link
Contributor Author

/merge

@ti-chi-bot
Copy link
Member

This pull request has been accepted and is ready to merge.

Commit hash: 9c0a541

@ti-chi-bot ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Mar 13, 2023
@ti-chi-bot ti-chi-bot merged commit fd91259 into pingcap:master Mar 13, 2023
@kennytm kennytm mentioned this pull request Feb 6, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@sleepymole sleepymole sleepymole approved these changes

@D3Hunter D3Hunter D3Hunter approved these changes

@GMHDBJD GMHDBJD Awaiting requested review from GMHDBJD

@Leavrth Leavrth Awaiting requested review from Leavrth

Assignees
No one assigned
Labels
release-note-none Denotes a PR that doesn't merit a release note. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. status/can-merge Indicates a PR has been approved by a committer. status/LGT2 Indicates that a PR has LGTM 2.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@lance6716 @ti-chi-bot @D3Hunter @sleepymole