Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

UCP: Speed up parse slow-log when query slow_query #15371

Closed
crazycs520 opened this issue Mar 14, 2020 · 13 comments · Fixed by #19139
Closed

UCP: Speed up parse slow-log when query slow_query #15371

crazycs520 opened this issue Mar 14, 2020 · 13 comments · Fixed by #19139
Labels
sig/execution SIG execution

Comments

@crazycs520
Copy link
Contributor

crazycs520 commented Mar 14, 2020 •
edited
Loading

Description

After Issue #15354 laster, I found the performance improvement is not significant. The Read slow log file and parse process are slow.

So, we still need to speed up it.

Score

Use a bench test to judge the score, the bench test should in the below situation:

  • compare the time cost of select count(*) from information_schema.slow_query
  • And the slow-log.log file is 300M.
  • Should do bench test in NVME or SSD.

Below is the score:

  • improve 10X -1000
  • improve 9X -900
  • improve 8X -800
    ......
  • improve 2X -200

At least has 2x performance improvement, the max score is 1000.

Mentor(s)

Contact the mentors: #tidb-challenge-program channel in TiDB Community Slack Workspace

Recommended Skills

  • Go

Learning Materials

  • use slowQueryRetriever search in TiDB code to see related implemention code.
@yzwqf
Copy link
Contributor

yzwqf commented Mar 23, 2020

/pick-up-challenge

@sre-bot
Copy link
Contributor

sre-bot commented Mar 23, 2020

@yzwqf sorry for this accident, we will fix it ASAP.
@you06 something wrong with this challenge issue, PTAL.

@you06
Copy link
Contributor

you06 commented Mar 23, 2020

@yzwqf the issue should be fixed and I noticed that you have 150 points in this repo, the medium level tasks required at least 200 points before picking up.
@crazycs520 I assign with 1000 points for this issue by default, you may use /award-point to adjust score in the pull request. eg. /award-point 600

@yzwqf
Copy link
Contributor

yzwqf commented Mar 23, 2020

@yzwqf the issue should be fixed and I noticed that you have 150 points in this repo, the medium level tasks required at least 200 points before picking up.
@crazycs520 I assign with 1000 points for this issue by default, you may use /award-point to adjust score in the pull request. eg. /award-point 600

it didn't make sense, would u please check #15021, #15036, #15401, #15031.

@you06
Copy link
Contributor

you06 commented Mar 23, 2020

@yzwqf I see, #15467 is not related with #15401, you may use UCP #xxx to UCP mention issue in PR description. I've fix the points issue, and you can pick this, congrats!

@yzwqf
Copy link
Contributor

yzwqf commented Mar 23, 2020

/pick-up-challenge

@sre-bot
Copy link
Contributor

sre-bot commented Mar 23, 2020

@yzwqf pick up issue success

@yzwqf
Copy link
Contributor

yzwqf commented Mar 23, 2020

@yzwqf I see, #15467 is not related with #15401, you may use UCP #xxx to UCP mention issue in PR description. I've fix the points issue, and you can pick this, congrats!

thx a lot

@yzwqf
Copy link
Contributor

yzwqf commented Mar 23, 2020

/give-up-challenge

@sre-bot
Copy link
Contributor

sre-bot commented Mar 23, 2020

@yzwqf give up issue success

@yufeiminds
Copy link
Contributor

/pick-up-challenge

@sre-bot
Copy link
Contributor

sre-bot commented May 22, 2020

@yufeiminds don't have enough score, pick up failed

Progress 0/200
You may pick up some easy issues first

@jyz0309
Copy link
Contributor

jyz0309 commented Aug 3, 2020

Alkaid will fix it this week :)

@jyz0309 jyz0309 mentioned this issue Aug 12, 2020
Merged
ti-srebot added a commit that referenced this issue Aug 25, 2020
@jyz0309 @crazycs520 @ti-srebot
*:Speed up parse slow-log when query slow_query #15371 (#19139)
c04bda6 
* speed up the slow log query

* speed up slow log query

* change the MaxRow num

* change channel location

* correct

* fix some bug

* fix error situation

* change ctx.Done() location

* update

* Update main.go

* change func name

* fix the len(rows)=0 bug

* remove else

* update slow_query_test

* update slow_query_test.go

* update slow_query_test.go

* update slow_query_test.go

* append offset var

* fix bug

* add notes

* remove fmt

* remove notes

* remove notes

* update

* update

* version back

* update offset

* update

* use len(log)==0 to check

* update

* update offset struct

* update offset

* defer recover

* update slow_query_test

* update offset

* remove fmt.print

* check error first

* update panic situation

* update failpoint test

* update failpoint test

* update panic test

* test

* test

* update test

* update failpoint name

* test

* update slow_query.go

* check err

* Code specification

* remove pointer

* update getLineIndex func

* update setFieldValue func

* debug

* update func name

Co-authored-by: crazycs <[email protected]>
Co-authored-by: ti-srebot <[email protected]>
@ti-srebot ti-srebot mentioned this issue Oct 21, 2020
Merged
ti-srebot added a commit that referenced this issue Oct 21, 2020
@ti-srebot
*:Speed up parse slow-log when query slow_query #15371 (#19139) (#20556)
ba9fe30 
Signed-off-by: ti-srebot <[email protected]>
Signed-off-by: jyz0309 <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
sig/execution SIG execution
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

*:Speed up parse slow-log when query slow_query #15371 jyz0309/tidb
7 participants
@zz-jason @yufeiminds @you06 @crazycs520 @sre-bot @yzwqf @jyz0309