infoschema/slow_query: fix token too long #10328
Conversation
Codecov Report
@@ Coverage Diff @@
## master #10328 +/- ##
================================================
+ Coverage 77.6692% 77.6875% +0.0183%
================================================
Files 411 411
Lines 85447 85450 +3
================================================
+ Hits 66366 66384 +18
+ Misses 14118 14111 -7
+ Partials 4963 4955 -8 |
Codecov Report
@@ Coverage Diff @@
## master #10328 +/- ##
================================================
+ Coverage 77.3236% 77.3471% +0.0235%
================================================
Files 412 412
Lines 85825 85839 +14
================================================
+ Hits 66363 66394 +31
+ Misses 14412 14402 -10
+ Partials 5050 5043 -7 |
infoschema/slow_log.go
Outdated
| if atomic.LoadUint64(&config.QueryLogMaxLenRecord) > (bufio.MaxScanTokenSize - 100) { | ||
| maxBuf := int(atomic.LoadUint64(&config.QueryLogMaxLenRecord) + 100) | ||
| scanner.Buffer([]byte{}, maxBuf) | ||
| } |
There was a problem hiding this comment.
Can we use bufio.Reader.ReadLine to completely avoid this error?
There was a problem hiding this comment.
We can use ReadLine to implement a new scanner
There was a problem hiding this comment.
Great idea. Other question, should we add a max-single-line-length-limit? If user executes a long query.
At first, I think if we don't limit the max-single-line-length, then parse slow log may consume a lot of memory and then may make TiDB-server killed by the system.
But after careful consideration, I think no need to add max-single-line-length, the first reason is we have log rotate, the other reason is executing the long query may make tidb-server killed in executing. 😂 What do you think? @eurekaka @winkyao
There was a problem hiding this comment.
https://dev.mysql.com/doc/refman/5.7/en/server-system-variables.html#sysvar_max_allowed_packet should limit the input query size?
|
/run-all-tests |
crazycs520
force-pushed
the
fix-slow-log-token-too-long
branch
from
ab58560 to
b4e610e
Compare
|
/run-all-tests |
|
@eurekaka PTAL |
|
/run-all-tests |
| lineByte = append(lineByte, tempLine...) | ||
|
|
||
| // Use the max value of max_allowed_packet to check the single line length. | ||
| if len(lineByte) > int(variable.MaxOfMaxAllowedPacket) { |
There was a problem hiding this comment.
Why using MaxOfMaxAllowedPacket instead of MaxAllowedPacket?
There was a problem hiding this comment.
Because MaxAllowedPacket may has been changed.
|
|
|
/run-all-tests |
|
/run-all-tests |
1 similar comment
|
/run-all-tests |
|
/run-all-tests |
2 similar comments
|
/run-all-tests |
|
/run-all-tests |
|
/run-all-tests |
What problem does this PR solve?
This PR try to fix If there is row length more than
bufio.MaxScanTokenSize, parse slow log will get error:bufio.Scanner: token too long.What is changed and how it works?
Adjust
bufio.Bufferbefore parse slow log and refine error msg.QueryLogMaxLenRecordto record the maxQueryLogMaxLen.Attention: If the user
set @@tidb_query_log_max_len=65536and write a long slow query to slow log, then the tidb-restart. Because theQueryLogMaxLenRecorddoesn't persistent Stores, so it will be theQueryLogMaxLendefault value2048, and then queryINFORMATION_SCHEMA.SLOW_QUERYwill also get error:"read file buffer overflow, please try to enlarge the variable 'tidb_query_log_max_len', but I think the error message is more clear now, and just enlargetidb_query_log_max_lenthen retry.Check List
Tests
Code changes
Side effects
Related changes