*: refactor encoding and uniform usages #30288

tangenta · 2021-11-30T13:36:05Z

What problem does this PR solve?

Issue Number:

close #30429
close #29685
close #30362

Problem Summary:

What is changed and how it works?

Encoding:

Encoding is changed to an interface. Implementations includes EncodingUTF8, EncodingASCII, EncodingLatin1, EncodingBin and EncodingGBK.
StringValidator is merged to Encoding. Encoding.Validate is the new interface.
NewEncoding() is renamed to FindEncoding() since there is no memory allocation anymore.
One more value nSrc is returned from Encode/Decode to indicate how many source bytes are consumed. This is useful in truncation.

Binary Literal Handling:

The built-in functions are classified into 3 kinds of funcProp(function property):
- funcPropNone: No property. The arguments are not specially handled.
- funcPropBinAware: The function depends on the binary representation of its string arguments. It will wrap each argument with to_binary().
- funcPropAuto: There are mulitple string arguments. Implicit cast can happen. It will wrap to_binary() or from_binary() according to charsets.
All the string-related functions are considered.

Built-in functions:

Invalid string characters are handled properly.
builtinCharSig(char()): it only returns null for invalid strings when the SQL mode is strict.
builtinConvertSig(convert()): it reuses the charset.Encoding.
builtinBitLengthSig(bit_length()): the special encoding is implemented through warping to_binary().
builtinOrdSig(ord()): it fallbacks to using the first byte instead of returning encoding error for invalid strings.
Encoding field is removed from lowerFunctionClass and upperFunctionClass without performance loss.

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No code

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

None

ti-chi-bot · 2021-11-30T13:36:06Z

[REVIEW NOTIFICATION]

This pull request has been approved by:

Defined2014
xiongjiwei

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

Defined2014 · 2021-12-01T07:35:50Z

I think this PR cloud close #29685. Maybe we could add those tests in test codes.

bb7133 · 2021-12-07T09:32:58Z

Is this PR WIP?

Also, please have a brief introduction to the refactors.

Defined2014 · 2021-12-07T09:53:05Z

Is this PR WIP?

Also, please have a brief introduction to the refactors.

Add one more linked issue #30362 (comment).

This PR also will close #30362

tangenta · 2021-12-07T12:21:48Z

Is this PR WIP?

Also, please have a brief introduction to the refactors.

@bb7133 Yes. It is WIP because I am still struggling with the failed tests. I will add the PR description as soon as possible.

expression/integration_test.go

sre-bot · 2021-12-20T04:10:26Z

Code Coverage Details: https://codecov.io/github/pingcap/tidb/commit/1aae0922be92312a5dd31639dfd3f21ff700e3a0

tangenta · 2021-12-20T07:10:53Z

/merge

ti-chi-bot · 2021-12-20T07:10:56Z

This pull request has been accepted and is ready to merge.

Commit hash: 4b7cc17

ti-chi-bot · 2021-12-20T07:11:09Z

@tangenta: Your PR was out of date, I have automatically updated it for you.

At the same time I will also trigger all tests for you:

/run-all-tests

If the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

* dumpling: fix default collation with upstream when dump database and table (#30292) * ddl: fix the enum default value by triming trailing space (#30356) * expression: migrate test-infra to testify for flag_simplify_test.go (#30407) * server: refine code logic in handleDownloadFile (#30422) * refine logic Signed-off-by: yisaer <[email protected]> * fix Signed-off-by: yisaer <[email protected]> * ddl: migrate test-infra to testify for ddl/table_test.go (#30267) * ddl: handle the error from `addBatchDDLJobs()` correctly (#30401) * br: fix the integration tests (#30423) * util, cmd: remove unused filesort (#30438) * *: update client-go for small backoff time (#30436) * server: Fix unstable tests with FakeAuthSwitch (#30287) * dumpling: fix dump failed when sequence exists (#30164) * *: replace compareDatum by compare (#30421) * lightning: fix gcs max key limit (#30393) * expression, parser: add built-in func is_uuid (#30318) * expression: migrate test-infra to testify for constant_fold_test.go (#30424) * executor: fix pipelined window invalid memory address (#30418) * makefile: add gotestsum for verify ci (#29848) * server: close sql rows to fix unstable test (#30306) * Makefile: add coverage record for BR and Dumpling (#30457) * executor: track the mem usage of IndexMergeReader (#30210) * infosync: close body when ReadAll encounters error (#30462) * planner: show accessed partition when explain mpp query over partition table (#30367) * *: Fix use of user identity in SHOW GRANTS + error messages (#30294) * ddl: add not null flag for auto_increment column (#30477) * expression: make some unstable test serial (#30323) * expression: migrate test-infra to testify for constant_propagation_test.go (#30430) * executor: stable test TestSetDDLReorgBatchSize and TestSetDDLReorgWorkerCnt (#30480) * statistics, util/ranger: add cardinality estimation trace for `GetRowCountBy...` (#30321) * *: skip mysql client goroutine leak detection in integration ddl (#30467) * executor,util: write slow query to slow log no matter what log level (#30461) * executor: enable index_merge used in transaction. (#29875) * logutil: add testcase for SlowQueryLogger.MaxDays/MaxSize/MaxBackups (#30316) * expression: fix data race in builtin_other_vec_generated_test.go (#30503) * expression: fix data race in the collationInfo (#30490) * planner/core, session: fix error message of wrong variable scope (#30510) * lightning: support Re/ReregisterMySQL by different tls name (#30463) * executor: TestBatchGetandPointGetwithHashPartition test typo (#29669) (#29671) * mockstore: improve log to avoid panic for nil pointer (#30513) * *: replace compareDatum by compare, PR 10 (#30456) * planner: Disable dynamic partition prune mode for all non-autocommit (#27532) (#30505) * expression: change the log level of an confusing log from warn to debug (#30484) * br: Check crypter.key valid before backup (#29991) * *: replace compareDatum by compare, PR 11 (#30465) * dumpling: fix default column collation with upstream when dump table (#30531) * server: fix prepared cursor select (#30285) * executor: HashJoinExec checks the buildError even if the probeSide is empty (#30471) * parser, expression: follow mysql, increase interval precedence (#30528) * makefile: set timeout 25m for make race (#30555) * planner: fix the unstable test TestAnalyzeGlobalStatsWithOpts/2 (#30576) * expression,types: Adjusts UNIX_TIMESTAMP() for non-existing DST values (#28739) (#30405) * br: add res.Body.close to avoid leak (#30545) * lightning: add back integration test lightning_error_summary (#30547) * sessionctx/variable: small refactor (split large file) (#30511) * ddl: let `admin cancel ddl jobs` run in a new transaction (#30549) * *: Retry when placement PutBundles failed (#30590) * dumpling: delete unit test in github actions (#30562) * *: support trace plan target='estimation' statement (#30491) * expression: migrate test-infra to testify for integration_test.go (#30548) * planner: support trace for min/max eliminate (#30441) * support min/max trace Signed-off-by: yisaer <[email protected]> * address the comment Signed-off-by: yisaer <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * br: remove cdclog in br (#30573) * *: show cmd to check if all needed histograms are loaded (#29672) * expression: clone repertoire when clone the scalar function (#30602) * *: use the real StateRemote interface implementation for cached table (#30066) * *: query failed after add index / timestamp out-of-range (#28424) (#29323) * planner: implement collecting predicate columns from logical plan (#29878) * *: show PK name when decoding the clustered index row key (#30623) * ddl/callback_test.go: migrate test-infra to testify (#30317) * *: Rename some names of placement ddl operation (#30622) * executor: fix data race in the index_lookup_hash_join (#30619) * ddl: remove unnecessary locking when adding an index (#29772) * server: try to make `TidbTestSuite` more stable (#30643) * *: Add some PD tests for placement and fix some bug found (#30621) * *: migrate sync.WaitGroup to util.WaitGroupWrapper (#30644) * planner: add trace for join eliminate rule (#30343) * executor: migrate test-infra to testify for executor/shuffle_test.go (#30514) * planner: make (*AccessPath).OnlyPointRange more succinct (#30520) * planner: add trace for join reorder (#30394) * executor: migrate test-infra to testify for executor/union_scan_test.go (#30525) * expression: make cast return error if cast binary literal to another character set (#30537) * *: update tikv client (#30670) * *: update sysutil in go.mod to fix panic when search log (#30523) * topsql: shouldn't evict the SQL meta, since the evicted SQL can be appear on Other components (TiKV) TopN records (#27050) * testify: migrate test-infra to testify for analyze_test.go (#30640) * util: replace compareDatum by compare, point part (#30575) * test: make all the tests run in serial (#30692) * statistics: add mutex for Handle.globalMap and Handle.feedback (#30550) * executor: fix regular expression in json so that it could match identifer start with '$' (#29750) * util/testkit/testkit.go: fix typo (#30638) * planner: Introduce a new global variable to control the historical statistics feature (#30646) * topsql: introduce datasink interface (#30662) * planner: unify the argument of stats functions to use SessionCtx instead of StatementContext (#30668) * metrics: fix the Max SafeTS Gap metrics (#30689) * lightning: Add source dir existence check for s3 (#30674) * golangci-lint: support durationcheck (#30027) * executor: fix data race on IndexHashJoin.cancelFunc (#30701) * sessionctx/variable: change tidb_store_limit to global only (#30522) * statistics: remove reassignment of Handle.pool in NewHandle (#30675) * br: fix some unstable unit test cases. (#30716) * bindinfo: fix the comment typo (#30616) * server: support decoding prepared string args to character_set_client (#30723) * expression: fix enum type join binary get wrong result (#30445) * cmd/explaintest: fix wrong result comparison for explain test (#30717) * parallel create tables in br * metrics: fix copr-cache metrics (#30712) * test: merge executor's serial tests to other tests (#30711) * statistics: avoid deadlock when create/drop extended stats and analyze at the same time (#30566) * ddl: add batch create table api Signed-off-by: xhe <[email protected]> * ddl: add unit tests Signed-off-by: xhe <[email protected]> * ddl: fix fmt Signed-off-by: xhe <[email protected]> * ddl: typo Co-authored-by: Arenatlx <[email protected]> * ddl: fix tests Signed-off-by: xhe <[email protected]> * ddl: rename to BatchCreateTableWithInfo Signed-off-by: xhe <[email protected]> * ddl: trace the error Signed-off-by: xhe <[email protected]> * ddl: comments Signed-off-by: xhe <[email protected]> * ddl: cancle the job right Signed-off-by: xhe <[email protected]> * ddl: cancel the job right 2 Signed-off-by: xhe <[email protected]> * ddl: report error if entry too large Signed-off-by: xhe <[email protected]> * ddl: report error when table is duplicated Signed-off-by: xhe <[email protected]> * ddl: go fmt Signed-off-by: xhe <[email protected]> * infoschema: improve batch memory perf Signed-off-by: xhe <[email protected]> * ddl: retain ID Signed-off-by: xhe <[email protected]> * sessionctx: fix the value of analyze_version when upgrading 4.x to 5.… (#30743) * ddl: reduce log frequency Signed-off-by: xhe <[email protected]> * ddl: fix tests Signed-off-by: xhe <[email protected]> * server: disable socket listener for `basicHTTPHandlerTestSuite` (#30680) * planner: support the plan cache aware of bindings (#30169) * planner: fix early set of plan's statisticsTable (#30754) * *: implement renew write lock lease for cached table (#30206) * *: Modify placement rule index to reserve some indexes for future work (#30737) * executor: add an unit test case for unreasonable invoking Close (#30696) * planner: fix wrong subquery's coercibility (#30750) * executor: add more testcases for index merge (#30497) * server: add grpc server config for a suitable behavior (#30774) * config, charset: make charset config not affected by collation config (#30572) * lightning: emit tidb log by change FilterCore to only allow matched packages (#30700) * topsql: a centralized place to generate tipb report data (#30781) * planner: add trace for partition pruning (#30546) * planner: refine collation handling for between (#30793) * test: merge serial tests in bindinfo, expression, parser and statistics (#30749) * br: update log description for split check (#30763) * *: replace compareDatum by compare, range part (#30710) * *: placement policy ref will be converted to direct options when recover or flashback table (#30705) * ddl: handle the incorrect number of placement followers (#30715) * ddl: revert "ddl: remove unnecessary locking when adding an index" (#30667) * br/pkg/task: migrate test-infra to testify (#30605) * *: fix the flen type datetime for union/case-when/control-funcs (#30588) * types, util: clean up compareDatum (#30815) * ddl: add helper function to set and query TiFlash's sync status (#30473) * dumpling: fix more dumpling log level query template (#30176) * parser: support `admin flush plan_cache` (#30747) * topsql: support multiple datasinks (#30808) * br: update permission, so tikv can write to folder when backup to local (#30396) * session: fix bootstrap to only persist global variables (#30593) close #28667 * docs/design: update collation compatibility issues in charsets doc (#30806) * executor: improve SET sysvar=DEFAULT handling (#29680) close #29670 * br: add error handling for group context cancel when restore file is corrupted (#30190) close #30135 * executor: buildWindow cannot call typeInfer twice (#30773) close #30402 * *: refactor encoding and uniform usages (#30288) * lightning: optimize region split check logic (#30428) close #30018 * br: ignore mock directory when gcov in br (#30586) * *: forbid set tiflash replica count for a placement table (#30844) close #30741 * execute: don't transform charset in internal sql (#30843) close #30789 * planner: update PlanBuilder.windowSpecs when building subquery (#30878) close #30804 * br: fix S3 backup endpoint suffix (#30530) close #30104 * lightning: make pre-check output message clearer (#30439) close #30395 * expression: wrap to_binary and from_binary for cast function's argument (#30706) * executor: fix bug when using IndexMerge in transaction (#30719) close #30685 * ddl: migrate test-infra to testify for ddl/foreign_key_test.go (#30853) close #29103 * expression: fix wrong retType for reverse function (#30829) close #30809 * planner: support trace topn push down (#30800) ref #29661 * github: add issue requirement to pull request template (#30817) close #30814 * fix merge issue * topsql: introduce stmtstats and sql execution count (#30277) * topsql: add pubsub datasink (#30860) * executor: fix the incorrect untouch used in optimistic transactions (#30447) close #30410 * expression, cmd: let crc32() support gbk (#30900) close #30898 * server: Add uptime status var and statistics (#29790) close #8842 * br: error log optimization (#29640) close #27015 * planner: fix wrong collation when rewrite in condition (#30492) close #30486 * planner: add extractor for tikv_region_peers (#30656) * fix issue that loss table restore * lightning: add back table empty check and add a switch config (#30887) close #27919 * br: improve backoff unit test (#30892) * *: add TxnManager to manage txn in session (#30574) * *: add TxnManager to manage txn in session * modify * add tests * move failpoint content to a single file * Makefile: add `t.Parallel` check to ensure tests are run in serial (#30869) * refactoring code * refactoring code * placement: remove isolationlevel (#30859) close #30858 * planner: revise the optimize trace output (#30882) * table: set the datum collation correctly in CastValue() (#30931) close #30930 * *: Use TxnManager.GetTxnInfoSchema() to get the txn infoschema (#30934) close #30933 * parser: add IsValid() to Encoding to speed up string validation for UTF-8 (#30937) close #30936 * planner: rename pstmtPlanCacheXX to PlanCacheXX (#30909) * table/tables: make CI TestCacheTableBasicReadAndWrite more stable (#30924) close #30922 * restore: use new ScatterRegions API (#30899) close #30425 * *: when placement conflicts with tiflash, cancel the job (#30945) * Makefile,tools: make CI great again! (#30828) close #30822 * br/pkg/membuf: remove global buffer pool (#29934) * ddl: add format error for incorrect dict syntax in the placement rule (#30919) close #30454 * planner: fix index merge plan when expr cannot be pushed to tikv (#30341) close #30200 * executor: display 'show create table' and INFOSCHEMA for cached table correctly (#30951) close #30950 * br: extend the timeout for scan region since 3 seconds is not enough (#30889) close #30720 * planner: remove bindSQL from planCacheKey to planCacheValue (#30916) * execution: refine precision of cast as decimal in agg func (#30805) * *: fix data race in the tikv_client (#30964) close #30658 * ddl: migrate test-infra to testify for ddl/db_partition_test.go (#30952) close #28635 * planner: fix `AccessPath.TableFilters` got modified unexpectedly (#30966) close #30965 * test: merge serial tests in ddl, infoschema, session, store, table, telemetry and types (#30874) * executor: fix the returned field count of the prepare statement (#30981) close #30971 * binlog: allow multiple ddl targets (#30904) * planner: trace predicate push down (#30902) ref #29661 * placement: give default 2 followers for non-sugar syntax (#31000) * flatten the json output (#30905) Signed-off-by: yisaer <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * test: control log level with environment variables (#30871) * planner: add usage of paging copr in optimizer (#30536) close #30578 * test: merge serial tests in cmd, planner, server, util (#31003) * planner: change predicateColumnCollector to columnStatsUsageCollector and collect histogram-needed columns (#30671) * executor: migrate test-infra to testify for distsql_test.go (#31023) close #28574 * remote uncessary package errors * reused the retry code from lightning * refactoring retryable * ddl: add batch create table api Signed-off-by: xhe <[email protected]> * ddl: add unit tests Signed-off-by: xhe <[email protected]> * ddl: fix fmt Signed-off-by: xhe <[email protected]> * br ddl code * parallel create tables in br * ddl: add batch create table api Signed-off-by: xhe <[email protected]> * ddl: add unit tests Signed-off-by: xhe <[email protected]> * ddl: fix fmt Signed-off-by: xhe <[email protected]> * ddl: typo Co-authored-by: Arenatlx <[email protected]> * ddl: fix tests Signed-off-by: xhe <[email protected]> * ddl: rename to BatchCreateTableWithInfo Signed-off-by: xhe <[email protected]> * ddl: trace the error Signed-off-by: xhe <[email protected]> * ddl: comments Signed-off-by: xhe <[email protected]> * ddl: cancle the job right Signed-off-by: xhe <[email protected]> * ddl: cancel the job right 2 Signed-off-by: xhe <[email protected]> * ddl: report error if entry too large Signed-off-by: xhe <[email protected]> * ddl: report error when table is duplicated Signed-off-by: xhe <[email protected]> * ddl: go fmt Signed-off-by: xhe <[email protected]> * infoschema: improve batch memory perf Signed-off-by: xhe <[email protected]> * ddl: retain ID Signed-off-by: xhe <[email protected]> * ddl: reduce log frequency Signed-off-by: xhe <[email protected]> * ddl: fix tests Signed-off-by: xhe <[email protected]> * ddl: remove retainID from the interface Signed-off-by: xhe <[email protected]> * ddl: fix tests Signed-off-by: xhe <[email protected]> * executor: fix rebasing problem Signed-off-by: xhe <[email protected]> Co-authored-by: WizardXiao <[email protected]> Co-authored-by: sylzd <[email protected]> Co-authored-by: tison <[email protected]> Co-authored-by: Song Gao <[email protected]> Co-authored-by: Weizhen Wang <[email protected]> Co-authored-by: tangenta <[email protected]> Co-authored-by: 3pointer <[email protected]> Co-authored-by: wjHuang <[email protected]> Co-authored-by: Lei Zhao <[email protected]> Co-authored-by: Daniël van Eeden <[email protected]> Co-authored-by: Jianjun Liao <[email protected]> Co-authored-by: unconsolable <[email protected]> Co-authored-by: Shenghui Wu <[email protected]> Co-authored-by: guo-shaoge <[email protected]> Co-authored-by: Ryan Leung <[email protected]> Co-authored-by: xufei <[email protected]> Co-authored-by: Morgan Tocker <[email protected]> Co-authored-by: Zhou Kunqin <[email protected]> Co-authored-by: 王超 <[email protected]> Co-authored-by: TonsnakeLin <[email protected]> Co-authored-by: Ehco <[email protected]> Co-authored-by: Mattias Jonsson <[email protected]> Co-authored-by: HuaiyuXu <[email protected]> Co-authored-by: Zak Zhao <[email protected]> Co-authored-by: xhe <[email protected]> Co-authored-by: Hangjie Mo <[email protected]> Co-authored-by: Yuanjia Zhang <[email protected]> Co-authored-by: glorv <[email protected]> Co-authored-by: djshow832 <[email protected]> Co-authored-by: Chunzhu Li <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> Co-authored-by: Xiaoju Wu <[email protected]> Co-authored-by: xiongjiwei <[email protected]> Co-authored-by: tiancaiamao <[email protected]> Co-authored-by: Yifan Xu <[email protected]> Co-authored-by: JmPotato <[email protected]> Co-authored-by: Zach <[email protected]> Co-authored-by: bb7133 <[email protected]> Co-authored-by: lvtu <[email protected]> Co-authored-by: crazycs <[email protected]> Co-authored-by: znhh6018 <[email protected]> Co-authored-by: eddie lin <[email protected]> Co-authored-by: dongjunduo <[email protected]> Co-authored-by: Zhenchi <[email protected]> Co-authored-by: wangggong <[email protected]> Co-authored-by: zhangjinpeng1987 <[email protected]> Co-authored-by: Jack Yu <[email protected]> Co-authored-by: Arenatlx <[email protected]> Co-authored-by: Yiding Cui <[email protected]> Co-authored-by: Chengpeng Yan <[email protected]> Co-authored-by: bestwoody <[email protected]> Co-authored-by: Calvin Neo <[email protected]> Co-authored-by: Lynn <[email protected]> Co-authored-by: Zhuhe Fang <[email protected]> Co-authored-by: Mini256 <[email protected]> Co-authored-by: Xiang Zhang <[email protected]> Co-authored-by: Yexiang Zhang <[email protected]> Co-authored-by: cfzjywxk <[email protected]> Co-authored-by: db <[email protected]> Co-authored-by: 山岚 <[email protected]> Co-authored-by: Yujie Xia <[email protected]> Co-authored-by: Yilong Li <[email protected]> Co-authored-by: tuuuuuu <[email protected]> Co-authored-by: qupeng <[email protected]> Co-authored-by: you06 <[email protected]>

* br: fix the integration tests (#30423) * util, cmd: remove unused filesort (#30438) * *: update client-go for small backoff time (#30436) * server: Fix unstable tests with FakeAuthSwitch (#30287) * dumpling: fix dump failed when sequence exists (#30164) * *: replace compareDatum by compare (#30421) * lightning: fix gcs max key limit (#30393) * expression, parser: add built-in func is_uuid (#30318) * expression: migrate test-infra to testify for constant_fold_test.go (#30424) * executor: fix pipelined window invalid memory address (#30418) * makefile: add gotestsum for verify ci (#29848) * server: close sql rows to fix unstable test (#30306) * Makefile: add coverage record for BR and Dumpling (#30457) * executor: track the mem usage of IndexMergeReader (#30210) * infosync: close body when ReadAll encounters error (#30462) * planner: show accessed partition when explain mpp query over partition table (#30367) * *: Fix use of user identity in SHOW GRANTS + error messages (#30294) * ddl: add not null flag for auto_increment column (#30477) * expression: make some unstable test serial (#30323) * expression: migrate test-infra to testify for constant_propagation_test.go (#30430) * executor: stable test TestSetDDLReorgBatchSize and TestSetDDLReorgWorkerCnt (#30480) * statistics, util/ranger: add cardinality estimation trace for `GetRowCountBy...` (#30321) * *: skip mysql client goroutine leak detection in integration ddl (#30467) * executor,util: write slow query to slow log no matter what log level (#30461) * executor: enable index_merge used in transaction. (#29875) * logutil: add testcase for SlowQueryLogger.MaxDays/MaxSize/MaxBackups (#30316) * expression: fix data race in builtin_other_vec_generated_test.go (#30503) * expression: fix data race in the collationInfo (#30490) * planner/core, session: fix error message of wrong variable scope (#30510) * lightning: support Re/ReregisterMySQL by different tls name (#30463) * executor: TestBatchGetandPointGetwithHashPartition test typo (#29669) (#29671) * mockstore: improve log to avoid panic for nil pointer (#30513) * *: replace compareDatum by compare, PR 10 (#30456) * planner: Disable dynamic partition prune mode for all non-autocommit (#27532) (#30505) * expression: change the log level of an confusing log from warn to debug (#30484) * br: Check crypter.key valid before backup (#29991) * *: replace compareDatum by compare, PR 11 (#30465) * dumpling: fix default column collation with upstream when dump table (#30531) * server: fix prepared cursor select (#30285) * executor: HashJoinExec checks the buildError even if the probeSide is empty (#30471) * parser, expression: follow mysql, increase interval precedence (#30528) * makefile: set timeout 25m for make race (#30555) * planner: fix the unstable test TestAnalyzeGlobalStatsWithOpts/2 (#30576) * expression,types: Adjusts UNIX_TIMESTAMP() for non-existing DST values (#28739) (#30405) * br: add res.Body.close to avoid leak (#30545) * lightning: add back integration test lightning_error_summary (#30547) * sessionctx/variable: small refactor (split large file) (#30511) * ddl: let `admin cancel ddl jobs` run in a new transaction (#30549) * *: Retry when placement PutBundles failed (#30590) * dumpling: delete unit test in github actions (#30562) * *: support trace plan target='estimation' statement (#30491) * expression: migrate test-infra to testify for integration_test.go (#30548) * planner: support trace for min/max eliminate (#30441) * support min/max trace Signed-off-by: yisaer <[email protected]> * address the comment Signed-off-by: yisaer <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * br: remove cdclog in br (#30573) * *: show cmd to check if all needed histograms are loaded (#29672) * expression: clone repertoire when clone the scalar function (#30602) * *: use the real StateRemote interface implementation for cached table (#30066) * *: query failed after add index / timestamp out-of-range (#28424) (#29323) * planner: implement collecting predicate columns from logical plan (#29878) * *: show PK name when decoding the clustered index row key (#30623) * ddl/callback_test.go: migrate test-infra to testify (#30317) * *: Rename some names of placement ddl operation (#30622) * executor: fix data race in the index_lookup_hash_join (#30619) * ddl: remove unnecessary locking when adding an index (#29772) * server: try to make `TidbTestSuite` more stable (#30643) * *: Add some PD tests for placement and fix some bug found (#30621) * *: migrate sync.WaitGroup to util.WaitGroupWrapper (#30644) * planner: add trace for join eliminate rule (#30343) * executor: migrate test-infra to testify for executor/shuffle_test.go (#30514) * planner: make (*AccessPath).OnlyPointRange more succinct (#30520) * planner: add trace for join reorder (#30394) * executor: migrate test-infra to testify for executor/union_scan_test.go (#30525) * expression: make cast return error if cast binary literal to another character set (#30537) * *: update tikv client (#30670) * *: update sysutil in go.mod to fix panic when search log (#30523) * topsql: shouldn't evict the SQL meta, since the evicted SQL can be appear on Other components (TiKV) TopN records (#27050) * testify: migrate test-infra to testify for analyze_test.go (#30640) * util: replace compareDatum by compare, point part (#30575) * test: make all the tests run in serial (#30692) * statistics: add mutex for Handle.globalMap and Handle.feedback (#30550) * executor: fix regular expression in json so that it could match identifer start with '$' (#29750) * util/testkit/testkit.go: fix typo (#30638) * planner: Introduce a new global variable to control the historical statistics feature (#30646) * topsql: introduce datasink interface (#30662) * planner: unify the argument of stats functions to use SessionCtx instead of StatementContext (#30668) * metrics: fix the Max SafeTS Gap metrics (#30689) * lightning: Add source dir existence check for s3 (#30674) * golangci-lint: support durationcheck (#30027) * executor: fix data race on IndexHashJoin.cancelFunc (#30701) * sessionctx/variable: change tidb_store_limit to global only (#30522) * statistics: remove reassignment of Handle.pool in NewHandle (#30675) * br: fix some unstable unit test cases. (#30716) * bindinfo: fix the comment typo (#30616) * server: support decoding prepared string args to character_set_client (#30723) * expression: fix enum type join binary get wrong result (#30445) * cmd/explaintest: fix wrong result comparison for explain test (#30717) * parallel create tables in br * metrics: fix copr-cache metrics (#30712) * test: merge executor's serial tests to other tests (#30711) * statistics: avoid deadlock when create/drop extended stats and analyze at the same time (#30566) * ddl: add batch create table api Signed-off-by: xhe <[email protected]> * ddl: add unit tests Signed-off-by: xhe <[email protected]> * ddl: fix fmt Signed-off-by: xhe <[email protected]> * ddl: typo Co-authored-by: Arenatlx <[email protected]> * ddl: fix tests Signed-off-by: xhe <[email protected]> * ddl: rename to BatchCreateTableWithInfo Signed-off-by: xhe <[email protected]> * ddl: trace the error Signed-off-by: xhe <[email protected]> * ddl: comments Signed-off-by: xhe <[email protected]> * ddl: cancle the job right Signed-off-by: xhe <[email protected]> * ddl: cancel the job right 2 Signed-off-by: xhe <[email protected]> * ddl: report error if entry too large Signed-off-by: xhe <[email protected]> * ddl: report error when table is duplicated Signed-off-by: xhe <[email protected]> * ddl: go fmt Signed-off-by: xhe <[email protected]> * infoschema: improve batch memory perf Signed-off-by: xhe <[email protected]> * ddl: retain ID Signed-off-by: xhe <[email protected]> * sessionctx: fix the value of analyze_version when upgrading 4.x to 5.… (#30743) * ddl: reduce log frequency Signed-off-by: xhe <[email protected]> * ddl: fix tests Signed-off-by: xhe <[email protected]> * server: disable socket listener for `basicHTTPHandlerTestSuite` (#30680) * planner: support the plan cache aware of bindings (#30169) * planner: fix early set of plan's statisticsTable (#30754) * *: implement renew write lock lease for cached table (#30206) * *: Modify placement rule index to reserve some indexes for future work (#30737) * executor: add an unit test case for unreasonable invoking Close (#30696) * planner: fix wrong subquery's coercibility (#30750) * executor: add more testcases for index merge (#30497) * server: add grpc server config for a suitable behavior (#30774) * config, charset: make charset config not affected by collation config (#30572) * lightning: emit tidb log by change FilterCore to only allow matched packages (#30700) * topsql: a centralized place to generate tipb report data (#30781) * planner: add trace for partition pruning (#30546) * planner: refine collation handling for between (#30793) * test: merge serial tests in bindinfo, expression, parser and statistics (#30749) * br: update log description for split check (#30763) * *: replace compareDatum by compare, range part (#30710) * *: placement policy ref will be converted to direct options when recover or flashback table (#30705) * ddl: handle the incorrect number of placement followers (#30715) * ddl: revert "ddl: remove unnecessary locking when adding an index" (#30667) * br/pkg/task: migrate test-infra to testify (#30605) * *: fix the flen type datetime for union/case-when/control-funcs (#30588) * types, util: clean up compareDatum (#30815) * ddl: add helper function to set and query TiFlash's sync status (#30473) * dumpling: fix more dumpling log level query template (#30176) * parser: support `admin flush plan_cache` (#30747) * topsql: support multiple datasinks (#30808) * br: update permission, so tikv can write to folder when backup to local (#30396) * session: fix bootstrap to only persist global variables (#30593) close #28667 * docs/design: update collation compatibility issues in charsets doc (#30806) * executor: improve SET sysvar=DEFAULT handling (#29680) close #29670 * br: add error handling for group context cancel when restore file is corrupted (#30190) close #30135 * executor: buildWindow cannot call typeInfer twice (#30773) close #30402 * *: refactor encoding and uniform usages (#30288) * lightning: optimize region split check logic (#30428) close #30018 * br: ignore mock directory when gcov in br (#30586) * *: forbid set tiflash replica count for a placement table (#30844) close #30741 * execute: don't transform charset in internal sql (#30843) close #30789 * planner: update PlanBuilder.windowSpecs when building subquery (#30878) close #30804 * br: fix S3 backup endpoint suffix (#30530) close #30104 * lightning: make pre-check output message clearer (#30439) close #30395 * expression: wrap to_binary and from_binary for cast function's argument (#30706) * executor: fix bug when using IndexMerge in transaction (#30719) close #30685 * ddl: migrate test-infra to testify for ddl/foreign_key_test.go (#30853) close #29103 * expression: fix wrong retType for reverse function (#30829) close #30809 * planner: support trace topn push down (#30800) ref #29661 * github: add issue requirement to pull request template (#30817) close #30814 * fix merge issue * topsql: introduce stmtstats and sql execution count (#30277) * topsql: add pubsub datasink (#30860) * executor: fix the incorrect untouch used in optimistic transactions (#30447) close #30410 * expression, cmd: let crc32() support gbk (#30900) close #30898 * server: Add uptime status var and statistics (#29790) close #8842 * br: error log optimization (#29640) close #27015 * planner: fix wrong collation when rewrite in condition (#30492) close #30486 * planner: add extractor for tikv_region_peers (#30656) * fix issue that loss table restore * lightning: add back table empty check and add a switch config (#30887) close #27919 * br: improve backoff unit test (#30892) * *: add TxnManager to manage txn in session (#30574) * *: add TxnManager to manage txn in session * modify * add tests * move failpoint content to a single file * Makefile: add `t.Parallel` check to ensure tests are run in serial (#30869) * refactoring code * refactoring code * placement: remove isolationlevel (#30859) close #30858 * planner: revise the optimize trace output (#30882) * table: set the datum collation correctly in CastValue() (#30931) close #30930 * *: Use TxnManager.GetTxnInfoSchema() to get the txn infoschema (#30934) close #30933 * parser: add IsValid() to Encoding to speed up string validation for UTF-8 (#30937) close #30936 * planner: rename pstmtPlanCacheXX to PlanCacheXX (#30909) * table/tables: make CI TestCacheTableBasicReadAndWrite more stable (#30924) close #30922 * restore: use new ScatterRegions API (#30899) close #30425 * *: when placement conflicts with tiflash, cancel the job (#30945) * Makefile,tools: make CI great again! (#30828) close #30822 * br/pkg/membuf: remove global buffer pool (#29934) * ddl: add format error for incorrect dict syntax in the placement rule (#30919) close #30454 * planner: fix index merge plan when expr cannot be pushed to tikv (#30341) close #30200 * executor: display 'show create table' and INFOSCHEMA for cached table correctly (#30951) close #30950 * br: extend the timeout for scan region since 3 seconds is not enough (#30889) close #30720 * planner: remove bindSQL from planCacheKey to planCacheValue (#30916) * execution: refine precision of cast as decimal in agg func (#30805) * *: fix data race in the tikv_client (#30964) close #30658 * ddl: migrate test-infra to testify for ddl/db_partition_test.go (#30952) close #28635 * planner: fix `AccessPath.TableFilters` got modified unexpectedly (#30966) close #30965 * test: merge serial tests in ddl, infoschema, session, store, table, telemetry and types (#30874) * executor: fix the returned field count of the prepare statement (#30981) close #30971 * binlog: allow multiple ddl targets (#30904) * planner: trace predicate push down (#30902) ref #29661 * placement: give default 2 followers for non-sugar syntax (#31000) * flatten the json output (#30905) Signed-off-by: yisaer <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * test: control log level with environment variables (#30871) * planner: add usage of paging copr in optimizer (#30536) close #30578 * test: merge serial tests in cmd, planner, server, util (#31003) * planner: change predicateColumnCollector to columnStatsUsageCollector and collect histogram-needed columns (#30671) * executor: migrate test-infra to testify for distsql_test.go (#31023) close #28574 * remote uncessary package errors * reused the retry code from lightning * refactoring retryable * ddl: add batch create table api Signed-off-by: xhe <[email protected]> * ddl: add unit tests Signed-off-by: xhe <[email protected]> * ddl: fix fmt Signed-off-by: xhe <[email protected]> * br ddl code * parallel create tables in br * ddl: add batch create table api Signed-off-by: xhe <[email protected]> * ddl: add unit tests Signed-off-by: xhe <[email protected]> * ddl: fix fmt Signed-off-by: xhe <[email protected]> * ddl: typo Co-authored-by: Arenatlx <[email protected]> * ddl: fix tests Signed-off-by: xhe <[email protected]> * ddl: rename to BatchCreateTableWithInfo Signed-off-by: xhe <[email protected]> * ddl: trace the error Signed-off-by: xhe <[email protected]> * ddl: comments Signed-off-by: xhe <[email protected]> * ddl: cancle the job right Signed-off-by: xhe <[email protected]> * ddl: cancel the job right 2 Signed-off-by: xhe <[email protected]> * ddl: report error if entry too large Signed-off-by: xhe <[email protected]> * ddl: report error when table is duplicated Signed-off-by: xhe <[email protected]> * ddl: go fmt Signed-off-by: xhe <[email protected]> * infoschema: improve batch memory perf Signed-off-by: xhe <[email protected]> * ddl: retain ID Signed-off-by: xhe <[email protected]> * ddl: reduce log frequency Signed-off-by: xhe <[email protected]> * ddl: fix tests Signed-off-by: xhe <[email protected]> * ddl: remove retainID from the interface Signed-off-by: xhe <[email protected]> * ddl: fix tests Signed-off-by: xhe <[email protected]> * executor: fix rebasing problem Signed-off-by: xhe <[email protected]> * sessionctx: enable IndexMerge by default (#30650) close #29597 * br: Enable lint `gosec` in br (#30895) close #30699 * planner: support 'admin flush plan cache' (#30370) * merge from tidb batch_1 Co-authored-by: 3pointer <[email protected]> Co-authored-by: wjHuang <[email protected]> Co-authored-by: Lei Zhao <[email protected]> Co-authored-by: Daniël van Eeden <[email protected]> Co-authored-by: sylzd <[email protected]> Co-authored-by: Jianjun Liao <[email protected]> Co-authored-by: unconsolable <[email protected]> Co-authored-by: tison <[email protected]> Co-authored-by: Shenghui Wu <[email protected]> Co-authored-by: tangenta <[email protected]> Co-authored-by: Weizhen Wang <[email protected]> Co-authored-by: guo-shaoge <[email protected]> Co-authored-by: Ryan Leung <[email protected]> Co-authored-by: xufei <[email protected]> Co-authored-by: Morgan Tocker <[email protected]> Co-authored-by: Zhou Kunqin <[email protected]> Co-authored-by: 王超 <[email protected]> Co-authored-by: TonsnakeLin <[email protected]> Co-authored-by: Ehco <[email protected]> Co-authored-by: Mattias Jonsson <[email protected]> Co-authored-by: HuaiyuXu <[email protected]> Co-authored-by: Zak Zhao <[email protected]> Co-authored-by: WizardXiao <[email protected]> Co-authored-by: xhe <[email protected]> Co-authored-by: Hangjie Mo <[email protected]> Co-authored-by: Yuanjia Zhang <[email protected]> Co-authored-by: glorv <[email protected]> Co-authored-by: djshow832 <[email protected]> Co-authored-by: Chunzhu Li <[email protected]> Co-authored-by: Song Gao <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> Co-authored-by: Xiaoju Wu <[email protected]> Co-authored-by: xiongjiwei <[email protected]> Co-authored-by: tiancaiamao <[email protected]> Co-authored-by: Yifan Xu <[email protected]> Co-authored-by: JmPotato <[email protected]> Co-authored-by: Zach <[email protected]> Co-authored-by: bb7133 <[email protected]> Co-authored-by: lvtu <[email protected]> Co-authored-by: crazycs <[email protected]> Co-authored-by: znhh6018 <[email protected]> Co-authored-by: eddie lin <[email protected]> Co-authored-by: dongjunduo <[email protected]> Co-authored-by: Zhenchi <[email protected]> Co-authored-by: wangggong <[email protected]> Co-authored-by: zhangjinpeng1987 <[email protected]> Co-authored-by: Jack Yu <[email protected]> Co-authored-by: Arenatlx <[email protected]> Co-authored-by: Yiding Cui <[email protected]> Co-authored-by: Chengpeng Yan <[email protected]> Co-authored-by: bestwoody <[email protected]> Co-authored-by: Calvin Neo <[email protected]> Co-authored-by: Lynn <[email protected]> Co-authored-by: Zhuhe Fang <[email protected]> Co-authored-by: Mini256 <[email protected]> Co-authored-by: Xiang Zhang <[email protected]> Co-authored-by: Yexiang Zhang <[email protected]> Co-authored-by: cfzjywxk <[email protected]> Co-authored-by: db <[email protected]> Co-authored-by: 山岚 <[email protected]> Co-authored-by: Yujie Xia <[email protected]> Co-authored-by: Yilong Li <[email protected]> Co-authored-by: tuuuuuu <[email protected]> Co-authored-by: qupeng <[email protected]> Co-authored-by: you06 <[email protected]>

expression, parser: refactor convert to reuse string validator

b8ff109

ti-chi-bot added release-note-none Denotes a PR that doesn't merit a release note. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Nov 30, 2021

tangenta changed the title ~~expression, parser: refactor convert to reuse string validator~~ expression, parser: refactor convert() to reuse string validator Nov 30, 2021

tangenta added 2 commits November 30, 2021 22:55

expression: fix import not used

50c7b05

expression: fix indent-error-flow error

53acd18

remove convert() from from_binary() list

5271d3a

Defined2014 mentioned this pull request Dec 2, 2021

expression: add check for utf8 charset when decode to avoid invalid character #29765

Closed

12 tasks

refactor and fix all the problems

63cc99e

ti-chi-bot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Dec 7, 2021

tangenta added 2 commits December 7, 2021 13:41

add some comment and fix linter

6036f17

Merge remote-tracking branch 'upstream/master' into refactor-convert

1ef42a5

tangenta changed the title ~~expression, parser: refactor convert() to reuse string validator~~ *: refactor encoding and uniform usages Dec 7, 2021

This was referenced Dec 7, 2021

expression: only wrap *_binary() builtin functions in new charset #30216

Closed

expression: make built-in function ord() backward compatible #30177

Closed

tangenta added 4 commits December 7, 2021 20:22

fix integration test and cleanup

bb37c41

complete the binary functions list and update test for Encoding.Validate

fb3158c

Merge remote-tracking branch 'upstream/master' into refactor-convert

b1e13ce

fix integration test TestConvert and TestCIWeightString

ef3ecff

pingcap deleted a comment from sre-bot Dec 7, 2021

tangenta requested review from zimulala, xiongjiwei and Defined2014 December 8, 2021 02:58

fix char() in strict mode and add test

6015aa7

tangenta added 3 commits December 17, 2021 16:07

cast string to charset_connection before inserting into blob fields

70ec0f2

address comment

fd35ea3

Merge remote-tracking branch 'upstream/master' into refactor-convert

3ee9236

ti-chi-bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Dec 17, 2021

tangenta added 4 commits December 17, 2021 16:41

fix build

aff0d02

not to treat 0x80 as the single gbk character

18befa0

reset names in explaintest to keep test stable

1cc6321

revert to_binary before insert for blob columns

7e82fe3

xiongjiwei reviewed Dec 20, 2021

View reviewed changes

expression/integration_test.go Outdated Show resolved Hide resolved

remove t.parallel()

4b7cc17

xiongjiwei approved these changes Dec 20, 2021

View reviewed changes

ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Dec 20, 2021

Defined2014 approved these changes Dec 20, 2021

View reviewed changes

ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Dec 20, 2021

ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Dec 20, 2021

Merge branch 'master' into refactor-convert

1aae092

ti-chi-bot merged commit ab35db1 into pingcap:master Dec 20, 2021

This was referenced Dec 22, 2021

"get old row failed" when replace into case-insensitive primary key #30930

Closed

BenchmarkInsertIntoSelect regression #30936

Closed

Yui-Song mentioned this pull request Dec 24, 2021

PR #30288 results in 5.4%-10% performance regression in read-heavy workloads #31014

Closed

tangenta mentioned this pull request Dec 28, 2021

encoding: skip utf8 charset validation in some cases #31061

Merged

12 tasks

zimulala mentioned this pull request Jan 12, 2022

Support GBK for string-related built-in functions #26816

Closed

16 tasks

solotzg mentioned this pull request Jun 22, 2022

HEX + CONVERT / INSERT with utf8 returns different values with MySQL #32599

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

*: refactor encoding and uniform usages #30288

*: refactor encoding and uniform usages #30288

tangenta commented Nov 30, 2021 •

edited

Loading

ti-chi-bot commented Nov 30, 2021 •

edited

Loading

Defined2014 commented Dec 1, 2021

bb7133 commented Dec 7, 2021

Defined2014 commented Dec 7, 2021

tangenta commented Dec 7, 2021

sre-bot commented Dec 20, 2021 •

edited

Loading

tangenta commented Dec 20, 2021

ti-chi-bot commented Dec 20, 2021

ti-chi-bot commented Dec 20, 2021

*: refactor encoding and uniform usages #30288

*: refactor encoding and uniform usages #30288

Conversation

tangenta commented Nov 30, 2021 • edited Loading

What problem does this PR solve?

What is changed and how it works?

Check List

Release note

ti-chi-bot commented Nov 30, 2021 • edited Loading

Defined2014 commented Dec 1, 2021

bb7133 commented Dec 7, 2021

Defined2014 commented Dec 7, 2021

tangenta commented Dec 7, 2021

sre-bot commented Dec 20, 2021 • edited Loading

tangenta commented Dec 20, 2021

ti-chi-bot commented Dec 20, 2021

ti-chi-bot commented Dec 20, 2021

tangenta commented Nov 30, 2021 •

edited

Loading

ti-chi-bot commented Nov 30, 2021 •

edited

Loading

sre-bot commented Dec 20, 2021 •

edited

Loading