lightning: let ignore columns be compatible with tidb backend #27850

glorv · 2021-09-07T04:24:43Z

What problem does this PR solve?

Let lightning config mydumper.ignore-columns be compatible with tidb backend.

Issue Number: close #xxx

Problem Summary:

What is changed and how it works?

Let the kv packend generated by encodeLoop contains the column names with ignore columns filtered instead of the raw column names.

Proposal: xxx

What's Changed:

How it Works:

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No code

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

None

ti-chi-bot · 2021-09-07T04:24:44Z

[REVIEW NOTIFICATION]

This pull request has been approved by:

gozssky
kennytm

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

…x-ignore-cols

kennytm · 2021-09-08T08:13:38Z

(i think you shouldn't add the "needs-cherry-pick-5.1" label, to avoid confusing the bot)

glorv · 2021-09-08T09:05:38Z

(i think you shouldn't add the "needs-cherry-pick-5.1" label, to avoid confusing the bot)

Done. Forgot v5.1 is in br repo.😅

sleepymole · 2021-11-22T08:27:11Z

br/pkg/lightning/backend/tidb/tidb.go

 	}

 	// TODO: since the column count doesn't exactly reflect the real column names, we only check the upper bound currently.
 	// See: tests/generated_columns/data/gencol.various_types.0.sql this sql has no columns, so encodeLoop will fill the
 	// column permutation with default, thus enc.columnCnt > len(row).
-	if len(row) > enc.columnCnt {
+	if len(row) < enc.columnCnt {


Could you please explain why len(row) should be greater than or equal to columnCnt(which is actual columenMaxIdx + 1)?

Suppose a table has 5 columns, namely a, b, c, d, e. We have a SQL statement in a source file like insert into t(a, d, c) values(xxx, xxx, xxx). Then the columnPermutation should be (0, -1, 2, 1, -1, -1), columnCnt = columnMaxIdx + 1 = 4. len(row) is 3 which is smaller than columnCnt, so in this scenario, data cannot be imported correctly.

Is my understanding correct？

@gozssky In this case, the columnMaxIdx should be 2, so columnCnt = 3.

This check is used to handle there are ignored columns. Giving a source file insert into t(a, d, c, f) values(xxx, xxx, xxx, xx), and ignore field f in lightning's config, this is still a valid source file. Then len(row) (4) will bigger than columnCnt (3).

BTW, there may be case that row count is smaller than column count if user manually create table schema has extra column with default value, e.g. updated timestamp default current_timestamp, then current lightning will result in an error.

Oh, I mistook i for idx 😂.

sleepymole · 2021-11-22T12:14:44Z

br/tests/lightning_duplicate_detection/run.sh

@@ -22,7 +22,7 @@ LOG_FILE1="$TEST_DIR/lightning-duplicate-detection1.log"
 LOG_FILE2="$TEST_DIR/lightning-duplicate-detection2.log"

 # let lightning run a bit slow to avoid some table in the first lightning finish too fast.
-export GO_FAILPOINTS="github.com/pingcap/tidb/br/pkg/lightning/restore/SlowDownImport=sleep(50)"
+export GO_FAILPOINTS="github.com/pingcap/tidb/br/pkg/lightning/restore/SlowDownImport=sleep(250)"


The test lightning_distributed_import also need to inscrease the timeout.

sleepymole

rest LGTM

…e-cols

glorv · 2021-11-23T02:20:52Z

/run-integration-br-test

glorv · 2021-11-23T10:00:37Z

/run-integration-br-test

…e-cols

…x-ignore-cols

glorv · 2021-11-23T10:59:49Z

/merge

ti-chi-bot · 2021-11-23T10:59:55Z

This pull request has been accepted and is ready to merge.

Commit hash: 58f035d

Signed-off-by: ti-srebot <[email protected]>

ti-srebot · 2021-11-23T11:12:30Z

cherry pick to release-5.2 in PR #30063

Signed-off-by: ti-srebot <[email protected]>

ti-srebot · 2021-11-23T11:14:14Z

cherry pick to release-5.3 in PR #30064

* executor: fix error msg of granting non-table level privilege (#29321) * server: fix bug https://asktug.com/t/topic/213082/11 (#29577) * *: replace compareDatum by compare (#30048) * executor, store: replace compareDatum by compare (#30044) * executor: migrate test-infra to testify for executor/write_test.go (#29953) * planner: add sub plan info of shuffleReceiver when query explain analyze (#27992) * expression: fix wrong flen for CastAsString funtion (#29563) * br: migrate test-infra to testify for redact (#29747) * ddl: stable create expression index test (#29990) * *: replace compareDatum by compare (#30060) * planner: fix the issue that binding cannot work when sql_select_limit is enabled (#29789) * br : migrate test-infra to testify for lightning/backend/tidb/tidb_test.go (#30042) * *: fix unstable test in placement gc cases (#30045) * lightning: let ignore columns be compatible with tidb backend (#27850) * statistics: replace compareDatum by compare (#30052) * br/pkg/mock: migrate test-infra to testify (#30034) * ddl: forbit alter table cache in system db (#29998) * topsql: distinguish the row and index operation type (#29044) * ddl, util/codec: fix add index OOM and prevent panic in logging (#29925) * br/pkg/utils: migrate tests to testify (#30032) * expression, sessionctx: support rand_seed1 and rand_seed2 sysvar (#29936) * *: track the memory usage of IndexJoin more accurate (#29068) * br : migrate test-infra to testify for lightning/backend/importer (#30073) * planner: fix panic when the join key is scalarFunction (#30002) * ddl: fix none info of `alter table placement` (#29929) * expression: cast charset according to the function's resulting charset (#29905) * *: replace compareDatum by compare and fix compare (#30090) * store/driver: Use BatchGet method supported in client-go. (#29860) * privilege: disable role privilege when it is revoked from an user (#30028) * partition: Show partition reformat (#29945) * planner: fix the unstable test `TestPartitionWithVariedDatasources` (#30139) * server: Add a `MockConn` to enable testing connections (#30119) * *: skip unstable test (#29433) * skip test Signed-off-by: yisaer <[email protected]> * skip unstable test Signed-off-by: yisaer <[email protected]> * table/tables: add `StateRemote` interface for the cached table (#29152) * planner: rebuild range when the range is empty (#30003) * planner: support dump file for trace plan statement (#30059) * planner: make clear for MaybeOverOptimized4PlanCache (#29782) * update: fix the updatable table name resolution in build update list (#30061) * expression : prevent function DATE_ADD/SUB_STRING_XXX pushed down to TiFlash (#30154) * *: remove unused profile memory tracker from global tracker (#30143) * expression, util/codec: fix wrongly eliminated conditions caused by `HashCode()` collision (#30120) * cmd, expression: fix convert binary string to gbk (#30004) * sessionctx/variable: make lc_time_names read only (#30084) * expression: fix misuse of datumsToConstants in GBK test (#30072) * plugin: add more tests to cover audit logs (#30165) * planner: consider prefix index column length in skyline pruning (#27527) * server: support download optimize trace file (#30150) * expression: add ut config (#30156) * ddl/ddl_test.go: refactor testTableInfo to return error (#30069) * *: skip a case in TestClusterTables/ForClusterServerInfo on mac M1 (#29933) * planner: do not add extra limit when handle the execute stmt (#30152) * ddl: migrate test-infra to testify for db_cache_test.go (#30117) * tables: fix data race in mockStatRemoteData (#30180) * server: Port missing in processlist (#30183) * server: add build not race flag (#30184) * unistore: get/batchGet/scan support read-through-lock (#29898) * topsql: fix nil pointer panic in stmtctx (#30181) * types: fix wrong str_to_date() microseconds with leading zeros are not converted correctly (#30122) * dumpling: Add support for `Create Placement` (#29724) * *: add cardinality estimation trace for `Selectivity` (#29883) * *: replace compareDatum by compare and ignore warnings is collate is empty (#30105) * planner: implement aggregation eliminate optimize trace (#30114) * server: Combined fix for authentication issues (#29738) * executer: fix data race (#30186) * dumpling : add a function for the variable call of dm (#30033) * planner: revise optimize trace logic (#30163) * statistics: fix unstable test case TestTraceCE (#30235) * lightning: avoid retry write rows and record duplicate rows in tidb backend (#30179) * executor: send a task with error to the resultCh when panic happen (#30214) * docs: update placement policy limits (#30202) * expression: Support GBK for builtin function AesEncrypt. (#29946) * executor: remove useless log (#30248) * docs: Update 2021-09-29-secure-bootstrap.md (#30243) * executor: fix unstable test of topsql (#30257) * copr: add paging API for streaming-like process (#29612) * *: forbid set TiFlash Replica for a table with unsupport charset (#30162) * privilege, session, server: consistently map user login to identity (#30204) * expression: fix flen for timestamp_add (#30213) * types: check values equals to NaN or Inf when convert to float (#30148) * parser: respect TiDB comment when DROP INDEX IF EXISTS (#30173) * ddl: disallow change columns from zero int to time (#25728) * * : statement summary should know the slow write is blocked on read lock lease (#30166) * planner: regard NULL as point when accessing composite index (#30244) * planner: Add trace for proj elimination rule (#30275) * privilege,session: Match loopback connections to 'localhost' accounts (#29995) (#30189) * ddltest: refactor logutil.InitLogger in ddltest to avoid data race (#30299) * load data: fix bug if load data with long content (#29222) * topsql: reduce data race of sql digest (#30296) * ddl: Do not consider the clustered index when checking the length of the secondary index (#29660) * *: update client-go to use resolveForRead (#30314) * ddl/ddl_algorithm_test.go: migrate test-infra to testify (#30268) * store: Add metrics for pd api call time (#30062) * *: replace compareDatum by compare and fix wrong optimize order by (#30273) * metrics/grafana: fix display for 'Start TSO Wait Duration' panel (#30311) * ddl: migrate test-infra to testify for ddl/options_test.go (#30269) * planner: fix inconsistent schema between UnionAll and child operator (#30231) * test: fix incorrect regexp pattern during migrating test (#30080) * br: add more precise check for lock file (#30218) * executor: replace should not change other rows when auto ID is out of range (#30301) * tidb-server: return 1 while failing to set cpu affinity (#30197) * executor: migrate test-infra to testify for executor/index_lookup_join_test.go (#30260) * expression: fix wrong result of greatest/least(mixed unsigned/signed int) (#30121) * types: casting JSON literal `null` to number types should fail (#30278) * executor: fix data race in oomtest (#30339) * parser: fix missing charset and collation of handle column (#30320) * expression: don't append null when param for char() is null (#30236) * *: add warn log for stale read (#30340) * parser: support multi bracket for subquery (#29656) * session, sessionctx/variable: fix validation recursion bug (#30293) * planner: Add trace for agg pushdown rule (#30262) * planner/core: fix a data race when building plan for cached table (#30355) * executor: avoid sum from avg overflow (#30010) * config: make EnableSlowLog atomic (#30346) * *: fix goroutine leak in ddl intergration test (#30369) * executor: make projection executor unparallel for insert/update/delete (#30290) * executor, util: reset offsets and nullBitMap for MutRow when setting new values (#30265) * expression: fix tidb can't alter table from other-type with null value to timestamp with NOT NULL attribute (#29664) * dumpling: fix default collation with upstream when dump database and table (#30292) * ddl: fix the enum default value by triming trailing space (#30356) * expression: migrate test-infra to testify for flag_simplify_test.go (#30407) * server: refine code logic in handleDownloadFile (#30422) * refine logic Signed-off-by: yisaer <[email protected]> * fix Signed-off-by: yisaer <[email protected]> * ddl: migrate test-infra to testify for ddl/table_test.go (#30267) * ddl: handle the error from `addBatchDDLJobs()` correctly (#30401) * br: fix the integration tests (#30423) * util, cmd: remove unused filesort (#30438) * *: update client-go for small backoff time (#30436) * server: Fix unstable tests with FakeAuthSwitch (#30287) * dumpling: fix dump failed when sequence exists (#30164) * *: replace compareDatum by compare (#30421) * lightning: fix gcs max key limit (#30393) * expression, parser: add built-in func is_uuid (#30318) * expression: migrate test-infra to testify for constant_fold_test.go (#30424) * executor: fix pipelined window invalid memory address (#30418) * makefile: add gotestsum for verify ci (#29848) * server: close sql rows to fix unstable test (#30306) * Makefile: add coverage record for BR and Dumpling (#30457) * executor: track the mem usage of IndexMergeReader (#30210) * infosync: close body when ReadAll encounters error (#30462) * planner: show accessed partition when explain mpp query over partition table (#30367) * *: Fix use of user identity in SHOW GRANTS + error messages (#30294) * ddl: add not null flag for auto_increment column (#30477) * expression: make some unstable test serial (#30323) * expression: migrate test-infra to testify for constant_propagation_test.go (#30430) * executor: stable test TestSetDDLReorgBatchSize and TestSetDDLReorgWorkerCnt (#30480) * statistics, util/ranger: add cardinality estimation trace for `GetRowCountBy...` (#30321) * *: skip mysql client goroutine leak detection in integration ddl (#30467) * executor,util: write slow query to slow log no matter what log level (#30461) * executor: enable index_merge used in transaction. (#29875) * logutil: add testcase for SlowQueryLogger.MaxDays/MaxSize/MaxBackups (#30316) * expression: fix data race in builtin_other_vec_generated_test.go (#30503) * expression: fix data race in the collationInfo (#30490) * planner/core, session: fix error message of wrong variable scope (#30510) * lightning: support Re/ReregisterMySQL by different tls name (#30463) * executor: TestBatchGetandPointGetwithHashPartition test typo (#29669) (#29671) * mockstore: improve log to avoid panic for nil pointer (#30513) * *: replace compareDatum by compare, PR 10 (#30456) * planner: Disable dynamic partition prune mode for all non-autocommit (#27532) (#30505) * expression: change the log level of an confusing log from warn to debug (#30484) * br: Check crypter.key valid before backup (#29991) * *: replace compareDatum by compare, PR 11 (#30465) * dumpling: fix default column collation with upstream when dump table (#30531) * server: fix prepared cursor select (#30285) * executor: HashJoinExec checks the buildError even if the probeSide is empty (#30471) * parser, expression: follow mysql, increase interval precedence (#30528) * makefile: set timeout 25m for make race (#30555) * planner: fix the unstable test TestAnalyzeGlobalStatsWithOpts/2 (#30576) * expression,types: Adjusts UNIX_TIMESTAMP() for non-existing DST values (#28739) (#30405) * br: add res.Body.close to avoid leak (#30545) * lightning: add back integration test lightning_error_summary (#30547) * sessionctx/variable: small refactor (split large file) (#30511) * ddl: let `admin cancel ddl jobs` run in a new transaction (#30549) * *: Retry when placement PutBundles failed (#30590) * dumpling: delete unit test in github actions (#30562) * *: support trace plan target='estimation' statement (#30491) * expression: migrate test-infra to testify for integration_test.go (#30548) * planner: support trace for min/max eliminate (#30441) * support min/max trace Signed-off-by: yisaer <[email protected]> * address the comment Signed-off-by: yisaer <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * br: remove cdclog in br (#30573) * *: show cmd to check if all needed histograms are loaded (#29672) * expression: clone repertoire when clone the scalar function (#30602) * *: use the real StateRemote interface implementation for cached table (#30066) * *: query failed after add index / timestamp out-of-range (#28424) (#29323) * planner: implement collecting predicate columns from logical plan (#29878) * *: show PK name when decoding the clustered index row key (#30623) * ddl/callback_test.go: migrate test-infra to testify (#30317) * *: Rename some names of placement ddl operation (#30622) * executor: fix data race in the index_lookup_hash_join (#30619) * ddl: remove unnecessary locking when adding an index (#29772) * server: try to make `TidbTestSuite` more stable (#30643) * *: Add some PD tests for placement and fix some bug found (#30621) * *: migrate sync.WaitGroup to util.WaitGroupWrapper (#30644) * planner: add trace for join eliminate rule (#30343) * executor: migrate test-infra to testify for executor/shuffle_test.go (#30514) * planner: make (*AccessPath).OnlyPointRange more succinct (#30520) * planner: add trace for join reorder (#30394) * executor: migrate test-infra to testify for executor/union_scan_test.go (#30525) * expression: make cast return error if cast binary literal to another character set (#30537) * *: update tikv client (#30670) * *: update sysutil in go.mod to fix panic when search log (#30523) * topsql: shouldn't evict the SQL meta, since the evicted SQL can be appear on Other components (TiKV) TopN records (#27050) * testify: migrate test-infra to testify for analyze_test.go (#30640) * util: replace compareDatum by compare, point part (#30575) * test: make all the tests run in serial (#30692) * statistics: add mutex for Handle.globalMap and Handle.feedback (#30550) * executor: fix regular expression in json so that it could match identifer start with '$' (#29750) * util/testkit/testkit.go: fix typo (#30638) * planner: Introduce a new global variable to control the historical statistics feature (#30646) * topsql: introduce datasink interface (#30662) * planner: unify the argument of stats functions to use SessionCtx instead of StatementContext (#30668) * metrics: fix the Max SafeTS Gap metrics (#30689) * lightning: Add source dir existence check for s3 (#30674) * golangci-lint: support durationcheck (#30027) * executor: fix data race on IndexHashJoin.cancelFunc (#30701) * sessionctx/variable: change tidb_store_limit to global only (#30522) * statistics: remove reassignment of Handle.pool in NewHandle (#30675) * br: fix some unstable unit test cases. (#30716) * bindinfo: fix the comment typo (#30616) * server: support decoding prepared string args to character_set_client (#30723) * expression: fix enum type join binary get wrong result (#30445) * cmd/explaintest: fix wrong result comparison for explain test (#30717) * parallel create tables in br * metrics: fix copr-cache metrics (#30712) * test: merge executor's serial tests to other tests (#30711) * statistics: avoid deadlock when create/drop extended stats and analyze at the same time (#30566) * ddl: add batch create table api Signed-off-by: xhe <[email protected]> * ddl: add unit tests Signed-off-by: xhe <[email protected]> * ddl: fix fmt Signed-off-by: xhe <[email protected]> * ddl: typo Co-authored-by: Arenatlx <[email protected]> * ddl: fix tests Signed-off-by: xhe <[email protected]> * ddl: rename to BatchCreateTableWithInfo Signed-off-by: xhe <[email protected]> * ddl: trace the error Signed-off-by: xhe <[email protected]> * ddl: comments Signed-off-by: xhe <[email protected]> * ddl: cancle the job right Signed-off-by: xhe <[email protected]> * ddl: cancel the job right 2 Signed-off-by: xhe <[email protected]> * ddl: report error if entry too large Signed-off-by: xhe <[email protected]> * ddl: report error when table is duplicated Signed-off-by: xhe <[email protected]> * ddl: go fmt Signed-off-by: xhe <[email protected]> * infoschema: improve batch memory perf Signed-off-by: xhe <[email protected]> * ddl: retain ID Signed-off-by: xhe <[email protected]> * sessionctx: fix the value of analyze_version when upgrading 4.x to 5.… (#30743) * ddl: reduce log frequency Signed-off-by: xhe <[email protected]> * ddl: fix tests Signed-off-by: xhe <[email protected]> * server: disable socket listener for `basicHTTPHandlerTestSuite` (#30680) * planner: support the plan cache aware of bindings (#30169) * planner: fix early set of plan's statisticsTable (#30754) * *: implement renew write lock lease for cached table (#30206) * *: Modify placement rule index to reserve some indexes for future work (#30737) * executor: add an unit test case for unreasonable invoking Close (#30696) * planner: fix wrong subquery's coercibility (#30750) Co-authored-by: unconsolable <[email protected]> Co-authored-by: zhangguangchao <[email protected]> Co-authored-by: wjHuang <[email protected]> Co-authored-by: Weizhen Wang <[email protected]> Co-authored-by: mmyj <[email protected]> Co-authored-by: Shenghui Wu <[email protected]> Co-authored-by: tison <[email protected]> Co-authored-by: Yuanjia Zhang <[email protected]> Co-authored-by: xiaolunzhou <[email protected]> Co-authored-by: 王超 <[email protected]> Co-authored-by: glorv <[email protected]> Co-authored-by: sylzd <[email protected]> Co-authored-by: Yexiang Zhang <[email protected]> Co-authored-by: tangenta <[email protected]> Co-authored-by: Yujie Xia <[email protected]> Co-authored-by: Hangjie Mo <[email protected]> Co-authored-by: HuaiyuXu <[email protected]> Co-authored-by: Song Gao <[email protected]> Co-authored-by: xiongjiwei <[email protected]> Co-authored-by: WangDeng <[email protected]> Co-authored-by: Mattias Jonsson <[email protected]> Co-authored-by: djshow832 <[email protected]> Co-authored-by: tiancaiamao <[email protected]> Co-authored-by: Chengpeng Yan <[email protected]> Co-authored-by: Arenatlx <[email protected]> Co-authored-by: Meng Xin <[email protected]> Co-authored-by: Zhou Kunqin <[email protected]> Co-authored-by: Youra Cho <[email protected]> Co-authored-by: Yifan Xu <[email protected]> Co-authored-by: José Clovis Ramírez de la Rosa <[email protected]> Co-authored-by: Xiang Zhang <[email protected]> Co-authored-by: Lei Zhao <[email protected]> Co-authored-by: Daniël van Eeden <[email protected]> Co-authored-by: docsir <[email protected]> Co-authored-by: Morgan Tocker <[email protected]> Co-authored-by: jakevin <[email protected]> Co-authored-by: 姬小野 <[email protected]> Co-authored-by: you06 <[email protected]> Co-authored-by: lance6716 <[email protected]> Co-authored-by: TonsnakeLin <[email protected]> Co-authored-by: imentu <[email protected]> Co-authored-by: Alkaid <[email protected]> Co-authored-by: lvtu <[email protected]> Co-authored-by: 3pointer <[email protected]> Co-authored-by: Zhuhe Fang <[email protected]> Co-authored-by: WizardXiao <[email protected]> Co-authored-by: Jianjun Liao <[email protected]> Co-authored-by: guo-shaoge <[email protected]> Co-authored-by: Ryan Leung <[email protected]> Co-authored-by: xufei <[email protected]> Co-authored-by: Ehco <[email protected]> Co-authored-by: Zak Zhao <[email protected]> Co-authored-by: xhe <[email protected]> Co-authored-by: Chunzhu Li <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> Co-authored-by: Xiaoju Wu <[email protected]> Co-authored-by: JmPotato <[email protected]> Co-authored-by: Zach <[email protected]> Co-authored-by: bb7133 <[email protected]> Co-authored-by: crazycs <[email protected]> Co-authored-by: znhh6018 <[email protected]> Co-authored-by: eddie lin <[email protected]> Co-authored-by: dongjunduo <[email protected]> Co-authored-by: Zhenchi <[email protected]> Co-authored-by: wangggong <[email protected]> Co-authored-by: zhangjinpeng1987 <[email protected]> Co-authored-by: Jack Yu <[email protected]> Co-authored-by: Arenatlx <[email protected]> Co-authored-by: Yiding Cui <[email protected]>

* executor: migrate test-infra to testify for executor/show_stats_test.go (#30000) * ddl/table_split_test.go: migrate test-infra to testify (#29960) * *: fix unstable test caused by TestRandomPanicAggConsume (#30035) * ddl/ddl_test.go: refactor testSchemaInfo to return error #29964 (#30008) * executor: remove useless parameter (#30043) * executor: fix error msg of granting non-table level privilege (#29321) * server: fix bug https://asktug.com/t/topic/213082/11 (#29577) * *: replace compareDatum by compare (#30048) * executor, store: replace compareDatum by compare (#30044) * executor: migrate test-infra to testify for executor/write_test.go (#29953) * planner: add sub plan info of shuffleReceiver when query explain analyze (#27992) * expression: fix wrong flen for CastAsString funtion (#29563) * br: migrate test-infra to testify for redact (#29747) * ddl: stable create expression index test (#29990) * *: replace compareDatum by compare (#30060) * planner: fix the issue that binding cannot work when sql_select_limit is enabled (#29789) * br : migrate test-infra to testify for lightning/backend/tidb/tidb_test.go (#30042) * *: fix unstable test in placement gc cases (#30045) * lightning: let ignore columns be compatible with tidb backend (#27850) * statistics: replace compareDatum by compare (#30052) * br/pkg/mock: migrate test-infra to testify (#30034) * ddl: forbit alter table cache in system db (#29998) * topsql: distinguish the row and index operation type (#29044) * ddl, util/codec: fix add index OOM and prevent panic in logging (#29925) * br/pkg/utils: migrate tests to testify (#30032) * expression, sessionctx: support rand_seed1 and rand_seed2 sysvar (#29936) * *: track the memory usage of IndexJoin more accurate (#29068) * br : migrate test-infra to testify for lightning/backend/importer (#30073) * planner: fix panic when the join key is scalarFunction (#30002) * ddl: fix none info of `alter table placement` (#29929) * expression: cast charset according to the function's resulting charset (#29905) * *: replace compareDatum by compare and fix compare (#30090) * store/driver: Use BatchGet method supported in client-go. (#29860) * privilege: disable role privilege when it is revoked from an user (#30028) * partition: Show partition reformat (#29945) * planner: fix the unstable test `TestPartitionWithVariedDatasources` (#30139) * server: Add a `MockConn` to enable testing connections (#30119) * *: skip unstable test (#29433) * skip test Signed-off-by: yisaer <[email protected]> * skip unstable test Signed-off-by: yisaer <[email protected]> * table/tables: add `StateRemote` interface for the cached table (#29152) * planner: rebuild range when the range is empty (#30003) * planner: support dump file for trace plan statement (#30059) * planner: make clear for MaybeOverOptimized4PlanCache (#29782) * update: fix the updatable table name resolution in build update list (#30061) * expression : prevent function DATE_ADD/SUB_STRING_XXX pushed down to TiFlash (#30154) * *: remove unused profile memory tracker from global tracker (#30143) * expression, util/codec: fix wrongly eliminated conditions caused by `HashCode()` collision (#30120) * cmd, expression: fix convert binary string to gbk (#30004) * sessionctx/variable: make lc_time_names read only (#30084) * expression: fix misuse of datumsToConstants in GBK test (#30072) * plugin: add more tests to cover audit logs (#30165) * planner: consider prefix index column length in skyline pruning (#27527) * server: support download optimize trace file (#30150) * expression: add ut config (#30156) * ddl/ddl_test.go: refactor testTableInfo to return error (#30069) * *: skip a case in TestClusterTables/ForClusterServerInfo on mac M1 (#29933) * planner: do not add extra limit when handle the execute stmt (#30152) * ddl: migrate test-infra to testify for db_cache_test.go (#30117) * tables: fix data race in mockStatRemoteData (#30180) * server: Port missing in processlist (#30183) * server: add build not race flag (#30184) * unistore: get/batchGet/scan support read-through-lock (#29898) * topsql: fix nil pointer panic in stmtctx (#30181) * types: fix wrong str_to_date() microseconds with leading zeros are not converted correctly (#30122) * dumpling: Add support for `Create Placement` (#29724) * *: add cardinality estimation trace for `Selectivity` (#29883) * *: replace compareDatum by compare and ignore warnings is collate is empty (#30105) * planner: implement aggregation eliminate optimize trace (#30114) * server: Combined fix for authentication issues (#29738) * executer: fix data race (#30186) * dumpling : add a function for the variable call of dm (#30033) * planner: revise optimize trace logic (#30163) * statistics: fix unstable test case TestTraceCE (#30235) * lightning: avoid retry write rows and record duplicate rows in tidb backend (#30179) * executor: send a task with error to the resultCh when panic happen (#30214) * docs: update placement policy limits (#30202) * expression: Support GBK for builtin function AesEncrypt. (#29946) * executor: remove useless log (#30248) * docs: Update 2021-09-29-secure-bootstrap.md (#30243) * executor: fix unstable test of topsql (#30257) * copr: add paging API for streaming-like process (#29612) * *: forbid set TiFlash Replica for a table with unsupport charset (#30162) * privilege, session, server: consistently map user login to identity (#30204) * expression: fix flen for timestamp_add (#30213) * types: check values equals to NaN or Inf when convert to float (#30148) * parser: respect TiDB comment when DROP INDEX IF EXISTS (#30173) * ddl: disallow change columns from zero int to time (#25728) * * : statement summary should know the slow write is blocked on read lock lease (#30166) * planner: regard NULL as point when accessing composite index (#30244) * planner: Add trace for proj elimination rule (#30275) * privilege,session: Match loopback connections to 'localhost' accounts (#29995) (#30189) * ddltest: refactor logutil.InitLogger in ddltest to avoid data race (#30299) * load data: fix bug if load data with long content (#29222) * topsql: reduce data race of sql digest (#30296) * ddl: Do not consider the clustered index when checking the length of the secondary index (#29660) * *: update client-go to use resolveForRead (#30314) * ddl/ddl_algorithm_test.go: migrate test-infra to testify (#30268) * store: Add metrics for pd api call time (#30062) * *: replace compareDatum by compare and fix wrong optimize order by (#30273) * metrics/grafana: fix display for 'Start TSO Wait Duration' panel (#30311) * ddl: migrate test-infra to testify for ddl/options_test.go (#30269) * planner: fix inconsistent schema between UnionAll and child operator (#30231) * test: fix incorrect regexp pattern during migrating test (#30080) * br: add more precise check for lock file (#30218) * executor: replace should not change other rows when auto ID is out of range (#30301) * tidb-server: return 1 while failing to set cpu affinity (#30197) * executor: migrate test-infra to testify for executor/index_lookup_join_test.go (#30260) * expression: fix wrong result of greatest/least(mixed unsigned/signed int) (#30121) * types: casting JSON literal `null` to number types should fail (#30278) * executor: fix data race in oomtest (#30339) * parser: fix missing charset and collation of handle column (#30320) * expression: don't append null when param for char() is null (#30236) * *: add warn log for stale read (#30340) * parser: support multi bracket for subquery (#29656) * session, sessionctx/variable: fix validation recursion bug (#30293) * planner: Add trace for agg pushdown rule (#30262) * planner/core: fix a data race when building plan for cached table (#30355) * executor: avoid sum from avg overflow (#30010) * config: make EnableSlowLog atomic (#30346) * *: fix goroutine leak in ddl intergration test (#30369) * executor: make projection executor unparallel for insert/update/delete (#30290) * executor, util: reset offsets and nullBitMap for MutRow when setting new values (#30265) * expression: fix tidb can't alter table from other-type with null value to timestamp with NOT NULL attribute (#29664) * dumpling: fix default collation with upstream when dump database and table (#30292) * ddl: fix the enum default value by triming trailing space (#30356) * expression: migrate test-infra to testify for flag_simplify_test.go (#30407) * server: refine code logic in handleDownloadFile (#30422) * refine logic Signed-off-by: yisaer <[email protected]> * fix Signed-off-by: yisaer <[email protected]> * ddl: migrate test-infra to testify for ddl/table_test.go (#30267) * ddl: handle the error from `addBatchDDLJobs()` correctly (#30401) * br: fix the integration tests (#30423) * util, cmd: remove unused filesort (#30438) * *: update client-go for small backoff time (#30436) * server: Fix unstable tests with FakeAuthSwitch (#30287) * dumpling: fix dump failed when sequence exists (#30164) * *: replace compareDatum by compare (#30421) * lightning: fix gcs max key limit (#30393) * expression, parser: add built-in func is_uuid (#30318) * expression: migrate test-infra to testify for constant_fold_test.go (#30424) * executor: fix pipelined window invalid memory address (#30418) * makefile: add gotestsum for verify ci (#29848) * server: close sql rows to fix unstable test (#30306) * Makefile: add coverage record for BR and Dumpling (#30457) * executor: track the mem usage of IndexMergeReader (#30210) * infosync: close body when ReadAll encounters error (#30462) * planner: show accessed partition when explain mpp query over partition table (#30367) * *: Fix use of user identity in SHOW GRANTS + error messages (#30294) * ddl: add not null flag for auto_increment column (#30477) * expression: make some unstable test serial (#30323) * expression: migrate test-infra to testify for constant_propagation_test.go (#30430) * executor: stable test TestSetDDLReorgBatchSize and TestSetDDLReorgWorkerCnt (#30480) * statistics, util/ranger: add cardinality estimation trace for `GetRowCountBy...` (#30321) * *: skip mysql client goroutine leak detection in integration ddl (#30467) * executor,util: write slow query to slow log no matter what log level (#30461) * executor: enable index_merge used in transaction. (#29875) * logutil: add testcase for SlowQueryLogger.MaxDays/MaxSize/MaxBackups (#30316) * expression: fix data race in builtin_other_vec_generated_test.go (#30503) * expression: fix data race in the collationInfo (#30490) * planner/core, session: fix error message of wrong variable scope (#30510) * lightning: support Re/ReregisterMySQL by different tls name (#30463) * executor: TestBatchGetandPointGetwithHashPartition test typo (#29669) (#29671) * mockstore: improve log to avoid panic for nil pointer (#30513) * *: replace compareDatum by compare, PR 10 (#30456) * planner: Disable dynamic partition prune mode for all non-autocommit (#27532) (#30505) * expression: change the log level of an confusing log from warn to debug (#30484) * br: Check crypter.key valid before backup (#29991) * *: replace compareDatum by compare, PR 11 (#30465) * dumpling: fix default column collation with upstream when dump table (#30531) * server: fix prepared cursor select (#30285) * executor: HashJoinExec checks the buildError even if the probeSide is empty (#30471) * parser, expression: follow mysql, increase interval precedence (#30528) * makefile: set timeout 25m for make race (#30555) * planner: fix the unstable test TestAnalyzeGlobalStatsWithOpts/2 (#30576) * expression,types: Adjusts UNIX_TIMESTAMP() for non-existing DST values (#28739) (#30405) * br: add res.Body.close to avoid leak (#30545) * lightning: add back integration test lightning_error_summary (#30547) * sessionctx/variable: small refactor (split large file) (#30511) * ddl: let `admin cancel ddl jobs` run in a new transaction (#30549) * *: Retry when placement PutBundles failed (#30590) * dumpling: delete unit test in github actions (#30562) * *: support trace plan target='estimation' statement (#30491) * expression: migrate test-infra to testify for integration_test.go (#30548) * planner: support trace for min/max eliminate (#30441) * support min/max trace Signed-off-by: yisaer <[email protected]> * address the comment Signed-off-by: yisaer <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * br: remove cdclog in br (#30573) * *: show cmd to check if all needed histograms are loaded (#29672) * expression: clone repertoire when clone the scalar function (#30602) * *: use the real StateRemote interface implementation for cached table (#30066) * *: query failed after add index / timestamp out-of-range (#28424) (#29323) * planner: implement collecting predicate columns from logical plan (#29878) * *: show PK name when decoding the clustered index row key (#30623) * ddl/callback_test.go: migrate test-infra to testify (#30317) * *: Rename some names of placement ddl operation (#30622) * executor: fix data race in the index_lookup_hash_join (#30619) * ddl: remove unnecessary locking when adding an index (#29772) * server: try to make `TidbTestSuite` more stable (#30643) * *: Add some PD tests for placement and fix some bug found (#30621) * *: migrate sync.WaitGroup to util.WaitGroupWrapper (#30644) * planner: add trace for join eliminate rule (#30343) * executor: migrate test-infra to testify for executor/shuffle_test.go (#30514) * planner: make (*AccessPath).OnlyPointRange more succinct (#30520) * planner: add trace for join reorder (#30394) * executor: migrate test-infra to testify for executor/union_scan_test.go (#30525) * expression: make cast return error if cast binary literal to another character set (#30537) * *: update tikv client (#30670) * *: update sysutil in go.mod to fix panic when search log (#30523) * topsql: shouldn't evict the SQL meta, since the evicted SQL can be appear on Other components (TiKV) TopN records (#27050) * testify: migrate test-infra to testify for analyze_test.go (#30640) * util: replace compareDatum by compare, point part (#30575) * test: make all the tests run in serial (#30692) * statistics: add mutex for Handle.globalMap and Handle.feedback (#30550) * executor: fix regular expression in json so that it could match identifer start with '$' (#29750) * util/testkit/testkit.go: fix typo (#30638) * planner: Introduce a new global variable to control the historical statistics feature (#30646) * topsql: introduce datasink interface (#30662) * planner: unify the argument of stats functions to use SessionCtx instead of StatementContext (#30668) * metrics: fix the Max SafeTS Gap metrics (#30689) * lightning: Add source dir existence check for s3 (#30674) * golangci-lint: support durationcheck (#30027) * executor: fix data race on IndexHashJoin.cancelFunc (#30701) * sessionctx/variable: change tidb_store_limit to global only (#30522) * statistics: remove reassignment of Handle.pool in NewHandle (#30675) * br: fix some unstable unit test cases. (#30716) * bindinfo: fix the comment typo (#30616) * server: support decoding prepared string args to character_set_client (#30723) * expression: fix enum type join binary get wrong result (#30445) * cmd/explaintest: fix wrong result comparison for explain test (#30717) * parallel create tables in br * metrics: fix copr-cache metrics (#30712) * test: merge executor's serial tests to other tests (#30711) * ddl: add batch create table api Signed-off-by: xhe <[email protected]> * ddl: add unit tests Signed-off-by: xhe <[email protected]> * ddl: fix fmt Signed-off-by: xhe <[email protected]> * ddl: typo Co-authored-by: Arenatlx <[email protected]> * ddl: fix tests Signed-off-by: xhe <[email protected]> * ddl: rename to BatchCreateTableWithInfo Signed-off-by: xhe <[email protected]> * ddl: trace the error Signed-off-by: xhe <[email protected]> * ddl: comments Signed-off-by: xhe <[email protected]> * ddl: cancle the job right Signed-off-by: xhe <[email protected]> * ddl: cancel the job right 2 Signed-off-by: xhe <[email protected]> * ddl: report error if entry too large Signed-off-by: xhe <[email protected]> * ddl: report error when table is duplicated Signed-off-by: xhe <[email protected]> * ddl: go fmt Signed-off-by: xhe <[email protected]> * infoschema: improve batch memory perf Signed-off-by: xhe <[email protected]> * ddl: retain ID Signed-off-by: xhe <[email protected]> * ddl: reduce log frequency Signed-off-by: xhe <[email protected]> * ddl: fix tests Signed-off-by: xhe <[email protected]> * fix merge issue * fix issue that loss table restore * refactoring code * refactoring code Co-authored-by: Weizhen Wang <[email protected]> Co-authored-by: wangggong <[email protected]> Co-authored-by: 王超 <[email protected]> Co-authored-by: José Clovis Ramírez de la Rosa <[email protected]> Co-authored-by: unconsolable <[email protected]> Co-authored-by: zhangguangchao <[email protected]> Co-authored-by: wjHuang <[email protected]> Co-authored-by: mmyj <[email protected]> Co-authored-by: Shenghui Wu <[email protected]> Co-authored-by: tison <[email protected]> Co-authored-by: Yuanjia Zhang <[email protected]> Co-authored-by: xiaolunzhou <[email protected]> Co-authored-by: glorv <[email protected]> Co-authored-by: sylzd <[email protected]> Co-authored-by: Yexiang Zhang <[email protected]> Co-authored-by: tangenta <[email protected]> Co-authored-by: Yujie Xia <[email protected]> Co-authored-by: Hangjie Mo <[email protected]> Co-authored-by: HuaiyuXu <[email protected]> Co-authored-by: Song Gao <[email protected]> Co-authored-by: xiongjiwei <[email protected]> Co-authored-by: WangDeng <[email protected]> Co-authored-by: Mattias Jonsson <[email protected]> Co-authored-by: djshow832 <[email protected]> Co-authored-by: tiancaiamao <[email protected]> Co-authored-by: Chengpeng Yan <[email protected]> Co-authored-by: Arenatlx <[email protected]> Co-authored-by: Meng Xin <[email protected]> Co-authored-by: Zhou Kunqin <[email protected]> Co-authored-by: Youra Cho <[email protected]> Co-authored-by: Yifan Xu <[email protected]> Co-authored-by: Xiang Zhang <[email protected]> Co-authored-by: Lei Zhao <[email protected]> Co-authored-by: Daniël van Eeden <[email protected]> Co-authored-by: docsir <[email protected]> Co-authored-by: Morgan Tocker <[email protected]> Co-authored-by: jakevin <[email protected]> Co-authored-by: 姬小野 <[email protected]> Co-authored-by: you06 <[email protected]> Co-authored-by: lance6716 <[email protected]> Co-authored-by: TonsnakeLin <[email protected]> Co-authored-by: imentu <[email protected]> Co-authored-by: Alkaid <[email protected]> Co-authored-by: lvtu <[email protected]> Co-authored-by: 3pointer <[email protected]> Co-authored-by: Zhuhe Fang <[email protected]> Co-authored-by: WizardXiao <[email protected]> Co-authored-by: Jianjun Liao <[email protected]> Co-authored-by: guo-shaoge <[email protected]> Co-authored-by: Ryan Leung <[email protected]> Co-authored-by: xufei <[email protected]> Co-authored-by: Ehco <[email protected]> Co-authored-by: Zak Zhao <[email protected]> Co-authored-by: xhe <[email protected]> Co-authored-by: Chunzhu Li <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> Co-authored-by: Xiaoju Wu <[email protected]> Co-authored-by: JmPotato <[email protected]> Co-authored-by: Zach <[email protected]> Co-authored-by: bb7133 <[email protected]> Co-authored-by: crazycs <[email protected]> Co-authored-by: znhh6018 <[email protected]> Co-authored-by: eddie lin <[email protected]> Co-authored-by: dongjunduo <[email protected]> Co-authored-by: Zhenchi <[email protected]> Co-authored-by: zhangjinpeng1987 <[email protected]> Co-authored-by: Jack Yu <[email protected]> Co-authored-by: Arenatlx <[email protected]>

#30064)

let ignore columns be compatible with tidb backend

cf67771

glorv added needs-cherry-pick-release-5.1 needs-cherry-pick-release-5.2 component/lightning This issue is related to Lightning of TiDB. labels Sep 7, 2021

glorv requested review from kennytm and 3pointer September 7, 2021 04:24

ti-chi-bot added release-note-none Denotes a PR that doesn't merit a release note. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Sep 7, 2021

glorv added 6 commits September 7, 2021 13:29

Merge branch 'master' into fix-ignore-cols

7241543

fix ut

c8e1f7f

Merge branch 'fix-ignore-cols' of ssh://github.com/glorv/tidb into fi…

7ded42f

…x-ignore-cols

Merge branch 'master' into fix-ignore-cols

c677ba0

fix

ddec21a

Merge branch 'fix-ignore-cols' of ssh://github.com/glorv/tidb into fi…

263d7ed

…x-ignore-cols

ti-chi-bot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Sep 8, 2021

Merge branch 'master' into fix-ignore-cols

466bd14

kennytm approved these changes Sep 8, 2021

View reviewed changes

ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Sep 8, 2021

fmt code

c66cdf5

glorv removed the needs-cherry-pick-release-5.1 label Sep 8, 2021

Merge branch 'master' into fix-ignore-cols

62ab23e

ti-chi-bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Sep 27, 2021

Merge branch 'master' into fix-ignore-cols

6fc3880

ti-chi-bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Sep 28, 2021

ti-chi-bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Oct 30, 2021

kennytm changed the title ~~br: let ignore columns be compatible with tidb backend~~ lightning: let ignore columns be compatible with tidb backend Nov 1, 2021

sleepymole reviewed Nov 22, 2021

View reviewed changes

sleepymole approved these changes Nov 22, 2021

View reviewed changes

ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Nov 22, 2021

glorv added 2 commits November 23, 2021 10:17

slow down write speed for lightning_distributed_import

afb8cf7

Merge branch 'master' of ssh://github.com/pingcap/tidb into fix-ignor…

88ba4bc

…e-cols

glorv added 3 commits November 23, 2021 11:53

fix unit test

5459b6a

fix test

80d22a0

Merge branch 'master' into fix-ignore-cols

a149d7f

glorv added 3 commits November 23, 2021 18:51

Merge branch 'master' of ssh://github.com/pingcap/tidb into fix-ignor…

8935105

…e-cols

Merge branch 'fix-ignore-cols' of ssh://github.com/glorv/tidb into fi…

7c10715

…x-ignore-cols

rollback change

58f035d

ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Nov 23, 2021

ti-chi-bot merged commit c687915 into pingcap:master Nov 23, 2021

glorv deleted the fix-ignore-cols branch November 23, 2021 11:12

ti-srebot pushed a commit to ti-srebot/tidb that referenced this pull request Nov 23, 2021

cherry pick pingcap#27850 to release-5.2

e8d4b10

Signed-off-by: ti-srebot <[email protected]>

ti-srebot mentioned this pull request Nov 23, 2021

lightning: let ignore columns be compatible with tidb backend (#27850) #30063

Open

12 tasks

ti-srebot pushed a commit to ti-srebot/tidb that referenced this pull request Nov 23, 2021

cherry pick pingcap#27850 to release-5.3

1bb05f8

Signed-off-by: ti-srebot <[email protected]>

ti-srebot mentioned this pull request Nov 23, 2021

lightning: let ignore columns be compatible with tidb backend (#27850) #30064

Merged

12 tasks

ti-chi-bot pushed a commit that referenced this pull request Feb 22, 2022

lightning: let ignore columns be compatible with tidb backend (#27850) (

4d1f787

#30064)

GMHDBJD mentioned this pull request Mar 24, 2022

lightning panic due to column count mismatch #33396

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

lightning: let ignore columns be compatible with tidb backend #27850

lightning: let ignore columns be compatible with tidb backend #27850

glorv commented Sep 7, 2021

ti-chi-bot commented Sep 7, 2021 •

edited

Loading

kennytm commented Sep 8, 2021

glorv commented Sep 8, 2021

sleepymole Nov 22, 2021

glorv Nov 22, 2021

sleepymole Nov 22, 2021

sleepymole Nov 22, 2021

sleepymole left a comment

glorv commented Nov 23, 2021

glorv commented Nov 23, 2021

glorv commented Nov 23, 2021

ti-chi-bot commented Nov 23, 2021

ti-srebot commented Nov 23, 2021

ti-srebot commented Nov 23, 2021

lightning: let ignore columns be compatible with tidb backend #27850

lightning: let ignore columns be compatible with tidb backend #27850

Conversation

glorv commented Sep 7, 2021

What problem does this PR solve?

What is changed and how it works?

Check List

Release note

ti-chi-bot commented Sep 7, 2021 • edited Loading

kennytm commented Sep 8, 2021

glorv commented Sep 8, 2021

sleepymole Nov 22, 2021

Choose a reason for hiding this comment

glorv Nov 22, 2021

Choose a reason for hiding this comment

sleepymole Nov 22, 2021

Choose a reason for hiding this comment

sleepymole Nov 22, 2021

Choose a reason for hiding this comment

sleepymole left a comment

Choose a reason for hiding this comment

glorv commented Nov 23, 2021

glorv commented Nov 23, 2021

glorv commented Nov 23, 2021

ti-chi-bot commented Nov 23, 2021

ti-srebot commented Nov 23, 2021

ti-srebot commented Nov 23, 2021

ti-chi-bot commented Sep 7, 2021 •

edited

Loading