Cherry pick remaining commits from raftstore-proxy-6.1 to raftstore-proxy #72

CalvinNeo · 2022-05-30T06:53:52Z

What problem does this PR solve?

Issue Number: close #xxx

Problem Summary:

What is changed and how it works?

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No code

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

None

…1269) * ctl: should check column name validation when scan data using ctl close tikv#11268 Signed-off-by: yuqi1129 <[email protected]> * fix format problem Signed-off-by: yuqi1129 <[email protected]> Co-authored-by: Xinye Tao <[email protected]>

…onsistency (tikv#10977) * Add mutation checking logic Signed-off-by: MyonKeminta <[email protected]> * Fix build Signed-off-by: MyonKeminta <[email protected]> * Fix usages of mutation Signed-off-by: MyonKeminta <[email protected]> * Update kvproto and renaming Signed-off-by: MyonKeminta <[email protected]> * Add a simple test Signed-off-by: MyonKeminta <[email protected]> * Fix incorrect assertion Signed-off-by: MyonKeminta <[email protected]> * Support check existance during pessimistic lock Signed-off-by: MyonKeminta <[email protected]> * Add assertion level support Signed-off-by: MyonKeminta <[email protected]> * Add tests Signed-off-by: MyonKeminta <[email protected]> * Add more test cases Signed-off-by: MyonKeminta <[email protected]> * Add test for check_existence of pessimistic_lock request Signed-off-by: MyonKeminta <[email protected]> * Fix build Signed-off-by: MyonKeminta <[email protected]> * ref tikv#11542 Signed-off-by: MyonKeminta <[email protected]> * Add a metric for counting different way of checking assertion, for reference about the performance impact Signed-off-by: MyonKeminta <[email protected]> Co-authored-by: MyonKeminta <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]>

Signed-off-by: Shuaipeng Yu <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]>

* close tikv#11660 Signed-off-by: xiongjiwei <[email protected]> * close tikv#11660 Signed-off-by: xiongjiwei <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]>

…kv#11216) * wip Signed-off-by: pingyu <[email protected]> * wip Signed-off-by: pingyu <[email protected]> * ref tikv#10974: Add API V2 check for RawKV and TxnKV requests (txn) Signed-off-by: pingyu <[email protected]> * ref tikv#10974: Add API V2 check for RawKV and TxnKV requests (txn) Signed-off-by: pingyu <[email protected]> * ref tikv#10974: Add API V2 check for RawKV and TxnKV requests (txn) Signed-off-by: pingyu <[email protected]> * ref tikv#10974: Add API V2 check for RawKV and TxnKV requests (txn) Signed-off-by: pingyu <[email protected]> * ref tikv#10974: Add API V2 check for RawKV and TxnKV requests (txn) Signed-off-by: pingyu <[email protected]> * ref tikv#10974: Add API V2 check for RawKV and TxnKV requests (txn) Signed-off-by: pingyu <[email protected]> * fix enable_ttl Signed-off-by: andylokandy <[email protected]> * simplify test Signed-off-by: andylokandy <[email protected]> * ref tikv#10974: Add API V2 check for RawKV and TxnKV requests (txn) Signed-off-by: pingyu <[email protected]> Co-authored-by: andylokandy <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]>

Signed-off-by: qupeng <[email protected]>

* close tikv#11642 Signed-off-by: qupeng <[email protected]> * a more comment Signed-off-by: qupeng <[email protected]>

* ctl: split into multiple modules ref tikv#10938 Signed-off-by: andylokandy <[email protected]> * refactor if let to match Signed-off-by: andylokandy <[email protected]> * address comment Signed-off-by: andylokandy <[email protected]> * address comment Signed-off-by: andylokandy <[email protected]> * address comment Signed-off-by: andylokandy <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]>

* close tikv#11676 Signed-off-by: devillove084 <[email protected]>

* Deprecation is planned for MIN,MAX,EPSLION,INFINITY,NEG_INFINITY of the (i|u|f)(8|16|32|64) types in standard. ref tikv#11678 ```release-note NONE ``` Signed-off-by: Harold Dost <[email protected]>

Signed-off-by: zhangjinpeng1987 <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]>

close tikv#11690 Signed-off-by: mornyx <[email protected]>

* file_system: support collect the read flow by proc Signed-off-by: MuZhou233 <[email protected]> * fix lint Signed-off-by: MuZhou233 <[email protected]> * fix clippy Signed-off-by: MuZhou233 <[email protected]> * Delete unused content & Adjust copyright text Signed-off-by: MuZhou233 <[email protected]> * Add bench_fetch_io_bytes Signed-off-by: MuZhou233 <[email protected]> * fix lint Signed-off-by: MuZhou233 <[email protected]> * Integrate thread_io Signed-off-by: MuZhou233 <[email protected]> * record bytes in request Signed-off-by: MuZhou233 <[email protected]> * fix logical bug Signed-off-by: MuZhou233 <[email protected]> * Update test write_bytes Signed-off-by: MuZhou233 <[email protected]> * Delete unused & Rename func Signed-off-by: MuZhou233 <[email protected]> * Add test for read & Update test for write Signed-off-by: MuZhou233 <[email protected]> * Add buffered thread io Signed-off-by: MuZhou233 <[email protected]> * Replace DashMap with ThreadLocal Signed-off-by: MuZhou233 <[email protected]> * Rename variables and function & Remove unused call Signed-off-by: MuZhou233 <[email protected]> * Change fetch logic. close tikv#10867 Signed-off-by: MuZhou233 <[email protected]> * Add AtomicIOBytes Signed-off-by: MuZhou233 <[email protected]> * Split ThreadIOSentinel Signed-off-by: MuZhou233 <[email protected]> * add THREAD_IO_TOTAL for metrics bytes fetcher Signed-off-by: MuZhou233 <[email protected]> * Adjust code style & fix Signed-off-by: MuZhou233 <[email protected]> * improve write_bytes test Signed-off-by: MuZhou233 <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]>

…ocks (tikv#11591) ref tikv#11452 Signed-off-by: Yilin Chen <[email protected]>

* add datasink trait Signed-off-by: Zhenchi <[email protected]> * RAII removes on_reporter_closing Signed-off-by: Zhenchi <[email protected]> * fmt Signed-off-by: Zhenchi <[email protected]> * fix build ref tikv#11691 Signed-off-by: Zhenchi <[email protected]> * datasink -> data_sink Signed-off-by: Zhenchi <[email protected]> * fmt Signed-off-by: Zhenchi <[email protected]> * fix build Signed-off-by: Zhenchi <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]>

* clippy: Fixing up clippy errors. * Fix crate references * Fix anonymous lifetimes references. ref tikv#4301 Signed-off-by: Harold Dost <[email protected]> * Test Revert of Iterations ref tikv#4301 Signed-off-by: Harold Dost <[email protected]> * clippy: Enable check for rust-2018-idioms ref tikv#4301 Signed-off-by: Harold Dost <[email protected]>

ref tikv#11691 Signed-off-by: Zhenchi <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]>

Signed-off-by: linning <[email protected]>

* Use generic for api version to reduce runtime branching ref tikv#10938 Signed-off-by: andylokandy <[email protected]> * fix test Signed-off-by: andylokandy <[email protected]> * add test Signed-off-by: andylokandy <[email protected]> * fix clippy Signed-off-by: andylokandy <[email protected]> * address comment Signed-off-by: andylokandy <[email protected]> * fmt Signed-off-by: andylokandy <[email protected]> * address comment Signed-off-by: andylokandy <[email protected]> * address comment Signed-off-by: andylokandy <[email protected]> * rename vars Signed-off-by: andylokandy <[email protected]> * improve vars Signed-off-by: andylokandy <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]>

* sst_importer: check api version when importing ref tikv#10938 Signed-off-by: Peng Guanwen <[email protected]> * Fix bugs Signed-off-by: Peng Guanwen <[email protected]> * fix bug of Iterator::{scan,scan_cf} Signed-off-by: Peng Guanwen <[email protected]> * remove redundant log Signed-off-by: Peng Guanwen <[email protected]> * Show the entire key Signed-off-by: Peng Guanwen <[email protected]> * Fix lint issue Signed-off-by: Peng Guanwen <[email protected]> * resolve conflict Signed-off-by: andylokandy <[email protected]> Co-authored-by: Andy Lok <[email protected]>

* br: sperate io and scan threads Signed-off-by: Yu Juncen <[email protected]> * br/stream: make clippy happy Signed-off-by: yujuncen <[email protected]> * br/stream: fix tests Signed-off-by: Yu Juncen <[email protected]> * backup: fix a dummy bug Signed-off-by: Yu Juncen <[email protected]> * backup: add config of io-threads Signed-off-by: yujuncen <[email protected]> * backup: ref tikv#11350: some minior change Signed-off-by: yujuncen <[email protected]> * backup: added some metrics Signed-off-by: yujuncen <[email protected]> * backup: better tuning concurrency, and added a metric Signed-off-by: yujuncen <[email protected]> * backup: added some slow log and metrics Signed-off-by: yujuncen <[email protected]> * backup: address comments Signed-off-by: yujuncen <[email protected]> * backup: fix build Signed-off-by: yujuncen <[email protected]> * Revert "backup: fix build" This reverts commit 74537be. Signed-off-by: yujuncen <[email protected]> * Revert "backup: address comments" This reverts commit 77d7575. Signed-off-by: yujuncen <[email protected]> * backup: address comments Signed-off-by: yujuncen <[email protected]> * backup: always set IO type to Export Signed-off-by: yujuncen <[email protected]> * backup:make clippy happy Signed-off-by: yujuncen <[email protected]> * backup: address comments Signed-off-by: yujuncen <[email protected]>

ref tikv#11662 Signed-off-by: Ryan Leung <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]>

* Update kvproto Signed-off-by: Peng Guanwen <[email protected]> * Return api-version for br ref tikv#10938 Signed-off-by: Peng Guanwen <[email protected]> * Reformat code Signed-off-by: Peng Guanwen <[email protected]> * Update components/external_storage/export/src/export.rs Signed-off-by: Peng Guanwen <[email protected]> Co-authored-by: Andy Lok <[email protected]> * format code Signed-off-by: Peng Guanwen <[email protected]> Co-authored-by: Andy Lok <[email protected]>

* limit thread_io target os Signed-off-by: MuZhou233 <[email protected]> * limit thread_io target os. typo. close tikv#11698 Signed-off-by: MuZhou233 <[email protected]> Co-authored-by: Yilin Chen <[email protected]>

Close tikv#10688 Patch fs2 to https://github.com/tabokie/fs2-rs/tree/tikv. In which, `posix_fallocate` is replaced with `fallocate` (danburkert/fs2-rs#42). * bypass file allocate implementation in fs2 Signed-off-by: tabokie <[email protected]>

Minimum legend isn't very useful, and it occupied valuable visual real estate. Remove some min legends from Grafana metrics, except for the following: - Level 0 chance - Number files at each level - File Count - Entry Count - Allocator Stats - Encrypted files - Encryption meta files size Ref tikv#11119 Signed-off-by: tabokie <[email protected]>

…ation (tikv#11708) * not return rows when there is no input for simple aggregation Signed-off-by: xufei <[email protected]> * close tikv#11735, and address comments Signed-off-by: xufei <[email protected]>

…ne by default (tikv#11699) * backup: clamp auto tune values Signed-off-by: yujuncen <[email protected]> * ref tikv#11000: change the default backup threads. Signed-off-by: yujuncen <[email protected]> * backup: run tasks in background Signed-off-by: yujuncen <[email protected]> * backup: run rustfmt Signed-off-by: yujuncen <[email protected]> * backup: set remain threads to 20% of vcpu Signed-off-by: yujuncen <[email protected]> * br-stream: fix build Signed-off-by: yujuncen <[email protected]>

* s3: add request metrics for s3 storage Signed-off-by: 3pointer <[email protected]> * s3: add grafana json Signed-off-by: 3pointer <[email protected]> * br: add config for s3 multi part upload Signed-off-by: 3pointer <[email protected]> * update comment Signed-off-by: 3pointer <[email protected]> * update comment Signed-off-by: 3pointer <[email protected]> * address comment Signed-off-by: 3pointer <[email protected]> * hidden the new config Signed-off-by: 3pointer <[email protected]> * only hidden config in config-template.toml Signed-off-by: 3pointer <[email protected]> * address comment Signed-off-by: 3pointer <[email protected]> * close tikv#11727 && format Signed-off-by: 3pointer <[email protected]> * fix Signed-off-by: 3pointer <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]>

* rsmeter: support multiple datasinks Signed-off-by: Zhenchi <[email protected]> * wrap config notifier Signed-off-by: Zhenchi <[email protected]> * add unit tests Signed-off-by: Zhenchi <[email protected]> * polish ref tikv#11691 Signed-off-by: Zhenchi <[email protected]> * address comments Signed-off-by: Zhenchi <[email protected]> * fix test Signed-off-by: Zhenchi <[email protected]> * fix outdated comments Signed-off-by: Zhenchi <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]>

Signed-off-by: CalvinNeo <[email protected]>

…kv#12315) close tikv#12278, ref tikv#12309 Fix test_delegate_subscribe_unsubscribe flaking test. Use the corresponding id, otherwise there will be failures due to concurrent runs of the test. (We will assign downstream ids concurrently). Signed-off-by: ti-srebot <[email protected]> Co-authored-by: 二手掉包工程师 <[email protected]> Co-authored-by: qupeng <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]>

Signed-off-by: CalvinNeo <[email protected]>

…6.0-try

Signed-off-by: CalvinNeo <[email protected]>

Signed-off-by: Wish <[email protected]>

…raftstore-proxy-6.0-try

Signed-off-by: mornyx <[email protected]>

…g on ARM (pingcap#61) Signed-off-by: mornyx <[email protected]>

Signed-off-by: CalvinNeo <[email protected]>

close tikv#11379, close tikv#11598, close tikv#12107 when memory usage is high water, add flow control on the append log entries. disable log entry cache evict functions. Signed-off-by: tier-cap <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> Signed-off-by: CalvinNeo <[email protected]>

…ine-ext into raftstore-proxy-6.0-try

Signed-off-by: CalvinNeo <[email protected]>

* raftstore: Introduce force leader state (tikv#11932) close tikv#6107, ref tikv#10483 Signed-off-by: Connor1996 <[email protected]> * raftstore: Wait ticks for hibernated peer when doing force leader (tikv#12364) ref tikv#10483 Force leader is rejected on a peer who is already a leader. For the hibernated leader, it doesn't step down to follower when quorum is missing due to not ticking election. So wait ticks in force leader process for hibernated peers to make sure election ticking is performed. Signed-off-by: Connor1996 <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * raftstore: Make unsafe recovery wait apply cover snapshot apply cases ref tikv#10483 (tikv#12308) ref tikv#10483 Signed-off-by: v01dstar <[email protected]> * raftstore: Execute recovery plan via raft (tikv#12022) Signed-off-by: Connor1996 <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> Co-authored-by: Yang Zhang <[email protected]>

Signed-off-by: mornyx <[email protected]>

Signed-off-by: CalvinNeo <[email protected]>

CLAassistant · 2022-05-30T06:54:04Z

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
5 out of 8 committers have signed the CLA.

✅ CalvinNeo
✅ mornyx
✅ breezewish
✅ purelind
✅ tier-cap
❌ ti-srebot
❌ tabokie
❌ Connor1996
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

…mergeagain

Signed-off-by: CalvinNeo <[email protected]>

yuqi1129 and others added 30 commits December 15, 2021 14:08

metrics: fix memory metrics (tikv#11656)

c5c030d

Signed-off-by: Shuaipeng Yu <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]>

coprocesser: refactor encoding and add gbk_chinese_ci map (tikv#11661)

444edc0

* close tikv#11660 Signed-off-by: xiongjiwei <[email protected]> * close tikv#11660 Signed-off-by: xiongjiwei <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]>

cdc: load uninlined value more effectively (tikv#11615)

45fac77

Signed-off-by: qupeng <[email protected]>

cdc: load old value with prefix seek if possible (tikv#11643)

1b0edd0

* close tikv#11642 Signed-off-by: qupeng <[email protected]> * a more comment Signed-off-by: qupeng <[email protected]>

Fix build issues on non-linux platforms (tikv#11671)

ed82712

* close tikv#11676 Signed-off-by: devillove084 <[email protected]>

all: Migrate from std:: types to primitives. (tikv#11677)

c2ca032

* Deprecation is planned for MIN,MAX,EPSLION,INFINITY,NEG_INFINITY of the (i|u|f)(8|16|32|64) types in standard. ref tikv#11678 ```release-note NONE ``` Signed-off-by: Harold Dost <[email protected]>

metrics alert: fix apply cpu alert (tikv#11672)

186bca2

Signed-off-by: zhangjinpeng1987 <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]>

resrc_mtr: introduce scanned keys statistics (tikv#11085)

8c24715

close tikv#11690 Signed-off-by: mornyx <[email protected]>

*: check epoch and validness of lock table when writing pessimistic l…

a249ec7

…ocks (tikv#11591) ref tikv#11452 Signed-off-by: Yilin Chen <[email protected]>

rsmeter: add centralized place to construct protobuf data (tikv#11696)

849a01d

ref tikv#11691 Signed-off-by: Zhenchi <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]>

tikv_util: update procfs to 0.12.0 (tikv#11703)

92da184

Signed-off-by: linning <[email protected]>

metrics: fix the metrics cannot be displayed (tikv#11710)

205e1b8

ref tikv#11662 Signed-off-by: Ryan Leung <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]>

file_system: limit thread_io target os (tikv#11715)

b62e381

* limit thread_io target os Signed-off-by: MuZhou233 <[email protected]> * limit thread_io target os. typo. close tikv#11698 Signed-off-by: MuZhou233 <[email protected]> Co-authored-by: Yilin Chen <[email protected]>

coprocessor: not return rows when there is no input for simple aggreg…

8cbc5f6

…ation (tikv#11708) * not return rows when there is no input for simple aggregation Signed-off-by: xufei <[email protected]> * close tikv#11735, and address comments Signed-off-by: xufei <[email protected]>

CalvinNeo and others added 23 commits April 8, 2022 17:46

fix

782c299

Signed-off-by: CalvinNeo <[email protected]>

fix

762f08f

Signed-off-by: CalvinNeo <[email protected]>

ignore some unsupported tests

317086a

Signed-off-by: CalvinNeo <[email protected]>

fix fmt

8035118

Signed-off-by: CalvinNeo <[email protected]>

disable more disk_full tests

a5ff078

Signed-off-by: CalvinNeo <[email protected]>

f

1e31964

Signed-off-by: CalvinNeo <[email protected]>

disable

03b8215

Signed-off-by: CalvinNeo <[email protected]>

remove stale peer

77923ca

Signed-off-by: CalvinNeo <[email protected]>

Merge remote-tracking branch 'tikv/release-6.0' into raftstore-proxy-…

a03434d

…6.0-try

pick pr 12301

88168c6

Signed-off-by: CalvinNeo <[email protected]>

Fix build when openssl v1.1 and v3 are both installed (pingcap#59)

0ef80c7

Signed-off-by: Wish <[email protected]>

Merge remote-tracking branch 'upstream/raftstore-proxy-6.0-try' into …

6cb40b3

…raftstore-proxy-6.0-try

Enable frame pointer (pingcap#60)

2cf6ff7

Signed-off-by: mornyx <[email protected]>

Upgrade pprof-rs to v0.9 (use frame-pointer), and enable CPU Profilin…

e46e887

…g on ARM (pingcap#61) Signed-off-by: mornyx <[email protected]>

reduce log

f307053

Signed-off-by: CalvinNeo <[email protected]>

Merge branch 'raftstore-proxy-6.0-try' of github.com:pingcap/tidb-eng…

e98f959

…ine-ext into raftstore-proxy-6.0-try

fix pd.rs of reporting to PD

fe64c1d

Signed-off-by: CalvinNeo <[email protected]>

fix

5bd9c73

Signed-off-by: CalvinNeo <[email protected]>

Hotfix pprof (pingcap#65)

4c10d3b

Signed-off-by: mornyx <[email protected]>

remove log

7578b81

Signed-off-by: CalvinNeo <[email protected]>

Merge remote-tracking branch 'upstream/raftstore-proxy-6.0-try' into …

7c69927

…mergeagain

CalvinNeo force-pushed the mergeagain branch from d229097 to 7c69927 Compare May 30, 2022 06:56

remove log

53daaf5

Signed-off-by: CalvinNeo <[email protected]>

CalvinNeo marked this pull request as ready for review May 30, 2022 06:57

breezewish approved these changes May 30, 2022

View reviewed changes

CalvinNeo merged commit fbd13ee into pingcap:raftstore-proxy May 30, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cherry pick remaining commits from raftstore-proxy-6.1 to raftstore-proxy #72

Cherry pick remaining commits from raftstore-proxy-6.1 to raftstore-proxy #72

CalvinNeo commented May 30, 2022

CLAassistant commented May 30, 2022 •

edited

Loading

Cherry pick remaining commits from raftstore-proxy-6.1 to raftstore-proxy #72

Cherry pick remaining commits from raftstore-proxy-6.1 to raftstore-proxy #72

Conversation

CalvinNeo commented May 30, 2022

What problem does this PR solve?

What is changed and how it works?

Check List

Release note

CLAassistant commented May 30, 2022 • edited Loading

CLAassistant commented May 30, 2022 •

edited

Loading