Don't write and print out a message when meet empty batch #158

CalvinNeo · 2022-09-05T06:21:49Z

What problem does this PR solve?

Issue Number: close pingcap/tiflash#5783

Problem Summary:

What is changed and how it works?

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No code

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

None

close tikv#9765 Signed-off-by: YangKeao <[email protected]>

Signed-off-by: tier-cap <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]>

close tikv#12161 Signed-off-by: tabokie <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]>

close tikv#12235, ref tikv#12236 Shouldn't report config related errors when bucket is disabled. Signed-off-by: youjiali1995 <[email protected]>

close tikv#12218 Use the results returned by the cpu limiter directly without extra processing Signed-off-by: Xintao <[email protected]>

Signed-off-by: linning <[email protected]>

…2152) close tikv#11915 tikv_worker_pending_task_total is a gauge. We should directly use its value to represent the pending tasks instead of using rate(). This PR removes the rate() on these metrics. Signed-off-by: Yilin Chen <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]>

Signed-off-by: 5kbpers <[email protected]>

close tikv#12234 Signed-off-by: mornyx <[email protected]> Co-authored-by: Zhenchi <[email protected]>

) Signed-off-by: Xintao <[email protected]>

…cost (tikv#12264) Signed-off-by: Xintao <[email protected]>

Signed-off-by: qupeng <[email protected]>

…eeded (tikv#12251) close tikv#12232 raftstore: only persist merge target if the merge is known to be succeeded Signed-off-by: linning <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]>

close tikv#12240 split the initializer from the endpoint Signed-off-by: hi-rustin <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]>

…ikv#12280) Signed-off-by: Yilin Chen <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]>

close tikv#12238, ref tikv#12238 Signed-off-by: tabokie <[email protected]> Co-authored-by: Xintao <[email protected]>

close tikv#12269, ref tikv#12269 Signed-off-by: tabokie <[email protected]>

close tikv#12198, close tikv#12202 update grpcio to fix UAF Signed-off-by: Jay Lee <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]>

…v#12270) close tikv#11439 Signed-off-by: SpadeA-Tang <[email protected]>

Signed-off-by: tabokie <[email protected]>

close tikv#11379, close tikv#11598, close tikv#12107 when memory usage is high water, add flow control on the append log entries. disable log entry cache evict functions. Signed-off-by: tier-cap <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]>

close tikv#12014 Signed-off-by: Jayice <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> Co-authored-by: disksing <[email protected]>

close tikv#12046 If a read index request travel too long in the network, a leader may receive the request it sent when it's still follower and break the strict order pending read assumption. This PR solves the problem by ignoring invalid read states. It also adds a defense check to ignore all ReadIndex messages when it's sent by current leader. Signed-off-by: Jay Lee <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]>

ref tikv#11965 * Add `causal_ts` component for causal timestamp generation & management. * Implement causal timestamp provider from "batch" TSO from PD (see `causal_ts::BatchTsoProvider`). * Append timestamp to keys of RawKV in `QueryObserver.pre_propose_query`. * Setup causal timestamp observer when API V2 enabled (comment out) Signed-off-by: pingyu <[email protected]> Signed-off-by: Andy Lok <[email protected]> Co-authored-by: Andy Lok <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]>

Signed-off-by: hi-rustin <[email protected]>

close tikv#12257 Signed-off-by: glorv <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]>

Signed-off-by: qi.xu <[email protected]>

…v#12328) Signed-off-by: SpadeA-Tang <[email protected]>

close tikv#12322 Use generic to reduce duplicated code and extract KvEngineFactory for reuse in tests and future usage. Signed-off-by: Jay Lee <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]>

ref tikv#11965, ref tikv#12207 Add APIVersion as generic parameter of `Storage`. Signed-off-by: pingyu <[email protected]>

Signed-off-by: CalvinNeo <[email protected]>

Signed-off-by: hehechen <[email protected]>

…#119) Signed-off-by: CalvinNeo <[email protected]>

Introduce crate proxy_server for a decouple of component/server, it will be used to start a proxy service. Introduce crate new-mock-engine-store as a new mock-engine-store plus a test_raftstore implementation for proxy. This will enable us to run mix-mode tests. Tests in tests/failpoints and tests/integrations are obsolete tests and will be removed once the refactor ends. New tests should be added in tests/proxy with support of new-mock-engine-store. Introduce crate engine_tiflash as a new KvEngine. It currently just works like former RocksEngine, unless we enable compat_new_proxy. Since some PRs have not been merged in TiKV, we can only enable this feature in some tests in tests/proxy. It can not be enabled in formal codes, since it can filter some writes. cherry pick newly merge TiKV PRs, aka on_empty_cmd. Fix some bugs. * f Signed-off-by: CalvinNeo <[email protected]> * fix Signed-off-by: CalvinNeo <[email protected]> * add engine_tiflash with this version of engine_rocks Signed-off-by: CalvinNeo <[email protected]> * fmt Signed-off-by: CalvinNeo <[email protected]> * add engine_tiflash actually Signed-off-by: CalvinNeo <[email protected]> * fix Signed-off-by: CalvinNeo <[email protected]> * fix some tests Signed-off-by: CalvinNeo <[email protected]> * f Signed-off-by: CalvinNeo <[email protected]> * it runs if we do_write everytime Signed-off-by: CalvinNeo <[email protected]> * add ensure_no_common_unrecognized_keys to config checker Signed-off-by: CalvinNeo <[email protected]> * reorg tests Signed-off-by: CalvinNeo <[email protected]> * raftstore: Implement coprocessor observer on_empty_cmd (tikv#12851) ref tikv#12849 Support new observers on_empty_cmd. Signed-off-by: CalvinNeo <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * fix tests Signed-off-by: CalvinNeo <[email protected]> * move proxy-related code in component/server into proxy_server Signed-off-by: CalvinNeo <[email protected]> * move codes in component/server/src/setup.rs into proxy_server Signed-off-by: CalvinNeo <[email protected]> * support new ffis in mock-engine-store and new-mock-engine-store Signed-off-by: CalvinNeo <[email protected]> * add disclaimer Signed-off-by: CalvinNeo <[email protected]> * enlength some test time Signed-off-by: CalvinNeo <[email protected]> * Remove pub fields of PdCluster, add feature to origin import Signed-off-by: CalvinNeo <[email protected]> * f Signed-off-by: CalvinNeo <[email protected]> * add Signed-off-by: CalvinNeo <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]>

* f Signed-off-by: CalvinNeo <[email protected]> * fix Signed-off-by: CalvinNeo <[email protected]> * add engine_tiflash with this version of engine_rocks Signed-off-by: CalvinNeo <[email protected]> * fmt Signed-off-by: CalvinNeo <[email protected]> * add engine_tiflash actually Signed-off-by: CalvinNeo <[email protected]> * fix Signed-off-by: CalvinNeo <[email protected]> * fix some tests Signed-off-by: CalvinNeo <[email protected]> * f Signed-off-by: CalvinNeo <[email protected]> * it runs if we do_write everytime Signed-off-by: CalvinNeo <[email protected]> * add ensure_no_common_unrecognized_keys to config checker Signed-off-by: CalvinNeo <[email protected]> * reorg tests Signed-off-by: CalvinNeo <[email protected]> * raftstore: Implement coprocessor observer on_empty_cmd (tikv#12851) ref tikv#12849 Support new observers on_empty_cmd. Signed-off-by: CalvinNeo <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * fix tests Signed-off-by: CalvinNeo <[email protected]> * move proxy-related code in component/server into proxy_server Signed-off-by: CalvinNeo <[email protected]> * move codes in component/server/src/setup.rs into proxy_server Signed-off-by: CalvinNeo <[email protected]> * support new ffis in mock-engine-store and new-mock-engine-store Signed-off-by: CalvinNeo <[email protected]> * add disclaimer Signed-off-by: CalvinNeo <[email protected]> * enlength some test time Signed-off-by: CalvinNeo <[email protected]> * f Signed-off-by: CalvinNeo <[email protected]> * fmt Signed-off-by: CalvinNeo <[email protected]> * fix tests Signed-off-by: CalvinNeo <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]>

We add support for new observer pre_exec in this PR. We will filter ComputeHash and VerifyHash now. We will firstly try flush then CompactLog if succeed.

Signed-off-by: CalvinNeo <[email protected]>

and unsafe backoff * raftstore: use force_send to send ApplyRes (tikv#13168) close tikv#13160 Use force_send to send ApplyRes Signed-off-by: 5kbpers <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> Co-authored-by: 5kbpers <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]>

close tikv#13217, ref tikv#13217 support cgroup limit memory in diagnostics service Signed-off-by: Lloyd-Pottiger <[email protected]> Co-authored-by: Lloyd-Pottiger <[email protected]> Co-authored-by: Lloyd-Pottiger <[email protected]> Co-authored-by: Calvin Neo <[email protected]>

) * raftstore: Implement observer on_compute_engine_size (tikv#12948) ref tikv#12849 Implement observer on_compute_engine_size Signed-off-by: CalvinNeo <[email protected]> Co-authored-by: Xinye Tao <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * inc gc time Signed-off-by: CalvinNeo <[email protected]> Co-authored-by: Xinye Tao <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]>

…cap#139)

…cap#146)

Signed-off-by: CalvinNeo <[email protected]>

CLAassistant · 2022-09-05T06:22:05Z

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
4 out of 7 committers have signed the CLA.

✅ Lloyd-Pottiger
✅ heibaijian
✅ CalvinNeo
✅ hehechen
❌ tonyxuqqi
❌ pingyu
❌ ti-srebot
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

Signed-off-by: CalvinNeo <[email protected]>

YangKeao and others added 30 commits March 19, 2022 16:58

add vdso to the pprof blocklist (tikv#12201)

9a43faf

close tikv#9765 Signed-off-by: YangKeao <[email protected]>

raftstore:Fix memory metrics overflow on raft msgs (tikv#12186)

75526df

Signed-off-by: tier-cap <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]>

do not force compact unreplicated entries (tikv#12181)

ac32d5c

close tikv#12161 Signed-off-by: tabokie <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]>

raftstore: validate bucket config only when it's enabled (tikv#12237)

7ff2b94

close tikv#12235, ref tikv#12236 Shouldn't report config related errors when bucket is disabled. Signed-off-by: youjiali1995 <[email protected]>

Quota limiter: make method to delay cpu more precise (tikv#12219)

b48ebc1

close tikv#12218 Use the results returned by the cpu limiter directly without extra processing Signed-off-by: Xintao <[email protected]>

fix shutdown panic (tikv#12244)

869b953

Signed-off-by: linning <[email protected]>

raftstore: report buckets to pd (tikv#12168)

b555984

Signed-off-by: 5kbpers <[email protected]>

res_meter: fix assertion failure (tikv#12252)

1b35044

close tikv#12234 Signed-off-by: mornyx <[email protected]> Co-authored-by: Zhenchi <[email protected]>

Update procinfo-rs to support pid status parse in kernel 5.x (tikv#12263

e089324

) Signed-off-by: Xintao <[email protected]>

Quota limiter: increase refill duration to avoid much context switch …

a9c1e59

…cost (tikv#12264) Signed-off-by: Xintao <[email protected]>

cdc: remove downstreams correctly when uninitialized (tikv#12262)

7c10093

Signed-off-by: qupeng <[email protected]>

raftstore: only persist merge target if the merge is known to be succ…

3364702

…eeded (tikv#12251) close tikv#12232 raftstore: only persist merge target if the merge is known to be succeeded Signed-off-by: linning <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]>

cdc: split the initializer from the endpoint (tikv#12241)

f3a2309

close tikv#12240 split the initializer from the endpoint Signed-off-by: hi-rustin <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]>

storage: send old value to CDC after in-memory pessimistic locking (t…

a3bf40b

…ikv#12280) Signed-off-by: Yilin Chen <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]>

server: fix issues with critical config check (tikv#12250)

db98134

close tikv#12238, ref tikv#12238 Signed-off-by: tabokie <[email protected]> Co-authored-by: Xintao <[email protected]>

server: do not rename directories when migrating raft data (tikv#12273)

11f571d

close tikv#12269, ref tikv#12269 Signed-off-by: tabokie <[email protected]>

*: update grpcio (tikv#12203)

4f3e720

close tikv#12198, close tikv#12202 update grpcio to fix UAF Signed-off-by: Jay Lee <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]>

Storage: remove unnecessary multi attribute in PointerGetter (tik…

0e5741a

…v#12270) close tikv#11439 Signed-off-by: SpadeA-Tang <[email protected]>

update rust-rocksdb (tikv#12301)

6fb61eb

Signed-off-by: tabokie <[email protected]>

config:consider config change in readstats (tikv#12164)

356774c

close tikv#12014 Signed-off-by: Jayice <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> Co-authored-by: disksing <[email protected]>

Fix test_delegate_subscribe_unsubscribe flaking test (tikv#12309)

fd2bdd2

Signed-off-by: hi-rustin <[email protected]>

quota limiter: support online config (tikv#12258)

bf09ca6

close tikv#12257 Signed-off-by: glorv <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]>

Raftstore: support bucket split/merge -- tikv#11759 (tikv#12132)

379180b

Signed-off-by: qi.xu <[email protected]>

txn: remove unnecessary length check in SnapshotStore::batch_get (tik…

0cfc9af

…v#12328) Signed-off-by: SpadeA-Tang <[email protected]>

*: simplify setup (tikv#12323)

923870f

close tikv#12322 Use generic to reduce duplicated code and extract KvEngineFactory for reuse in tests and future usage. Signed-off-by: Jay Lee <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]>

storage: add APIVersion as generic parameter (tikv#12324)

ed7e1aa

ref tikv#11965, ref tikv#12207 Add APIVersion as generic parameter of `Storage`. Signed-off-by: pingyu <[email protected]>

CalvinNeo and others added 25 commits July 4, 2022 15:28

fix mock-engine-store bugs

993b09e

Signed-off-by: CalvinNeo <[email protected]>

disable some check

45194f2

Signed-off-by: CalvinNeo <[email protected]>

fix test merge failure

d436191

Signed-off-by: CalvinNeo <[email protected]>

introduce only-decription mode (pingcap#118)

6ea4d60

Signed-off-by: hehechen <[email protected]>

Introduce new FFIs: fn_need_flush_data and fn_try_flush_data (pingcap…

42ede65

…#119) Signed-off-by: CalvinNeo <[email protected]>

Migrate pre exec (pingcap#123)

3bd934f

We add support for new observer pre_exec in this PR. We will filter ComputeHash and VerifyHash now. We will firstly try flush then CompactLog if succeed.

Fix commands are filtered without no post_exec to catch (pingcap#130)

cdd5996

Pass index and term to pre_exec (pingcap#133)

ebdaccf

Fix RegionEpoch Error and CheckWaitIndex takes too long (pingcap#134)

ae2fe14

Re-enable debug service (pingcap#131)

714b6ce

Handle destroy peer by observer (pingcap#132)

2eea34c

Signed-off-by: CalvinNeo <[email protected]>

Avoid set addr/engine-addr and HackedLockManager in src/node.rs (ping…

70de535

…cap#139)

Fix error parse of ProxyConfig (pingcap#144)

3ed58cc

Implement Proxy's own status_server (pingcap#141)

d21b9e8

Support post_exec on Raftstore Proxy (pingcap#140)

0703d9f

Use proxy's own config checker (pingcap#153)

2feed33

Decouple apply snapshot by introducing pre(post)_apply_snapshot (ping…

6686c34

…cap#146)

Remove engine store server helper in raftstore (pingcap#148)

1edf78e

Reclaim TiKV's component server and status server (pingcap#152)

6ff4158

f

6083975

Signed-off-by: CalvinNeo <[email protected]>

CalvinNeo and others added 3 commits September 5, 2022 15:02

Decouple engine_store_ffi and raftstore (pingcap#157)

d855fa2

f

f1beee2

Signed-off-by: CalvinNeo <[email protected]>

Merge branch 'raftstore-proxy-6.2' into fix-empty-batch-proxy

e73c203

CalvinNeo closed this Sep 5, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Don't write and print out a message when meet empty batch #158

Don't write and print out a message when meet empty batch #158

CalvinNeo commented Sep 5, 2022

CLAassistant commented Sep 5, 2022 •

edited

Loading

Don't write and print out a message when meet empty batch #158

Don't write and print out a message when meet empty batch #158

Conversation

CalvinNeo commented Sep 5, 2022

What problem does this PR solve?

What is changed and how it works?

Check List

Release note

CLAassistant commented Sep 5, 2022 • edited Loading

CLAassistant commented Sep 5, 2022 •

edited

Loading