Balance fix #3668

liwenhui-soul · 2022-01-07T14:25:41Z

What type of PR is this?

bug
feature
enhancement

What problem(s) does this PR solve?

Issue(s) number:

Description:

How do you solve it?

Special notes for your reviewer, ex. impact of this fix, design document, etc:

Checklist:

Tests:

Unit test(positive and negative cases)
Function test
Performance test
N/A

Affects:

Documentation affected (Please add the label if documentation needs to be modified.)
Incompatibility (If it breaks the compatibility, please describe it and add the label.）
If it's needed to cherry-pick (If cherry-pick to some branches is required, please label the destination version(s).)
Performance impacted: Consumes more CPU/Memory

Release notes:

Please confirm whether to be reflected in release notes and how to describe:

ex. Fixed the bug .....

Sophie-Xie · 2022-01-10T00:51:07Z

Which issues does this PR solve?

HarrisChu · 2022-01-10T01:31:33Z

fix #3654

src/meta/processors/job/BalanceJobExecutor.cpp

critical27

Generally LGTM, maybe add some comments

critical27 · 2022-01-14T09:03:39Z

src/meta/processors/job/JobManager.cpp

@@ -144,7 +144,7 @@ void JobManager::scheduleThread() {

 // @return: true if all task dispatched, else false
 bool JobManager::runJobInternal(const JobDescription& jobDesc, JbOp op) {
-  std::lock_guard<std::mutex> lk(muJobFinished_);
+  std::lock_guard<std::recursive_mutex> lk(muJobFinished_);


Why we need to use recursive_mutex here?

if all balance tasks success and the job is stopped before job status updated, the balance tasks will get muJobFinished_ through call back in the same thread

I got it. Please add some comments. Or ,aybe trigger the callback of BalancePlan in another thread is more clear.

have added comment at its declared place

critical27 · 2022-01-20T08:44:13Z

Close #3672

critical27

Generally LGTM

critical27 · 2022-01-20T08:48:32Z

src/kvstore/raftex/RaftexService.cpp

+        folly::RWSpinLock::WriteHolder wh(partsLock_);
+        parts_.erase(std::make_pair(part->spaceId(), part->partitionId()));
+      })
+      .via(getThreadManager().get())


Maybe add some comments here. Do you think we should use a global executor here?

…thread

critical27

Good job

* fix core balance in zone removing the only host in zone * fix balance plan that make redundant part * fix deadlock when runInternal and finish callback called in the same thread * fix transleader return TERM_OUT_OF_DATE * continue running balance task from last status * fix deadlock when removing part Co-authored-by: Sophie <[email protected]>

* drop host only when it registe heartbeat (#3739) * Fix race of create tag and edge (#3735) * fix rece * polish * fix review * Add version for `show hosts` (#3702) * version * tck * pytest * pytest Co-authored-by: cpw <[email protected]> * Update client version to v3.0.0 (#3741) * Update python package name to v3 * Update client version white list * Update go mod to v3 * Change the python package name used in tests * Update client version white list * Fix create fulltext index failed (#3747) * fix bug: the same tagId/edgetype under different spaces, failed to create fulltext indexes * fix bug: the same tagId/edgetype under different spaces, failed to create fulltext indexes * fix writternBy delete by optimizer (#3746) * fix bugs * add tck features * style Co-authored-by: cpw <[email protected]> * fix with & optimizer rule error (#3736) * fix error * fix collapseProjectRule Co-authored-by: cpw <[email protected]> * fix upgrade bug (#3772) Co-authored-by: Sophie <[email protected]> * fix 3727 (#3737) Co-authored-by: Doodle <[email protected]> * Balance fix (#3668) * fix core balance in zone removing the only host in zone * fix balance plan that make redundant part * fix deadlock when runInternal and finish callback called in the same thread * fix transleader return TERM_OUT_OF_DATE * continue running balance task from last status * fix deadlock when removing part Co-authored-by: Sophie <[email protected]> * Add snapshot for prefix iterate. (#3717) * Add snapshot for prefix iterate. * Add header comment * Add happy test * Fix lint * Modify engine snap argument position * Fix other subclass * Fix code Co-authored-by: yaphet <[email protected]> Co-authored-by: Sophie <[email protected]> Co-authored-by: Doodle <[email protected]> * suppress undesired ASAN error (#3705) * suppress undesired error * fix unstable test case * fix upgrade (#3787) Co-authored-by: Doodle <[email protected]> * Create CODEOWNERS (#3770) Confirm with sophie.xie * disable data balance (#3756) Co-authored-by: Sophie <[email protected]> * Fix iterator with snapshot(parameter position) (#3785) Co-authored-by: Harris.Chu <[email protected]> Co-authored-by: Sophie <[email protected]> Co-authored-by: yaphet <[email protected]> Co-authored-by: jakevin <[email protected]> Co-authored-by: cpw <[email protected]> Co-authored-by: Yichen Wang <[email protected]> Co-authored-by: panda-sheep <[email protected]> Co-authored-by: endy.li <[email protected]> Co-authored-by: jimingquan <[email protected]> Co-authored-by: hs.zhang <[email protected]> Co-authored-by: Doodle <[email protected]> Co-authored-by: liwenhui-soul <[email protected]> Co-authored-by: Alex Xing <[email protected]> Co-authored-by: Harris.Chu <[email protected]>

@panda-sheep

* add cmake git hooks options * githooks option * include what you use * make name follow convention * linker support lld, which is the fastest linker. If ci install lld when pull request , will set lld as default linker. * log rotate * delete werror * pragma once => ifndef * pragma once => ifndef * Werror bison * format code * fix issue 3601 (#3666) * fix meta upgrade for multi instance (#3734) * drop host only when it registe heartbeat (#3739) * Fix race of create tag and edge (#3735) * fix rece * polish * fix review * Add version for `show hosts` (#3702) * version * tck * pytest * pytest Co-authored-by: cpw <[email protected]> * Enhancement/set expression depth by flag (#3722) * Set expression depth limit by flag. * Format. Co-authored-by: Yichen Wang <[email protected]> * Update client version to v3.0.0 (#3741) * Update python package name to v3 * Update client version white list * Update go mod to v3 * Change the python package name used in tests * Update client version white list * Fix create fulltext index failed (#3747) * fix bug: the same tagId/edgetype under different spaces, failed to create fulltext indexes * fix bug: the same tagId/edgetype under different spaces, failed to create fulltext indexes * add comments and adjust log level for meta/processors/admin (#3662) * add comments and adjust log level for meta admin processors * change agent hb log level Co-authored-by: Doodle <[email protected]> * fix writternBy delete by optimizer (#3746) * fix bugs * add tck features * style Co-authored-by: cpw <[email protected]> * fix with & optimizer rule error (#3736) * fix error * fix collapseProjectRule Co-authored-by: cpw <[email protected]> * fix upgrade bug (#3772) Co-authored-by: Sophie <[email protected]> * fix 3727 (#3737) Co-authored-by: Doodle <[email protected]> * Balance fix (#3668) * fix core balance in zone removing the only host in zone * fix balance plan that make redundant part * fix deadlock when runInternal and finish callback called in the same thread * fix transleader return TERM_OUT_OF_DATE * continue running balance task from last status * fix deadlock when removing part Co-authored-by: Sophie <[email protected]> * Eliminte the useless row collect. (#3363) * Eliminte the useless row collect. * Remove comment. * Format. * Rebase. Co-authored-by: cpw <[email protected]> Co-authored-by: Sophie <[email protected]> * Add snapshot for prefix iterate. (#3717) * Add snapshot for prefix iterate. * Add header comment * Add happy test * Fix lint * Modify engine snap argument position * Fix other subclass * Fix code Co-authored-by: yaphet <[email protected]> Co-authored-by: Sophie <[email protected]> Co-authored-by: Doodle <[email protected]> * Read lock zone when create space. (#3693) Co-authored-by: yaphet <[email protected]> Co-authored-by: Sophie <[email protected]> * suppress undesired ASAN error (#3705) * suppress undesired error * fix unstable test case * 1. add comments and change log level for meta (#3709) 2. remove some unused meta kv files 3. remove unused tests 4. remove unused kv relative tests in metaclient Co-authored-by: Doodle <[email protected]> * fix upgrade (#3787) Co-authored-by: Doodle <[email protected]> * Create CODEOWNERS (#3770) Confirm with sophie.xie * disable data balance (#3756) Co-authored-by: Sophie <[email protected]> * Fix iterator with snapshot(parameter position) (#3785) Co-authored-by: Harris.Chu <[email protected]> Co-authored-by: Sophie <[email protected]> * disable meta upgrade from v1 (#3798) * Add gflag validator for client_idle_timeout_secs (#3807) Co-authored-by: Sophie <[email protected]> * disable zone commands (#3776) Co-authored-by: cpw <[email protected]> * Add ci for standalone compile and ut. (#3637) * Add standalone workflow * remove tmp note * Add meta v2 thrift obj for sa * Add tck test * Fix port and dist tck * skip some test on sa * Fix merge * Fix add host Co-authored-by: Sophie <[email protected]> * Fix the promise/future. (#3814) Co-authored-by: jimingquan <[email protected]> * add comments & accumulate bug fix for TOSS (#3643) * 1. add comments & adjust some log level 2. accumulate bug fix for TOSS 2.1 one space may be affected by another space. when recover prime edges. 2.2 when call kvstore->get(), if leaderLeaseInvalid, it will report leader_change, change it to leader_lease_failed. 2.3 print all execution trace in ChainProcessor::finish(), make much more grep friendly. 2.4 some function of memory lock is hard to understand (forceLock, forceUnlock) , change to "setAutoUnlock(bool)" modify update fix compile error adjust finish() adjust finish() add rcPrepare_ = succeeded adjust delete local processor adjust log use rcPrepare, rcRemote rcCommit fix resume add edges adjust UT disable ioThreadPoolForMeta_ make int64 vid printable adjust some log to vlog 2 only print prepare succeeded result only print readable in finish() of resume processor adjust some log some log not print term desc class name in uuid debug add some more log looks like factory may fail? add leader lease error code add check when get lock report leader lease do some clean change table prefix to NebulaKeyUtil Style change some print disable UT for short clean compile error disable pool monitor * unify add/delete finish() as update * not stable will add it back later * fix a clang warning * fix a clang10 warning 2 Co-authored-by: Sophie <[email protected]> * disable add hosts into zone (#3825) Co-authored-by: Sophie <[email protected]> * Check active zone before create space (#3822) Co-authored-by: Sophie <[email protected]> * abandon upgrader for old version (#3818) Co-authored-by: Sophie <[email protected]> * put dataVersionKey on create space for storage (#3817) * put dataVersionKey on create space for storage * address some comments Co-authored-by: Doodle <[email protected]> Co-authored-by: Sophie <[email protected]> * Update PR template (#3809) Co-authored-by: Sophie <[email protected]> * fix meta http replace host handler (#3794) * fix meta http replace host handler * fix http replace test and add zone test Co-authored-by: Sophie <[email protected]> * Fix `Match cannot be followed by use space` (#3748) * fix bug * add tck * remove set * fix * rm header * fix * remove changes in match * fix * enhance annotation * fix review * fix * fix review * remove redundant Co-authored-by: Sophie <[email protected]> Co-authored-by: cpw <[email protected]> * support toSet function (#3594) add test Co-authored-by: Sophie <[email protected]> * Enhance date parser (#3797) * parse date * enhance datetime * add test * fix review * rename Co-authored-by: Sophie <[email protected]> * raft: fix always loop when no log need to send (#3819) Co-authored-by: Sophie <[email protected]> Co-authored-by: Doodle <[email protected]> * Update CODEOWNERS (#3835) * Update CODEOWNERS * Update CODEOWNERS Confirm with sophie.xie * refactor AdminClient (#3781) * refactor AdminClient * fix test case, unify all getAdminAddrFromStoreAddr into AdminClient * add more test cases * address @panda-sheep's comment * Feature/optimizer trait match (#3769) * Add trait for pattern match. * Implement match plan by trait and reduce some reduncant rules. * Remove unused header. * Fix warning. * Let optimize rule decide collection of matched nodes. Co-authored-by: kyle.cao <[email protected]> Co-authored-by: Sophie <[email protected]> * Simplify datetime tostring. (#3712) * Simplify datetime tostring. * Remove unused code. * Fix error. Co-authored-by: Sophie <[email protected]> * pragma once not allowd (#3842) Co-authored-by: Sophie <[email protected]> * refactor LOG and add comment for meta, meta/processors/job (#3686) * refactor LOG and add comment for meta, meta/processors/job * refactor LOG and add comment for meta/processors/zone Co-authored-by: Doodle <[email protected]> * replace STRING to name_label for zone's name (#3780) Co-authored-by: Doodle <[email protected]> * Add test for rc. (#3847) * Add test for rc. * Fix cxx. Confirm with sophie.xie * cpp lint Co-authored-by: Yee <[email protected]> Co-authored-by: Sophie <[email protected]> Co-authored-by: cpw <[email protected]> Co-authored-by: hs.zhang <[email protected]> Co-authored-by: yaphet <[email protected]> Co-authored-by: jakevin <[email protected]> Co-authored-by: shylock <[email protected]> Co-authored-by: Yichen Wang <[email protected]> Co-authored-by: panda-sheep <[email protected]> Co-authored-by: pengwei.song <[email protected]> Co-authored-by: Doodle <[email protected]> Co-authored-by: endy.li <[email protected]> Co-authored-by: jimingquan <[email protected]> Co-authored-by: liwenhui-soul <[email protected]> Co-authored-by: Alex Xing <[email protected]> Co-authored-by: Harris.Chu <[email protected]> Co-authored-by: [email protected] <[email protected]> Co-authored-by: kyle.cao <[email protected]>

@panda-sheep

* add cmake git hooks options * githooks option * include what you use * make name follow convention * linker support lld, which is the fastest linker. If ci install lld when pull request , will set lld as default linker. * log rotate * delete werror * pragma once => ifndef * pragma once => ifndef * Werror bison * format code * fix issue 3601 (vesoft-inc#3666) * fix meta upgrade for multi instance (vesoft-inc#3734) * drop host only when it registe heartbeat (vesoft-inc#3739) * Fix race of create tag and edge (vesoft-inc#3735) * fix rece * polish * fix review * Add version for `show hosts` (vesoft-inc#3702) * version * tck * pytest * pytest Co-authored-by: cpw <[email protected]> * Enhancement/set expression depth by flag (vesoft-inc#3722) * Set expression depth limit by flag. * Format. Co-authored-by: Yichen Wang <[email protected]> * Update client version to v3.0.0 (vesoft-inc#3741) * Update python package name to v3 * Update client version white list * Update go mod to v3 * Change the python package name used in tests * Update client version white list * Fix create fulltext index failed (vesoft-inc#3747) * fix bug: the same tagId/edgetype under different spaces, failed to create fulltext indexes * fix bug: the same tagId/edgetype under different spaces, failed to create fulltext indexes * add comments and adjust log level for meta/processors/admin (vesoft-inc#3662) * add comments and adjust log level for meta admin processors * change agent hb log level Co-authored-by: Doodle <[email protected]> * fix writternBy delete by optimizer (vesoft-inc#3746) * fix bugs * add tck features * style Co-authored-by: cpw <[email protected]> * fix with & optimizer rule error (vesoft-inc#3736) * fix error * fix collapseProjectRule Co-authored-by: cpw <[email protected]> * fix upgrade bug (vesoft-inc#3772) Co-authored-by: Sophie <[email protected]> * fix 3727 (vesoft-inc#3737) Co-authored-by: Doodle <[email protected]> * Balance fix (vesoft-inc#3668) * fix core balance in zone removing the only host in zone * fix balance plan that make redundant part * fix deadlock when runInternal and finish callback called in the same thread * fix transleader return TERM_OUT_OF_DATE * continue running balance task from last status * fix deadlock when removing part Co-authored-by: Sophie <[email protected]> * Eliminte the useless row collect. (vesoft-inc#3363) * Eliminte the useless row collect. * Remove comment. * Format. * Rebase. Co-authored-by: cpw <[email protected]> Co-authored-by: Sophie <[email protected]> * Add snapshot for prefix iterate. (vesoft-inc#3717) * Add snapshot for prefix iterate. * Add header comment * Add happy test * Fix lint * Modify engine snap argument position * Fix other subclass * Fix code Co-authored-by: yaphet <[email protected]> Co-authored-by: Sophie <[email protected]> Co-authored-by: Doodle <[email protected]> * Read lock zone when create space. (vesoft-inc#3693) Co-authored-by: yaphet <[email protected]> Co-authored-by: Sophie <[email protected]> * suppress undesired ASAN error (vesoft-inc#3705) * suppress undesired error * fix unstable test case * 1. add comments and change log level for meta (vesoft-inc#3709) 2. remove some unused meta kv files 3. remove unused tests 4. remove unused kv relative tests in metaclient Co-authored-by: Doodle <[email protected]> * fix upgrade (vesoft-inc#3787) Co-authored-by: Doodle <[email protected]> * Create CODEOWNERS (vesoft-inc#3770) Confirm with sophie.xie * disable data balance (vesoft-inc#3756) Co-authored-by: Sophie <[email protected]> * Fix iterator with snapshot(parameter position) (vesoft-inc#3785) Co-authored-by: Harris.Chu <[email protected]> Co-authored-by: Sophie <[email protected]> * disable meta upgrade from v1 (vesoft-inc#3798) * Add gflag validator for client_idle_timeout_secs (vesoft-inc#3807) Co-authored-by: Sophie <[email protected]> * disable zone commands (vesoft-inc#3776) Co-authored-by: cpw <[email protected]> * Add ci for standalone compile and ut. (vesoft-inc#3637) * Add standalone workflow * remove tmp note * Add meta v2 thrift obj for sa * Add tck test * Fix port and dist tck * skip some test on sa * Fix merge * Fix add host Co-authored-by: Sophie <[email protected]> * Fix the promise/future. (vesoft-inc#3814) Co-authored-by: jimingquan <[email protected]> * add comments & accumulate bug fix for TOSS (vesoft-inc#3643) * 1. add comments & adjust some log level 2. accumulate bug fix for TOSS 2.1 one space may be affected by another space. when recover prime edges. 2.2 when call kvstore->get(), if leaderLeaseInvalid, it will report leader_change, change it to leader_lease_failed. 2.3 print all execution trace in ChainProcessor::finish(), make much more grep friendly. 2.4 some function of memory lock is hard to understand (forceLock, forceUnlock) , change to "setAutoUnlock(bool)" modify update fix compile error adjust finish() adjust finish() add rcPrepare_ = succeeded adjust delete local processor adjust log use rcPrepare, rcRemote rcCommit fix resume add edges adjust UT disable ioThreadPoolForMeta_ make int64 vid printable adjust some log to vlog 2 only print prepare succeeded result only print readable in finish() of resume processor adjust some log some log not print term desc class name in uuid debug add some more log looks like factory may fail? add leader lease error code add check when get lock report leader lease do some clean change table prefix to NebulaKeyUtil Style change some print disable UT for short clean compile error disable pool monitor * unify add/delete finish() as update * not stable will add it back later * fix a clang warning * fix a clang10 warning 2 Co-authored-by: Sophie <[email protected]> * disable add hosts into zone (vesoft-inc#3825) Co-authored-by: Sophie <[email protected]> * Check active zone before create space (vesoft-inc#3822) Co-authored-by: Sophie <[email protected]> * abandon upgrader for old version (vesoft-inc#3818) Co-authored-by: Sophie <[email protected]> * put dataVersionKey on create space for storage (vesoft-inc#3817) * put dataVersionKey on create space for storage * address some comments Co-authored-by: Doodle <[email protected]> Co-authored-by: Sophie <[email protected]> * Update PR template (vesoft-inc#3809) Co-authored-by: Sophie <[email protected]> * fix meta http replace host handler (vesoft-inc#3794) * fix meta http replace host handler * fix http replace test and add zone test Co-authored-by: Sophie <[email protected]> * Fix `Match cannot be followed by use space` (vesoft-inc#3748) * fix bug * add tck * remove set * fix * rm header * fix * remove changes in match * fix * enhance annotation * fix review * fix * fix review * remove redundant Co-authored-by: Sophie <[email protected]> Co-authored-by: cpw <[email protected]> * support toSet function (vesoft-inc#3594) add test Co-authored-by: Sophie <[email protected]> * Enhance date parser (vesoft-inc#3797) * parse date * enhance datetime * add test * fix review * rename Co-authored-by: Sophie <[email protected]> * raft: fix always loop when no log need to send (vesoft-inc#3819) Co-authored-by: Sophie <[email protected]> Co-authored-by: Doodle <[email protected]> * Update CODEOWNERS (vesoft-inc#3835) * Update CODEOWNERS * Update CODEOWNERS Confirm with sophie.xie * refactor AdminClient (vesoft-inc#3781) * refactor AdminClient * fix test case, unify all getAdminAddrFromStoreAddr into AdminClient * add more test cases * address @panda-sheep's comment * Feature/optimizer trait match (vesoft-inc#3769) * Add trait for pattern match. * Implement match plan by trait and reduce some reduncant rules. * Remove unused header. * Fix warning. * Let optimize rule decide collection of matched nodes. Co-authored-by: kyle.cao <[email protected]> Co-authored-by: Sophie <[email protected]> * Simplify datetime tostring. (vesoft-inc#3712) * Simplify datetime tostring. * Remove unused code. * Fix error. Co-authored-by: Sophie <[email protected]> * pragma once not allowd (vesoft-inc#3842) Co-authored-by: Sophie <[email protected]> * refactor LOG and add comment for meta, meta/processors/job (vesoft-inc#3686) * refactor LOG and add comment for meta, meta/processors/job * refactor LOG and add comment for meta/processors/zone Co-authored-by: Doodle <[email protected]> * replace STRING to name_label for zone's name (vesoft-inc#3780) Co-authored-by: Doodle <[email protected]> * Add test for rc. (vesoft-inc#3847) * Add test for rc. * Fix cxx. Confirm with sophie.xie * cpp lint Co-authored-by: Yee <[email protected]> Co-authored-by: Sophie <[email protected]> Co-authored-by: cpw <[email protected]> Co-authored-by: hs.zhang <[email protected]> Co-authored-by: yaphet <[email protected]> Co-authored-by: jakevin <[email protected]> Co-authored-by: shylock <[email protected]> Co-authored-by: Yichen Wang <[email protected]> Co-authored-by: panda-sheep <[email protected]> Co-authored-by: pengwei.song <[email protected]> Co-authored-by: Doodle <[email protected]> Co-authored-by: endy.li <[email protected]> Co-authored-by: jimingquan <[email protected]> Co-authored-by: liwenhui-soul <[email protected]> Co-authored-by: Alex Xing <[email protected]> Co-authored-by: Harris.Chu <[email protected]> Co-authored-by: [email protected] <[email protected]> Co-authored-by: kyle.cao <[email protected]>

@panda-sheep

* add cmake git hooks options * githooks option * include what you use * make name follow convention * linker support lld, which is the fastest linker. If ci install lld when pull request , will set lld as default linker. * log rotate * delete werror * pragma once => ifndef * pragma once => ifndef * Werror bison * format code * fix issue 3601 (vesoft-inc#3666) * fix meta upgrade for multi instance (vesoft-inc#3734) * drop host only when it registe heartbeat (vesoft-inc#3739) * Fix race of create tag and edge (vesoft-inc#3735) * fix rece * polish * fix review * Add version for `show hosts` (vesoft-inc#3702) * version * tck * pytest * pytest Co-authored-by: cpw <[email protected]> * Enhancement/set expression depth by flag (vesoft-inc#3722) * Set expression depth limit by flag. * Format. Co-authored-by: Yichen Wang <[email protected]> * Update client version to v3.0.0 (vesoft-inc#3741) * Update python package name to v3 * Update client version white list * Update go mod to v3 * Change the python package name used in tests * Update client version white list * Fix create fulltext index failed (vesoft-inc#3747) * fix bug: the same tagId/edgetype under different spaces, failed to create fulltext indexes * fix bug: the same tagId/edgetype under different spaces, failed to create fulltext indexes * add comments and adjust log level for meta/processors/admin (vesoft-inc#3662) * add comments and adjust log level for meta admin processors * change agent hb log level Co-authored-by: Doodle <[email protected]> * fix writternBy delete by optimizer (vesoft-inc#3746) * fix bugs * add tck features * style Co-authored-by: cpw <[email protected]> * fix with & optimizer rule error (vesoft-inc#3736) * fix error * fix collapseProjectRule Co-authored-by: cpw <[email protected]> * fix upgrade bug (vesoft-inc#3772) Co-authored-by: Sophie <[email protected]> * fix 3727 (vesoft-inc#3737) Co-authored-by: Doodle <[email protected]> * Balance fix (vesoft-inc#3668) * fix core balance in zone removing the only host in zone * fix balance plan that make redundant part * fix deadlock when runInternal and finish callback called in the same thread * fix transleader return TERM_OUT_OF_DATE * continue running balance task from last status * fix deadlock when removing part Co-authored-by: Sophie <[email protected]> * Eliminte the useless row collect. (vesoft-inc#3363) * Eliminte the useless row collect. * Remove comment. * Format. * Rebase. Co-authored-by: cpw <[email protected]> Co-authored-by: Sophie <[email protected]> * Add snapshot for prefix iterate. (vesoft-inc#3717) * Add snapshot for prefix iterate. * Add header comment * Add happy test * Fix lint * Modify engine snap argument position * Fix other subclass * Fix code Co-authored-by: yaphet <[email protected]> Co-authored-by: Sophie <[email protected]> Co-authored-by: Doodle <[email protected]> * Read lock zone when create space. (vesoft-inc#3693) Co-authored-by: yaphet <[email protected]> Co-authored-by: Sophie <[email protected]> * suppress undesired ASAN error (vesoft-inc#3705) * suppress undesired error * fix unstable test case * 1. add comments and change log level for meta (vesoft-inc#3709) 2. remove some unused meta kv files 3. remove unused tests 4. remove unused kv relative tests in metaclient Co-authored-by: Doodle <[email protected]> * fix upgrade (vesoft-inc#3787) Co-authored-by: Doodle <[email protected]> * Create CODEOWNERS (vesoft-inc#3770) Confirm with sophie.xie * disable data balance (vesoft-inc#3756) Co-authored-by: Sophie <[email protected]> * Fix iterator with snapshot(parameter position) (vesoft-inc#3785) Co-authored-by: Harris.Chu <[email protected]> Co-authored-by: Sophie <[email protected]> * disable meta upgrade from v1 (vesoft-inc#3798) * Add gflag validator for client_idle_timeout_secs (vesoft-inc#3807) Co-authored-by: Sophie <[email protected]> * disable zone commands (vesoft-inc#3776) Co-authored-by: cpw <[email protected]> * Add ci for standalone compile and ut. (vesoft-inc#3637) * Add standalone workflow * remove tmp note * Add meta v2 thrift obj for sa * Add tck test * Fix port and dist tck * skip some test on sa * Fix merge * Fix add host Co-authored-by: Sophie <[email protected]> * Fix the promise/future. (vesoft-inc#3814) Co-authored-by: jimingquan <[email protected]> * add comments & accumulate bug fix for TOSS (vesoft-inc#3643) * 1. add comments & adjust some log level 2. accumulate bug fix for TOSS 2.1 one space may be affected by another space. when recover prime edges. 2.2 when call kvstore->get(), if leaderLeaseInvalid, it will report leader_change, change it to leader_lease_failed. 2.3 print all execution trace in ChainProcessor::finish(), make much more grep friendly. 2.4 some function of memory lock is hard to understand (forceLock, forceUnlock) , change to "setAutoUnlock(bool)" modify update fix compile error adjust finish() adjust finish() add rcPrepare_ = succeeded adjust delete local processor adjust log use rcPrepare, rcRemote rcCommit fix resume add edges adjust UT disable ioThreadPoolForMeta_ make int64 vid printable adjust some log to vlog 2 only print prepare succeeded result only print readable in finish() of resume processor adjust some log some log not print term desc class name in uuid debug add some more log looks like factory may fail? add leader lease error code add check when get lock report leader lease do some clean change table prefix to NebulaKeyUtil Style change some print disable UT for short clean compile error disable pool monitor * unify add/delete finish() as update * not stable will add it back later * fix a clang warning * fix a clang10 warning 2 Co-authored-by: Sophie <[email protected]> * disable add hosts into zone (vesoft-inc#3825) Co-authored-by: Sophie <[email protected]> * Check active zone before create space (vesoft-inc#3822) Co-authored-by: Sophie <[email protected]> * abandon upgrader for old version (vesoft-inc#3818) Co-authored-by: Sophie <[email protected]> * put dataVersionKey on create space for storage (vesoft-inc#3817) * put dataVersionKey on create space for storage * address some comments Co-authored-by: Doodle <[email protected]> Co-authored-by: Sophie <[email protected]> * Update PR template (vesoft-inc#3809) Co-authored-by: Sophie <[email protected]> * fix meta http replace host handler (vesoft-inc#3794) * fix meta http replace host handler * fix http replace test and add zone test Co-authored-by: Sophie <[email protected]> * Fix `Match cannot be followed by use space` (vesoft-inc#3748) * fix bug * add tck * remove set * fix * rm header * fix * remove changes in match * fix * enhance annotation * fix review * fix * fix review * remove redundant Co-authored-by: Sophie <[email protected]> Co-authored-by: cpw <[email protected]> * support toSet function (vesoft-inc#3594) add test Co-authored-by: Sophie <[email protected]> * Enhance date parser (vesoft-inc#3797) * parse date * enhance datetime * add test * fix review * rename Co-authored-by: Sophie <[email protected]> * raft: fix always loop when no log need to send (vesoft-inc#3819) Co-authored-by: Sophie <[email protected]> Co-authored-by: Doodle <[email protected]> * Update CODEOWNERS (vesoft-inc#3835) * Update CODEOWNERS * Update CODEOWNERS Confirm with sophie.xie * refactor AdminClient (vesoft-inc#3781) * refactor AdminClient * fix test case, unify all getAdminAddrFromStoreAddr into AdminClient * add more test cases * address @panda-sheep's comment * Feature/optimizer trait match (vesoft-inc#3769) * Add trait for pattern match. * Implement match plan by trait and reduce some reduncant rules. * Remove unused header. * Fix warning. * Let optimize rule decide collection of matched nodes. Co-authored-by: kyle.cao <[email protected]> Co-authored-by: Sophie <[email protected]> * Simplify datetime tostring. (vesoft-inc#3712) * Simplify datetime tostring. * Remove unused code. * Fix error. Co-authored-by: Sophie <[email protected]> * pragma once not allowd (vesoft-inc#3842) Co-authored-by: Sophie <[email protected]> * refactor LOG and add comment for meta, meta/processors/job (vesoft-inc#3686) * refactor LOG and add comment for meta, meta/processors/job * refactor LOG and add comment for meta/processors/zone Co-authored-by: Doodle <[email protected]> * replace STRING to name_label for zone's name (vesoft-inc#3780) Co-authored-by: Doodle <[email protected]> * Add test for rc. (vesoft-inc#3847) * Add test for rc. * Fix cxx. Confirm with sophie.xie * cpp lint Co-authored-by: Yee <[email protected]> Co-authored-by: Sophie <[email protected]> Co-authored-by: cpw <[email protected]> Co-authored-by: hs.zhang <[email protected]> Co-authored-by: yaphet <[email protected]> Co-authored-by: jakevin <[email protected]> Co-authored-by: shylock <[email protected]> Co-authored-by: Yichen Wang <[email protected]> Co-authored-by: panda-sheep <[email protected]> Co-authored-by: pengwei.song <[email protected]> Co-authored-by: Doodle <[email protected]> Co-authored-by: endy.li <[email protected]> Co-authored-by: jimingquan <[email protected]> Co-authored-by: liwenhui-soul <[email protected]> Co-authored-by: Alex Xing <[email protected]> Co-authored-by: Harris.Chu <[email protected]> Co-authored-by: [email protected] <[email protected]> Co-authored-by: kyle.cao <[email protected]> Co-authored-by: yuehua.jia <[email protected]> Co-authored-by: Yee <[email protected]> Co-authored-by: Sophie <[email protected]> Co-authored-by: cpw <[email protected]> Co-authored-by: hs.zhang <[email protected]> Co-authored-by: yaphet <[email protected]> Co-authored-by: jakevin <[email protected]> Co-authored-by: shylock <[email protected]> Co-authored-by: Yichen Wang <[email protected]> Co-authored-by: panda-sheep <[email protected]> Co-authored-by: pengwei.song <[email protected]> Co-authored-by: Doodle <[email protected]> Co-authored-by: endy.li <[email protected]> Co-authored-by: jimingquan <[email protected]> Co-authored-by: liwenhui-soul <[email protected]> Co-authored-by: Alex Xing <[email protected]> Co-authored-by: Harris.Chu <[email protected]> Co-authored-by: [email protected] <[email protected]> Co-authored-by: kyle.cao <[email protected]>

Sophie-Xie added the cherry-pick-v3.0 PR: need cherry-pick to this version label Jan 10, 2022

Sophie-Xie added the ready-for-testing PR: ready for the CI test label Jan 10, 2022

Sophie-Xie linked an issue Jan 10, 2022 that may be closed by this pull request

balance remove error #3654

Closed

kikimo reviewed Jan 11, 2022

View reviewed changes

src/meta/processors/job/BalanceJobExecutor.cpp Show resolved Hide resolved

liwenhui-soul force-pushed the balance branch 2 times, most recently from 8d9109e to 335cda9 Compare January 12, 2022 09:52

This was referenced Jan 13, 2022

Extra parts found after performing balance operation #3657

Closed

balance data failed - Transfer leader failed, status Unknown code -3503 #3672

Closed

A job cannot be restarted after it is stopped #3701

Closed

Sophie-Xie linked an issue Jan 13, 2022 that may be closed by this pull request

balance data failed - Transfer leader failed, status Unknown code -3503 #3672

Closed

critical27 reviewed Jan 14, 2022

View reviewed changes

This was linked to issues Jan 17, 2022

Extra parts found after performing balance operation #3657

Closed

A job cannot be restarted after it is stopped #3701

Closed

liwenhui-soul force-pushed the balance branch 14 times, most recently from 2aca63c to d10d446 Compare January 19, 2022 04:24

critical27 previously approved these changes Jan 20, 2022

View reviewed changes

critical27 reviewed Jan 20, 2022

View reviewed changes

liwenhui-soul dismissed critical27’s stale review via 9e7d915 January 20, 2022 09:41

liwenhui-soul force-pushed the balance branch from d10d446 to 9e7d915 Compare January 20, 2022 09:41

liwenhui-soul requested a review from liuyu85cn January 20, 2022 09:44

liuyu85cn previously approved these changes Jan 20, 2022

View reviewed changes

liwenhui-soul added 6 commits January 20, 2022 17:47

fix core balance in zone removing the only host in zone

e9e8efb

fix balance plan that make redundant part

f817477

fix deadlock when runInternal and finish callback called in the same …

41d81ff

…thread

fix transleader return TERM_OUT_OF_DATE

e414eb9

continue running balance task from last status

3f80ed7

fix deadlock when removing part

b9318ef

liwenhui-soul dismissed liuyu85cn’s stale review via b9318ef January 20, 2022 09:47

liwenhui-soul force-pushed the balance branch from 9e7d915 to b9318ef Compare January 20, 2022 09:47

liuyu85cn approved these changes Jan 20, 2022

View reviewed changes

Merge branch 'master' into balance

7ea9f07

critical27 approved these changes Jan 20, 2022

View reviewed changes

critical27 merged commit 9a39fe4 into vesoft-inc:master Jan 20, 2022

Sophie-Xie mentioned this pull request Jan 21, 2022

balance leader command keep running forever #3636

Closed

Sophie-Xie linked an issue Jan 21, 2022 that may be closed by this pull request

cannot exit metad leader if the job is running #3749

Closed

jamieliu1023 mentioned this pull request Jan 22, 2022

Weekly Report 2022-01-21 vesoft-inc/nebula-community#86

Closed

This was referenced Jan 24, 2022

Balance data always failed one task #3671

Closed

Storage go crazy while balancing data #3664

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Balance fix #3668

Balance fix #3668

liwenhui-soul commented Jan 7, 2022

Sophie-Xie commented Jan 10, 2022

HarrisChu commented Jan 10, 2022

critical27 left a comment

critical27 Jan 14, 2022

liwenhui-soul Jan 14, 2022

critical27 Jan 14, 2022

liwenhui-soul Jan 17, 2022 •

edited

Loading

critical27 commented Jan 20, 2022

critical27 left a comment

critical27 Jan 20, 2022

liwenhui-soul Jan 20, 2022

critical27 left a comment

Balance fix #3668

Balance fix #3668

Conversation

liwenhui-soul commented Jan 7, 2022

What type of PR is this?

What problem(s) does this PR solve?

How do you solve it?

Special notes for your reviewer, ex. impact of this fix, design document, etc:

Checklist:

Release notes:

Sophie-Xie commented Jan 10, 2022

HarrisChu commented Jan 10, 2022

critical27 left a comment

Choose a reason for hiding this comment

critical27 Jan 14, 2022

Choose a reason for hiding this comment

liwenhui-soul Jan 14, 2022

Choose a reason for hiding this comment

critical27 Jan 14, 2022

Choose a reason for hiding this comment

liwenhui-soul Jan 17, 2022 • edited Loading

Choose a reason for hiding this comment

critical27 commented Jan 20, 2022

critical27 left a comment

Choose a reason for hiding this comment

critical27 Jan 20, 2022

Choose a reason for hiding this comment

liwenhui-soul Jan 20, 2022

Choose a reason for hiding this comment

critical27 left a comment

Choose a reason for hiding this comment

liwenhui-soul Jan 17, 2022 •

edited

Loading