Skip to content

Commit

Permalink
Cherry pick from nebula: go performance (vesoft-inc#2769)
Browse files Browse the repository at this point in the history 
* change batch size (vesoft-inc#5550)

* change batch size

* add flag to config

* Fix simple case (vesoft-inc#5551)

* m2n simple case

* add test case

---------

Co-authored-by: Sophie <[email protected]>

---------

Co-authored-by: jimingquan <[email protected]>
  • Loading branch information
@Sophie-Xie @nevermore3
Sophie-Xie and nevermore3 authored May 15, 2023
1 parent eef0513 commit 63c7e26
Show file tree
Hide file tree
Showing 6 changed files with 69 additions and 50 deletions.
2 changes: 2 additions & 0 deletions conf/nebula-graphd.conf.default
View file
Original file line number Diff line number Diff line change
Expand Up @@ -180,3 +180,5 @@
--min_batch_size=8192
# if true, return directly without go through RPC
--optimize_appendvertices=false
# number of paths constructed by each thread
--path_batch_size=10000
2 changes: 2 additions & 0 deletions conf/nebula-graphd.conf.production
View file
Original file line number Diff line number Diff line change
Expand Up @@ -179,3 +179,5 @@
--min_batch_size=8192
# if true, return directly without go through RPC
--optimize_appendvertices=false
# number of paths constructed by each thread
--path_batch_size=10000
2 changes: 1 addition & 1 deletion src/graph/executor/algo/AllPathsExecutor.cpp
View file
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ DEFINE_uint32(
100,
"the number of vids to expand, when this threshold is exceeded, use heuristic expansion");
DEFINE_uint32(path_threshold_ratio, 2, "threshold for heuristics expansion");
DEFINE_uint32(path_batch_size, 50000, "number of paths constructed by each thread");
DEFINE_uint32(path_batch_size, 10000, "number of paths constructed by each thread");

namespace nebula {
namespace graph {
Expand Down
19 changes: 13 additions & 6 deletions src/graph/planner/ngql/GoPlanner.cpp
View file
Original file line number Diff line number Diff line change
Expand Up @@ -311,12 +311,19 @@ StatusOr<SubPlan> GoPlanner::transform(AstContext* astCtx) {
for (auto node : varPtr->writtenBy) {
preRootNode_ = node;
}

auto argNode = Argument::make(qctx, from.runtimeVidName);
argNode->setColNames({from.runtimeVidName});
argNode->setInputVertexRequired(false);
goCtx_->vidsVar = argNode->outputVar();
startNode_ = argNode;
if (goCtx_->isSimple) {
// go from 'xxx' over edge yield distinct edge._dst as id |
// go from $-.id over edge yield distinct edge._dst
// above scenario, the second statement does not need the argument operator
startNode_ = preRootNode_;
goCtx_->vidsVar = varName;
} else {
auto argNode = Argument::make(qctx, from.runtimeVidName);
argNode->setColNames({from.runtimeVidName});
argNode->setInputVertexRequired(false);
goCtx_->vidsVar = argNode->outputVar();
startNode_ = argNode;
}
}
auto& steps = goCtx_->steps;
if (!steps.isMToN() && steps.steps() == 0) {
Expand Down
3 changes: 1 addition & 2 deletions src/graph/validator/GoValidator.cpp
View file
Original file line number Diff line number Diff line change
Expand Up @@ -292,8 +292,7 @@ bool GoValidator::checkDstPropOrVertexExist(const Expression* expr) {
}

bool GoValidator::isSimpleCase() {
if (!goCtx_->limits.empty() || !goCtx_->distinct || goCtx_->filter ||
goCtx_->from.fromType != FromType::kInstantExpr) {
if (!goCtx_->limits.empty() || !goCtx_->distinct || goCtx_->filter != nullptr) {
return false;
}
// Check if the yield cluase uses:
Expand Down
91 changes: 50 additions & 41 deletions tests/tck/features/go/SimpleCase.feature
View file
Original file line number Diff line number Diff line change
Expand Up @@ -366,23 +366,20 @@ Feature: Simple case
| count(*) |
| 28 |
And the execution plan should be:
| id | name | dependencies | operator info |
| 17 | Aggregate | 15 | |
| 15 | Minus | 13,14 | |
| 13 | Project | 16 | |
| 16 | PassThrough | 12 | |
| 12 | Dedup | 11 | |
| 11 | Project | 10 | |
| 10 | HashInnerJoin | 5,9 | |
| 5 | Project | 4 | |
| 4 | Dedup | 3 | |
| 3 | ExpandAll | 2 | |
| 2 | Expand | 0 | |
| 0 | Start | | |
| 9 | ExpandAll | 8 | |
| 8 | Expand | 7 | |
| 7 | Argument | | |
| 14 | Project | 16 | |
| id | name | dependencies | operator info |
| 14 | Aggregate | 12 | |
| 12 | Minus | 10,11 | |
| 10 | Project | 13 | |
| 13 | PassThrough | 9 | |
| 9 | Project | 8 | |
| 8 | Dedup | 7 | |
| 7 | Expand | 5 | |
| 5 | Project | 4 | |
| 4 | Dedup | 3 | |
| 3 | ExpandAll | 2 | |
| 2 | Expand | 0 | |
| 0 | Start | | |
| 11 | Project | 13 | |

Scenario: other simple case
When profiling query:
Expand All @@ -393,18 +390,15 @@ Feature: Simple case
| count(*) |
| 0 |
And the execution plan should be:
| id | name | dependencies | operator info |
| 12 | Aggregate | 11 | |
| 11 | Dedup | 10 | |
| 10 | Project | 9 | |
| 9 | HashInnerJoin | 4,8 | |
| 4 | Project | 3 | |
| 3 | Dedup | 2 | |
| 2 | Expand | 0 | |
| 0 | Start | | |
| 8 | ExpandAll | 7 | |
| 7 | Expand | 6 | |
| 6 | Argument | | |
| id | name | dependencies | operator info |
| 9 | Aggregate | 8 | |
| 8 | Project | 7 | |
| 7 | Dedup | 6 | |
| 6 | Expand | 4 | |
| 4 | Project | 3 | |
| 3 | Dedup | 2 | |
| 2 | Expand | 0 | |
| 0 | Start | | |
When profiling query:
"""
GO 1 STEP FROM "Tony Parker" OVER * YIELD distinct id($$) as id| GO 3 STEP FROM $-.id OVER * YIELD distinct id($$) | YIELD COUNT(*)
Expand All @@ -413,18 +407,33 @@ Feature: Simple case
| COUNT(*) |
| 22 |
And the execution plan should be:
| id | name | dependencies | operator info |
| 12 | Aggregate | 11 | |
| 11 | Dedup | 10 | |
| 10 | Project | 9 | |
| 9 | HashInnerJoin | 4,8 | |
| 4 | Project | 3 | |
| 3 | Dedup | 2 | |
| 2 | Expand | 0 | |
| 0 | Start | | |
| 8 | ExpandAll | 7 | |
| 7 | Expand | 6 | |
| 6 | Argument | | |
| id | name | dependencies | operator info |
| 9 | Aggregate | 8 | |
| 8 | Project | 7 | |
| 7 | Dedup | 6 | |
| 6 | Expand | 4 | |
| 4 | Project | 3 | |
| 3 | Dedup | 2 | |
| 2 | Expand | 0 | |
| 0 | Start | | |
When profiling query:
"""
GO 1 STEP FROM "Tony Parker" OVER * YIELD distinct id($$) as id| GO 2 to 4 STEP FROM $-.id OVER * YIELD distinct id($$) | YIELD COUNT(*)
"""
Then the result should be, in any order, with relax comparison:
| COUNT(*) |
| 26 |
And the execution plan should be:
| id | name | dependencies | operator info |
| 10 | Aggregate | 9 | |
| 9 | Project | 8 | |
| 8 | Dedup | 7 | |
| 7 | ExpandAll | 6 | |
| 6 | Expand | 4 | |
| 4 | Project | 3 | |
| 3 | Dedup | 2 | |
| 2 | Expand | 0 | |
| 0 | Start | | |

Scenario: could not be optimied cases
When profiling query:
Expand Down

0 comments on commit 63c7e26

Please sign in to comment.