Bfs optimize vertex-out-out query #1359

javeme · 2021-02-19T12:06:32Z

No description provided.

javeme · 2021-02-20T12:50:00Z

cassandra ci:

rocksdb ci:

javeme · 2021-02-24T03:06:19Z

rocksdb ci:

javeme · 2021-02-24T05:21:44Z

cassandra:

[ERROR] Tests run: 684, Failures: 2, Errors: 0, Skipped: 21, Time elapsed: 481.763 s <<< FAILURE! - in com.baidu.hugegraph.core.CoreTestSuite
[ERROR] testQueryEdgesWithLimitOnMultiLevel(com.baidu.hugegraph.core.EdgeCoreTest)  Time elapsed: 0.359 s  <<< FAILURE!
java.lang.AssertionError: expected:<4> but was:<3>
	at com.baidu.hugegraph.core.EdgeCoreTest.testQueryEdgesWithLimitOnMultiLevel(EdgeCoreTest.java:1999)
[ERROR] testQueryEdgesAdjacentVerticesWithLimitAndFilterProp(com.baidu.hugegraph.core.EdgeCoreTest)  Time elapsed: 1.056 s  <<< FAILURE!
java.lang.AssertionError: expected:<9> but was:<11>
	at com.baidu.hugegraph.core.EdgeCoreTest.testQueryEdgesAdjacentVerticesWithLimitAndFilterProp(EdgeCoreTest.java:2398)

mysql:

[ERROR] Tests run: 684, Failures: 2, Errors: 0, Skipped: 26, Time elapsed: 314.164 s <<< FAILURE! - in com.baidu.hugegraph.core.CoreTestSuite
[ERROR] testQueryEdgesWithLimitOnMultiLevel(com.baidu.hugegraph.core.EdgeCoreTest)  Time elapsed: 0.273 s  <<< FAILURE!
java.lang.AssertionError: expected:<4> but was:<3>
	at com.baidu.hugegraph.core.EdgeCoreTest.testQueryEdgesWithLimitOnMultiLevel(EdgeCoreTest.java:1999)
[ERROR] testQueryEdgesAdjacentVerticesWithLimitAndFilterProp(com.baidu.hugegraph.core.EdgeCoreTest)  Time elapsed: 0.312 s  <<< FAILURE!
java.lang.AssertionError: expected:<9> but was:<11>
	at com.baidu.hugegraph.core.EdgeCoreTest.testQueryEdgesAdjacentVerticesWithLimitAndFilterProp(EdgeCoreTest.java:2398)

rocksdb:

[ERROR] Tests run: 684, Failures: 2, Errors: 2, Skipped: 13, Time elapsed: 286.605 s <<< FAILURE! - in com.baidu.hugegraph.core.CoreTestSuite
[ERROR] testQueryEdgesWithLimitOnMultiLevel(com.baidu.hugegraph.core.EdgeCoreTest)  Time elapsed: 0.121 s  <<< FAILURE!
java.lang.AssertionError: expected:<4> but was:<3>
	at com.baidu.hugegraph.core.EdgeCoreTest.testQueryEdgesWithLimitOnMultiLevel(EdgeCoreTest.java:1999)
[ERROR] testQueryEdgesWithLimitOnSuperVertexAndFilterProp(com.baidu.hugegraph.core.EdgeCoreTest)  Time elapsed: 0.704 s  <<< ERROR!
java.lang.IllegalArgumentException: Too many conditions(10001) each query
	at com.baidu.hugegraph.core.EdgeCoreTest.testQueryEdgesWithLimitOnSuperVertexAndFilterProp(EdgeCoreTest.java:2553)
[ERROR] testQueryEdgesAdjacentVerticesWithLimitAndFilterProp(com.baidu.hugegraph.core.EdgeCoreTest)  Time elapsed: 0.37 s  <<< FAILURE!
java.lang.AssertionError: expected:<9> but was:<11>
	at com.baidu.hugegraph.core.EdgeCoreTest.testQueryEdgesAdjacentVerticesWithLimitAndFilterProp(EdgeCoreTest.java:2398)
[ERROR] testReloadAndQueryWithProperty(com.baidu.hugegraph.core.RamTableTest)  Time elapsed: 0.258 s  <<< ERROR!
java.lang.ClassCastException: java.util.LinkedList cannot be cast to com.baidu.hugegraph.backend.id.Id
	at com.baidu.hugegraph.core.RamTableTest.testReloadAndQueryWithProperty(RamTableTest.java:456)

Linary · 2021-02-25T08:25:19Z

...aph-core/src/main/java/com/baidu/hugegraph/traversal/optimize/HugeVertexStepWithoutPath.java

+    protected Traverser.Admin<E> processNextStart() {
+        /* Override super.processNextStart() */
+        if (this.parents == null) {
+            Function<Object, Traverser.Admin<Vertex>> nextFunc = null;


use java.util.Function

Linary · 2021-02-25T08:26:01Z

...aph-core/src/main/java/com/baidu/hugegraph/traversal/optimize/HugeVertexStepWithoutPath.java

+    protected Traverser.Admin<E> processNextStart() {
+        /* Override super.processNextStart() */
+        if (this.parents == null) {
+            Function<Object, Traverser.Admin<Vertex>> nextFunc = null;


remove = null, it's redudant

Linary · 2021-02-25T08:26:54Z

...aph-core/src/main/java/com/baidu/hugegraph/traversal/optimize/HugeVertexStepWithoutPath.java

+            }
+
+            // TODO: use BatchMapperIterator
+            List<Traverser.Admin<Vertex>> vertexes = new ArrayList<>();


replace with vertices

Linary · 2021-02-25T08:34:34Z

...aph-core/src/main/java/com/baidu/hugegraph/traversal/optimize/HugeVertexStepWithoutPath.java

+
+        BatchConditionQuery batchQuery = new BatchConditionQuery(HugeType.EDGE);
+
+        for (Traverser.Admin<Vertex> t : traversers) {


t -> traverser

Linary · 2021-04-19T08:30:25Z

hugegraph-core/src/main/java/com/baidu/hugegraph/backend/query/BatchConditionQuery.java

+        } else {
+            E.checkArgument(this.in.key().equals(key),
+                            "Invalid key '%s'", key);
+            E.checkArgument(keysEquals(query),


Linary · 2021-04-19T08:30:34Z

hugegraph-core/src/main/java/com/baidu/hugegraph/backend/query/BatchConditionQuery.java

+
+    public void mergeToIN(ConditionQuery query, HugeKeys key) {
+        Object value = query.condition(key);
+


remove empty line

Linary · 2021-04-19T08:36:46Z

hugegraph-core/src/main/java/com/baidu/hugegraph/backend/query/BatchConditionQuery.java

+                continue;
+            }
+            Object key = r.key();
+            if (!keys.contains(key)) {


if (!keys.contains(key)) { }

seems unnecessary, keys and key are both from this.relations()

Linary · 2021-04-19T08:39:07Z

hugegraph-core/src/main/java/com/baidu/hugegraph/backend/query/BatchConditionQuery.java

+        }
+    }
+
+    protected boolean keysEquals(ConditionQuery query) {


method name can be better, because it compared both keys and values

github-actions · 2021-06-19T21:13:28Z

Due to the lack of activity, the current pr is marked as stale and will be closed after 180 days, any update will remove the stale label

github-actions · 2021-10-07T21:12:21Z

Due to the lack of activity, the current pr is marked as stale and will be closed after 180 days, any update will remove the stale label

Change-Id: I1b3c2e942cbfdc9f51cfc43a3ff5aa61b62e22d0

Change-Id: I4e53af1bc276a892c6d0d59985f925a4e7fb775b

Change-Id: I5966568d023413a692667db212c398d795aaf714

Change-Id: Ief8e5ab3d2e50ecabb03c63e515eabe4d08252d6

Change-Id: I52ee68642b03127ac498aa8a32254c0999ff73d3

Change-Id: Ie2104984bfe8402187119876be47a102663996ea

Change-Id: Ibef23b66de60d901502b8cf5b836da3cb9e3c22c

Change-Id: I7366d240fb762fdb9ca36c60344c3082586fd24c

Change-Id: I92d01e9beb699e4585e03b8e8392c8d65138d939

Change-Id: I9334d87b55deefca1963403697beb8aef246c30a

Change-Id: Ie003fc8e1a5fa3befac78afe777830370b246aad

…OOM) Change-Id: I34fe7bd6c37d93d2743978e8a5b7b54871044a49

Change-Id: Ie6f9a9ece555b908ef2004dca1440e613a138ca9

Change-Id: I56435b4f512826b0c40dc93317e8f6096bc9c45b

Change-Id: I70290333e221be87fda4cb332fc9e8e234271e4d

codecov · 2021-10-26T06:27:01Z

Codecov Report

Merging #1359 (acb7ddd) into master (7512587) will decrease coverage by 0.02%.
The diff coverage is 82.06%.

@@             Coverage Diff              @@
##             master    #1359      +/-   ##
============================================
- Coverage     63.26%   63.24%   -0.03%     
- Complexity     6703     6759      +56     
============================================
  Files           418      421       +3     
  Lines         34454    34642     +188     
  Branches       4764     4797      +33     
============================================
+ Hits          21799    21910     +111     
- Misses        10407    10459      +52     
- Partials       2248     2273      +25

Impacted Files	Coverage Δ
.../java/com/baidu/hugegraph/backend/query/Query.java	`86.00% <ø> (ø)`
...baidu/hugegraph/backend/store/BackendFeatures.java	`0.00% <ø> (ø)`
...aidu/hugegraph/backend/store/hbase/HbaseTable.java	`78.75% <0.00%> (ø)`
...u/hugegraph/backend/store/mysql/MysqlFeatures.java	`0.00% <0.00%> (ø)`
...aidu/hugegraph/backend/store/mysql/MysqlTable.java	`0.00% <0.00%> (ø)`
.../hugegraph/backend/store/rocksdb/RocksDBTable.java	`70.99% <0.00%> (ø)`
...h/backend/store/scylladb/ScyllaDBTablesWithMV.java	`96.03% <0.00%> (ø)`
...a/com/baidu/hugegraph/backend/query/Condition.java	`79.42% <66.66%> (-0.14%)`	⬇️
...raph/traversal/optimize/HugeVertexStepByBatch.java	`72.72% <72.72%> (ø)`
...m/baidu/hugegraph/backend/tx/GraphTransaction.java	`79.89% <75.00%> (+0.02%)`	⬆️
... and 49 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7512587...acb7ddd. Read the comment docs.

javeme · 2021-10-26T09:00:34Z

cassandra：

Error:  Failures: 
Error:    EdgeCoreTest.testQueryEdgesAdjacentVerticesWithLimitAndFilterProp:2398 expected:<9> but was:<11>
Error:    EdgeCoreTest.testQueryEdgesWithLimitOnMultiLevel:1999 expected:<4> but was:<3>
[INFO] 
Error:  Tests run: 713, Failures: 2, Errors: 0, Skipped: 21

testQueryEdgesWithLimitOnMultiLevel:
we request with duplicated conditions in BatchConditionQuery[v[8:java-1], v[8:java-3], v[8:java-1], v[8:java-2]], but duplicated results may be removed by backend store.

testQueryEdgesAdjacentVerticesWithLimitAndFilterProp:
boolean firstLouise = graph.traversal().V(louise, james).outE("write", "look").outV().next().equals(louise);
firstLouise == false

scylladb:

com.baidu.hugegraph.core.EdgeCoreTest.testQueryEdgesWithLimitOnSuperVertexAndFilterProp(EdgeCoreTest.java:2553)
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /127.0.0.1:9042 (com.datastax.driver.core.exceptions.ServerError: 
An unexpected error occurred server side on /127.0.0.1:9042: partition key cartesian product size 1000 is greater than maximum 100))

Change-Id: I5bd87d3dd4210568fe84c4d7154d6db5e0dea0a0

Change-Id: I77083eed68a77d385f03321ec267821fd4cc0c31

imbajin · 2021-11-03T06:51:45Z

hugegraph-core/src/main/java/com/baidu/hugegraph/traversal/optimize/HugeVertexStepByBatch.java

+
+    private static final long serialVersionUID = -3609787815053052222L;
+
+    private final int batchSize = (int) Query.QUERY_BATCH;


single usage can be move to line 61? after if (this.batchIterator == null) {

imbajin · 2021-11-03T06:59:21Z

hugegraph-core/src/main/java/com/baidu/hugegraph/traversal/optimize/HugeVertexStepByBatch.java

+        if (queryVertex) {
+            this.iterator = (Iterator<E>) this.vertices(traversers);
+        } else {
+            assert queryEdge;


seems same with line 105

yes, it's just for comment

Change-Id: I2fa0849c16757a6a02959e797385570d0f6876d2

Linary requested changes Feb 25, 2021

View reviewed changes

javeme force-pushed the bfs-optimize-query-vertex-out-out-with-in-and branch 2 times, most recently from 72e1556 to 35a306d Compare February 25, 2021 11:23

javeme force-pushed the bfs-optimize-query-vertex-out-out-with-in-and branch from 35a306d to 4f1327d Compare April 17, 2021 16:34

Linary requested changes Apr 19, 2021

View reviewed changes

github-actions bot added the inactive label Jun 19, 2021

imbajin added improvement General improvement and removed inactive labels Sep 7, 2021

github-actions bot added the inactive label Oct 7, 2021

imbajin added feature New feature and removed inactive feature New feature labels Oct 8, 2021

javeme added 13 commits October 26, 2021 14:09

not-commit: optimize-query-vertex-out-out-with-in-and

321c874

Change-Id: I1b3c2e942cbfdc9f51cfc43a3ff5aa61b62e22d0

add HugeVertexStep.constructEdgesQuery()

2c03127

Change-Id: I4e53af1bc276a892c6d0d59985f925a4e7fb775b

fix errors to run example

abb145d

Change-Id: I5966568d023413a692667db212c398d795aaf714

fix cassandra IN query

63cf90f

Change-Id: Ief8e5ab3d2e50ecabb03c63e515eabe4d08252d6

fix batch query missing limit and page info

3f25729

Change-Id: I52ee68642b03127ac498aa8a32254c0999ff73d3

add feature supportsQueryWithInCondition()

1fa91f7

Change-Id: Ie2104984bfe8402187119876be47a102663996ea

improve TableSerializer.writeQueryEdgeCondition()

9023df9

Change-Id: Ibef23b66de60d901502b8cf5b836da3cb9e3c22c

impprove flatten IN

3a62411

Change-Id: I7366d240fb762fdb9ca36c60344c3082586fd24c

rename BatchQuery to BatchConditionQuery

def4bdb

Change-Id: I92d01e9beb699e4585e03b8e8392c8d65138d939

fix query from RamTable not flatten

d43748c

Change-Id: I9334d87b55deefca1963403697beb8aef246c30a

small improve

93ffd6c

Change-Id: Ie003fc8e1a5fa3befac78afe777830370b246aad

use BatchMapperIterator to fetch intermediate results(by batch avoid …

7f905a7

…OOM) Change-Id: I34fe7bd6c37d93d2743978e8a5b7b54871044a49

rename to HugeVertexStepByBatch and just enable when needed

d937c96

Change-Id: Ie6f9a9ece555b908ef2004dca1440e613a138ca9

javeme added 2 commits October 26, 2021 14:11

set batch list capacity

38aae46

Change-Id: I56435b4f512826b0c40dc93317e8f6096bc9c45b

tiny fix

bd83132

Change-Id: I70290333e221be87fda4cb332fc9e8e234271e4d

javeme force-pushed the bfs-optimize-query-vertex-out-out-with-in-and branch from 20055fe to bd83132 Compare October 26, 2021 06:18

javeme added 2 commits October 26, 2021 18:54

adapt test cases

e2b6422

Change-Id: I5bd87d3dd4210568fe84c4d7154d6db5e0dea0a0

set QUERY_BATCH = 100 for scylladb limit

2348b48

Change-Id: I77083eed68a77d385f03321ec267821fd4cc0c31

zhoney previously approved these changes Oct 27, 2021

View reviewed changes

imbajin reviewed Nov 3, 2021

View reviewed changes

tiny improve

acb7ddd

Change-Id: I2fa0849c16757a6a02959e797385570d0f6876d2

javeme dismissed zhoney’s stale review via acb7ddd November 3, 2021 07:34

imbajin approved these changes Nov 3, 2021

View reviewed changes

zhoney approved these changes Nov 3, 2021

View reviewed changes

zhoney merged commit 43b8ae7 into master Nov 3, 2021

zhoney deleted the bfs-optimize-query-vertex-out-out-with-in-and branch November 3, 2021 11:09

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bfs optimize vertex-out-out query #1359

Bfs optimize vertex-out-out query #1359

javeme commented Feb 19, 2021

javeme commented Feb 20, 2021 •

edited

Loading

javeme commented Feb 24, 2021

javeme commented Feb 24, 2021 •

edited

Loading

Linary Feb 25, 2021

Linary Feb 25, 2021

Linary Feb 25, 2021

Linary Feb 25, 2021

Linary Apr 19, 2021

Linary Apr 19, 2021

Linary Apr 19, 2021

Linary Apr 19, 2021 •

edited

Loading

github-actions bot commented Jun 19, 2021

github-actions bot commented Oct 7, 2021

codecov bot commented Oct 26, 2021 •

edited

Loading

javeme commented Oct 26, 2021 •

edited

Loading

imbajin Nov 3, 2021

imbajin Nov 3, 2021

javeme Nov 3, 2021


		BatchConditionQuery batchQuery = new BatchConditionQuery(HugeType.EDGE);

		for (Traverser.Admin<Vertex> t : traversers) {


		public void mergeToIN(ConditionQuery query, HugeKeys key) {
		Object value = query.condition(key);


		private static final long serialVersionUID = -3609787815053052222L;

		private final int batchSize = (int) Query.QUERY_BATCH;

Bfs optimize vertex-out-out query #1359

Bfs optimize vertex-out-out query #1359

Conversation

javeme commented Feb 19, 2021

javeme commented Feb 20, 2021 • edited Loading

javeme commented Feb 24, 2021

javeme commented Feb 24, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Linary Apr 19, 2021 • edited Loading

Choose a reason for hiding this comment

github-actions bot commented Jun 19, 2021

github-actions bot commented Oct 7, 2021

codecov bot commented Oct 26, 2021 • edited Loading

Codecov Report

javeme commented Oct 26, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

javeme commented Feb 20, 2021 •

edited

Loading

javeme commented Feb 24, 2021 •

edited

Loading

Linary Apr 19, 2021 •

edited

Loading

codecov bot commented Oct 26, 2021 •

edited

Loading

javeme commented Oct 26, 2021 •

edited

Loading