Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: optimising adjacency edge queries #2242

Merged
merged 31 commits into from
Nov 29, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
31 commits
Select commit Hold shift + click to select a range
3559ace
refact: move dirs into hugegraph-server
VGalaxies Aug 1, 2023
9068cef
refact: create dirs for hugegraph-pd and hugegraph-store
VGalaxies Aug 2, 2023
fd161f2
refact: init root pom for all modules
VGalaxies Aug 4, 2023
49a1acf
fix: fix parent of submodules in hugegraph-server
VGalaxies Aug 4, 2023
fca91b2
fix: path
msgui Aug 13, 2023
8465927
fix: conf path
msgui Aug 13, 2023
3b791ee
fix: 1.10 -> 1.11
msgui Aug 13, 2023
ae90ba6
Revert "fix: 1.10 -> 1.11"
msgui Aug 14, 2023
ce5ddbf
fix: version 1.0.0 -> 1.5.0
msgui Aug 14, 2023
8a7ed6a
fix:
msgui Aug 14, 2023
242786b
check
msgui Aug 16, 2023
2e401de
fix: SERVER_DIR
msgui Aug 16, 2023
67de746
fix
msgui Aug 16, 2023
edb77ef
fix
msgui Aug 16, 2023
764dd89
fix: SERVER_DIR
msgui Aug 16, 2023
9a2f2e6
Merge pull request #5 from msgui/fix_CI
msgui Aug 17, 2023
4f68641
test
msgui Sep 9, 2023
d582a51
Merge pull request #14 from msgui/new-fix-depedency-review
msgui Sep 9, 2023
80cf036
Merge branch 'master' of https://github.com/msgui/incubator-hugegraph
msgui Sep 13, 2023
2b241dd
Merge branch 'master' of https://github.com/msgui/incubator-hugegraph
msgui Oct 12, 2023
5b5560f
Merge branch 'master' of https://github.com/msgui/incubator-hugegraph
msgui Nov 28, 2023
5db1cfb
feat: optimising adjacency edge queries
msgui Jul 11, 2023
32f44c0
fix: "source" -> "target"
msgui Jul 11, 2023
33108fe
fix: resolve reviews
msgui Jul 12, 2023
7a2a61a
fix: lack of support for sort keys
msgui Jul 25, 2023
89b34b3
chore
msgui Aug 6, 2023
43d024a
Merge branch 'master' into edge_queries_optimize
msgui Nov 28, 2023
ea6cd18
update pom.xml
msgui Nov 28, 2023
531f389
Update EdgeExistenceTraverser.java
imbajin Nov 28, 2023
e08b25e
chore: add summary for edge-existence api
msgui Nov 28, 2023
c0e2ea6
tiny improve
imbajin Nov 28, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion hugegraph-server/hugegraph-api/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -188,7 +188,7 @@
</manifest>
<manifestEntries>
<!-- TODO: update it -->
<Implementation-Version>0.69.0.0</Implementation-Version>
<Implementation-Version>0.71.0.0</Implementation-Version>
</manifestEntries>
</archive>
</configuration>
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,84 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with this
* work for additional information regarding copyright ownership. The ASF
* licenses this file to You under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
* WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
* License for the specific language governing permissions and limitations
* under the License.
*/

package org.apache.hugegraph.api.traversers;

import static org.apache.hugegraph.traversal.algorithm.HugeTraverser.DEFAULT_LIMIT;

import java.util.Iterator;

import org.apache.hugegraph.HugeGraph;
import org.apache.hugegraph.backend.id.Id;
import org.apache.hugegraph.core.GraphManager;
import org.apache.hugegraph.structure.HugeVertex;
import org.apache.hugegraph.traversal.algorithm.EdgeExistenceTraverser;
import org.apache.hugegraph.util.E;
import org.apache.hugegraph.util.Log;
import org.apache.tinkerpop.gremlin.structure.Edge;
import org.slf4j.Logger;

import com.codahale.metrics.annotation.Timed;

import io.swagger.v3.oas.annotations.Operation;
import io.swagger.v3.oas.annotations.tags.Tag;
import jakarta.inject.Singleton;
import jakarta.ws.rs.DefaultValue;
import jakarta.ws.rs.GET;
import jakarta.ws.rs.Path;
import jakarta.ws.rs.PathParam;
import jakarta.ws.rs.Produces;
import jakarta.ws.rs.QueryParam;
import jakarta.ws.rs.core.Context;

@Path("graphs/{graph}/traversers/edgeexist")
@Singleton
@Tag(name = "EdgeExistenceAPI")
msgui marked this conversation as resolved.
Show resolved Hide resolved
public class EdgeExistenceAPI extends TraverserAPI {

Check warning on line 50 in hugegraph-server/hugegraph-api/src/main/java/org/apache/hugegraph/api/traversers/EdgeExistenceAPI.java

View check run for this annotation

Codecov / codecov/patch

hugegraph-server/hugegraph-api/src/main/java/org/apache/hugegraph/api/traversers/EdgeExistenceAPI.java#L50

Added line #L50 was not covered by tests

private static final Logger LOG = Log.logger(EdgeExistenceAPI.class);

Check warning on line 52 in hugegraph-server/hugegraph-api/src/main/java/org/apache/hugegraph/api/traversers/EdgeExistenceAPI.java

View check run for this annotation

Codecov / codecov/patch

hugegraph-server/hugegraph-api/src/main/java/org/apache/hugegraph/api/traversers/EdgeExistenceAPI.java#L52

Added line #L52 was not covered by tests
private static final String DEFAULT_EMPTY = "";

@GET
@Timed
@Produces(APPLICATION_JSON_WITH_CHARSET)
@Operation(summary = "get edges from 'source' to 'target' vertex")
public String get(@Context GraphManager manager,
@PathParam("graph") String graph,
@QueryParam("source") String source,
@QueryParam("target") String target,
@QueryParam("label") String edgeLabel,
@QueryParam("sort_values")
@DefaultValue(DEFAULT_EMPTY) String sortValues,
@QueryParam("limit")
@DefaultValue(DEFAULT_LIMIT) long limit) {
LOG.debug("Graph [{}] get edgeExistence with " +

Check warning on line 68 in hugegraph-server/hugegraph-api/src/main/java/org/apache/hugegraph/api/traversers/EdgeExistenceAPI.java

View check run for this annotation

Codecov / codecov/patch

hugegraph-server/hugegraph-api/src/main/java/org/apache/hugegraph/api/traversers/EdgeExistenceAPI.java#L68

Added line #L68 was not covered by tests
"source '{}', target '{}', edgeLabel '{}', sortValue '{}', limit '{}'",
graph, source, target, edgeLabel, sortValues, limit);

Check warning on line 70 in hugegraph-server/hugegraph-api/src/main/java/org/apache/hugegraph/api/traversers/EdgeExistenceAPI.java

View check run for this annotation

Codecov / codecov/patch

hugegraph-server/hugegraph-api/src/main/java/org/apache/hugegraph/api/traversers/EdgeExistenceAPI.java#L70

Added line #L70 was not covered by tests

E.checkArgumentNotNull(source, "The source can't be null");
E.checkArgumentNotNull(target, "The target can't be null");

Check warning on line 73 in hugegraph-server/hugegraph-api/src/main/java/org/apache/hugegraph/api/traversers/EdgeExistenceAPI.java

View check run for this annotation

Codecov / codecov/patch

hugegraph-server/hugegraph-api/src/main/java/org/apache/hugegraph/api/traversers/EdgeExistenceAPI.java#L72-L73

Added lines #L72 - L73 were not covered by tests

Id sourceId = HugeVertex.getIdValue(source);
Id targetId = HugeVertex.getIdValue(target);
HugeGraph hugegraph = graph(manager, graph);
EdgeExistenceTraverser traverser = new EdgeExistenceTraverser(hugegraph);
Iterator<Edge> edges = traverser.queryEdgeExistence(sourceId, targetId, edgeLabel,

Check warning on line 79 in hugegraph-server/hugegraph-api/src/main/java/org/apache/hugegraph/api/traversers/EdgeExistenceAPI.java

View check run for this annotation

Codecov / codecov/patch

hugegraph-server/hugegraph-api/src/main/java/org/apache/hugegraph/api/traversers/EdgeExistenceAPI.java#L75-L79

Added lines #L75 - L79 were not covered by tests
sortValues, limit);

return manager.serializer(hugegraph).writeEdges(edges, false);

Check warning on line 82 in hugegraph-server/hugegraph-api/src/main/java/org/apache/hugegraph/api/traversers/EdgeExistenceAPI.java

View check run for this annotation

Codecov / codecov/patch

hugegraph-server/hugegraph-api/src/main/java/org/apache/hugegraph/api/traversers/EdgeExistenceAPI.java#L82

Added line #L82 was not covered by tests
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ public final class ApiVersion {
* [0.2] HugeGraph-527: First add the version to the hugegraph module
* [0.3] HugeGraph-525: Add versions check of components and api
* [0.4] HugeGraph-162: Add schema builder to separate client and inner interface.
* [0.5] HugeGraph-498: Support three kind of id strategy
* [0.5] HugeGraph-498: Support three kinds of id strategy
* <p>
* version 0.3:
* <p>
Expand All @@ -41,12 +41,12 @@ public final class ApiVersion {
* <p>
* version 0.4:
* [0.11] HugeGraph-938: Remove useless index-names field in VL/EL API
* [0.12] HugeGraph-589: Add schema id for all schema element
* [0.12] HugeGraph-589: Add schema id for all schema elements
* [0.13] HugeGraph-956: Support customize string/number id strategy
* <p>
* version 0.5:
* [0.14] HugeGraph-1085: Add enable_label_index to VL/EL
* [0.15] HugeGraph-1105: Support paging for large amounts of records
* [0.15] HugeGraph-1105: Support paging for large numbers of records
* [0.16] HugeGraph-944: Support rest shortest path, k-out, k-neighbor
* [0.17] HugeGraph-944: Support rest shortest path, k-out, k-neighbor
* [0.18] HugeGraph-81: Change argument "checkVertex" to "check_vertex"
Expand Down Expand Up @@ -75,7 +75,7 @@ public final class ApiVersion {
* [0.34] Issue-307: Let VertexAPI use simplified property serializer
* [0.35] Issue-287: Support pagination when do index query
* [0.36] Issue-360: Support paging for scan api
* [0.37] Issue-391: Add skip_super_node for shortest path
* [0.37] Issue-391: Add skip_super_node for the shortest path
* [0.38] Issue-274: Add personal-rank and neighbor-rank RESTful API
* <p>
* version 0.10:
Expand Down Expand Up @@ -114,17 +114,18 @@ public final class ApiVersion {
* [0.67] Issue-1065: Support dynamically add/remove graph
* [0.68] Issue-1763: Support adamic-adar & resource-allocation API
* [0.69] Issue-1748: Support Cypher query RESTful API
* [0.70] Issue-2242: Optimising adjacency edge queries
* [0.70] PR-2242: Add edge-existence RESTful API
* [0.71] PR-2286: Support Arthas API & Metric API prometheus format
*/

/**
* The second parameter of Version.of() is for IDE running without JAR
* Note: Also update the version number in hugegraph-api/pom.xml
*/
public static final Version VERSION = Version.of(ApiVersion.class, "0.71");

public static void check() {
// Check version of hugegraph-core. Firstly do check from version 0.3
// Check the version of hugegraph-core. Do first check from version 0.3
VersionUtil.check(CoreVersion.VERSION, "1.0", "1.6", CoreVersion.NAME);
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with this
* work for additional information regarding copyright ownership. The ASF
* licenses this file to You under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
* WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
* License for the specific language governing permissions and limitations
* under the License.
*/

package org.apache.hugegraph.traversal.algorithm;

import java.util.Iterator;

import org.apache.hugegraph.HugeGraph;
import org.apache.hugegraph.backend.id.Id;
import org.apache.hugegraph.backend.query.ConditionQuery;
import org.apache.hugegraph.iterator.FilterIterator;
import org.apache.hugegraph.schema.EdgeLabel;
import org.apache.hugegraph.type.HugeType;
import org.apache.hugegraph.type.define.Directions;
import org.apache.hugegraph.type.define.HugeKeys;
import org.apache.tinkerpop.gremlin.structure.Edge;

public class EdgeExistenceTraverser extends HugeTraverser {

public EdgeExistenceTraverser(HugeGraph graph) {
super(graph);
}

Check warning on line 36 in hugegraph-server/hugegraph-core/src/main/java/org/apache/hugegraph/traversal/algorithm/EdgeExistenceTraverser.java

View check run for this annotation

Codecov / codecov/patch

hugegraph-server/hugegraph-core/src/main/java/org/apache/hugegraph/traversal/algorithm/EdgeExistenceTraverser.java#L35-L36

Added lines #L35 - L36 were not covered by tests

public Iterator<Edge> queryEdgeExistence(Id sourceId, Id targetId, String label,
String sortValues, long limit) {
// If no label provided, fallback to a slow query by filtering
if (label == null || label.isEmpty()) {
return queryByNeighbors(sourceId, targetId, limit);

Check warning on line 42 in hugegraph-server/hugegraph-core/src/main/java/org/apache/hugegraph/traversal/algorithm/EdgeExistenceTraverser.java

View check run for this annotation

Codecov / codecov/patch

hugegraph-server/hugegraph-core/src/main/java/org/apache/hugegraph/traversal/algorithm/EdgeExistenceTraverser.java#L42

Added line #L42 was not covered by tests
}

Id edgeLabelId = getEdgeLabelId(label);
EdgeLabel edgeLabel = graph().edgeLabel(edgeLabelId);
ConditionQuery conditionQuery = new ConditionQuery(HugeType.EDGE);
conditionQuery.eq(HugeKeys.OWNER_VERTEX, sourceId);
conditionQuery.eq(HugeKeys.OTHER_VERTEX, targetId);
conditionQuery.eq(HugeKeys.LABEL, edgeLabelId);
conditionQuery.eq(HugeKeys.DIRECTION, Directions.OUT);
conditionQuery.limit(limit);

Check warning on line 52 in hugegraph-server/hugegraph-core/src/main/java/org/apache/hugegraph/traversal/algorithm/EdgeExistenceTraverser.java

View check run for this annotation

Codecov / codecov/patch

hugegraph-server/hugegraph-core/src/main/java/org/apache/hugegraph/traversal/algorithm/EdgeExistenceTraverser.java#L45-L52

Added lines #L45 - L52 were not covered by tests

if (edgeLabel.existSortKeys()) {
conditionQuery.eq(HugeKeys.SORT_VALUES, sortValues);

Check warning on line 55 in hugegraph-server/hugegraph-core/src/main/java/org/apache/hugegraph/traversal/algorithm/EdgeExistenceTraverser.java

View check run for this annotation

Codecov / codecov/patch

hugegraph-server/hugegraph-core/src/main/java/org/apache/hugegraph/traversal/algorithm/EdgeExistenceTraverser.java#L55

Added line #L55 was not covered by tests
} else {
conditionQuery.eq(HugeKeys.SORT_VALUES, "");

Check warning on line 57 in hugegraph-server/hugegraph-core/src/main/java/org/apache/hugegraph/traversal/algorithm/EdgeExistenceTraverser.java

View check run for this annotation

Codecov / codecov/patch

hugegraph-server/hugegraph-core/src/main/java/org/apache/hugegraph/traversal/algorithm/EdgeExistenceTraverser.java#L57

Added line #L57 was not covered by tests
}
return graph().edges(conditionQuery);

Check warning on line 59 in hugegraph-server/hugegraph-core/src/main/java/org/apache/hugegraph/traversal/algorithm/EdgeExistenceTraverser.java

View check run for this annotation

Codecov / codecov/patch

hugegraph-server/hugegraph-core/src/main/java/org/apache/hugegraph/traversal/algorithm/EdgeExistenceTraverser.java#L59

Added line #L59 was not covered by tests
}

private Iterator<Edge> queryByNeighbors(Id sourceId, Id targetId, long limit) {
return new FilterIterator<>(edgesOfVertex(sourceId, Directions.OUT, (Id) null, limit),
edge -> targetId.equals(edge.inVertex().id()));

Check warning on line 64 in hugegraph-server/hugegraph-core/src/main/java/org/apache/hugegraph/traversal/algorithm/EdgeExistenceTraverser.java

View check run for this annotation

Codecov / codecov/patch

hugegraph-server/hugegraph-core/src/main/java/org/apache/hugegraph/traversal/algorithm/EdgeExistenceTraverser.java#L63-L64

Added lines #L63 - L64 were not covered by tests
}
}
Loading