feat(api&core): in oltp apis, add statistics info and support full info about vertices and edges #2262
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ApiMeasure will count the number of vertices and edges traversed at runtime, and the time the api takes to execute
…rface * JsonSerializer: return measure information in api response * Serializer: fit the feature that returns complete information about vertices and edges
…d Support full information about vertices and edges Statistics information: * add vertexIterCounter and edgeIterCounter in HugeTraverser.java to track traversed vertices and edges at run time * modify all oltp restful apis to add statistics information in response Full information about vertices and edges: * add 'with_vertex' and 'with_edge' parameter option in apis * modify oltp apis to support vertex and edge information in api response * add EdgeRecord in HugeTraverser.java to record edges at run time and generate the edge information returned in api response * modify Path and PathSet in HugeTraverser.java to support full edge information storage * modify all traversers to support track of edge information at run time
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
...graph-core/src/main/java/org/apache/hugegraph/traversal/algorithm/SameNeighborTraverser.java
Fixed
Show resolved
Hide resolved
Codecov Report
@@ Coverage Diff @@
## master #2262 +/- ##
============================================
- Coverage 68.63% 65.08% -3.56%
- Complexity 977 981 +4
============================================
Files 498 498
Lines 40684 41240 +556
Branches 5681 5738 +57
============================================
- Hits 27922 26839 -1083
- Misses 10056 11746 +1690
+ Partials 2706 2655 -51
... and 72 files with indirect coverage changes 📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more |
javeme
reviewed
Jul 30, 2023
hugegraph-api/src/main/java/org/apache/hugegraph/api/traversers/KneighborAPI.java
Outdated
Show resolved
Hide resolved
...ph-core/src/main/java/org/apache/hugegraph/traversal/algorithm/CollectionPathsTraverser.java
Show resolved
Hide resolved
...e/src/main/java/org/apache/hugegraph/traversal/algorithm/MultiNodeShortestPathTraverser.java
Show resolved
Hide resolved
| return manager.serializer(g).writePaths("crosspoints", paths, true); | ||
| measure.addIterCount(traverser.vertexIterCounter.get(), | ||
| traverser.edgeIterCounter.get()); | ||
| return manager.serializer(g, measure.getResult()) |
There was a problem hiding this comment.
prefer to update the serializer(g) method and set default an ApiMeasure
There was a problem hiding this comment.
Rejected:
ApiMeasurershould be passed in from an outside api implementation so that it can carry data- not all apis need
ApiMeasurerand keeping theApiMeasurernull for those APIs can preventApiMeasurerfrom being serialized
hugegraph-api/src/main/java/org/apache/hugegraph/api/traversers/CustomizedCrosspointsAPI.java
Outdated
Show resolved
Hide resolved
hugegraph-api/src/main/java/org/apache/hugegraph/api/traversers/CustomizedCrosspointsAPI.java
Outdated
Show resolved
Hide resolved
hugegraph-api/src/main/java/org/apache/hugegraph/api/traversers/AllShortestPathsAPI.java
Outdated
Show resolved
Hide resolved
DanGuge
force-pushed
the
feature_measure_and_VE_info
branch
from
d468e18 to
e671963
Compare
6 tasks
zyxxoo
reviewed
Aug 11, 2023
| } | ||
|
|
||
| public Map<String, Object> measures() { | ||
| measures.put(COST, System.currentTimeMillis() - timeStart); |
There was a problem hiding this comment.
compute duration use System.nanoTime perfered
| } else if (current instanceof MutableLong) { | ||
| MutableLong currentMutableLong = (MutableLong) current; | ||
| currentMutableLong.add(value); | ||
| } else if (current instanceof Long) { |
There was a problem hiding this comment.
measures.computeIfAbsent(key, MutableLong.new).add(value)
| private static CollectionFactory collectionFactory; | ||
| private final HugeGraph graph; | ||
| // for apimeasure | ||
| public AtomicLong edgeIterCounter = new AtomicLong(0); |
There was a problem hiding this comment.
Wouldn't it be better for this class to have no side effects?
There was a problem hiding this comment.
HugeTraverser is the root class of all traverser algorithms, and there is a need for all sub-traverser algorithms to count the measuring statistics during traversering. So I think of two ways to solve this problems:
- Give HugeTraverser the ability to track the measuring statistics;
- Track the measuing statistics in all traverser functions which the Traverser APIs would call
I think the first one is better & more elegant and this is the current implementation.
imbajin
reviewed
Aug 11, 2023
.gitignore
Outdated
Comment on lines
10
to
11
| swagger-ui-* | ||
| hugegraph-dist/dist.sh |
There was a problem hiding this comment.
Thanks, but we need fix/remove it in the dist-script rather than exclude them in git (will be fixed it in another PR) @VGalaxies
There was a problem hiding this comment.
ok, fixed and rollback change
11 tasks
javeme
reviewed
Aug 12, 2023
| public static final String JSON = MediaType.APPLICATION_JSON_TYPE | ||
| .getSubtype(); | ||
|
|
||
| .getSubtype(); |
There was a problem hiding this comment.
can we keep the origin style
| @@ -224,8 +220,7 @@ protected void addCount(String key, long value) { | |||
| if (current == null) { | |||
| measures.put(key, new MutableLong(value)); | |||
| } else if (current instanceof MutableLong) { | |||
| MutableLong currentMutableLong = (MutableLong) current; | |||
| currentMutableLong.add(value); | |||
| ((MutableLong) measures.computeIfAbsent(key, MutableLong::new)).add(value); | |||
| } else if (current instanceof Long) { | |||
There was a problem hiding this comment.
add else branch then throw an exception
| @Path("graphs/{graph}/traversers/customizedpaths") | ||
| @Singleton | ||
| @Tag(name = "CustomizedPathsAPI") | ||
| public class CustomizedPathsAPI extends API { | ||
|
|
||
| private static final Logger LOG = Log.logger(CustomizedPathsAPI.class); | ||
|
|
||
| private static List<WeightedEdgeStep> step(HugeGraph graph, | ||
| PathRequest req) { |
There was a problem hiding this comment.
also keep request name style?
| HugeGraph g = graph(manager, graph); | ||
| Iterator<Vertex> sources = request.sources.vertices(g); | ||
| E.checkArgument(sources != null && sources.hasNext(), | ||
| "The source vertices can't be empty"); | ||
|
|
||
| FusiformSimilarityTraverser traverser = | ||
| new FusiformSimilarityTraverser(g); | ||
| new FusiformSimilarityTraverser(g); |
| E.checkArgument(!request.withVertex && !request.withPath, | ||
| "Can't return vertex or path when count only"); | ||
| E.checkArgument(!request.withVertex && !request.withPath && !request.withEdge, | ||
| "Can't return vertex or path or edge when count only"); |
There was a problem hiding this comment.
Can't return vertex, edge or path...
| public class CustomizedCrosspointsTraverser extends HugeTraverser { | ||
|
|
||
| private final EdgeRecord edgeRecord; |
| public class CustomizePathsTraverser extends HugeTraverser { | ||
| private final EdgeRecord edgeRecord; |
There was a problem hiding this comment.
expect a blank line, and prefer to rename to edgeResults.
| @@ -52,10 +50,11 @@ public abstract class PathTraverser { | |||
| protected Set<HugeTraverser.Path> paths; | |||
|
|
|||
| protected TraverseStrategy traverseStrategy; | |||
| protected HugeTraverser.EdgeRecord edgeRecord; | |||
There was a problem hiding this comment.
just import EdgeRecord to keep the same style?
| return paths; | ||
| } | ||
|
|
||
| private class Traverser { | ||
|
|
||
| private final ShortestPathRecords record; | ||
| private final EdgeRecord edgeRecord; |
There was a problem hiding this comment.
rename to pathResults and edgeResults
| result.add(nodes.get(random)); | ||
| } | ||
| return result; | ||
| public EdgeRecord getEdgeRecord() { |
* rename edgeRecord to edgeResults * fix Request class code style in SameNeighborsAPI.java
|
Good job, thanks for ur contributuion & reaction 'll check this pr carefully later, @zyxxoo could ensure it now? |
zyxxoo
approved these changes
Aug 18, 2023
| for (Path path : paths) { | ||
| List<Id> vertices = path.vertices(); | ||
| int length = vertices.size(); | ||
| if (intersection.contains(vertices.get(length - 1))) { |
There was a problem hiding this comment.
Do you mean to leave this static method in its original place?
I just formatted it automatically. The content of this method is the same as the original, and only the location has changed.
There was a problem hiding this comment.
I will check it later（merge it first）
zyxxoo
approved these changes
Aug 18, 2023
DanGuge
added a commit
to DanGuge/incubator-hugegraph
that referenced
this pull request
Sep 25, 2023
…fo about vertices and edges (apache#2262) * chore: improve gitignore file * feat: add ApiMeasure to collect runtime data ApiMeasure will count the number of vertices and edges traversed at runtime, and the time the api takes to execute * feat: Add ApiMeasure to JsonSerializer and Modify the Serializer interface * JsonSerializer: return measure information in api response * Serializer: fit the feature that returns complete information about vertices and edges * refactor: format code based on hugegraph-style.xml * feat: Add statistics information in all oltp restful apis response and Support full information about vertices and edges Statistics information: * add vertexIterCounter and edgeIterCounter in HugeTraverser.java to track traversed vertices and edges at run time * modify all oltp restful apis to add statistics information in response Full information about vertices and edges: * add 'with_vertex' and 'with_edge' parameter option in apis * modify oltp apis to support vertex and edge information in api response * add EdgeRecord in HugeTraverser.java to record edges at run time and generate the edge information returned in api response * modify Path and PathSet in HugeTraverser.java to support full edge information storage * modify all traversers to support track of edge information at run time * fix: numeric cast * fix: Jaccard Similarity api test * fix: adjust the code style and naming convention * Empty commit * Empty commit * fix: 1. change System.currentTimeMillis() to System.nanoTime(); 2. modify addCount() * fix: rollback change in .gitignore * fix: rollback ServerOptions.java code style * fix: rollback API.java code style and add exception in else branch * fix: fix code style * fix: name style & code style * rename edgeRecord to edgeResults * fix Request class code style in SameNeighborsAPI.java
VGalaxies
pushed a commit
to VGalaxies/incubator-hugegraph
that referenced
this pull request
Nov 10, 2023
…fo about vertices and edges (apache#2262) * chore: improve gitignore file * feat: add ApiMeasure to collect runtime data ApiMeasure will count the number of vertices and edges traversed at runtime, and the time the api takes to execute * feat: Add ApiMeasure to JsonSerializer and Modify the Serializer interface * JsonSerializer: return measure information in api response * Serializer: fit the feature that returns complete information about vertices and edges * refactor: format code based on hugegraph-style.xml * feat: Add statistics information in all oltp restful apis response and Support full information about vertices and edges Statistics information: * add vertexIterCounter and edgeIterCounter in HugeTraverser.java to track traversed vertices and edges at run time * modify all oltp restful apis to add statistics information in response Full information about vertices and edges: * add 'with_vertex' and 'with_edge' parameter option in apis * modify oltp apis to support vertex and edge information in api response * add EdgeRecord in HugeTraverser.java to record edges at run time and generate the edge information returned in api response * modify Path and PathSet in HugeTraverser.java to support full edge information storage * modify all traversers to support track of edge information at run time * fix: numeric cast * fix: Jaccard Similarity api test * fix: adjust the code style and naming convention * Empty commit * Empty commit * fix: 1. change System.currentTimeMillis() to System.nanoTime(); 2. modify addCount() * fix: rollback change in .gitignore * fix: rollback ServerOptions.java code style * fix: rollback API.java code style and add exception in else branch * fix: fix code style * fix: name style & code style * rename edgeRecord to edgeResults * fix Request class code style in SameNeighborsAPI.java
imbajin
pushed a commit
that referenced
this pull request
Nov 10, 2023
…fo about vertices and edges (#2262) * chore: improve gitignore file * feat: add ApiMeasure to collect runtime data ApiMeasure will count the number of vertices and edges traversed at runtime, and the time the api takes to execute * feat: Add ApiMeasure to JsonSerializer and Modify the Serializer interface * JsonSerializer: return measure information in api response * Serializer: fit the feature that returns complete information about vertices and edges * refactor: format code based on hugegraph-style.xml * feat: Add statistics information in all oltp restful apis response and Support full information about vertices and edges Statistics information: * add vertexIterCounter and edgeIterCounter in HugeTraverser.java to track traversed vertices and edges at run time * modify all oltp restful apis to add statistics information in response Full information about vertices and edges: * add 'with_vertex' and 'with_edge' parameter option in apis * modify oltp apis to support vertex and edge information in api response * add EdgeRecord in HugeTraverser.java to record edges at run time and generate the edge information returned in api response * modify Path and PathSet in HugeTraverser.java to support full edge information storage * modify all traversers to support track of edge information at run time * fix: numeric cast * fix: Jaccard Similarity api test * fix: adjust the code style and naming convention * Empty commit * Empty commit * fix: 1. change System.currentTimeMillis() to System.nanoTime(); 2. modify addCount() * fix: rollback change in .gitignore * fix: rollback ServerOptions.java code style * fix: rollback API.java code style and add exception in else branch * fix: fix code style * fix: name style & code style * rename edgeRecord to edgeResults * fix Request class code style in SameNeighborsAPI.java
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Purpose of the PR
Main Changes
Statistics information:
add vertexIterCounter and edgeIterCounter in HugeTraverser.java to track traversed vertices and edges at run time
modify all oltp apis to add statistics information in response
modify Serializer.java and JsonSerializer.java to support statistics infomation serialization
design document
Full information about vertices and edges:
add 'with_vertex' and 'with_edge' parameter option in oltp apis
modify oltp apis to support vertex and edge information in api response
add EdgeRecord in HugeTraverser.java to record edges at run time and generate the edge information returned in api response
modify Path and PathSet in HugeTraverser.java to support full edge information storage
modify all traversers to support track of edge information at run time
modify Serializer.java and JsonSerializer.java to support full infomation serialization about vertices and edges
design document
Implementation overview
N: No Need
✓: Supported
Verifying these changes
Kout PostandSingle Source Shortest Path Getare shown belowInitialize the Graph
Kout Post
with_vertex=false, with_edge=false
{ "source": "1:marko", "step": { "direction": "BOTH", "labels": ["knows", "created"], "properties": { "weight": "P.gt(0.1)" }, "max_degree": 10000, "skip_degree": 100000 }, "max_depth": 1, "nearest": true, "limit": 10000, "with_vertex": false, "with_path": true, "with_edge": false }{ "kout": [ "1:vadas", "1:josh", "2:lop" ], "size": 3, "paths": [ { "objects": [ "1:marko", "2:lop" ] }, { "objects": [ "1:marko", "1:josh" ] }, { "objects": [ "1:marko", "1:vadas" ] } ], "vertices": [ "1:marko", "1:josh", "1:vadas", "2:lop" ], "edges": [ "S1:marko>1>20160110>S1:vadas", "S1:marko>2>>S2:lop", "S1:marko>1>20130220>S1:josh" ], "measure": { "edge_iterations": 3, "vertice_iterations": 1, "cost": 72 } }with_vertex=true, with_edge=true
{ "source": "1:marko", "step": { "direction": "BOTH", "labels": ["knows", "created"], "properties": { "weight": "P.gt(0.1)" }, "max_degree": 10000, "skip_degree": 100000 }, "max_depth": 1, "nearest": true, "limit": 10000, "with_vertex": true, "with_path": true, "with_edge": true }{ "kneighbor": [ "1:vadas", "1:josh", "2:lop", "1:peter", "2:ripple" ], "size": 5, "paths": [ { "objects": [ "1:marko", "2:lop" ] }, { "objects": [ "1:marko", "2:lop", "1:peter" ] }, { "objects": [ "1:marko", "1:josh" ] }, { "objects": [ "1:marko", "1:vadas" ] }, { "objects": [ "1:marko", "1:josh", "2:ripple" ] } ], "vertices": [ { "id": "2:ripple", "label": "software", "type": "vertex", "properties": { "name": "ripple", "lang": "java", "price": 199 } }, { "id": "1:marko", "label": "person", "type": "vertex", "properties": { "name": "marko", "age": 29, "city": "Beijing" } }, { "id": "1:josh", "label": "person", "type": "vertex", "properties": { "name": "josh", "age": 32, "city": "Beijing" } }, { "id": "1:vadas", "label": "person", "type": "vertex", "properties": { "name": "vadas", "age": 27, "city": "Hongkong" } }, { "id": "1:peter", "label": "person", "type": "vertex", "properties": { "name": "peter", "age": 35, "city": "Shanghai" } }, { "id": "2:lop", "label": "software", "type": "vertex", "properties": { "name": "lop", "lang": "java", "price": 328 } } ], "edges": [ { "id": "S1:josh>2>>S2:ripple", "label": "created", "type": "edge", "outV": "1:josh", "outVLabel": "person", "inV": "2:ripple", "inVLabel": "software", "properties": { "weight": 1.0, "date": "20171210" } }, { "id": "S1:marko>1>20160110>S1:vadas", "label": "knows", "type": "edge", "outV": "1:marko", "outVLabel": "person", "inV": "1:vadas", "inVLabel": "person", "properties": { "weight": 0.5, "date": "20160110" } }, { "id": "S1:marko>2>>S2:lop", "label": "created", "type": "edge", "outV": "1:marko", "outVLabel": "person", "inV": "2:lop", "inVLabel": "software", "properties": { "weight": 0.4, "date": "20171210" } }, { "id": "S1:marko>1>20130220>S1:josh", "label": "knows", "type": "edge", "outV": "1:marko", "outVLabel": "person", "inV": "1:josh", "inVLabel": "person", "properties": { "weight": 1.0, "date": "20130220" } }, { "id": "S1:peter>2>>S2:lop", "label": "created", "type": "edge", "outV": "1:peter", "outVLabel": "person", "inV": "2:lop", "inVLabel": "software", "properties": { "weight": 0.2, "date": "20170324" } } ], "measure": { "edge_iterations": 12, "vertice_iterations": 12, "cost": 34 } }Single Source Shortest Path Get
{ "paths": { "1:peter": { "weight": 0.6000000000000001, "vertices": [ "1:marko", "2:lop", "1:peter" ] }, "2:lop": { "weight": 0.4, "vertices": [ "1:marko", "2:lop" ] }, "1:josh": { "weight": 0.8, "vertices": [ "1:marko", "2:lop", "1:josh" ] }, "2:ripple": { "weight": 1.8, "vertices": [ "1:marko", "2:lop", "1:josh", "2:ripple" ] }, "1:vadas": { "weight": 0.5, "vertices": [ "1:marko", "1:vadas" ] } }, "vertices": [ "1:peter", "2:lop", "1:josh", "2:ripple", "1:marko", "1:vadas" ], "edges": [ "S1:josh>2>>S2:ripple", "S1:marko>1>20160110>S1:vadas", "S1:josh>2>>S2:lop", "S1:marko>2>>S2:lop", "S1:peter>2>>S2:lop" ], "measure": { "edge_iterations": 12, "vertice_iterations": 6, "cost": 16 } }{ "paths": { "1:peter": { "weight": 0.6000000000000001, "vertices": [ "1:marko", "2:lop", "1:peter" ] }, "2:lop": { "weight": 0.4, "vertices": [ "1:marko", "2:lop" ] }, "1:josh": { "weight": 0.8, "vertices": [ "1:marko", "2:lop", "1:josh" ] }, "2:ripple": { "weight": 1.8, "vertices": [ "1:marko", "2:lop", "1:josh", "2:ripple" ] }, "1:vadas": { "weight": 0.5, "vertices": [ "1:marko", "1:vadas" ] } }, "vertices": [ { "id": "1:peter", "label": "person", "type": "vertex", "properties": { "name": "peter", "age": 35, "city": "Shanghai" } }, { "id": "2:lop", "label": "software", "type": "vertex", "properties": { "name": "lop", "lang": "java", "price": 328 } }, { "id": "1:josh", "label": "person", "type": "vertex", "properties": { "name": "josh", "age": 32, "city": "Beijing" } }, { "id": "2:ripple", "label": "software", "type": "vertex", "properties": { "name": "ripple", "lang": "java", "price": 199 } }, { "id": "1:marko", "label": "person", "type": "vertex", "properties": { "name": "marko", "age": 29, "city": "Beijing" } }, { "id": "1:vadas", "label": "person", "type": "vertex", "properties": { "name": "vadas", "age": 27, "city": "Hongkong" } } ], "edges": [ { "id": "S1:josh>2>>S2:ripple", "label": "created", "type": "edge", "outV": "1:josh", "outVLabel": "person", "inV": "2:ripple", "inVLabel": "software", "properties": { "weight": 1.0, "date": "20171210" } }, { "id": "S1:marko>1>20160110>S1:vadas", "label": "knows", "type": "edge", "outV": "1:marko", "outVLabel": "person", "inV": "1:vadas", "inVLabel": "person", "properties": { "weight": 0.5, "date": "20160110" } }, { "id": "S1:josh>2>>S2:lop", "label": "created", "type": "edge", "outV": "1:josh", "outVLabel": "person", "inV": "2:lop", "inVLabel": "software", "properties": { "weight": 0.4, "date": "20091111" } }, { "id": "S1:marko>2>>S2:lop", "label": "created", "type": "edge", "outV": "1:marko", "outVLabel": "person", "inV": "2:lop", "inVLabel": "software", "properties": { "weight": 0.4, "date": "20171210" } }, { "id": "S1:peter>2>>S2:lop", "label": "created", "type": "edge", "outV": "1:peter", "outVLabel": "person", "inV": "2:lop", "inVLabel": "software", "properties": { "weight": 0.2, "date": "20170324" } } ], "measure": { "edge_iterations": 12, "vertice_iterations": 12, "cost": 2 } }Does this PR potentially affect the following parts?
Documentation Status
Doc - TODODoc - DoneDoc - No Need