feat(algorithm): support biased second order random walk #280

diaohancai · 2023-11-08T13:54:32Z

Purpose of the PR

implement [Feature] random walk algorithm support biased second order random walk #279
follow-up feat(algorithm): support random walk in computer #274 (V1 version)

Main Changes

The current random walk algorithm requires 2 additional features.

Biased random walk.
Second order random walk.

Add the following parameters:

    private String weightProperty;
    private Double defaultWeight;
    private Double minWeightThreshold;
    private Double maxWeightThreshold;
 
    private Double returnFactor;
    private Double inOutFactor;

String weightProperty. To implement biased random walk. The higher the weight, the higher the probability of walking.
Double defaultWeight. Provide a default value if the weight is null.
Double minWeightThreshold. Truncate when weight is less than the threshold to avoid too small weight.
Double maxWeightThreshold. Truncate when weight exceeds the threshold to avoid overweighting.
Double returnFactor. Controls the probability of re-walk to a previously walked vertex.
Double inOutFactor. Controls whether to walk inward or outward.

For more details about returnFactor and inOutFactor, please refer to the paper《node2vec: scalable feature learning for networks》.

Verifying these changes

Trivial rework / code cleanup without any test coverage. (No Need)
Already covered by existing tests, such as (please modify tests here).
Need tests and can be verified as follows.
Unit test: org.apache.hugegraph.computer.algorithm.sampling.RandomWalkTest

Does this PR potentially affect the following parts?

Documentation Status

Doc - TODO
Doc - Done
Doc - No Need

codecov · 2023-11-09T08:48:51Z

Codecov Report

Attention: 29 lines in your changes are missing coverage. Please review.

Comparison is base (ff85e34) 85.03% compared to head (7d6e8ca) 84.99%.
Report is 4 commits behind head on master.

❗ Current head 7d6e8ca differs from pull request most recent head 94bd35b. Consider uploading reports for the commit 94bd35b to get more accurate results

Files	Patch %	Lines
...egraph/computer/algorithm/sampling/RandomWalk.java	78.86%	11 Missing and 15 partials ⚠️
...hugegraph/computer/core/graph/value/ListValue.java	33.33%	1 Missing and 1 partial ⚠️
...computer/algorithm/sampling/RandomWalkMessage.java	95.65%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##             master     #280      +/-   ##
============================================
- Coverage     85.03%   84.99%   -0.04%     
- Complexity     3246     3291      +45     
============================================
  Files           345      349       +4     
  Lines         12298    12472     +174     
  Branches       1102     1129      +27     
============================================
+ Hits          10458    10601     +143     
- Misses         1315     1328      +13     
- Partials        525      543      +18

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

javeme

some tiny comments

javeme · 2023-11-11T04:55:40Z

...ter-algorithm/src/main/java/org/apache/hugegraph/computer/algorithm/sampling/RandomWalk.java

+        }
+
+        // weight threshold truncation
+        if ((Double) weight.value() < this.minWeightThreshold) {


can we call weight.doubleValue() here?

Property value may not be a numeric value.

javeme · 2023-11-11T04:56:39Z

...ter-algorithm/src/main/java/org/apache/hugegraph/computer/algorithm/sampling/RandomWalk.java

+    }
+
+    /**
+     * get edge weight by weight property


"Get the weight of a edge by its weight property"

javeme · 2023-11-11T04:56:53Z

...ter-algorithm/src/main/java/org/apache/hugegraph/computer/algorithm/sampling/RandomWalk.java

+    /**
+     * get edge weight by weight property
+     */
+    private Value getWeight(Edge edge) {


prefer getEdgeWeight()

javeme · 2023-11-11T05:00:11Z

...ter-algorithm/src/main/java/org/apache/hugegraph/computer/algorithm/sampling/RandomWalk.java

        }
-        LOG.info("[RandomWalk] algorithm param, {}: {}", OPTION_WALK_LENGTH, walkLength);
+        LOG.info("[RandomWalk] algorithm param, {}: {}", OPTION_WALK_LENGTH, this.walkLength);


we can add a common method like logAlgorithmParam(name, value), and just use the this.name() as logged algorithm name

Now, I feel that the logs of algorithm param here should be placed in the framework. I'll remove thoes logs.

diaohancai · 2023-11-13T06:37:23Z

some tiny comments

Thank you for your guidance.

javeme · 2023-11-13T12:12:29Z

...ter-algorithm/src/main/java/org/apache/hugegraph/computer/algorithm/sampling/RandomWalk.java

        }

        // weight threshold truncation
-        if ((Double) weight.value() < this.minWeightThreshold) {
+        DoubleValue weight = (DoubleValue) property;


based on a double value like:

double weight = this.defaultWeight;

weight = property..doubleValue() if checked ok

do truncation

return weight

javeme · 2023-11-13T12:13:31Z

...ter-algorithm/src/main/java/org/apache/hugegraph/computer/algorithm/sampling/RandomWalk.java

-    private Double calculateWeight(Id preVertexId, IdList preVertexAdjacenceIdList,
-                                   Id nextVertexId, Value weight) {
+    private Double calculateEdgeWeight(Id preVertexId, IdList preVertexAdjacenceIdList,
+                                       Id nextVertexId, double weight) {


also keep double finalWeight and return double?

javeme · 2023-11-13T12:20:28Z

...ter-algorithm/src/main/java/org/apache/hugegraph/computer/algorithm/sampling/RandomWalk.java

+            double weight = this.getEdgeWeight(edge);
+            Double finalWeight = this.calculateEdgeWeight(preVertexId, preVertexAdjacenceIdList,
+                                                          edge.targetId(), weight);
+            weightList.add(finalWeight);


can we add a mark here: TODO: improve to avoid OOM

javeme · 2023-11-13T12:49:25Z

...ter-algorithm/src/main/java/org/apache/hugegraph/computer/algorithm/sampling/RandomWalk.java

-        Edge selectedEdge = null;
-        int randomNum = random.nextInt(edges.size());
+    private Edge randomSelectEdge(Id preVertexId, IdList preVertexAdjacenceIdList, Edges edges) {
+        List<Double> weightList = new ArrayList<>();


can we add a mark here: TODO: use primitive array instead, like DoubleArray, in order to reduce memory fragmentation generated during calculations
the same to https://github.com/search?q=repo%3Aapache%2Fincubator-hugegraph-computer+path%3A%2F%5Ecomputer-algorithm%5C%2F%2F++new+ArrayList&type=code

You mean double[]? I'll submit a new issue for this and try to fix it.

yes, but we can use third-party libraries such as eclipse collections: https://eclipse.dev/collections/javadoc/11.1.0/org/eclipse/collections/api/list/primitive/DoubleList.html

some libs of primitive collections:

eclipse-collections: https://github.com/eclipse/eclipse-collections

hppc: https://github.com/carrotsearch/hppc

fastutil: https://github.com/vigna/fastutil

we also used them in https://github.com/apache/incubator-hugegraph/blob/25301f62288293e53c852f9c4eeb116a3a201594/hugegraph-server/hugegraph-core/src/main/java/org/apache/hugegraph/util/collection/CollectionFactory.java#L43

javeme · 2023-11-18T06:38:43Z

...ter-algorithm/src/main/java/org/apache/hugegraph/computer/algorithm/sampling/RandomWalk.java

-        Edge selectedEdge = null;
-        int randomNum = random.nextInt(edges.size());
+    private Edge randomSelectEdge(Id preVertexId, IdList preVertexAdjacenceIdList, Edges edges) {
+        List<Double> weightList = new ArrayList<>();


yes, but we can use third-party libraries such as eclipse collections: https://eclipse.dev/collections/javadoc/11.1.0/org/eclipse/collections/api/list/primitive/DoubleList.html

some libs of primitive collections:

eclipse-collections: https://github.com/eclipse/eclipse-collections

hppc: https://github.com/carrotsearch/hppc

fastutil: https://github.com/vigna/fastutil

we also used them in https://github.com/apache/incubator-hugegraph/blob/25301f62288293e53c852f9c4eeb116a3a201594/hugegraph-server/hugegraph-core/src/main/java/org/apache/hugegraph/util/collection/CollectionFactory.java#L43

javeme · 2023-11-18T06:41:03Z

...hm/src/main/java/org/apache/hugegraph/computer/algorithm/path/rings/filter/SpreadFilter.java

@@ -128,6 +128,8 @@ private static List<Serializable> expressions(
        if (filter.size() == 0) {
            return PASS;
        }
+        // TODO: use primitive array instead, like DoubleArray,


we only need to mark lists about algorithms.

javeme · 2023-11-18T06:46:43Z

...gorithm/src/main/java/org/apache/hugegraph/computer/algorithm/sampling/RandomWalkOutput.java

    }

    @Override
    protected List<String> value(Vertex vertex) {
        IdListList value = vertex.value();
+        // TODO: use primitive array instead, like DoubleArray,
+        //  in order to reduce memory fragmentation generated during calculations


List<String> doesn't require replacement(because there is no string[] type), and the propValues doesn't look like very large.

javeme

LGTM

imbajin

THX~

diaohancai · 2023-11-22T02:35:28Z

THX~

Thank you for your guidance.

diaohancai and others added 10 commits November 5, 2023 17:56

chore: option rename

9b7ec3e

feat: define second order random walk algorithm param

7d760ee

chore: code format

5d8eb98

feat: add defaultWeight param

3e4da8d

feat: implement second order random walk algorithm

a2b6efc

feat: ListValue support getFirst

0c14381

bug: null point

5914bff

chore: supplementary test parameters

77d3752

optimize: change weightThreshold parameter to Double type

70d8f0f

optimize: weight threshold truncation

548148e

diaohancai changed the title ~~Feat second order random walk~~ feat(algorithm): support biased second order random walk Nov 8, 2023

coderzc requested review from Linary, javeme, imbajin and coderzc November 9, 2023 08:28

javeme reviewed Nov 11, 2023

View reviewed changes

diaohancai added 2 commits November 13, 2023 16:06

optimize: method optimize

b4b94b1

chore: nullableKeys test

7d6e8ca

diaohancai requested a review from javeme November 13, 2023 08:58

javeme reviewed Nov 13, 2023

View reviewed changes

diaohancai added 2 commits November 15, 2023 15:22

refactor: double value simplification

18f1260

chore: add todo note

9a93341

diaohancai requested a review from javeme November 15, 2023 07:31

fix: can not load edge properties

37d2aa7

javeme reviewed Nov 18, 2023

View reviewed changes

chore: remove TODO

8cc1734

diaohancai requested a review from javeme November 20, 2023 08:32

javeme approved these changes Nov 21, 2023

View reviewed changes

imbajin approved these changes Nov 22, 2023

View reviewed changes

Merge branch 'master' into pr/280

94bd35b

imbajin merged commit 2be0c28 into apache:master Dec 4, 2023
6 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(algorithm): support biased second order random walk #280

feat(algorithm): support biased second order random walk #280

diaohancai commented Nov 8, 2023 •

edited by imbajin

Loading

codecov bot commented Nov 9, 2023 •

edited

Loading

javeme left a comment

javeme Nov 11, 2023

diaohancai Nov 13, 2023

javeme Nov 11, 2023

javeme Nov 11, 2023

javeme Nov 11, 2023

diaohancai Nov 13, 2023

diaohancai commented Nov 13, 2023

javeme Nov 13, 2023

javeme Nov 13, 2023

javeme Nov 13, 2023

javeme Nov 13, 2023

diaohancai Nov 15, 2023

javeme Nov 18, 2023

diaohancai Nov 20, 2023

javeme Nov 18, 2023

javeme Nov 18, 2023

javeme Nov 18, 2023

javeme left a comment

imbajin left a comment

diaohancai commented Nov 22, 2023

feat(algorithm): support biased second order random walk #280

feat(algorithm): support biased second order random walk #280

Conversation

diaohancai commented Nov 8, 2023 • edited by imbajin Loading

Purpose of the PR

Main Changes

Verifying these changes

Does this PR potentially affect the following parts?

Documentation Status

codecov bot commented Nov 9, 2023 • edited Loading

Codecov Report

javeme left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

diaohancai commented Nov 13, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

awesome

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

javeme left a comment

Choose a reason for hiding this comment

imbajin left a comment

Choose a reason for hiding this comment

diaohancai commented Nov 22, 2023

diaohancai commented Nov 8, 2023 •

edited by imbajin

Loading

codecov bot commented Nov 9, 2023 •

edited

Loading