[improve][build] Improve package & cli & build image #200

coderzc · 2022-11-24T10:57:36Z

We can quick run the `PageRank` algorithm by this PR on the local.

Start the hugegraph-sever and etcd first and then execute the following commands.

> mvn clean package -am -DskipTests -ntp

> cd hugegraph-computer-*/

> bin/start-computer.sh -d local -r master

> bin/start-computer.sh -d local -r worker

The effect is as follows:

View the results

Open OLAP index query for server:

PUT http://localhost:8080/graphs/hugegraph/graph_read_mode
"ALL"

Query page_rank propertie value:

> curl "http://localhost:8080/graphs/hugegraph/graph/vertices?page&limit=3" | gunzip

{"vertices":[{"id":"test","label":"person","type":"vertex","properties":{"name":"asd","age":10,"addr":"asa","weight":10.0,"iid":661554156602722581,"page_rank":0.02159763313609468}},{"id":"okram","label":"person","type":"vertex","properties":{"name":"Marko A. Rodriguez","age":29,"addr":"Santa Fe, New Mexico","weight":1.0,"page_rank":0.02159763313609468}},{"id":"dalaro","label":"person","type":"vertex","properties":{"name":"Dan LaRocque ","age":0,"addr":"","weight":1.0,"page_rank":0.02159763313609468}}],"page": "B4VqYXZlbWUAAAAAAAAAAw=="}

codecov · 2022-11-24T11:11:26Z

Codecov Report

Merging #200 (1c2e8e6) into master (5320190) will increase coverage by 0.08%.
The diff coverage is 75.00%.

@@             Coverage Diff              @@
##             master     #200      +/-   ##
============================================
+ Coverage     85.73%   85.81%   +0.08%     
- Complexity     3236     3240       +4     
============================================
  Files           344      344              
  Lines         12106    12111       +5     
  Branches       1097     1099       +2     
============================================
+ Hits          10379    10393      +14     
+ Misses         1195     1188       -7     
+ Partials        532      530       -2

Impacted Files	Coverage Δ
...egraph/computer/core/util/ComputerContextUtil.java	`87.09% <0.00%> (ø)`
...idu/hugegraph/computer/core/util/ShutdownHook.java	`66.66% <ø> (+22.22%)`	⬆️
.../hugegraph/computer/core/worker/WorkerService.java	`90.47% <ø> (ø)`
...r/k8s/operator/controller/ComputerJobDeployer.java	`86.06% <0.00%> (ø)`
...va/com/baidu/hugegraph/computer/k8s/Constants.java	`90.90% <ø> (ø)`
...gegraph/computer/k8s/config/KubeDriverOptions.java	`96.42% <ø> (ø)`
...idu/hugegraph/computer/dist/HugeGraphComputer.java	`92.45% <100.00%> (+0.78%)`	⬆️
...k8s/operator/controller/ComputerJobController.java	`88.69% <0.00%> (+1.71%)`	⬆️

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

javeme · 2022-11-24T14:05:21Z

computer-dist/src/main/java/com/baidu/hugegraph/computer/dist/HugeGraphComputer.java

@@ -69,7 +70,7 @@ public static void main(String[] args) throws IOException,
        setUncaughtExceptionHandler();
        loadClass();
        registerOptions();
-        ComputerContext context = parseContext(args[0]);
+        ComputerContext context = parseContext(args[0], role);


can separate the add-role commit separated PR?

If we don't fix it, the master and worker will not be able to use the same configuration file, It's best to stay in this PR

* address comment * improve pom and options

coderzc · 2022-11-25T04:30:01Z

ci failed：@imbajin

Error: -25 04:09:47 [Thread-2] [ERROR] c.b.h.c.s.i.SenderIntegrateTest - Failed to execute master service
java.lang.ExceptionInInitializerError: null
	at org.apache.hugegraph.driver.HugeClient.<clinit>(HugeClient.java:35) ~[hugegraph-client-1.0.0.jar:2.0.1.0]
	at org.apache.hugegraph.driver.HugeClientBuilder.build(HugeClientBuilder.java:67) ~[hugegraph-client-1.0.0.jar:2.0.1.0]
	at com.baidu.hugegraph.computer.core.input.hg.HugeInputSplitFetcher.<init>(HugeInputSplitFetcher.java:45) ~[classes/:?]
	at com.baidu.hugegraph.computer.core.input.InputSourceFactory.createInputSplitFetcher(InputSourceFactory.java:37) ~[classes/:?]
	at com.baidu.hugegraph.computer.core.input.MasterInputManager.init(MasterInputManager.java:45) ~[classes/:?]
	at com.baidu.hugegraph.computer.core.manager.Managers.initAll(Managers.java:61) ~[classes/:?]
	at com.baidu.hugegraph.computer.core.master.MasterService.initManagers(MasterService.java:301) ~[classes/:?]
	at com.baidu.hugegraph.computer.core.master.MasterService.init(MasterService.java:99) ~[classes/:?]
	at com.baidu.hugegraph.computer.suite.integrate.SenderIntegrateTest.initMaster(SenderIntegrateTest.java:297) ~[classes/:?]
	at com.baidu.hugegraph.computer.suite.integrate.SenderIntegrateTest.lambda$testOneWorkerWithBusyClient$4(SenderIntegrateTest.java:223) ~[classes/:?]
	at java.lang.Thread.run(Thread.java:829) ~[?:?]
Caused by: java.lang.IllegalStateException: The version 1.0.0 of 'hugegraph-common' is not in [2.1, 2.2)
	at com.google.common.base.Preconditions.checkState(Preconditions.java:532) ~[guava-30.0-jre.jar:?]
	at org.apache.hugegraph.util.E.checkState(E.java:68) ~[hugegraph-common-1.0.0.jar:1.0.0]
	at org.apache.hugegraph.util.VersionUtil.check(VersionUtil.java:66) ~[hugegraph-common-1.0.0.jar:1.0.0]
	at org.apache.hugegraph.version.ClientVersion.check(ClientVersion.java:38) ~[hugegraph-client-1.0.0.jar:2.0.1.0]
	at org.apache.hugegraph.version.ClientVersion.<clinit>(ClientVersion.java:29) ~[hugegraph-client-1.0.0.jar:2.0.1.0]
	... 11 more

javeme · 2022-11-25T10:54:58Z

.github/workflows/publish_images.yml

@@ -19,13 +19,11 @@ jobs:
          map: |
            {
              "push": {
-                "FRAMEWORK_IMAGE_URL": "hugegraph/hugegraph-computer-framework:latest",
-                "ALGORITHM_IMAGE_URL": "hugegraph/hugegraph-builtin-algorithms:latest",
+                "COMPUTER_IMAGE_URL": "hugegraph/hugegraph-computer:latest",


do you merge builtin-algorithm image into hugegraph-computer image?

Yes，I think the two images are a little redundant. The builtin-algorithm.jar only 800kb.

coderzc added the build label Nov 24, 2022

coderzc added this to the 1.0.0 milestone Nov 24, 2022

coderzc requested review from zyxxoo and imbajin November 24, 2022 10:58

coderzc self-assigned this Nov 24, 2022

improve package

d5b18b0

coderzc force-pushed the improve_package branch from b034998 to d5b18b0 Compare November 24, 2022 12:29

javeme reviewed Nov 24, 2022

View reviewed changes

coderzc added 4 commits November 25, 2022 00:52

Refactor computer: abstract computer-api from computer-core (#196)

1b4fb7f

* address comment * improve pom and options

fix package

8e97cd0

Merge branch 'master' into improve_package

6cb6019

improve build image

394b1b9

coderzc force-pushed the improve_package branch from cc9a39c to 394b1b9 Compare November 24, 2022 17:06

improve start-computer.sh

76f4282

coderzc changed the title ~~[improve][build] Improve package~~ [improve][build] Improve package and cli Nov 24, 2022

coderzc changed the title ~~[improve][build] Improve package and cli~~ [improve][build] Improve package & cli & build image Nov 24, 2022

improve log file

bdf2d8b

coderzc added the improvement label Nov 24, 2022

improve start-computer.sh

67b7f9f

coderzc force-pushed the improve_package branch from eb2b22d to 67b7f9f Compare November 24, 2022 19:46

coderzc requested a review from javeme November 25, 2022 02:08

coderzc mentioned this pull request Nov 25, 2022

[Summary] computer release v1.0 todo list #188

Closed

14 tasks

coderzc force-pushed the improve_package branch 2 times, most recently from b0b6524 to 7fe8bdd Compare November 25, 2022 03:06

improve build image

cb5aaf2

coderzc force-pushed the improve_package branch from 7fe8bdd to cb5aaf2 Compare November 25, 2022 03:07

coderzc added the cli label Nov 25, 2022

trigger maven cache invalidation

39714ea

coderzc force-pushed the improve_package branch from f959792 to 39714ea Compare November 25, 2022 03:59

Update ci.yml

1c2e8e6

javeme approved these changes Nov 25, 2022

View reviewed changes

imbajin approved these changes Nov 25, 2022

View reviewed changes

imbajin merged commit adf6801 into master Nov 25, 2022

imbajin deleted the improve_package branch November 25, 2022 15:30

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[improve][build] Improve package & cli & build image #200

[improve][build] Improve package & cli & build image #200

coderzc commented Nov 24, 2022 •

edited

Loading

codecov bot commented Nov 24, 2022 •

edited

Loading

javeme Nov 24, 2022

coderzc Nov 24, 2022

coderzc commented Nov 25, 2022 •

edited

Loading

javeme Nov 25, 2022

coderzc Nov 25, 2022 •

edited

Loading

javeme Nov 25, 2022

[improve][build] Improve package & cli & build image #200

[improve][build] Improve package & cli & build image #200

Conversation

coderzc commented Nov 24, 2022 • edited Loading

We can quick run the PageRank algorithm by this PR on the local.

The effect is as follows:

View the results

codecov bot commented Nov 24, 2022 • edited Loading

Codecov Report

javeme Nov 24, 2022

Choose a reason for hiding this comment

coderzc Nov 24, 2022

Choose a reason for hiding this comment

coderzc commented Nov 25, 2022 • edited Loading

javeme Nov 25, 2022

Choose a reason for hiding this comment

coderzc Nov 25, 2022 • edited Loading

Choose a reason for hiding this comment

javeme Nov 25, 2022

Choose a reason for hiding this comment

coderzc commented Nov 24, 2022 •

edited

Loading

We can quick run the `PageRank` algorithm by this PR on the local.

codecov bot commented Nov 24, 2022 •

edited

Loading

coderzc commented Nov 25, 2022 •

edited

Loading

coderzc Nov 25, 2022 •

edited

Loading