feat: support Cassandra with docker-compose in server #2307

aroundabout · 2023-09-09T04:41:00Z

…840)

Purpose of the PR

subtask of [Feature] Support build docker image from Dockerfile (Task Summary) #840 : support Cassandra(Docker) as backend (compose with HugeGraph)
Now support use docker to start hugegraph with cassandra backend
problem like [Question] docker挂载数据目录时，启动失败 #2298 can also be solved.

Main Changes

change the dockerfile, adding the shell to wait for storage backend and use a docker-entrypoint.sh to manage the starting process.
delete a deprecated class in gremlin-console.sh (reference: doc of ScriptExecutor)
add a healthy check in docker-compose
add an example folder where we can put all the template docker-compose.yml here
add *swagger-ui* in gitignore, which appears after you compile the source code locally.

Detailed doc of the pr is here: https://hugegraph.feishu.cn/wiki/BBrgwmytNi0DiQkvDj1cpNbFn5b
Some test case: https://hugegraph.feishu.cn/docx/CBcydSr6CokDpexlZAQcH4Icnlb#TbYrdpl3io1c9NxT4vLcDTn8nhg

Verifying these changes

Trivial rework / code cleanup without any test coverage. (No Need)
Already covered by existing tests, such as (please modify tests here).
Need tests and can be verified as follows:
- xxx

Does this PR potentially affect the following parts?

Documentation Status

Doc - TODO
Doc - Done
Doc - No Need

…pache#840)

codecov · 2023-09-09T04:51:44Z

Codecov Report

Merging #2307 (da46a0f) into master (869fc81) will increase coverage by 4.44%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##             master    #2307      +/-   ##
============================================
+ Coverage     63.85%   68.29%   +4.44%     
- Complexity      684      987     +303     
============================================
  Files           505      505              
  Lines         41902    41902              
  Branches       5817     5817              
============================================
+ Hits          26756    28619    +1863     
+ Misses        12455    10513    -1942     
- Partials       2691     2770      +79

see 94 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

imbajin · 2023-09-09T11:13:04Z

.gitignore

@@ -83,3 +83,5 @@ hs_err_pid*
 .mtj.tmp/
 # blueJ files
 *.ctxt
+
+*swagger-ui*


seems no need to add here due to the master code has fixed it (#2277)

and also need handle the CI error (if we used refined github plugin)

imbajin · 2023-09-09T11:13:09Z

Dockerfile


 EXPOSE 8080
 VOLUME /hugegraph

 ENTRYPOINT ["/usr/bin/dumb-init", "--"]
-CMD ["./bin/start-hugegraph.sh", "-d false -j $JAVA_OPTS -g zgc"]
+CMD ["./bin/docker-entrypoint.sh"]


maybe we shouldn't put the docker files in general path?

https://github.com/JanusGraph/janusgraph/tree/master/janusgraph-dist/docker
maybe I can try to move the docker related file in a split folder in hugegraph-dist, like janusgraph?

imbajin · 2023-09-09T11:14:13Z

hugegraph-dist/src/assembly/static/bin/docker-entrypoint.sh

+
+./bin/init-store.sh
+
+./bin/start-hugegraph.sh -d false -j "$JAVA_OPTS" -g zgc


why the params here? could we get the value of $JAVA_OPTS?

JAVA_OPTS is an env var which is set in dockerfile. And this shell is only used in dockerfile, hence I think this way is available.

imbajin · 2023-09-09T11:17:20Z

hugegraph-dist/docker/example/docker-compose-cassandra.yml

+    depends_on:
+      - cassandra
+    healthcheck:
+      test: ["CMD", "bin/gremlin-console.sh", "--" ,"-e", "scripts/remote-connect.groovy"]


we still need -e?

https://tinkerpop.apache.org/docs/3.6.2-SNAPSHOT/reference/#execution-mode
-e is for the gremlin console to use the execution mode to execute scripts.

imbajin · 2023-09-09T11:17:34Z

hugegraph-dist/docker/example/docker-compose-cassandra.yml

+      retries: 3
+
+  cassandra:
+    image: cassandra:3.11


try C* 4 for it

tested, no problem

hugegraph-dist/docker/example/docker-compose-cassandra.yml

Co-authored-by: imbajin <[email protected]>

hugegraph-dist/docker/example/docker-compose-cassandra.yml

javeme · 2023-10-12T16:18:44Z

hugegraph-dist/docker/scripts/try-storage.groovy

+RegisterUtil.registerPalo()
+RegisterUtil.registerPostgresql()
+
+graph = HugeFactory.open('./conf/graphs/hugegraph.properties')


can we rename this file to detect-storage.groovy?

Sure. And "detect" seems better than "try". I have applied the change.

javeme · 2023-10-15T05:20:26Z

hugegraph-dist/src/assembly/static/bin/wait-storage.sh

+TOP="$(cd "$BIN"/../ && pwd)"
+GRAPH_PROP="$TOP/conf/graphs/hugegraph.properties"
+HUGE_STORAGE_TIMEOUT_S=120
+TRY_STORAGE="$TOP/scripts/detect-storage.groovy"


also update TRY_STORAGE

javeme · 2023-10-15T05:21:15Z

hugegraph-dist/src/assembly/static/bin/wait-storage.sh

+BIN=$(abs_path)
+TOP="$(cd "$BIN"/../ && pwd)"
+GRAPH_PROP="$TOP/conf/graphs/hugegraph.properties"
+HUGE_STORAGE_TIMEOUT_S=120


prefer WAIT_STORAGE_TIMEOUT_S

javeme · 2023-10-15T05:21:53Z

hugegraph-dist/src/assembly/static/bin/wait-storage.sh

+
+BIN=$(abs_path)
+TOP="$(cd "$BIN"/../ && pwd)"
+GRAPH_PROP="$TOP/conf/graphs/hugegraph.properties"


prefer GRAPH_CONF

The changes have been applied on the var name

javeme

LGTM

hugegraph-dist/src/assembly/static/bin/wait-storage.sh

1. change the dockerfile, adding the shell to wait for storage backend and use a docker-entrypoint.sh to manage the starting process. 2. delete a deprecated class in gremlin-console.sh (reference: [doc of ScriptExecutor](https://tinkerpop.apache.org/javadocs/3.2.3/full/org/apache/tinkerpop/gremlin/groovy/jsr223/ScriptExecutor.html)) 3. add a healthy check in docker-compose 4. add an example folder where we can put all the template docker-compose.yml here 5. add `*swagger-ui*` in gitignore, which appears after you compile the source code locally. --------- Co-authored-by: imbajin <[email protected]>

feat: support Cassandra(Docker) as backend (compose with HugeGraph)(a…

9e5f1ee

…pache#840)

imbajin reviewed Sep 9, 2023

View reviewed changes

aroundabout and others added 2 commits September 9, 2023 19:43

Update hugegraph-dist/docker/example/docker-compose-cassandra.yml

3be461b

Co-authored-by: imbajin <[email protected]>

Update hugegraph-dist/docker/example/docker-compose-cassandra.yml

137d004

imbajin added ci-cd Build or deploy cassandra Cassandra backend labels Sep 11, 2023

imbajin mentioned this pull request Sep 15, 2023

[Feature] Support build docker image from Dockerfile (Task Summary) #840

Open

17 tasks

aroundabout commented Sep 19, 2023

View reviewed changes

hugegraph-dist/docker/example/docker-compose-cassandra.yml Outdated Show resolved Hide resolved

Update hugegraph-dist/docker/example/docker-compose-cassandra.yml

2fdd40f

aroundabout commented Sep 19, 2023

View reviewed changes

hugegraph-dist/docker/example/docker-compose-cassandra.yml Outdated Show resolved Hide resolved

Update hugegraph-dist/docker/example/docker-compose-cassandra.yml

ffa1779

imbajin requested a review from javeme October 12, 2023 15:50

javeme reviewed Oct 12, 2023

View reviewed changes

aroundabout and others added 2 commits October 13, 2023 14:13

Merge branch 'apache:master' into docker_cassandra

a01d0e4

change detect script's filename

d924508

aroundabout requested review from imbajin and javeme October 13, 2023 11:03

javeme reviewed Oct 15, 2023

View reviewed changes

change var name in wait storage script

5e4c8e6

aroundabout requested a review from javeme October 15, 2023 11:45

aroundabout and others added 2 commits October 22, 2023 18:17

Merge branch 'apache:master' into docker_cassandra

94a6351

fix: some reference url and 3rd-party file license

da46a0f

javeme approved these changes Oct 22, 2023

View reviewed changes

imbajin changed the title ~~feat: support Cassandra(Docker) as backend (compose with HugeGraph)(#…~~ feat: support Cassandra in docker-compose with server Oct 23, 2023

imbajin requested changes Oct 23, 2023

View reviewed changes

hugegraph-dist/src/assembly/static/bin/wait-storage.sh Show resolved Hide resolved

imbajin approved these changes Oct 23, 2023

View reviewed changes

imbajin merged commit d4b95ca into apache:master Oct 23, 2023
18 of 21 checks passed

imbajin changed the title ~~feat: support Cassandra in docker-compose with server~~ feat: support Cassandra with docker-compose in server Oct 23, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: support Cassandra with docker-compose in server #2307

feat: support Cassandra with docker-compose in server #2307

aroundabout commented Sep 9, 2023

codecov bot commented Sep 9, 2023 •

edited

Loading

imbajin Sep 9, 2023

imbajin Sep 9, 2023 •

edited

Loading

imbajin Sep 9, 2023

aroundabout Sep 9, 2023

imbajin Sep 9, 2023

aroundabout Sep 9, 2023 •

edited

Loading

imbajin Sep 9, 2023

aroundabout Sep 9, 2023

imbajin Sep 9, 2023

aroundabout Sep 19, 2023

javeme Oct 12, 2023

aroundabout Oct 13, 2023

javeme Oct 15, 2023

javeme Oct 15, 2023

javeme Oct 15, 2023

aroundabout Oct 15, 2023

javeme left a comment


		./bin/init-store.sh

		./bin/start-hugegraph.sh -d false -j "$JAVA_OPTS" -g zgc

feat: support Cassandra with docker-compose in server #2307

feat: support Cassandra with docker-compose in server #2307

Conversation

aroundabout commented Sep 9, 2023

Purpose of the PR

Main Changes

Verifying these changes

Does this PR potentially affect the following parts?

Documentation Status

codecov bot commented Sep 9, 2023 • edited Loading

Codecov Report

Choose a reason for hiding this comment

imbajin Sep 9, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aroundabout Sep 9, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

javeme left a comment

Choose a reason for hiding this comment

codecov bot commented Sep 9, 2023 •

edited

Loading

imbajin Sep 9, 2023 •

edited

Loading

aroundabout Sep 9, 2023 •

edited

Loading