Add juicefsruntime dataload function #1539

ldd91 · 2022-03-18T07:01:00Z

Ⅰ. Describe what this PR does

Add juicefs dataload function

Ⅱ. Does this pull request fix one issue?

Ⅲ. List the added test cases (unit test/integration test) if any, please explain if no tests are needed.

Ⅳ. Describe how to verify it

Ⅴ. Special notes for reviews

charts/fluid-dataloader/juicefs/values.yaml

* Add helm lint check in Github workflow Signed-off-by: TrafalgarZZZ <[email protected]> * Fix jindofs helm lint Signed-off-by: TrafalgarZZZ <[email protected]> Signed-off-by: zwwhdls <[email protected]>

Signed-off-by: zwwhdls <[email protected]>

* Add pd documents, To #38946668 Signed-off-by: cheyang <[email protected]> * Add pd documents, To #38946668 Signed-off-by: cheyang <[email protected]> * Add pd documents, To #38946668 Signed-off-by: cheyang <[email protected]> Signed-off-by: zwwhdls <[email protected]>

Signed-off-by: zwwhdls <[email protected]>

Signed-off-by: cheyang <[email protected]> Signed-off-by: zwwhdls <[email protected]>

) * Add remount check for hostpath mount during sync Signed-off-by: nizifan <[email protected]> * Fix CI issue & rename function Signed-off-by: nizifan <[email protected]> * Add ut & fix potential leak of ufstoupdate.toadd Signed-off-by: nizifan <[email protected]> * improve test coverage & address comments - add new line Signed-off-by: nizifan <[email protected]> * Fix ci issue Signed-off-by: nizifan <[email protected]> * Only perform remount operation when mounttime ealier than master starting time Signed-off-by: nizifan <[email protected]> * Fix ci issue Signed-off-by: nizifan <[email protected]> * add ut Signed-off-by: nizifan <[email protected]> * rename function Signed-off-by: nizifan <[email protected]> Co-authored-by: nizifan <[email protected]> Signed-off-by: zwwhdls <[email protected]>

…tive#1385) * JindoFS fuse recover Signed-off-by: frankleaf <[email protected]> * refine Signed-off-by: frankleaf <[email protected]> * refine Signed-off-by: frankleaf <[email protected]> * refine Signed-off-by: frankleaf <[email protected]> * refine Signed-off-by: frankleaf <[email protected]> * refine Signed-off-by: frankleaf <[email protected]> Signed-off-by: zwwhdls <[email protected]>

…tive#1388) Signed-off-by: mahao <[email protected]> Signed-off-by: zwwhdls <[email protected]>

* Fix format, To #38946668 Signed-off-by: cheyang <[email protected]> * Add volume, To #38946668 Signed-off-by: cheyang <[email protected]> Signed-off-by: zwwhdls <[email protected]>

* Refactor Fluid CSI Plugin Signed-off-by: TrafalgarZZZ <[email protected]> * Go fmt Signed-off-by: TrafalgarZZZ <[email protected]> * Option to enable fuse mount point recovery Signed-off-by: TrafalgarZZZ <[email protected]> * Refactor Fluid CSI Plugin - Move package 'mountinfo' to utils - Move package 'fuse' to plugin - Add registry functions Signed-off-by: TrafalgarZZZ <[email protected]> * Minor fix for fuse recover period Signed-off-by: TrafalgarZZZ <[email protected]> * Remove duplicate license Signed-off-by: TrafalgarZZZ <[email protected]> * Declare registraion funcs to enable/disable feature gates Signed-off-by: TrafalgarZZZ <[email protected]> * Add tests Signed-off-by: TrafalgarZZZ <[email protected]> * Add tests Signed-off-by: TrafalgarZZZ <[email protected]> * Fix copyright Signed-off-by: TrafalgarZZZ <[email protected]> Signed-off-by: zwwhdls <[email protected]>

…loudnative#1397) Signed-off-by: cheyang <[email protected]> Signed-off-by: zwwhdls <[email protected]>

Signed-off-by: ssz1997 <[email protected]> Signed-off-by: zwwhdls <[email protected]>

* Support serverless, To #38946668 Signed-off-by: cheyang <[email protected]> Fix ut, To #38946668 Signed-off-by: cheyang <[email protected]> * Signed-off-by: cheyang <[email protected]> Add mountPropagationHostToContainer, To #37688693 Add testcase, To #37688693 Signed-off-by: cheyang <[email protected]> Signed-off-by: zwwhdls <[email protected]>

* Fix docker build for cento8, To #37688693 Signed-off-by: cheyang <[email protected]> * Fix docker build for cento8, To #37688693 Signed-off-by: cheyang <[email protected]> Signed-off-by: zwwhdls <[email protected]>

* Build docker image, To #37688693 Signed-off-by: cheyang <[email protected]> * Build docker image, To #37688693 Signed-off-by: cheyang <[email protected]> Signed-off-by: zwwhdls <[email protected]>

* Support Alluxio, To #37688693 Signed-off-by: cheyang <[email protected]> * Support Alluxio, To #37688693 Signed-off-by: cheyang <[email protected]> * Support Alluxio, To #37688693 Signed-off-by: cheyang <[email protected]> Signed-off-by: zwwhdls <[email protected]>

* Support namespace is empty, To #37688693 Signed-off-by: cheyang <[email protected]> * Support namespace is empty, To #37688693 Signed-off-by: cheyang <[email protected]> * Support namespace is empty, To #37688693 Signed-off-by: cheyang <[email protected]> * Support namespace is empty, To #37688693 Signed-off-by: cheyang <[email protected]> * Support namespace is empty, To #37688693 Signed-off-by: cheyang <[email protected]> * Support namespace is empty, To #37688693 Signed-off-by: cheyang <[email protected]> * Support namespace is empty, To #37688693 Signed-off-by: cheyang <[email protected]> * Support namespace is empty, To #37688693 Signed-off-by: cheyang <[email protected]> * Support namespace is empty, To #37688693 Signed-off-by: cheyang <[email protected]> * Support namespace is empty, To #37688693 Signed-off-by: cheyang <[email protected]> * Support namespace is empty, To #37688693 Signed-off-by: cheyang <[email protected]> * Support namespace is empty, To #37688693 Signed-off-by: cheyang <[email protected]> Signed-off-by: zwwhdls <[email protected]>

Signed-off-by: cheyang <[email protected]> Signed-off-by: zwwhdls <[email protected]>

…-cloudnative#1417) * Fix redundant type from array, slice, or map composite literal, To #37688693 Signed-off-by: cheyang <[email protected]> * Fix redundant type from array, slice, or map composite literal, To #37688693 Signed-off-by: cheyang <[email protected]> Signed-off-by: zwwhdls <[email protected]>

Signed-off-by: TrafalgarZZZ <[email protected]> Signed-off-by: zwwhdls <[email protected]>

Signed-off-by: cheyang <[email protected]> Signed-off-by: zwwhdls <[email protected]>

* change goosefs fuse default parameters; fix redundant type from array, slice, or map composite literal; fix when ha mode journal type Signed-off-by: xieydd <[email protected]> * fix comments Signed-off-by: xieydd <[email protected]> Signed-off-by: zwwhdls <[email protected]>

Signed-off-by: cheyang <[email protected]> Signed-off-by: zwwhdls <[email protected]>

Signed-off-by: zwwhdls <[email protected]>

…udnative#1432) Signed-off-by: cheyang <[email protected]> Signed-off-by: zwwhdls <[email protected]>

…ative#1428) * fix clean cache linux version error Signed-off-by: xieydd <[email protected]> * update goosefs default image to 1.2.0 Signed-off-by: xieydd <[email protected]> * fix comment Signed-off-by: xieydd <[email protected]> * fix timeout option for ubuntu and alpine base image Signed-off-by: xieydd <[email protected]> * fix timeout option for ubuntu and alpine base image Signed-off-by: xieydd <[email protected]> Signed-off-by: zwwhdls <[email protected]>

) Signed-off-by: cheyang <[email protected]> Signed-off-by: zwwhdls <[email protected]>

* Fix update dataload status error Signed-off-by: Ruofeng Lei <[email protected]> * Fix some incorrect usage of `ctx.Name` Signed-off-by: Ruofeng Lei <[email protected]> Signed-off-by: zwwhdls <[email protected]>

* update committer and maintainer file Signed-off-by: RongGu <[email protected]> * update Governance file to add committer list file Signed-off-by: RongGu <[email protected]> * fix typo Signed-off-by: RongGu <[email protected]> Signed-off-by: zwwhdls <[email protected]>

Signed-off-by: zwwhdls <[email protected]>

* Add samples, To #40056722 Signed-off-by: cheyang <[email protected]> * Add samples, To #40056722 Signed-off-by: cheyang <[email protected]> * Update dockerfile, To #40056722 Signed-off-by: cheyang <[email protected]> * Update dockerfile, To #40056722 Signed-off-by: cheyang <[email protected]> * Build docker image for env in csi plugin, To #37688693 Signed-off-by: cheyang <[email protected]> Signed-off-by: zwwhdls <[email protected]>

Signed-off-by: zwwhdls <[email protected]>

codecov · 2022-03-18T08:01:32Z

Codecov Report

Merging #1539 (c254c0a) into master (3d632b9) will increase coverage by 0.08%.
The diff coverage is 76.29%.

@@            Coverage Diff             @@
##           master    #1539      +/-   ##
==========================================
+ Coverage   70.68%   70.76%   +0.08%     
==========================================
  Files         237      238       +1     
  Lines       12998    13127     +129     
==========================================
+ Hits         9187     9289     +102     
- Misses       2830     2843      +13     
- Partials      981      995      +14

Impacted Files	Coverage Δ
pkg/ddc/juicefs/health_check.go	`77.23% <ø> (+1.23%)`	⬆️
pkg/ddc/juicefs/utils.go	`84.12% <ø> (+2.38%)`	⬆️
pkg/ddc/juicefs/data_load.go	`76.06% <76.06%> (-23.94%)`	⬇️
pkg/ddc/juicefs/cacheinfo_parser.go	`77.77% <77.77%> (ø)`
pkg/ddc/jindo/dataset.go	`69.44% <0.00%> (ø)`
pkg/ddc/alluxio/worker.go	`73.77% <0.00%> (ø)`
pkg/ddc/jindo/replicas.go	`65.00% <0.00%> (ø)`
pkg/ddc/jindo/load_data.go	`64.76% <0.00%> (ø)`
pkg/ddc/alluxio/load_data.go	`82.95% <0.00%> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3d632b9...c254c0a. Read the comment docs.

cheyang · 2022-03-18T09:16:57Z

pkg/ddc/juicefs/cacheinfo_parser.go

+)
+
+// G
+func GetCacheInfoFromConfigmap(client client.Client, name string, namespace string) (cacheinfo map[string]string, err error) {


Do you want to write comments for function GetCacheInfoFromConfigmap?

cheyang · 2022-03-18T09:20:59Z

charts/fluid/fluid/templates/role/dataset/rbac.yaml

@@ -49,9 +49,12 @@ rules:
  - apiGroups:
      - ""
    resources:
-      - events


Why is events missing?

events bas been involved in line22.

cheyang · 2022-03-18T09:23:41Z

charts/fluid/fluid/templates/role/dataset/rbac.yaml

@@ -61,6 +64,16 @@ rules:
      - statefulsets/status
    verbs:
      - '*'
+  - apiGroups:


Is necessary to enlarge the RBAC authorization of dataset controller?

update: create rbac in juicefs controller.

Signed-off-by: zwwhdls <[email protected]>

cheyang · 2022-03-19T04:31:07Z

charts/fluid-dataloader/juicefs/templates/configmap.yaml

+
+          for((j=0;j<${#paths[@]};j++)) do
+            echo -e "juicefs warmup on $pod ${paths[j]} starts"
+            /usr/local/bin/kubectl -n $ns exec -it $pod -- $COMMAND


Will you add timeout to avoid the job hang?

cheyang · 2022-03-19T04:32:31Z

Please fix no check issue. thanks.

pkg/ddc/juicefs/data_load.go:208:25: Error return value of `fileUtils.UnMount` is not checked (errcheck)
	defer fileUtils.UnMount(mountPath)

Signed-off-by: zwwhdls <[email protected]>

cheyang · 2022-03-19T10:34:33Z

charts/fluid-dataloader/juicefs/templates/configmap.yaml

-            /usr/local/bin/kubectl -n $ns exec -it $pod -- juicefs warmup $MOUNTPATH${paths[j]}
-            /usr/local/bin/kubectl -n $ns exec -it $pod -- umount $MOUNTPATH
+            /usr/local/bin/kubectl -n $ns exec -it $pod --request-timeout=10s -- $COMMAND
+            /usr/local/bin/kubectl -n $ns exec -it $pod --request-timeout=1m -- juicefs warmup $MOUNTPATH${paths[j]}


I think the timeout of warmup behavior should be configurable. By the way, is 1Minute enough for juicefs warm scenario?

Signed-off-by: zwwhdls <[email protected]>

cheyang · 2022-03-21T11:53:52Z

charts/juicefs/templates/worker/statefuleset.yaml

+        - name: script
+          configMap:
+            name: {{ template "juicefs.fullname" . }}-script
+            defaultMode: 0777


Why not setting 0755?

0755 is ok, done

Signed-off-by: zwwhdls <[email protected]>

cheyang reviewed Mar 18, 2022

View reviewed changes

charts/fluid-dataloader/juicefs/values.yaml Outdated Show resolved Hide resolved

TrafalgarZZZ and others added 29 commits March 18, 2022 15:48

Add helm lint check in Github workflow (fluid-cloudnative#1384)

1964301

* Add helm lint check in Github workflow Signed-off-by: TrafalgarZZZ <[email protected]> * Fix jindofs helm lint Signed-off-by: TrafalgarZZZ <[email protected]> Signed-off-by: zwwhdls <[email protected]>

add juicefsruntime dataload

323f39e

Signed-off-by: zwwhdls <[email protected]>

fix test

c57c026

Signed-off-by: zwwhdls <[email protected]>

not check mount ready in stageVolume (fluid-cloudnative#1390)

2e58404

Signed-off-by: zwwhdls <[email protected]>

Build docker image for csi plugin, To #38946668 (fluid-cloudnative#1391)

e69c034

Signed-off-by: cheyang <[email protected]> Signed-off-by: zwwhdls <[email protected]>

unify the file format under cmd, alluxio jindo dataset (fluid-cloudna…

d10384c

…tive#1388) Signed-off-by: mahao <[email protected]> Signed-off-by: zwwhdls <[email protected]>

Docker image build for alluxio master restart (fluid-cloudnative#1393)

e02a0e4

* Fix format, To #38946668 Signed-off-by: cheyang <[email protected]> * Add volume, To #38946668 Signed-off-by: cheyang <[email protected]> Signed-off-by: zwwhdls <[email protected]>

Build docker images for refactoring csi plugin, To #38946668 (fluid-c…

f352d7e

…loudnative#1397) Signed-off-by: cheyang <[email protected]> Signed-off-by: zwwhdls <[email protected]>

fix diagnose (fluid-cloudnative#1402)

248c461

Signed-off-by: ssz1997 <[email protected]> Signed-off-by: zwwhdls <[email protected]>

Fix building docker image for init user (fluid-cloudnative#1408)

fb4be87

* Fix docker build for cento8, To #37688693 Signed-off-by: cheyang <[email protected]> * Fix docker build for cento8, To #37688693 Signed-off-by: cheyang <[email protected]> Signed-off-by: zwwhdls <[email protected]>

Build docker image (fluid-cloudnative#1409)

4aae2c4

* Build docker image, To #37688693 Signed-off-by: cheyang <[email protected]> * Build docker image, To #37688693 Signed-off-by: cheyang <[email protected]> Signed-off-by: zwwhdls <[email protected]>

Support namespace is empty, To #37688693 (fluid-cloudnative#1415)

3c94ac7

Signed-off-by: cheyang <[email protected]> Signed-off-by: zwwhdls <[email protected]>

Fix dataload cannot be cleaned up bug (fluid-cloudnative#1421)

6f8d5e3

Signed-off-by: TrafalgarZZZ <[email protected]> Signed-off-by: zwwhdls <[email protected]>

Add testcase, To #37688693 (fluid-cloudnative#1423)

49024f8

Signed-off-by: cheyang <[email protected]> Signed-off-by: zwwhdls <[email protected]>

Build docker image for goose, To #37688693 (fluid-cloudnative#1425)

3810f2c

Signed-off-by: cheyang <[email protected]> Signed-off-by: zwwhdls <[email protected]>

support juicefs in serverless (fluid-cloudnative#1427)

fffba9f

Signed-off-by: zwwhdls <[email protected]>

Build docker image for juicefs on serverless, To #37688693 (fluid-clo…

bcd6bc9

…udnative#1432) Signed-off-by: cheyang <[email protected]> Signed-off-by: zwwhdls <[email protected]>

Build docker image for Goosefs bug, To #37688693 (fluid-cloudnative#1434

b74f652

) Signed-off-by: cheyang <[email protected]> Signed-off-by: zwwhdls <[email protected]>

Fix update dataload status error (fluid-cloudnative#1433)

747ef31

* Fix update dataload status error Signed-off-by: Ruofeng Lei <[email protected]> * Fix some incorrect usage of `ctx.Name` Signed-off-by: Ruofeng Lei <[email protected]> Signed-off-by: zwwhdls <[email protected]>

RongGu and others added 9 commits March 18, 2022 15:48

fix pod/exec rbac in application controller (fluid-cloudnative#1518)

a5ceee1

Signed-off-by: zwwhdls <[email protected]>

fix application controller image (fluid-cloudnative#1519)

0b471a0

Signed-off-by: zwwhdls <[email protected]>

warmup in each fuse pod

b2d9f14

Signed-off-by: zwwhdls <[email protected]>

fix test

67667cb

Signed-off-by: zwwhdls <[email protected]>

update default fuse image

3f758b3

Signed-off-by: zwwhdls <[email protected]>

add label in job

fde4b07

Signed-off-by: zwwhdls <[email protected]>

fix DCO

ba97ed9

Signed-off-by: zwwhdls <[email protected]>

zwwhdls force-pushed the add_jfs_dataload branch from 6b05bc1 to ba97ed9 Compare March 18, 2022 07:50

zwwhdls added 2 commits March 18, 2022 15:57

fix conflict

17500b7

Signed-off-by: zwwhdls <[email protected]>

update typo

51a3e6d

Signed-off-by: zwwhdls <[email protected]>

cheyang reviewed Mar 18, 2022

View reviewed changes

zwwhdls added 2 commits March 18, 2022 22:52

update dataload rbac to juicefs controller

8d61be8

Signed-off-by: zwwhdls <[email protected]>

update warmup to worker

e22f938

Signed-off-by: zwwhdls <[email protected]>

cheyang requested a review from yangyuliufeng March 19, 2022 04:26

cheyang reviewed Mar 19, 2022

View reviewed changes

add timeout & deal with error

d40a378

Signed-off-by: zwwhdls <[email protected]>

cheyang requested a review from TrafalgarZZZ March 19, 2022 10:30

cheyang reviewed Mar 19, 2022

View reviewed changes

set timeout in dataload options

0914e31

Signed-off-by: zwwhdls <[email protected]>

cheyang reviewed Mar 21, 2022

View reviewed changes

update configmap volume mode 0755 & add unit test

c254c0a

Signed-off-by: zwwhdls <[email protected]>

cheyang approved these changes Mar 22, 2022

View reviewed changes

cheyang merged commit da5f89d into fluid-cloudnative:master Mar 22, 2022

RongGu mentioned this pull request Feb 14, 2023

[Community] Nominating Dongdong Lv (GithubID: ldd91) for Fluid committer fluid-cloudnative/community#49

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add juicefsruntime dataload function #1539

Add juicefsruntime dataload function #1539

ldd91 commented Mar 18, 2022

codecov bot commented Mar 18, 2022 •

edited

Loading

cheyang Mar 18, 2022 •

edited

Loading

zwwhdls Mar 19, 2022

cheyang Mar 18, 2022

zwwhdls Mar 19, 2022

cheyang Mar 18, 2022

zwwhdls Mar 22, 2022 •

edited

Loading

cheyang Mar 19, 2022

zwwhdls Mar 22, 2022

cheyang commented Mar 19, 2022

cheyang Mar 19, 2022

zwwhdls Mar 22, 2022

cheyang Mar 21, 2022

zwwhdls Mar 22, 2022

Add juicefsruntime dataload function #1539

Add juicefsruntime dataload function #1539

Conversation

ldd91 commented Mar 18, 2022

Ⅰ. Describe what this PR does

Ⅱ. Does this pull request fix one issue?

Ⅲ. List the added test cases (unit test/integration test) if any, please explain if no tests are needed.

Ⅳ. Describe how to verify it

Ⅴ. Special notes for reviews

codecov bot commented Mar 18, 2022 • edited Loading

Codecov Report

cheyang Mar 18, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zwwhdls Mar 22, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cheyang commented Mar 19, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Mar 18, 2022 •

edited

Loading

cheyang Mar 18, 2022 •

edited

Loading

zwwhdls Mar 22, 2022 •

edited

Loading