Initial implementation to add external health monitor controller and agent #3

NickrenREN · 2020-03-06T03:43:21Z

What type of PR is this?
/kind feature

What this PR does / why we need it:
This PR adds an external health monitor controller and an external health monitor agent.

Which issue(s) this PR fixes:
Fixes #1
Fixes #2

Does this PR introduce a user-facing change?:

Adds an external health monitor controller and an external health monitor agent.

NickrenREN · 2020-03-06T03:44:54Z

Ignore this PR first please.

It is far away from completion

Just a WIP PR to show progress.

xing-yang · 2020-03-30T18:10:47Z

/test continuous-integration/travis-ci/pr

xing-yang · 2020-04-23T13:55:24Z

cmd/csi-external-health-monitor-controller/main.go

+		os.Exit(1)
+	}
+
+	supportGetVolume, err := supportGetVolume(ctx, csiConn)


We should check LIST_VOLUMES support first. If both LIST_VOLUMES and VOLUME_CONDITION are supported, we should use that to retrieve volume health. This will have much better performance than making individual calls to GetVolume.

If LIST_VOLUMES is not support, then we check the combination of GET_VOLUME and VOLUME_CONDITION.

xing-yang · 2020-04-23T13:59:25Z

cmd/csi-external-health-monitor-controller/main.go

+}
+
+// TODO: move this to csi-lib-utils
+// TODO: do we need to consider ControllerServiceCapability_RPC_LIST_VOLUMES_VOLUME_HEALTH ?


oh, I added a comment above that LIST_VOLUMES should be checked and it should be the preferred way to retrieve information in batch.

Also you may be using an earlier version of the spec? LIST_VOLUMES_VOLUME_HEALTH is removed. Now we just have one VOLUME_CONDITION which is used for both LIST_VOLUMES and GET_VOLUME.

xing-yang · 2020-05-03T02:51:13Z

cmd/csi-external-health-monitor-agent/main.go

+	showVersion     = flag.Bool("version", false, "Show version.")
+	timeout         = flag.Duration("timeout", 15*time.Second, "Timeout for waiting for attaching or detaching the volume.")
+	workerThreads   = flag.Uint("worker-threads", 10, "Number of pv monitor worker threads")
+	kubeletRootPath = flag.String("kubelet-root-path", "/var/lib/kubelet", "The root path pf kubelet.")


xing-yang · 2020-05-03T02:52:04Z

cmd/csi-external-health-monitor-agent/main.go

+	monitorInterval = flag.Duration("monitor-interval", 1*time.Minute, "Interval for controller to check volumes health condition.")
+
+	kubeconfig      = flag.String("kubeconfig", "", "Absolute path to the kubeconfig file. Required only when running out of cluster.")
+	resync          = flag.Duration("resync", 10*time.Minute, "Resync interval of the controller.")


Can you also add retryIntervalStart and retryIntervalMax?

https://github.com/kubernetes-csi/external-attacher/blob/master/cmd/csi-attacher/main.go#L61

xing-yang · 2020-05-03T03:00:28Z

cmd/csi-external-health-monitor-agent/main.go

+	metricsManager.SetDriverName(storageDriver)
+	metricsManager.StartMetricsEndpoint(*metricsAddress, *metricsPath)
+
+	supportGetNodeVolumeHealth, err := supportGetNodeVolumeHealth(ctx, csiConn)


It's renamed to VOLUME_CONDITION in CSI spec, so change to supportNodeGetVolumeCondition

xing-yang · 2020-05-03T03:00:47Z

cmd/csi-external-health-monitor-agent/main.go

+		os.Exit(1)
+	}
+	if !supportGetNodeVolumeHealth {
+		klog.V(2).Infof("CSI driver does not support VolumeHealth Node Capability, exiting")


VolumeCondition

xing-yang · 2020-05-03T03:04:06Z

cmd/csi-external-health-monitor-agent/main.go

+			continue
+		}
+		t := rpc.GetType()
+		if t == csi.NodeServiceCapability_RPC_GET_VOLUME_STATS {


NodeServiceCapability_RPC_VOLUME_CONDITION

xing-yang · 2020-05-03T03:06:12Z

cmd/csi-external-health-monitor-controller/main.go

+
+	metricsAddress = flag.String("metrics-address", "", "The TCP network address where the prometheus metrics endpoint will listen (example: `:8080`). The default is empty string, which means metrics endpoint is disabled.")
+	metricsPath    = flag.String("metrics-path", "/metrics", "The HTTP path where prometheus metrics will be exposed. Default is `/metrics`.")
+)


Can you also add retryIntervalStart and retryIntervalMax?

xing-yang · 2020-05-03T03:08:10Z

cmd/csi-external-health-monitor-controller/main.go

+}
+
+// TODO: move this to csi-lib-utils
+func supportGetVolumeCondition(ctx context.Context, csiConn *grpc.ClientConn) (supportGetVolumeHealth bool, err error) {


supportControllerGetVolumeCondition

Add one for ListVolumes

xing-yang · 2020-05-03T14:51:48Z

cmd/csi-external-health-monitor-controller/main.go

+		os.Exit(1)
+	}
+
+	supportGetVolume, err := supportGetVolume(ctx, csiConn)


supportControllerGetVolume

xing-yang · 2020-05-03T14:52:39Z

cmd/csi-external-health-monitor-controller/main.go

+}
+
+// TODO: move this to csi-lib-utils
+func supportGetVolume(ctx context.Context, csiConn *grpc.ClientConn) (supportGetVolume bool, err error) {


supportControllerGetVolume

xing-yang · 2020-05-03T15:00:29Z

deploy/kubernetes/external-health-monitor-agent/rbac.yaml

+  - apiGroups: [""]
+    resources: ["pods"]
+    verbs: ["get", "list", "watch"]
+#Secret permission is optional.


I don't think we need secrets yet for get and list.

xing-yang · 2020-05-03T15:04:03Z

deploy/kubernetes/external-health-monitor-controller/deployment.yaml

+              mountPath: /var/lib/csi/sockets/pluginproxy/
+
+        - name: mock-driver
+          image: quay.io/k8scsi/mock-driver:canary


Are you going to submit a PR in csi-test to implement volume health in the mock-driver? Otherwise we can't test this?

I will test this on my local env first, and can send a PR in csi-test if needed

xing-yang · 2020-05-03T15:04:57Z

deploy/kubernetes/external-health-monitor-controller/rbac.yaml

+  - apiGroups: [""]
+    resources: ["pods"]
+    verbs: ["get", "list", "watch"]
+#Secret permission is optional.


I don't think we need Secrets for this.

xing-yang · 2020-05-03T15:06:38Z

go.mod

+go 1.12
+
+require (
+	github.com/container-storage-interface/spec v1.2.0-rc1.0.20200415022618-e129a75169c1


Does this have the latest changes we need for VolumeCondition? Should we use master for now?

xing-yang · 2020-05-03T15:07:04Z

go.mod

+	golang.org/x/tools v0.0.0-20190524140312-2c0ae7006135 // indirect
+	google.golang.org/grpc v1.28.0
+	honnef.co/go/tools v0.0.0-20190523083050-ea95bdfd59fc // indirect
+	k8s.io/api v0.17.0


Please update everything to v0.18.0

xing-yang · 2020-05-03T15:10:10Z

pkg/agent/pv-monitor-agent/event_handler.go

+		// TODO: support inline csi volumes
+		for _, volume := range pod.Spec.Volumes {
+			if volume.PersistentVolumeClaim != nil {
+				pvc, err := agent.pvcLister.PersistentVolumeClaims(pod.Namespace).Get(volume.PersistentVolumeClaim.ClaimName)


I think with v0.18.0, you'll need to have a context and options in all these API calls as well.

xing-yang · 2020-05-03T15:23:15Z

pkg/controller/pv-monitor-controller/event_handler.go

@@ -0,0 +1,65 @@
+/*


Do we need a sub-folder "pv-monitor-controller" under "controller"? Should the files just be directly under "controller"?

just in case we want to add new controllers

We'll change the directory structure when there's a need for new controllers.
For now, please remove extra layer of directory,

xing-yang · 2020-05-03T15:23:50Z

pkg/agent/pv-monitor-agent/event_handler.go

@@ -0,0 +1,74 @@
+/*


Do we need a sub-folder "pv-monitor-agent" under "agent"? Should the files just be directly under "agent"?

xing-yang · 2020-06-14T14:39:34Z

cmd/csi-external-health-monitor-controller/main.go

+	volumeListAndAddInterval = flag.Duration("volume-list-add-interval", 5*time.Minute, "Time interval for listing volumes and add them to queue")
+	nodeListAndAddInterval   = flag.Duration("node-list-add-interval", 5*time.Minute, "Time interval for listing nodess and add them to queue")
+	workerThreads            = flag.Uint("worker-threads", 10, "Number of pv monitor worker threads")
+	enableNodeWatcher        = flag.Bool("enable-node-watcher", false, "whether we want to enable node watcher, node watcher will only have effects on local PVs now, it may be useful for block storages too, will take this into account later.")


it may be useful for block storages too, will take this into account later

"block storage"? Do you mean "remote storage"?
If this only works for local pv now, how do you verify and stop user from enabling this feature if it is not local pv?

xing-yang · 2020-06-14T15:16:16Z

cmd/csi-external-health-monitor-controller/main.go

+	showVersion              = flag.Bool("version", false, "Show version.")
+	timeout                  = flag.Duration("timeout", 15*time.Second, "Timeout for waiting for attaching or detaching the volume.")
+	listVolumesInterval      = flag.Duration("list-volumes-interval", 5*time.Minute, "Time interval for calling ListVolumes RPC to check volumes' health condition")
+	volumeListAndAddInterval = flag.Duration("volume-list-add-interval", 5*time.Minute, "Time interval for listing volumes and add them to queue")


This option is used when ListVolumes is not supported? Should it be called GetVolumeAndAddInterval instead?

xing-yang · 2020-06-14T15:22:12Z

pkg/agent/pv_monitor_agent.go

 	<-stopCh
 }

-func (agent *PVMonitorAgent) atLeastOneFieldIsEmpty(podWithPVItem *PodWithPVItem) bool {
+func (agent *PVMonitorAgent) isPodWithPVItemInValid(podWithPVItem *PodWithPVItem) bool {


s/InValid/Invalid

xing-yang · 2020-06-14T15:23:27Z

pkg/agent/pv_monitor_agent.go

 	}

-	klog.V(4).Infof("Started PV processing %q", podWithPV.pvName)
+	klog.V(6).Infof("Started PV processing %q", podWithPV.pvName)


We usually use V(5) for debugging. Can you change V(6) to V(5)?

fengzixu · 2020-06-27T02:30:51Z

After we merged the mock-driver PR, I try to deploy external-health-monitor in my personal cluster(v1.15.3). I found it cannot work normally.

I just found two bugs because the controller and agent always were panic after they started for a while

Agent has leader-election flag, but it wasn't defined
- https://github.com/kubernetes-csi/external-health-monitor/pull/3/files#diff-3e503dea03bcf5f280dbdc9258f91635R25
pvcListerSynced was initialized wrongly. It caused panic of agent and controller
- https://github.com/kubernetes-csi/external-health-monitor/pull/3/files#diff-ec7c9594995d70465840d4fed7b1bd21R103
- https://github.com/kubernetes-csi/external-health-monitor/pull/3/files#diff-811bef6041bfee6175e08819d2f9d10aR137

I will try to modify it in my own experimental env first for testing.

@NickrenREN @xing-yang

fengzixu · 2020-06-27T02:40:34Z

After we merged the mock-driver PR, I try to deploy external-health-monitor in my personal cluster(v1.15.3). I found it cannot work normally.

I just found two bugs because the controller and agent always were panic after they started for a while

Agent has leader-election flag, but it wasn't defined

https://github.com/kubernetes-csi/external-health-monitor/pull/3/files#diff-3e503dea03bcf5f280dbdc9258f91635R25

pvcListerSynced was initialized wrongly. It caused panic of agent and controller

https://github.com/kubernetes-csi/external-health-monitor/pull/3/files#diff-ec7c9594995d70465840d4fed7b1bd21R103

https://github.com/kubernetes-csi/external-health-monitor/pull/3/files#diff-811bef6041bfee6175e08819d2f9d10aR137

I will try to modify it in my own experimental env first for testing.

@NickrenREN @xing-yang

After I changed, I think agent and controller can work fine

diff --git a/pkg/agent/pv_monitor_agent.go b/pkg/agent/pv_monitor_agent.go
index cf8f38c..d8e2252 100644
--- a/pkg/agent/pv_monitor_agent.go
+++ b/pkg/agent/pv_monitor_agent.go
@@ -100,7 +100,7 @@ func NewPVMonitorAgent(client kubernetes.Interface, driverName string, conn *grp

        // PVC informer
        agent.pvcLister = pvcInformer.Lister()
-       agent.pvListerSynced = pvcInformer.Informer().HasSynced
+       agent.pvcListerSynced = pvcInformer.Informer().HasSynced

        // Pod informer
        podInformer.Informer().AddEventHandler(cache.ResourceEventHandlerFuncs{
diff --git a/pkg/controller/pv_monitor_controller.go b/pkg/controller/pv_monitor_controller.go
index ec9ad28..db847fd 100644
--- a/pkg/controller/pv_monitor_controller.go
+++ b/pkg/controller/pv_monitor_controller.go
@@ -134,6 +134,7 @@ func NewPVMonitorController(client kubernetes.Interface, conn *grpc.ClientConn,

        // PVC informer
        ctrl.pvcLister = pvcInformer.Lister()
+       ctrl.pvcListerSynced = pvcInformer.Informer().HasSynced
        ctrl.pvListerSynced = pvcInformer.Informer().HasSynced

        // Pod informer
(END)

[root@experiment-lab001-xuran:~/workspace/csi-test/external-health-monitor]# kubectl get pods
NAME                                                      READY   STATUS    RESTARTS   AGE
csi-external-health-monitor-agent-mcxqn                   2/2     Running   0          4m8s
csi-external-health-monitor-agent-sjcgm                   2/2     Running   0          3m58s
csi-external-health-monitor-controller-6bddc8c88c-6ngcq   2/2     Running   0          111s
csi-external-health-monitor-controller-6bddc8c88c-bvm89   2/2     Running   0          104s
csi-external-health-monitor-controller-6bddc8c88c-ls26g   2/2     Running   0          2m1s

xing-yang · 2020-06-27T03:53:13Z

Thanks @fengzixu for testing and fixing the bugs!

Hi @NickrenREN, do you want to fix these bugs on your PR or do you want @fengzixu to submit a PR on top of yours?

fengzixu · 2020-06-27T07:39:54Z

In addition to bugs, I mentioned above, I finished the basic test of external-health-monitor using the csi-test. So, let me summarize the testing report as below.

Test Cases

Check if volmue-health-monitro-controller can call the ControllerListVolume or ControllerGetVolume gPRC interface. And Check if the response of ControllerListVolume or ControllerGetVolume gPRC interface satisfy the csi-specification
Test one simple negative case to check response of ControllerListVolume

Cluster Info

Version: 1.16.6
Control Plane: 3
Worker: 2

What I did

Clone https://github.com/kubernetes-csi/external-health-monitor and https://github.com/kubernetes-csi/csi-test
Build container images csi-external-health-monitor-controller and csi-external-health-monitor-agent and mocker-driver
Push images to my personal images repo: https://quay.io/user/fengzixu
Change the image tag of csi-external-health-monitor-controller and csi-external-health-monitor-agent and mocker-driver to my own images
Process the command `kubectl create -f deploy/kubernetes/xxx" to deploy the external-health-monitor
process the command kubectl logs csi-external-health-monitor-controller-5866488ddc-jxdb6 -c mock-driver -f to detect the log

Result

Case1

I run kubectl logs <controller_pod_name> -c mock-driver

// List
gRPCCall: {"Method":"/csi.v1.Controller/ListVolumes","Request":{},"Response":{"entries":[{"volume":{"capacity_bytes":107374182400,"volume_id":"1","volume_context":{"name":"Mock Volume 1"}},"status":{"published_node_ids":["io.kubernetes.storage.mock"],"volume_condition":{}}},{"volume":{"capacity_bytes":107374182400,"volume_id":"2","volume_context":{"name":"Mock Volume 2"}},"status":{"published_node_ids":["io.kubernetes.storage.mock"],"volume_condition":{}}},{"volume":{"capacity_bytes":107374182400,"volume_id":"3","volume_context":{"name":"Mock Volume 3"}},"status":{"published_node_ids":["io.kubernetes.storage.mock"],"volume_condition":{}}}]},"Error":"","FullError":null}

// Get
gRPCCall: {"Method":"/csi.v1.Controller/ControllerGetVolume","Request":{"volume_id":"1"},"Response":{"volume":{"capacity_bytes":107374182400,"volume_id":"1","volume_context":{"name":"Mock Volume 1"}},"status":{"published_node_ids":["io.kubernetes.storage.mock"],"volume_condition":{}}},"Error":"","FullError":null}

Case2

Add some temporary code to test a simple negative case

// CheckControllerListVolumeStatuses checks volumes health condition by ListVolumes
func (checker *PVHealthConditionChecker) CheckControllerListVolumeStatuses() error {
        ctx, cancel := context.WithTimeout(context.Background(), checker.timeout)
        defer cancel()

        result, err := checker.csiPVHandler.ControllerListVolumeConditions(ctx)
        if err != nil {
                return err
        }

        // Positive Case
        _, err = checker.csiPVHandler.ControllerGetVolumeCondition(ctx, "1")
        if err != nil {
                return err
        }

        // Negative Case
        _, err = checker.csiPVHandler.ControllerGetVolumeCondition(ctx, "Mock Volume 10")
        if err != nil {
                return err
        }

Result

// Positive
gRPCCall: {"Method":"/csi.v1.Controller/ControllerGetVolume","Request":{"volume_id":"1"},"Response":{"volume":{"capacity_bytes":107374182400,"volume_id":"1","volume_context":{"name":"Mock Volume 1"}},"status":{"published_node_ids":["io.kubernetes.storage.mock"],"volume_condition":{}}},"Error":"","FullError":null}

// Negative
gRPCCall: {"Method":"/csi.v1.Controller/ControllerGetVolume","Request":{"volume_id":"Mock Volume 10"},"Response":{"status":{"volume_condition":{"abnormal":true,"message":"volume not found"}}},"Error":"rpc error: code = NotFound desc = Mock Volume 10","FullError":{"code":5,"message":"Mock Volume 10"}}

Conclustion

In the simple way, for the basic function of external-health-monitor, I think it's fine. We can merge this initial PR fist after we solved all existing comments.

As I said before, this PR of csi-test kubernetes-csi/csi-test#268 only for testing the basic function of external-health-monitor. Just checking if it can invoke the ControllerListVolume and ControllerGetVolume functions and get the desired response that satisfy csi-specification.

What we will do

After we merged this PR, I think we can do some tasks asap

1. make the code base more clean
2. add more positive and negative test cases in csi-test for external-health-monitor
3. discuss the plan introduce external-health-monitor to 1.19 before code freeze

fengzixu · 2020-06-27T07:48:05Z

@NickrenREN @xing-yang If you are available, please check my comment

fengzixu · 2020-06-27T08:00:33Z

I fixed bugs that cause panic though submit the PR to @NickrenREN 's repository

NickrenREN#1

Maybe @NickrenREN can merge it and update this PR

NickrenREN · 2020-06-28T02:52:41Z

@fengzixu thanks a lot for doing this.
@xing-yang fengzixu has send a PR for this, i have merged that.

xing-yang · 2020-06-30T02:48:57Z

deploy/kubernetes/external-health-monitor-controller/deployment.yaml

+      serviceAccount: csi-external-health-monitor-controller
+      containers:
+        - name: csi-external-health-monitor-controller
+          image: quay.io/k8scsi/csi-external-health-monitor-controller:canary


We are going to move the image repo to a different place so images for this repo won't be here any more.
To avoid confusion, please change this to:
image:

We can update this after this PR is merged and all necessary release-tools changes are merged.

Can we merge this PR now?

I'm still reviewing this.

fengzixu · 2020-07-03T15:23:07Z

I forget one patch about the clusterrole of Event resource. I think controller needs it. So I fixed it.

NickrenREN#2

@NickrenREN @xing-yang

xing-yang · 2020-07-03T18:42:31Z

@NickrenREN Can you squash the commits?

Looks good to me other than that. I'll merge after you've squashed them.

xing-yang · 2020-07-04T14:09:02Z

I forget one patch about the clusterrole of Event resource. I think controller needs it. So I fixed it.
NickrenREN#2

@fengzixu, If Nick merges your PR into his branch, it will be merged together with this PR. Otherwise you can submit a separate PR so it can be merged after this PR is merged.

1. remove the leader-election from argument list of agent 2. assign correct value to pvcListerSynced

NickrenREN · 2020-07-06T03:37:03Z

@xing-yang done
@fengzixu Sorry, i forget to merge your second PR, could you send it to this repo directly after this one get merged ?

fengzixu · 2020-07-06T03:51:36Z

@xing-yang done
@fengzixu Sorry, i forget to merge your second PR, could you send it to this repo directly after this one get merged ?

Sure

xing-yang · 2020-07-06T12:28:53Z

/lgtm
/approve

k8s-ci-robot · 2020-07-06T12:29:09Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: NickrenREN, xing-yang

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [NickrenREN,xing-yang]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot requested review from saad-ali and xing-yang March 6, 2020 03:43

k8s-ci-robot added approved Indicates a PR has been approved by an approver from all required OWNERS files. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Mar 6, 2020

NickrenREN removed request for xing-yang and saad-ali March 6, 2020 03:45

xing-yang mentioned this pull request Mar 6, 2020

PV Health Monitor kubernetes/enhancements#1432

Open

9 tasks

NickrenREN force-pushed the initial-implementation branch 2 times, most recently from 86e741f to 4bcef6a Compare March 9, 2020 17:06

This was referenced Mar 9, 2020

Implement volume health monitoring controller #1

Closed

Implement volume health monitoring agent #2

Closed

NickrenREN force-pushed the initial-implementation branch 2 times, most recently from e92257d to e20cae9 Compare March 11, 2020 07:16

NickrenREN force-pushed the initial-implementation branch 2 times, most recently from af812e9 to e353fb1 Compare March 11, 2020 16:21

NickrenREN force-pushed the initial-implementation branch from e353fb1 to e437396 Compare April 17, 2020 04:32

NickrenREN changed the title ~~WIP: Initial implementation~~ Initial implementation Apr 23, 2020

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Apr 23, 2020

xing-yang reviewed Apr 23, 2020

View reviewed changes

xing-yang reviewed May 3, 2020

View reviewed changes

xing-yang reviewed Jun 14, 2020

View reviewed changes

xing-yang mentioned this pull request Jun 17, 2020

master: update release-tools #14

Merged

xing-yang reviewed Jun 30, 2020

View reviewed changes

k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. and removed do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. labels Jul 3, 2020

xing-yang changed the title ~~Initial implementation~~ Initial implementation to add external health monitor controller and agent Jul 4, 2020

NickrenREN and others added 4 commits July 6, 2020 11:12

initial implementation

df1e416

update dependencies

a3ce254

Add .prow.sh

bb8bac3

fix: fix the bug that cases panic of controller and agent

76fc3fd

1. remove the leader-election from argument list of agent 2. assign correct value to pvcListerSynced

NickrenREN force-pushed the initial-implementation branch from 2961012 to 76fc3fd Compare July 6, 2020 03:15

k8s-ci-robot assigned xing-yang Jul 6, 2020

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jul 6, 2020

k8s-ci-robot merged commit 826049d into kubernetes-csi:master Jul 6, 2020

Initial implementation to add external health monitor controller and agent #3

Initial implementation to add external health monitor controller and agent #3

Conversation

NickrenREN commented Mar 6, 2020 • edited by xing-yang Loading

NickrenREN commented Mar 6, 2020 • edited Loading

xing-yang commented Mar 30, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fengzixu commented Jun 27, 2020 • edited Loading

fengzixu commented Jun 27, 2020 • edited Loading

xing-yang commented Jun 27, 2020

fengzixu commented Jun 27, 2020 • edited Loading

Test Cases

Cluster Info

What I did

Result

Case1

Case2

Conclustion

What we will do

fengzixu commented Jun 27, 2020

fengzixu commented Jun 27, 2020

NickrenREN commented Jun 28, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fengzixu commented Jul 3, 2020

xing-yang commented Jul 3, 2020

xing-yang commented Jul 4, 2020

NickrenREN commented Jul 6, 2020

fengzixu commented Jul 6, 2020

xing-yang commented Jul 6, 2020

k8s-ci-robot commented Jul 6, 2020

NickrenREN commented Mar 6, 2020 •

edited by xing-yang

Loading

NickrenREN commented Mar 6, 2020 •

edited

Loading

fengzixu commented Jun 27, 2020 •

edited

Loading

fengzixu commented Jun 27, 2020 •

edited

Loading

fengzixu commented Jun 27, 2020 •

edited

Loading