add Kubernetes End to End tests to the NFS Driver #10

mathu97 · 2019-04-15T19:41:34Z

What type of PR is this?

/kind feature

What this PR does / why we need it:
This PR adds End to End tests to the NFS plugin. This is done by importing in-tree Kubernetes CSI storage E2E tests. The NFS CSI plugin could be an example for other CSI plugin authors who also want to leverage the in-tree storage tests for their plugins.

Which issue(s) this PR fixes:
This would be a stable example for the CSI e2e testing documentation

Special notes for your reviewer:

This PR updates the driver from Kubernetes-1.12 to Kubernetes-1.14

Does this PR introduce a user-facing change?:

Users would be able to run End To End tests on the driver by using the command: go test -v ./cmd/nfsplugin/ -ginkgo.v -ginkgo.progress --kubeconfig=/var/run/kubernetes/admin.kubeconfig -timeout=0

k8s-ci-robot · 2019-04-15T19:41:48Z

Hi @mathu97. Thanks for your PR.

I'm waiting for a kubernetes-csi or kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

mathu97 · 2019-04-15T19:43:19Z

@wongma7 @msau42 please take a look, when you can 👍

msau42 · 2019-04-15T20:35:53Z

/ok-to-test

msau42 · 2019-04-15T20:36:03Z

/assign @wongma7

k8s-ci-robot · 2019-04-15T20:36:05Z

@msau42: GitHub didn't allow me to assign the following users: wongma7.

Note that only kubernetes-csi members and repo collaborators can be assigned and that issues/PRs can only have 10 assignees at the same time.
For more information please see the contributor guide

In response to this:

/assign @wongma7

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

msau42 · 2019-04-15T20:37:28Z

Gopkg.toml

@@ -68,10 +68,18 @@
  name = "github.com/json-iterator/go"
  version = "1.1.4"

+[[override]]
+  name = "k8s.io/client-go"
+  branch = "master"


do we need master or can kubernetes-1.14.0 work?

I was having an issue similar to this, but using master solved it. I'll look more into it to see if we can avoid using master.

msau42 · 2019-04-15T20:37:55Z

Gopkg.toml

 [[override]]
  name = "gopkg.in/square/go-jose.v2"
  version = "2.1.7"

+[[override]]
+  source = "https://github.com/fsnotify/fsnotify/archive/v1.4.7.tar.gz"


is this actually needed?

when using dep ensure, I was getting a panic: version queue is empty error, and adding this solved it. Reference: golang/dep#1799

msau42 · 2019-04-15T20:41:00Z

test/csi-volumes.go

+	testsuites.InitVolumeIOTestSuite,
+	testsuites.InitVolumeModeTestSuite,
+	testsuites.InitSubPathTestSuite,
+	testsuites.InitProvisioningTestSuite,


Would it be useful to define a global variable in the test framework that contains all the test suites? That way drivers don't need to manually copy them?

Yes I think that would be useful. Would there be any case where a driver would not want to run some of these test suites?

It shouldn't need to. The test cases are supposed to check for driver capabilities and only run the tests that the driver can support.

We also have Snapshot and MultiVolume test suites, although they don't seem applicable to this nfs driver. Should be include them anyway and comment them out?

Ya sure, since this would be used as an example right. Also, I think MultiVolume testsuite is not in 1.14?

I guess it doesn't matter as it'll still be commented out.

msau42 · 2019-04-15T20:42:39Z

test/nfs-testdriver.go

+	}
+
+	//Install the nfs driver from the manifests
+	cleanup, err := config.Framework.CreateFromManifests(nil, n.manifests...)


Instead of creating a new nfs driver for every test case, can we bring up the nfs driver once in the BeforeSuite()?

Currently the testdriver automatically creates a new namespace with a randomly generated name for each testcase, and I think the driver needs to be in that same namespace for it to be accessible by the tests. This would be hard to do in the BeforeSuite() if we don't know what the randomly generated namespace is called.

I think if you don't use CreateFromManifests, you can just bring up the driver directly in some default namespace. This is how we configure pd csi driver to run e2e externally and not bring up the driver for every test:

https://github.com/kubernetes/kubernetes/blob/51db0fe21a816b0818df571ad70386864d83c722/test/e2e/storage/drivers/csi.go#L444

Ya that makes sense. So we would bring up the driver in a default namespace, instead of creating a new namespace for each test case right?

Yeah the driver could be brought up in the default namespace. Test case application pods will still come up in their own namespaces though

for now auto-deploying whatever is in the yaml folder every BeforeSuite is fine. If the user wants to test a different image version they can edit the yamls. In the future maybe there will be a flag for whether the test should deploy the driver or not if it already exists

@msau42 it's possible, like e.g. hostpath driver had some hardcoded paths that were changed so the yaml changed

oh, I misspoke didn't mean before every test i mean before every testsuite. We agree on that already

It looks like this is still bringing up the nfs driver for every test case?

Sorry, I still haven't pushed it.

msau42 · 2019-04-15T20:43:14Z

Gopkg.toml

@@ -42,22 +42,22 @@

 [[constraint]]
  name = "k8s.io/kubernetes"
-  version = "v1.12.0"
+  version = "v1.14.0"


Can you put the vendor updates into a separate commit?

msau42 · 2019-04-15T20:47:21Z

pkg/nfs/controllerserver.go

+
+func getControllerServer(csiDriver *csicommon.CSIDriver) ControllerServer {
+	return ControllerServer{
+		csicommon.NewDefaultControllerServer(csiDriver),


I don't think we've updated csicommon for csi spec 1.1 (and we don't intend to. We want to deprecate csicommon).

Are we still trying to move csicommon to kubernetes-csi/csi-lib-common?

No we're just going to deprecate it. It doesn't seem to provide much value.

okay, also should we stick to csi spec 1.0?

1.1 is fine too. The resizing stuff will be unimplemented

msau42 · 2019-04-15T20:48:44Z

README.md

+1) Stand up a local cluster `ALLOW_PRIVILEGED=1 hack/local-up-cluster.sh`
+2) Build the nfs driver by running `make`
+3) Create NFS Driver Image, where the image tag would be whatever that is required by your YAML deployment files        `docker build -t quay.io/k8scsi/nfsplugin:v1.0.0 .`
+4) Run E2E Tests using the following command: `go test -v ./cmd/nfsplugin/ -ginkgo.v -ginkgo.progress --kubeconfig=/var/run/kubernetes/admin.kubeconfig -timeout=0`


How does someone run the e2e tests with their own private images?

you would need to add the required YAML files for that private image in deploy/kubernetes, as those are the files that the e2e test uses to deploy the driver. I'll add that to the documentation.

I don't see the updated documentation pointing to which yamls need to be changed

should we build local image on each node? since the image is not publish to quay.io, if the plugin will run on each node, then should build or import the image to all nodes?

just pushed the updated docs

msau42 · 2019-04-23T20:52:52Z

test/csi-volumes.go

+	testsuites.InitVolumeIOTestSuite,
+	testsuites.InitVolumeModeTestSuite,
+	testsuites.InitSubPathTestSuite,
+	testsuites.InitProvisioningTestSuite,


We also have Snapshot and MultiVolume test suites, although they don't seem applicable to this nfs driver. Should be include them anyway and comment them out?

msau42 · 2019-04-23T20:54:59Z

test/nfs-testdriver.go

+
+func (n *nfsDriver) SkipUnsupportedTest(pattern testpatterns.TestPattern) {
+	if pattern.VolType == testpatterns.DynamicPV {
+		framework.Skipf("NFS Driver does not support dynamic provisioning -- skipping")


in the future it may be cool to add something like the nfs-client provisioner to the sample driver, and then we can unlock these test cases as well

msau42 · 2019-04-23T20:56:33Z

test/nfs-testdriver.go

+	}
+
+	//Install the nfs driver from the manifests
+	cleanup, err := config.Framework.CreateFromManifests(nil, n.manifests...)


It looks like this is still bringing up the nfs driver for every test case?

msau42 · 2019-04-23T20:58:17Z

README.md

+1) Stand up a local cluster `ALLOW_PRIVILEGED=1 hack/local-up-cluster.sh`
+2) Build the nfs driver by running `make`
+3) Create NFS Driver Image, where the image tag would be whatever that is required by your YAML deployment files        `docker build -t quay.io/k8scsi/nfsplugin:v1.0.0 .`
+4) Run E2E Tests using the following command: `go test -v ./cmd/nfsplugin/ -ginkgo.v -ginkgo.progress --kubeconfig=/var/run/kubernetes/admin.kubeconfig -timeout=0`


I don't see the updated documentation pointing to which yamls need to be changed

mathu97 · 2019-05-02T14:53:32Z

@msau42 let me know what you think about the new changes

msau42 · 2019-05-02T23:13:48Z

Sorry for the delay. Thanks for working on this!

/lgtm
/approve

k8s-ci-robot · 2019-05-02T23:14:05Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: mathu97, msau42

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [msau42]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

check vendor directory

a0f195c Merge pull request kubernetes-csi#106 from msau42/fix-canary 7100c12 Only set staging registry when running canary job b3c65f9 Merge pull request kubernetes-csi#99 from msau42/add-release-process e53f3e8 Merge pull request kubernetes-csi#103 from msau42/fix-canary d129462 Document new method for adding CI jobs are new K8s versions e73c2ce Use staging registry for canary tests 2c09846 Add cleanup instructions to release-notes generation 60e1cd3 Merge pull request kubernetes-csi#98 from pohly/kubernetes-1-19-fixes 0979c09 prow.sh: fix E2E suite for Kubernetes >= 1.18 3b4a2f1 prow.sh: fix installing Go for Kubernetes 1.19.0 1fbb636 Merge pull request kubernetes-csi#97 from pohly/go-1.15 82d108a switch to Go 1.15 d8a2530 Merge pull request kubernetes-csi#95 from msau42/add-release-process 843bddc Add steps on promoting release images 0345a83 Merge pull request kubernetes-csi#94 from linux-on-ibm-z/bump-timeout 1fdf2d5 cloud build: bump timeout in Prow job 41ec6d1 Merge pull request kubernetes-csi#93 from animeshk08/patch-1 5a54e67 filter-junit: Fix gofmt error 0676fcb Merge pull request kubernetes-csi#92 from animeshk08/patch-1 36ea4ff filter-junit: Fix golint error f5a4203 Merge pull request kubernetes-csi#91 from cyb70289/arm64 43e50d6 prow.sh: enable building arm64 image 0d5bd84 Merge pull request kubernetes-csi#90 from pohly/k8s-staging-sig-storage 3df86b7 cloud build: k8s-staging-sig-storage c5fd961 Merge pull request kubernetes-csi#89 from pohly/cloud-build-binfmt db0c2a7 cloud build: initialize support for running commands in Dockerfile be902f4 Merge pull request kubernetes-csi#88 from pohly/multiarch-windows-fix 340e082 build.make: optional inclusion of Windows in multiarch images 5231f05 build.make: properly declare push-multiarch 4569f27 build.make: fix push-multiarch ambiguity 17dde9e Merge pull request kubernetes-csi#87 from pohly/cloud-build bd41690 cloud build: initial set of shared files 9084fec Merge pull request kubernetes-csi#81 from msau42/add-release-process 6f2322e Update patch release notes generation command 0fcc3b1 Merge pull request kubernetes-csi#78 from ggriffiths/fix_csi_snapshotter_rbac_version_set d8c76fe Support local snapshot RBAC for pull jobs c1bdf5b Merge pull request kubernetes-csi#80 from msau42/add-release-process ea1f94a update release tools instructions 152396e Merge pull request kubernetes-csi#77 from ggriffiths/snapshotter201_update 7edc146 Update snapshotter to version 2.0.1 4cf843f Merge pull request kubernetes-csi#76 from pohly/build-targets 3863a0f build for multiple platforms only in CI, add s390x 8322a7d Merge pull request kubernetes-csi#72 from pohly/hostpath-update 7c5a89c prow.sh: use 1.3.0 hostpath driver for testing b8587b2 Merge pull request kubernetes-csi#71 from wozniakjan/test-vet fdb3218 Change 'make test-vet' to call 'go vet' d717c8c Merge pull request kubernetes-csi#69 from pohly/test-driver-config a1432bc Merge pull request kubernetes-csi#70 from pohly/kubelet-feature-gates 5f74333 prow.sh: also configure feature gates for kubelet 84f78b1 prow.sh: generic driver installation 3c34b4f Merge pull request kubernetes-csi#67 from windayski/fix-link fa90abd fix incorrect link ff3cc3f Merge pull request kubernetes-csi#54 from msau42/add-release-process ac8a021 Document the process for releasing a new sidecar 23be652 Merge pull request kubernetes-csi#65 from msau42/update-hostpath 6582f2f Update hostpath driver version to get fix for connection-timeout 4cc9174 Merge pull request kubernetes-csi#64 from ggriffiths/snapshotter_2_version_update 8191eab Update snapshotter to version v2.0.0 3c463fb Merge pull request kubernetes-csi#61 from msau42/enable-snapshots 8b0316c Fix overriding of junit results by using unique names for each e2e run 5f444b8 Merge pull request kubernetes-csi#60 from saad-ali/updateHostpathVersion af9549b Update prow hostpath driver version to 1.3.0-rc2 f6c74b3 Merge pull request kubernetes-csi#57 from ggriffiths/version_gt_kubernetes_fix fc80975 Fix version_gt to work with kubernetes prefix 9f1f3dd Merge pull request kubernetes-csi#56 from msau42/enable-snapshots b98b2ae Enable snapshot tests in 1.17 to be run in non-alpha jobs. 9ace020 Merge pull request kubernetes-csi#52 from msau42/update-readme 540599b Merge pull request kubernetes-csi#53 from msau42/fix-make a4e6299 fix syntax for ppc64le build 771ca6f Merge pull request kubernetes-csi#49 from ggriffiths/prowsh_improve_version_gt d7c69d2 Merge pull request kubernetes-csi#51 from msau42/enable-multinode 4ad6949 Improve snapshot pod running checks and improve version_gt 53888ae Improve README by adding an explicit Kubernetes dependency section 9a7a685 Create a kind cluster with two worker nodes so that the topology feature can be tested. Test cases that test accessing volumes from multiple nodes need to be skipped 4ff2f5f Merge pull request kubernetes-csi#50 from darkowlzz/kind-0.6.0 80bba1f Use kind v0.6.0 6d674a7 Merge pull request kubernetes-csi#47 from Pensu/multi-arch 8adde49 Merge pull request kubernetes-csi#45 from ggriffiths/snapshot_beta_crds 003c14b Add snapshotter CRDs after cluster setup a41f386 Merge pull request kubernetes-csi#46 from mucahitkurt/kind-cluster-cleanup 1eaaaa1 Delete kind cluster after tests run. 83a4ef1 Adding build for ppc64le 4fcafec Merge pull request kubernetes-csi#43 from pohly/system-pod-logging f41c135 prow.sh: also log output of system containers ee22a9c Merge pull request kubernetes-csi#42 from pohly/use-vendor-dir 8067845 travis.yml: also use vendor directory 23df4ae prow.sh: use vendor directory if available a53bd4c Merge pull request kubernetes-csi#41 from pohly/go-version c8a1c4a better handling of Go version 5e773d2 update CI to use Go 1.13.3 f419d74 Merge pull request kubernetes-csi#40 from msau42/add-1.16 e0fde8c Add new variables for 1.16 and remove 1.13 adf00fe Merge pull request kubernetes-csi#36 from msau42/full-clone f1697d2 Do full git clones in travis. Shallow clones are causing test-subtree errors when the depth is exactly 50. 2c81919 Merge pull request kubernetes-csi#34 from pohly/go-mod-tidy 518d6af Merge pull request kubernetes-csi#35 from ddebroy/winbld2 2d6b3ce Build Windows only for amd64 c1078a6 go-get-kubernetes.sh: automate Kubernetes dependency handling 194289a update Go mod support 0affdf9 Merge pull request kubernetes-csi#33 from gnufied/enable-hostpath-expansion 6208f6a Enable hostpath expansion 6ecaa76 Merge pull request kubernetes-csi#30 from msau42/fix-windows ea2f1b5 build windows binaries with .exe suffix 2d33550 Merge pull request kubernetes-csi#29 from mucahitkurt/create-2-node-kind-cluster a8ea8bc create 2-node kind cluster since topology support is added to hostpath driver df8530d Merge pull request kubernetes-csi#27 from pohly/dep-vendor-check 35ceaed prow.sh: install dep if needed f85ab5a Merge pull request kubernetes-csi#26 from ddebroy/windows1 9fba09b Add rule for building Windows binaries 0400867 Merge pull request kubernetes-csi#25 from msau42/fix-master-jobs dc0a5d8 Update kind to v0.5.0 aa85b82 Merge pull request kubernetes-csi#23 from msau42/fix-master-jobs f46191d Kubernetes master changed the way that releases are tagged, which needed changes to kind. There are 3 changes made to prow.sh: 1cac3af Merge pull request kubernetes-csi#22 from msau42/add-1.15-jobs 0c0dc30 prow.sh: tag master images with a large version number f4f73ce Merge pull request kubernetes-csi#21 from msau42/add-1.15-jobs 4e31f07 Change default hostpath driver name to hostpath.csi.k8s.io 4b6fa4a Update hostpath version for sidecar testing to v1.2.0-rc2 ecc7918 Update kind to v0.4.0. This requires overriding Kubernetes versions with specific patch versions that kind 0.4.0 supports. Also, feature gate setting is only supported on 1.15+ due to kind.sigs.k8s.io/v1alpha3 and kubeadm.k8s.io/v1beta2 dependencies. a6f21d4 Add variables for 1.15 db8abb6 Merge pull request kubernetes-csi#20 from pohly/test-driver-config b2f4e05 prow.sh: flexible test driver config 0399988 Merge pull request kubernetes-csi#19 from pohly/go-mod-vendor 066143d build.make: allow repos to use 'go mod' for vendoring 0bee749 Merge pull request kubernetes-csi#18 from pohly/go-version e157b6b update to Go 1.12.4 88dc9a4 Merge pull request kubernetes-csi#17 from pohly/prow 0fafc66 prow.sh: skip sanity testing if component doesn't support it bcac1c1 Merge pull request kubernetes-csi#16 from pohly/prow 0b10f6a prow.sh: update csi-driver-host-path 0c2677e Merge pull request kubernetes-csi#15 from pengzhisun/master ff9bce4 Replace 'return' to 'exit' to fix shellcheck error c60f382 Merge pull request kubernetes-csi#14 from pohly/prow 7aaac22 prow.sh: remove AllAlpha=all, part II 6617773 Merge pull request kubernetes-csi#13 from pohly/prow cda2fc5 prow.sh: avoid AllAlpha=true 546d550 prow.sh: debug failing KinD cluster creation 9b0d9cd build.make: skip shellcheck if Docker is not available aa45a1c prow.sh: more efficient execution of individual tests f3d1d2d prow.sh: fix hostpath driver version check 31dfaf3 prow.sh: fix running of just "alpha" tests f501443 prow.sh: AllAlpha=true for unknown Kubernetes versions 95ae9de Merge pull request kubernetes-csi#9 from pohly/prow d87eccb prow.sh: switch back to upstream csi-driver-host-path 6602d38 prow.sh: different E2E suite depending on Kubernetes version 741319b prow.sh: improve building Kubernetes from source 29545bb prow.sh: take Go version from Kubernetes source 429581c prow.sh: pull Go version from travis.yml 0a0fd49 prow.sh: comment clarification 2069a0a Merge pull request kubernetes-csi#11 from pohly/verify-shellcheck 55212ff initial Prow test job 6c7ba1b build.make: integrate shellcheck into "make test" b2d25d4 verify-shellcheck.sh: make it usable in csi-release-tools 3b6af7b Merge pull request kubernetes-csi#12 from pohly/local-e2e-suite 104a1ac build.make: avoid unit-testing E2E test suite 34010e7 Merge pull request kubernetes-csi#10 from pohly/vendor-check e6db50d check vendor directory fb13c51 verify-shellcheck.sh: import from Kubernetes 94fc1e3 build.make: avoid unit-testing E2E test suite 849db0a Merge pull request kubernetes-csi#8 from pohly/subtree-check-relax cc564f9 verify-subtree.sh: relax check and ignore old content git-subtree-dir: release-tools git-subtree-split: a0f195c

k8s-ci-robot added kind/feature Categorizes issue or PR as related to a new feature. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Apr 15, 2019

k8s-ci-robot requested review from msau42 and saad-ali April 15, 2019 19:41

k8s-ci-robot added needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Apr 15, 2019

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Apr 15, 2019

msau42 reviewed Apr 15, 2019

View reviewed changes

Mathusan Selvarajah added 4 commits April 23, 2019 15:42

update vendor files to kube 1.14

04b68cc

add e2e test files

f55eb4f

update nfs driver for kube 1.14

959bfe4

add instructions on how to run e2e tests

7e584f4

mathu97 force-pushed the nfs-e2e-tests branch from 3e8d50c to 7e584f4 Compare April 23, 2019 19:48

msau42 reviewed Apr 23, 2019

View reviewed changes

create e2e test binary and update docs

193faa0

mathu97 force-pushed the nfs-e2e-tests branch from 1210c70 to 193faa0 Compare April 25, 2019 13:06

k8s-ci-robot assigned msau42 May 2, 2019

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label May 2, 2019

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label May 2, 2019

k8s-ci-robot merged commit 2c5c9bc into kubernetes-csi:master May 2, 2019

pohly pushed a commit to pohly/csi-driver-nfs that referenced this pull request Jun 7, 2019

Merge pull request kubernetes-csi#10 from pohly/vendor-check

34010e7

check vendor directory

mkimuram mentioned this pull request Jun 25, 2019

add Kubernetes End to End tests to the ISCSI Driver kubernetes-csi/csi-driver-iscsi#17

Closed

mkimuram mentioned this pull request Nov 7, 2019

Add custom e2e job to prow.sh kubernetes-csi/csi-release-tools#44

Closed

pohly mentioned this pull request Oct 12, 2020

master: update release-tools #60

Merged

add Kubernetes End to End tests to the NFS Driver #10

add Kubernetes End to End tests to the NFS Driver #10

Conversation

mathu97 commented Apr 15, 2019

k8s-ci-robot commented Apr 15, 2019

mathu97 commented Apr 15, 2019 • edited Loading

msau42 commented Apr 15, 2019

msau42 commented Apr 15, 2019

k8s-ci-robot commented Apr 15, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mathu97 Apr 23, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mathu97 commented May 2, 2019

msau42 commented May 2, 2019

k8s-ci-robot commented May 2, 2019

mathu97 commented Apr 15, 2019 •

edited

Loading

mathu97 Apr 23, 2019 •

edited

Loading