Run upstream e2e test suites with migration

[wongma7]

Is this a bug fix or adding new feature? new feature

What is this PR about? / Why do we need it? As AWS migration will move to beta in 1.16, we should test the driver.

The test does driver := drivers.InitAwsDriver() which means tests will refer to the in-tree driver, i.e. they'll create PVs that say awselasticblockstore instead of csi. When migration is working, the CSI driver should be getting calls, not the in-tree driver. The "storage.migratedPlugins" flag signals tests to verify this, but it doesn't work for kops clusters because of faulty master node detection kubernetes/kubernetes#80177. The tests themselves are all the typical storage suites.

TODO: manually verify that the CSI driver is getting calls after all tests are run

The tests are put in their own e2e-migration directory so as not to pollute the rest of the repo. "storage.migratedPlugins" is only available in 1.16 alpha versions. Even if the flag doesn't work right, we should run the 1.16 version of tests.

This requires kubernetes/test-infra#13860.

There are known failures that can't be fixed on the driver side. I will add skips and comments for each of them.

What testing is done? ran this on my own AWS account. Will try to run in CI account after test-infra PR merges.

[wongma7]

/test pull-aws-ebs-csi-driver-migration-test-latest

[wongma7]

Summarizing 7 Failures:
[Fail] [ebs-csi-migration] EBS CSI Migration [Driver: aws] [Testpattern: Dynamic PV (default fs)] subPath [It] should support readOnly directory specified in the volumeMount 
/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/tests/e2e-migration/vendor/k8s.io/kubernetes/test/e2e/framework/util.go:1626
[Fail] [ebs-csi-migration] EBS CSI Migration [Driver: aws] [Testpattern: Dynamic PV (default fs)] volume-expand [It] should not allow expansion of pvcs without AllowVolumeExpansion property 
/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/tests/e2e-migration/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/base.go:280
[Fail] [ebs-csi-migration] EBS CSI Migration [Driver: aws] [Testpattern: Dynamic PV (block volmode)(allowExpansion)] volume-expand [It] Verify if offline PVC expansion works 
/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/tests/e2e-migration/vendor/k8s.io/kubernetes/test/e2e/framework/util.go:1306
[Fail] [ebs-csi-migration] EBS CSI Migration [Driver: aws] [Testpattern: Dynamic PV (block volmode)(allowExpansion)] volume-expand [It] should resize volume when PVC is edited while pod is using it 
/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/tests/e2e-migration/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/volume_expand.go:249
[Fail] [ebs-csi-migration] EBS CSI Migration [Driver: aws] [Testpattern: Dynamic PV (block volmode)] volume-expand [It] should not allow expansion of pvcs without AllowVolumeExpansion property 
/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/tests/e2e-migration/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/base.go:280
[Fail] [ebs-csi-migration] EBS CSI Migration [Driver: aws] [Testpattern: Dynamic PV (default fs)] provisioning [It] should provision storage with mount options 
/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/tests/e2e-migration/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/provisioning.go:507
[Fail] [ebs-csi-migration] EBS CSI Migration [Driver: aws] [Testpattern: Dynamic PV (ext3)] volumes [It] should store data 
/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/tests/e2e-migration/vendor/k8s.io/kubernetes/test/e2e/framework/volume/fixtures.go:538
Ran 37 of 210 Specs in 4571.035 seconds
FAIL! -- 30 Passed | 7 Failed | 0 Pending | 173 Skipped
--- FAIL: TestEBSCSI (4571.04s) 

These are all known failures except should support readOnly directory specified in the volumeMount... might just be a flake?

[wongma7]

/test pull-aws-ebs-csi-driver-migration-test-latest

[coveralls]

Pull Request Test Coverage Report for Build 778

• 0 of 0 changed or added relevant lines in 0 files are covered.
• No unchanged relevant lines lost coverage.
• Overall coverage remained the same at 71.1%

---

Totals
Change from base Build 769:	0.0%
Covered Lines:	1176
Relevant Lines:	1654

---

💛 - Coveralls

[wongma7]

/test pull-aws-ebs-csi-driver-migration-test-latest

[wongma7]

[Fail] [ebs-csi-migration] EBS CSI Migration [Driver: aws] [Testpattern: Dynamic PV (ext3)] volumes [It] should store data
wait for #335 to merge first, then it should pass

[leakingtapan]

/test pull-aws-ebs-csi-driver-migration-test-latest

rerun the test to see if #335 fixes one of failed test

[leakingtapan]

single e2e test failed due to apiserver timeout:

[ebs-csi-e2e] [single-az] Dynamic Provisioning
�[90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/tests/e2e/dynamic_provisioning.go:40�[0m
  �[91m�[1mshould create multiple PV objects, bind to PVCs and attach all to different pods on the same node [It]�[0m
  �[90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/tests/e2e/dynamic_provisioning.go:269�[0m

  �[91mExpected error:
      <*url.Error | 0xc000a66c00>: {
          Op: "Get",
          URL: "https://api-test-cluster-11027-k8-n6e2io-1281221233.us-west-2.elb.amazonaws.com/api/v1/namespaces/e2e-tests-ebs-5w9qn/persistentvolumeclaims/pvc-slfnp",
          Err: {
              Op: "dial",
              Net: "tcp",
              Source: nil,
              Addr: {IP: "4&\xacN", Port: 443, Zone: ""},
              Err: {},
          },
      }
      Get https://api-test-cluster-11027-k8-n6e2io-1281221233.us-west-2.elb.amazonaws.com/api/v1/namespaces/e2e-tests-ebs-5w9qn/persistentvolumeclaims/pvc-slfnp: dial tcp 52.38.172.78:443: i/o timeout
  not to have occurred�[0m

  /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/tests/e2e/testsuites/testsuites.go:273

[wongma7]

Passed after ~1hr, I've squashed and added the test-grid/$ARTIFACTS then we can review

[wongma7]

TODO for me: oops

[wongma7]

Bumping repo-wide to 1.16 broke something

pkg/driver/node.go:464:26: d.mounter.ExistsPath undefined (type Mounter has no field or method ExistsPath) 

Safer to have a submodule (go.mod under tests/e2e-migration) anyway, will work on that.

[wongma7]

/test pull-aws-ebs-csi-driver-migration-test-latest

[wongma7]

/test pull-aws-ebs-csi-driver-migration-test-latest

[wongma7]

/test pull-aws-ebs-csi-driver-migration-test-latest

[wongma7]

/test pull-aws-ebs-csi-driver-migration-test-latest

[wongma7]

/test pull-aws-ebs-csi-driver-migration-test-latest

[k8s-ci-robot]

@wongma7: The following test failed, say /retest to rerun them all:

Test name	Commit	Details	Rerun command
pull-aws-ebs-csi-driver-migration-test-latest	ec56a59	link	/test pull-aws-ebs-csi-driver-migration-test-latest

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[wongma7]

https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_aws-ebs-csi-driver/341/pull-aws-ebs-csi-driver-migration-test-latest/1165056033802948609
A couple failures to look into,
volumeMode should not mount / map unused volumes in a pod
This is a k8s test, no value in testing EBS or any drivers against it, we can skip it.
[ebs-csi-migration] EBS CSI Migration [Driver: aws] [Testpattern: Dynamic PV (default fs)(allowExpansion)] volume-expand Verify if offline PVC expansion works
TODO
[ebs-csi-migration] EBS CSI Migration [Driver: aws] [Testpattern: Dynamic PV (block volmode)] volumes should store data
TODO

Let's merge this and get the periodic kubernetes/test-infra#14077 running, I can create another dummy PR to investigate the 2 failures later.

[leakingtapan]

Two failures are due to SSH key issue:

[ebs-csi-migration]  EBS CSI Migration [Driver: aws] [Testpattern: Dynamic PV (filesystem  volmode)] volumeMode should not mount / map unused volumes in a pod expand_less | 1m4s
-- | --
/home/prow/go/pkg/mod/k8s.io/[email protected]/test/e2e/storage/testsuites/volumemode.go:285 Unexpected error:     <*errors.errorString \| 0xc000d058b0>: {         s: "error checking directory /var/lib/kubelet/pods/fa8f95e7-3ad4-4ca9-b4f6-b1b886c1d05b/volumes on node 54.70.94.199:22: error getting signer for provider aws: 'error reading SSH key /root/.ssh/kube_aws_rsa: 'open /root/.ssh/kube_aws_rsa: no such file or directory''",     }     error checking directory /var/lib/kubelet/pods/fa8f95e7-3ad4-4ca9-b4f6-b1b886c1d05b/volumes on node 54.70.94.199:22: error getting signer for provider aws: 'error reading SSH key /root/.ssh/kube_aws_rsa: 'open /root/.ssh/kube_aws_rsa: no such file or directory'' occurred /home/prow/go/pkg/mod/k8s.io/[email protected]/test/e2e/storage/testsuites/volumemode.go:317

[ebs-csi-migration]  EBS CSI Migration [Driver: aws] [Testpattern: Dynamic PV (block  volmode)] volumeMode should not mount / map unused volumes in a pod expand_less | 45s
-- | --
/home/prow/go/pkg/mod/k8s.io/[email protected]/test/e2e/storage/testsuites/volumemode.go:285 Unexpected error:     <*errors.errorString \| 0xc00071f8e0>: {         s: "error checking directory /var/lib/kubelet/pods/2a5b4968-7b80-4ffa-b0b1-65f524bbce07/volumes on node 54.70.94.199:22: error getting signer for provider aws: 'error reading SSH key /root/.ssh/kube_aws_rsa: 'open /root/.ssh/kube_aws_rsa: no such file or directory''",     }     error checking directory /var/lib/kubelet/pods/2a5b4968-7b80-4ffa-b0b1-65f524bbce07/volumes on node 54.70.94.199:22: error getting signer for provider aws: 'error reading SSH key /root/.ssh/kube_aws_rsa: 'open /root/.ssh/kube_aws_rsa: no such file or directory'' occurred /home/prow/go/pkg/mod/k8s.io/[email protected]/test/e2e/storage/testsuites/volumemode.go:317

One test is due to volume expand:

[ebs-csi-migration]  EBS CSI Migration [Driver: aws] [Testpattern: Dynamic PV (default  fs)(allowExpansion)] volume-expand Verify if offline PVC expansion  works expand_less | 6m2s
-- | --
/home/prow/go/pkg/mod/k8s.io/[email protected]/test/e2e/storage/testsuites/volume_expand.go:153 while recreating pod for resizing Unexpected error:     <*errors.errorString \| 0xc000d05770>: {         s: "pod \"security-context-12895dc8-3f32-4a6f-80e7-e9b8136762ea\" is not Running: timed out waiting for the condition",     }     pod "security-context-12895dc8-3f32-4a6f-80e7-e9b8136762ea" is not Running: timed out waiting for the condition occurred /home/prow/go/pkg/mod/k8s.io/[email protected]/test/e2e/storage/testsuites/volume_expand.go:208

One test is due to pod deletion for block volume test:

[ebs-csi-migration] EBS CSI Migration [Driver: aws] [Testpattern: Dynamic PV (block volmode)] volumes should store data expand_less | 11m8s
-- | --
/home/prow/go/pkg/mod/k8s.io/[email protected]/test/e2e/storage/testsuites/volumes.go:145 Aug 24 01:39:40.188: Failed to create client pod: timed out waiting for the condition /home/prow/go/pkg/mod/k8s.io/[email protected]/test/e2e/framework/volume/fixtures.go:551

[leakingtapan]

merge to enable periodic job.
/lgtm
/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: leakingtapan, wongma7

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [leakingtapan]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment