unable to apply addons to target cluster: EOF

[jazzdan]

/kind bug

What steps did you take and what happened:

1. Followed developer workflow pairing with @chuckha
2. Ran:

 clusterctl create cluster -v 3     --provider aws -m ./cmd/clusterctl/examples/aws/out/machines.yaml     -c ./cmd/clusterctl/examples/aws/out/cluster.yaml     -p ./cmd/clusterctl/examples/aws/out/provider-components-dev.yaml     -a ./cmd/clusterctl/examples/aws/out/addons.yaml         -e $(kind get kubeconfig-path)

3. Observed that it failed to create the cluster. The log output was:

I0304 17:25:37.904781   17378 createbootstrapcluster.go:27] Creating bootstrap cluster
I0304 17:25:37.906786   17378 clusterdeployer.go:70] Applying Cluster API stack to bootstrap cluster
I0304 17:25:37.906800   17378 applyclusterapicomponents.go:26] Applying Cluster API Provider Components
I0304 17:25:37.906807   17378 clusterclient.go:536] Waiting for kubectl apply...
I0304 17:25:38.668258   17378 clusterclient.go:565] Waiting for Cluster v1alpha resources to become available...
I0304 17:25:38.687717   17378 clusterclient.go:578] Waiting for Cluster v1alpha resources to be listable...
I0304 17:25:38.713285   17378 clusterdeployer.go:75] Provisioning target cluster via bootstrap cluster
I0304 17:25:38.720274   17378 applycluster.go:36] Creating cluster object test1 in namespace "default"
I0304 17:25:38.726977   17378 clusterdeployer.go:84] Creating control plane controlplane-0 in namespace "default"
I0304 17:25:38.732389   17378 applymachines.go:36] Creating machines in namespace "default"
I0304 17:25:38.748728   17378 clusterclient.go:589] Waiting for Machine controlplane-0 to become ready...
I0304 17:25:48.751387   17378 clusterclient.go:589] Waiting for Machine controlplane-0 to become ready...
I0304 17:25:58.751388   17378 clusterclient.go:589] Waiting for Machine controlplane-0 to become ready...
I0304 17:26:08.751382   17378 clusterclient.go:589] Waiting for Machine controlplane-0 to become ready...
I0304 17:26:18.751426   17378 clusterclient.go:589] Waiting for Machine controlplane-0 to become ready...
I0304 17:26:28.751418   17378 clusterclient.go:589] Waiting for Machine controlplane-0 to become ready...
I0304 17:26:38.751436   17378 clusterclient.go:589] Waiting for Machine controlplane-0 to become ready...
I0304 17:26:48.751456   17378 clusterclient.go:589] Waiting for Machine controlplane-0 to become ready...
I0304 17:26:58.751388   17378 clusterclient.go:589] Waiting for Machine controlplane-0 to become ready...
I0304 17:27:08.751386   17378 clusterclient.go:589] Waiting for Machine controlplane-0 to become ready...
I0304 17:27:18.751389   17378 clusterclient.go:589] Waiting for Machine controlplane-0 to become ready...
I0304 17:27:28.751413   17378 clusterclient.go:589] Waiting for Machine controlplane-0 to become ready...
I0304 17:27:38.751457   17378 clusterclient.go:589] Waiting for Machine controlplane-0 to become ready...
I0304 17:27:48.751422   17378 clusterclient.go:589] Waiting for Machine controlplane-0 to become ready...
I0304 17:27:48.754397   17378 clusterdeployer.go:89] Updating bootstrap cluster object for cluster test1 in namespace "default" with control plane endpoint running on controlplane-0
I0304 17:27:48.774059   17378 clusterdeployer.go:94] Creating target cluster
I0304 17:27:48.788581   17378 getkubeconfig.go:44] Getting target cluster kubeconfig.
I0304 17:27:48.788650   17378 getkubeconfig.go:60] Waiting for kubeconfig on controlplane-0 to become ready...
I0304 17:27:49.012086   17378 applyaddons.go:25] Applying Addons
I0304 17:27:49.012120   17378 clusterclient.go:536] Waiting for kubectl apply...
W0304 17:27:49.172441   17378 clusterclient.go:552] Waiting for kubectl apply... unknown error couldn't kubectl apply, output: Unable to connect to the server: EOF
: exit status 1
I0304 17:27:49.172667   17378 createbootstrapcluster.go:36] Cleaning up bootstrap cluster.
F0304 17:27:49.172686   17378 create_cluster.go:60] unable to apply addons to target cluster: couldn't kubectl apply, output: Unable to connect to the server: EOF
: exit status 1

What did you expect to happen:

I expected a cluster to be created.

Anything else you would like to add:

Environment:

• Cluster-api-provider-aws version:
  Built with bazel off of master 5bf5275
• Kubernetes version: (use kubectl version):

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.7", GitCommit:"0c38c362511b20a098d7cd855f1314dad92c2780", GitTreeState:"clean", BuildDate:"2018-08-20T10:09:03Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.3", GitCommit:"721bfa751924da8d1680787490c54b9179b1fed0", GitTreeState:"clean", BuildDate:"2019-02-23T00:15:17Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}

• OS (e.g. from /etc/os-release):

$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 18.04.2 LTS
Release:	18.04
Codename:	bionic

[vincepri]

I've seen this same issue using an older version of KIND. I'm honestly not sure if it's related but it went away after I rebuilt KIND from master. Does this happen every time? It also seems your kubectl version is a little outdated.

[chuckha]

we were using latest of kind

[vincepri]

Could you try with minikube bootstrapper? At least we'd be able to rule out KIND as possible cause.

[vincepri]

/lifecycle active

Need to update cluster-api dep.