azure: re-enable cloud-provider #84

philips · 2017-03-20T22:17:15Z

Currently the cloud provider is crashing the kubelet. We need to figure out:

What config is it expecting to read?
What is the correct recovery path for nil config? AWS has a nil config recovery path in their plugin entry point.

Mar 20 21:59:57 tectonic-master-000000 kubelet-wrapper[2628]: I0320 21:59:57.218818    2628 server.go:312] acquiring file lock on "/var/run/lock/kubelet.lock"
Mar 20 21:59:57 tectonic-master-000000 kubelet-wrapper[2628]: I0320 21:59:57.219230    2628 server.go:317] watching for inotify events for: /var/run/lock/kubelet.lock
Mar 20 21:59:57 tectonic-master-000000 kubelet-wrapper[2628]: I0320 21:59:57.219609    2628 feature_gate.go:189] feature gates: map[]
[Unit]
Mar 20 21:59:57 tectonic-master-000000 kubelet-wrapper[2628]: panic: runtime error: invalid memory address or nil pointer dereference [recovered]
Mar 20 21:59:57 tectonic-master-000000 kubelet-wrapper[2628]:         panic: runtime error: invalid memory address or nil pointer dereference
Mar 20 21:59:57 tectonic-master-000000 kubelet-wrapper[2628]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0xa7a394]
Mar 20 21:59:57 tectonic-master-000000 kubelet-wrapper[2628]: goroutine 1 [running]:
Mar 20 21:59:57 tectonic-master-000000 kubelet-wrapper[2628]: panic(0x2ea92a0, 0xc420014030)
Mar 20 21:59:57 tectonic-master-000000 kubelet-wrapper[2628]:         /usr/local/go/src/runtime/panic.go:500 +0x1a1
Mar 20 21:59:57 tectonic-master-000000 kubelet-wrapper[2628]: io/ioutil.readAll.func1(0xc4208fe698)
Mar 20 21:59:57 tectonic-master-000000 kubelet-wrapper[2628]:         /usr/local/go/src/io/ioutil/ioutil.go:30 +0x13e
Mar 20 21:59:57 tectonic-master-000000 kubelet-wrapper[2628]: panic(0x2ea92a0, 0xc420014030)
Mar 20 21:59:57 tectonic-master-000000 kubelet-wrapper[2628]:         /usr/local/go/src/runtime/panic.go:458 +0x243
Mar 20 21:59:57 tectonic-master-000000 kubelet-wrapper[2628]: bytes.(*Buffer).ReadFrom(0xc4208fe5e8, 0x0, 0x0, 0xc4202a8c00, 0x0, 0x200)
Mar 20 21:59:57 tectonic-master-000000 kubelet-wrapper[2628]:         /usr/local/go/src/bytes/buffer.go:176 +0x134
Mar 20 21:59:57 tectonic-master-000000 kubelet-wrapper[2628]: io/ioutil.readAll(0x0, 0x0, 0x200, 0x0, 0x0, 0x0, 0x0, 0x0)
Mar 20 21:59:57 tectonic-master-000000 kubelet-wrapper[2628]:         /usr/local/go/src/io/ioutil/ioutil.go:33 +0x147
Mar 20 21:59:57 tectonic-master-000000 kubelet-wrapper[2628]: io/ioutil.ReadAll(0x0, 0x0, 0x5c76860, 0x3435360, 0x5c76860, 0x3435e80, 0x0)
Mar 20 21:59:57 tectonic-master-000000 kubelet-wrapper[2628]:         /usr/local/go/src/io/ioutil/ioutil.go:42 +0x3e
Mar 20 21:59:57 tectonic-master-000000 kubelet-wrapper[2628]: k8s.io/kubernetes/pkg/cloudprovider/providers/azure.NewCloud(0x0, 0x0, 0x7ffebead2f65, 0x5, 0xc4200018f0, 0x1)
Mar 20 21:59:57 tectonic-master-000000 kubelet-wrapper[2628]:         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/cloudprovider/providers/azure
Mar 20 21:59:57 tectonic-master-000000 kubelet-wrapper[2628]: k8s.io/kubernetes/pkg/cloudprovider.GetCloudProvider(0x7ffebead2f65, 0x5, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
Mar 20 21:59:57 tectonic-master-000000 kubelet-wrapper[2628]:         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/cloudprovider/plugins.go:85 +
Mar 20 21:59:57 tectonic-master-000000 kubelet-wrapper[2628]: k8s.io/kubernetes/pkg/cloudprovider.InitCloudProvider(0x7ffebead2f65, 0x5, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
Mar 20 21:59:57 tectonic-master-000000 kubelet-wrapper[2628]:         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/cloudprovider/plugins.go:111
Mar 20 21:59:57 tectonic-master-000000 kubelet-wrapper[2628]: k8s.io/kubernetes/cmd/kubelet/app.run(0xc4207eb800, 0x0, 0x496ec8, 0xc4208ffa08)
Mar 20 21:59:57 tectonic-master-000000 kubelet-wrapper[2628]:         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubelet/app/server.go:362 +0x
Mar 20 21:59:57 tectonic-master-000000 kubelet-wrapper[2628]: k8s.io/kubernetes/cmd/kubelet/app.Run(0xc4207eb800, 0x0, 0xc4208ffa18, 0xc4208ffa28)
Mar 20 21:59:57 tectonic-master-000000 kubelet-wrapper[2628]:         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubelet/app/server.go:270 +0x
Mar 20 21:59:57 tectonic-master-000000 kubelet-wrapper[2628]: main.NewKubelet.func1(0xc420c26870, 0xc42019ef00, 0x0, 0xf, 0x0, 0x0)
Mar 20 21:59:57 tectonic-master-000000 kubelet-wrapper[2628]:         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/hyperkube/kubelet.go:37 +0x33
Mar 20 21:59:57 tectonic-master-000000 kubelet-wrapper[2628]: main.(*HyperKube).Run(0xc42040ac60, 0xc42000a610, 0xf, 0xf, 0x0, 0x0)
Mar 20 21:59:57 tectonic-master-000000 kubelet-wrapper[2628]:         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/hyperkube/hyperkube.go:179 +0
Mar 20 21:59:57 tectonic-master-000000 kubelet-wrapper[2628]: main.(*HyperKube).RunToExit(0xc42040ac60, 0xc42000a600, 0x10, 0x10)
Mar 20 21:59:57 tectonic-master-000000 kubelet-wrapper[2628]:         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/hyperkube/hyperkube.go:189 +0
Mar 20 21:59:57 tectonic-master-000000 kubelet-wrapper[2628]: main.main()
Mar 20 21:59:57 tectonic-master-000000 kubelet-wrapper[2628]:         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/hyperkube/main.go:46 +0x784
Mar 20 21:59:57 tectonic-master-000000 systemd[1]: kubelet.service: Main process exited, code=exited, status=2/INVALIDARGUMENT

The text was updated successfully, but these errors were encountered:

philips · 2017-03-20T22:20:08Z

We have made the azure cloud provider optional for the time being: #83

s-urbaniak · 2017-03-21T15:34:41Z

Upstream PR/discussion: kubernetes/kubernetes#42576

s-urbaniak · 2017-03-21T15:39:58Z

upstream just recently added some documentation as go comments for the expected yaml structure, see https://github.com/kubernetes/kubernetes/blob/af1389e/pkg/cloudprovider/providers/azure/azure.go#L36-L83

I'll try to make this work.

coresolve · 2017-03-22T18:34:16Z

Just hit this as well at seattle hackfest. Trying to create a svc.

NM Just realized I would need to create config as well.

cehoffman · 2017-05-11T15:50:24Z

I have cloud-config generation working for the kubelet, but the bootkube module also needs this configuration (I think). The apiserver has the cloud-provider cli flag, so it would make sense to me it also needs this file.

What is the recommendation for passing this file through? Should it be done as a host mount of the cloud configuration the kubelet is using or stored as a secret and mounted that way?

cehoffman · 2017-05-13T00:26:48Z

I've got the cloud provider working in all the components now. I decided to add the cloud config to the secret config of apiserver and controller-manager and used the host cloud config during the bootstrap manifest phase.

I have an unfortunately rather large PR coming soon for the Azure platform that addresses this issue, #464, #198, ability to use an existing dns zone, #93 (although I removed scale sets because of the limitation), and #39. The reason for the size is I was only recently given the ok from my employer to release the changes.

discordianfish · 2017-06-06T10:18:13Z

I need this fixed too and can help out but first we need to answer @cehoffman's question on where to put this.
@s-urbaniak: Do you have thoughts?

@cehoffman How did you implement this? Can you extract the changes for the cloud provider config? If not, it would be great if you can share your changes as-is, I'm happy to help extracting the bits.

s-urbaniak · 2017-06-06T10:55:19Z

@discordianfish @cehoffman Thanks a lot for helping out, this is much appreciated! My vote for this is to provide the cloud-config via a config map.

This would make the configuration introspectable and potentially changeable during runtime (/cc @Quentin-M @aaronlevy @alexsomesan for thoughts on this).

The canonical location would be in https://github.com/coreos/tectonic-installer/tree/4b8e586/modules/bootkube/resources/manifests and the wiring of the data would be declared in this block

tectonic-installer/modules/bootkube/assets.tf

Lines 52 to 109 in 4b8e586

    
           resource "template_dir" "bootkube" { 
        
             source_dir      = "${path.module}/resources/manifests" 
        
             destination_dir = "./generated/manifests" 
        
             vars { 
        
               hyperkube_image        = "${var.container_images["hyperkube"]}" 
        
               pod_checkpointer_image = "${var.container_images["pod_checkpointer"]}" 
        
               kubedns_image          = "${var.container_images["kubedns"]}" 
        
               kubednsmasq_image      = "${var.container_images["kubednsmasq"]}" 
        
               kubedns_sidecar_image  = "${var.container_images["kubedns_sidecar"]}" 
        
               flannel_image          = "${var.container_images["flannel"]}" 
        
               # Choose the etcd endpoints to use. 
        
               # 1. If experimental mode is enabled (self-hosted etcd), then use 
        
               # var.etcd_service_ip. 
        
               # 2. Else if no etcd TLS certificates are provided, i.e. we bootstrap etcd 
        
               # nodes ourselves (using http), then use insecure http var.etcd_endpoints. 
        
               # 3. Else (if etcd TLS certific are provided), then use the secure https 
        
               # var.etcd_endpoints. 
        
               etcd_servers = "${ 
        
                 var.experimental_enabled  
        
                   ? format("http://%s:2379", cidrhost(var.service_cidr, 15)) 
        
                   : data.null_data_source.etcd.outputs.no_certs 
        
                     ? join(",", formatlist("http://%s:2379", var.etcd_endpoints)) 
        
                     : join(",", formatlist("https://%s:2379", var.etcd_endpoints)) 
        
                 }" 
        
               etcd_ca_flag   = "${data.null_data_source.etcd.outputs.ca_flag}" 
        
               etcd_cert_flag = "${data.null_data_source.etcd.outputs.cert_flag}" 
        
               etcd_key_flag  = "${data.null_data_source.etcd.outputs.key_flag}" 
        
               etcd_service_ip           = "${cidrhost(var.service_cidr, 15)}" 
        
               bootstrap_etcd_service_ip = "${cidrhost(var.service_cidr, 200)}" 
        
               cloud_provider = "${var.cloud_provider}" 
        
               cluster_cidr        = "${var.cluster_cidr}" 
        
               service_cidr        = "${var.service_cidr}" 
        
               kube_dns_service_ip = "${cidrhost(var.service_cidr, 10)}" 
        
               advertise_address   = "${var.advertise_address}" 
        
               anonymous_auth      = "${var.anonymous_auth}" 
        
               oidc_issuer_url     = "${var.oidc_issuer_url}" 
        
               oidc_client_id      = "${var.oidc_client_id}" 
        
               oidc_username_claim = "${var.oidc_username_claim}" 
        
               oidc_groups_claim   = "${var.oidc_groups_claim}" 
        
               ca_cert            = "${base64encode(var.ca_cert == "" ? join(" ", tls_self_signed_cert.kube-ca.*.cert_pem) : var.ca_cert)}" 
        
               apiserver_key      = "${base64encode(tls_private_key.apiserver.private_key_pem)}" 
        
               apiserver_cert     = "${base64encode(tls_locally_signed_cert.apiserver.cert_pem)}" 
        
               serviceaccount_pub = "${base64encode(tls_private_key.service-account.public_key_pem)}" 
        
               serviceaccount_key = "${base64encode(tls_private_key.service-account.private_key_pem)}" 
        
               etcd_ca_cert     = "${base64encode(file(data.null_data_source.etcd.outputs.ca_path))}" 
        
               etcd_client_cert = "${base64encode(file(data.null_data_source.etcd.outputs.cert_path))}" 
        
               etcd_client_key  = "${base64encode(file(data.null_data_source.etcd.outputs.key_path))}" 
        
             } 
        
           }

.

discordianfish · 2017-06-06T11:02:37Z

@s-urbaniak Great, thanks for the pointers. I'll start playing with this now.

cehoffman · 2017-06-06T15:44:37Z

@discordianfish the cloud-config changes are in #690 and isolated to the bootkube module and a new cloud-config module in azure if I remember right

discordianfish · 2017-06-06T16:19:55Z

Ah great, I've just submitted a PR which addresses only the cloud-provider config for now. Still have to test it and probably fix some issues along the way though

philips mentioned this issue Mar 20, 2017

Azure: Make cloud_provider templatable. Set to empty-string for now. #83

Merged

philips added the platform/azure label Mar 20, 2017

s-urbaniak added the tectonic/terraform label Apr 25, 2017

cehoffman mentioned this issue May 15, 2017

Azure features #690

Closed

This was referenced May 16, 2017

vmware: support vsphere cloud provider #51

Closed

platforms/*: install the default storage classes #711

Closed

discordianfish mentioned this issue Jun 6, 2017

changes to bootkube does not update cluster #992

Closed

discordianfish mentioned this issue Jun 6, 2017

Add support for cloud provider config on azure #999

Closed

sym3tri added the migrate-issue label Aug 23, 2017

sym3tri closed this as completed Sep 5, 2017

justaugustus mentioned this issue Nov 21, 2017

[WIP] Reimplement Azure Scale Sets #2429

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

azure: re-enable cloud-provider #84

azure: re-enable cloud-provider #84

philips commented Mar 20, 2017

philips commented Mar 20, 2017

s-urbaniak commented Mar 21, 2017

s-urbaniak commented Mar 21, 2017

coresolve commented Mar 22, 2017 •

edited

Loading

cehoffman commented May 11, 2017

cehoffman commented May 13, 2017

discordianfish commented Jun 6, 2017

s-urbaniak commented Jun 6, 2017

discordianfish commented Jun 6, 2017

cehoffman commented Jun 6, 2017 •

edited

Loading

discordianfish commented Jun 6, 2017

azure: re-enable cloud-provider #84

azure: re-enable cloud-provider #84

Comments

philips commented Mar 20, 2017

philips commented Mar 20, 2017

s-urbaniak commented Mar 21, 2017

s-urbaniak commented Mar 21, 2017

coresolve commented Mar 22, 2017 • edited Loading

cehoffman commented May 11, 2017

cehoffman commented May 13, 2017

discordianfish commented Jun 6, 2017

s-urbaniak commented Jun 6, 2017

discordianfish commented Jun 6, 2017

cehoffman commented Jun 6, 2017 • edited Loading

discordianfish commented Jun 6, 2017

coresolve commented Mar 22, 2017 •

edited

Loading

cehoffman commented Jun 6, 2017 •

edited

Loading