Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Cluster Autoscaler for Exoscale - FailedScheduling #4782

Closed
rpbaquing-cloudmade opened this issue Apr 3, 2022 · 5 comments · Fixed by #5017
Closed

Cluster Autoscaler for Exoscale - FailedScheduling #4782

rpbaquing-cloudmade opened this issue Apr 3, 2022 · 5 comments · Fixed by #5017
Labels
area/cluster-autoscaler kind/bug Categorizes issue or PR as related to a bug.

Comments

@rpbaquing-cloudmade
Copy link

Which component are you using?: cluster-autoscaler

What version of the component are you using?: 9.16.2

Component version: latest

What k8s version are you using (kubectl version)?: 1.23

kubectl version Output
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.4", GitCommit:"e6c093d87ea4cbb530a7b2ae91e54c0842d8308a", GitTreeState:"clean", BuildDate:"2022-02-16T12:30:48Z", GoVersion:"go1.17.6", Compiler:"gc", Platform:"darwin/arm64"}
Server Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.3", GitCommit:"816c97ab8cff8a1c72eccca1026f7820e93e0d25", GitTreeState:"clean", BuildDate:"2022-01-25T21:19:12Z", GoVersion:"go1.17.6", Compiler:"gc", Platform:"linux/amd64"}

What environment is this in?: exoscale

What did you expect to happen?: to deploy cluster-autoscaler

What happened instead?: the pod got stuck in pending status.

Warning  FailedScheduling  79s   default-scheduler  0/1 nodes are available: 1 node(s) didn't match Pod's node affinity/selector.
Warning  FailedScheduling  11s   default-scheduler  0/1 nodes are available: 1 node(s) didn't match Pod's node affinity/selector.

How to reproduce it (as minimally and precisely as possible):

  1. Clone the repository - kubernetes/autoscaler
  2. Go to autoscaler/cluster-autoscaler/cloudprovider/exoscale/examples
  3. Create the secret required for the credentials in Exoscale using the generate-secret.sh script
  4. Apply the manifests cluster-autoscaler-run-on-control-plane.yaml to deploy cluster-autoscaler.

Anything else we need to know?:

I was expecting it will tolerate the taints in the control plane, but I don't think it is the case here.
@rpbaquing-cloudmade rpbaquing-cloudmade added the kind/bug Categorizes issue or PR as related to a bug. label Apr 3, 2022
@skol101
Copy link

skol101 commented Apr 29, 2022

@jbartosik any update on the issue?

@jbartosik
Copy link
Collaborator

Hi, I work on VPA. I also label issues to make it easier to find issues I want to check. I think @MaciekPytel is point of contact for CA

@PhilippeChepy
Copy link
Contributor

Hello,

On my view, the deployment provided as an example is missing a proper toleration field. If you are deploying your cluster with kubeadm, you probably want to update the Deployment for something like that instead:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: cluster-autoscaler
  namespace: kube-system
  labels:
    app: cluster-autoscaler
spec:
  replicas: 1
  selector:
    matchLabels:
      app: cluster-autoscaler
  template:
    metadata:
      labels:
        app: cluster-autoscaler
      annotations:
        prometheus.io/scrape: 'true'
        prometheus.io/port: '8085'
    spec:
      serviceAccountName: cluster-autoscaler
      containers:
        - name: cluster-autoscaler
          image: exoscale/cluster-autoscaler:latest
          imagePullPolicy: "Always"
          resources:
            limits:
              cpu: 100m
              memory: 300Mi
            requests:
              cpu: 100m
              memory: 300Mi
          command:
            - /cluster-autoscaler
            - --cloud-provider=exoscale
            - --stderrthreshold=info
            - --cordon-node-before-terminating
            #- --scale-down-delay-after-add=30s
            #- --scale-down-unneeded-time=30s
            #- --unremovable-node-recheck-timeout=30s
          env:
          - name: EXOSCALE_API_KEY
            valueFrom:
              secretKeyRef:
                key: api-key
                name: exoscale-api-credentials
          - name: EXOSCALE_API_SECRET
            valueFrom:
              secretKeyRef:
                key: api-secret
                name: exoscale-api-credentials
          - name: EXOSCALE_ZONE
            valueFrom:
              secretKeyRef:
                key: api-zone
                name: exoscale-api-credentials
          volumeMounts:
            - name: ssl-certs
              mountPath: /etc/ssl/certs/ca-certificates.crt
              readOnly: true
      volumes:
        - name: ssl-certs
          hostPath:
            path: "/etc/ssl/certs/ca-certificates.crt"
      tolerations:
        - key: "node-role.kubernetes.io/control-plane"
          operator: "Equal"
          value: "true"
          effect: "NoSchedule"

Please note that node-role.kubernetes.io/control-plane was node-role.kubernetes.io/master in previous releases (before kubeadm v1.24.x).

@PhilippeChepy PhilippeChepy mentioned this issue Jul 12, 2022
Merged
@rpbaquing-cloudmade
Copy link
Author

@PhilippeChepy thank you. I redeployed with the updated toleration. However, the pod deployed in control plane is stuck in pending status.

So I tried the other manifest "cluster-autoscaler.yaml" to deploy it in the worker node but the pod is on crashloopbackoff status.

See pod details:

Name:         cluster-autoscaler-6d59f4446f-dzfkj
Namespace:    kube-system
Priority:     0
Node:         pool-ad719-wdnxk/91.92.200.58
Start Time:   Thu, 14 Jul 2022 20:50:04 +0800
Labels:       app=cluster-autoscaler
              pod-template-hash=6d59f4446f
Annotations:  cni.projectcalico.org/containerID: 6be9df61d59cb6a5e3d040b6e69f2723f4ac499677db6a1305684cae243ce79f
              cni.projectcalico.org/podIP: 192.168.230.114/32
              cni.projectcalico.org/podIPs: 192.168.230.114/32
              kubernetes.io/psp: privileged
              prometheus.io/port: 8085
              prometheus.io/scrape: true
Status:       Running
IP:           192.168.230.114
IPs:
  IP:           192.168.230.114
Controlled By:  ReplicaSet/cluster-autoscaler-6d59f4446f
Containers:
  cluster-autoscaler:
    Container ID:  containerd://8d46c9eadcc4644790cf1d98faefa8938f4db2b7c19f02e5e860bc249c2330af
    Image:         k8s.gcr.io/autoscaling/cluster-autoscaler:v1.24.0
    Image ID:      k8s.gcr.io/autoscaling/cluster-autoscaler@sha256:5bd22353ae7f30c9abfaa08189281367ef47ea1b3d09eb13eb26bd13de241e72
    Port:          <none>
    Host Port:     <none>
    Command:
      /cluster-autoscaler
      --cloud-provider=exoscale
      --stderrthreshold=info
      --cordon-node-before-terminating
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    255
      Started:      Thu, 14 Jul 2022 20:53:41 +0800
      Finished:     Thu, 14 Jul 2022 20:53:43 +0800
    Ready:          False
    Restart Count:  5
    Limits:
      cpu:     100m
      memory:  300Mi
    Requests:
      cpu:     100m
      memory:  300Mi
    Environment:
      EXOSCALE_API_KEY:                                    <set to the key 'api-key' in secret 'exoscale-api-credentials'>     Optional: false
      EXOSCALE_API_SECRET:                                 <set to the key 'api-secret' in secret 'exoscale-api-credentials'>  Optional: false
      EXOSCALE_ZONE:                                       <set to the key 'api-zone' in secret 'exoscale-api-credentials'>    Optional: false
      NEW_RELIC_METADATA_KUBERNETES_CLUSTER_NAME:          exoscale
      NEW_RELIC_METADATA_KUBERNETES_NODE_NAME:              (v1:spec.nodeName)
      NEW_RELIC_METADATA_KUBERNETES_NAMESPACE_NAME:        kube-system (v1:metadata.namespace)
      NEW_RELIC_METADATA_KUBERNETES_POD_NAME:              cluster-autoscaler-6d59f4446f-dzfkj (v1:metadata.name)
      NEW_RELIC_METADATA_KUBERNETES_CONTAINER_NAME:        cluster-autoscaler
      NEW_RELIC_METADATA_KUBERNETES_CONTAINER_IMAGE_NAME:  k8s.gcr.io/autoscaling/cluster-autoscaler:v1.24.0
      NEW_RELIC_METADATA_KUBERNETES_DEPLOYMENT_NAME:       cluster-autoscaler
    Mounts:
      /etc/ssl/certs/ca-certificates.crt from ssl-certs (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-vxkb7 (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  ssl-certs:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/ssl/certs/ca-certificates.crt
    HostPathType:  
  kube-api-access-vxkb7:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Guaranteed
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                   From               Message
  ----     ------     ----                  ----               -------
  Normal   Scheduled  5m6s                  default-scheduler  Successfully assigned kube-system/cluster-autoscaler-6d59f4446f-dzfkj to pool-ad719-wdnxk
  Normal   Pulled     5m5s                  kubelet            Successfully pulled image "k8s.gcr.io/autoscaling/cluster-autoscaler:v1.24.0" in 246.001724ms
  Normal   Pulled     4m45s                 kubelet            Successfully pulled image "k8s.gcr.io/autoscaling/cluster-autoscaler:v1.24.0" in 337.525213ms
  Normal   Pulled     4m27s                 kubelet            Successfully pulled image "k8s.gcr.io/autoscaling/cluster-autoscaler:v1.24.0" in 260.107221ms
  Normal   Created    3m57s (x4 over 5m5s)  kubelet            Created container cluster-autoscaler
  Normal   Started    3m57s (x4 over 5m5s)  kubelet            Started container cluster-autoscaler
  Normal   Pulled     3m57s                 kubelet            Successfully pulled image "k8s.gcr.io/autoscaling/cluster-autoscaler:v1.24.0" in 302.832971ms
  Normal   Pulling    3m3s (x5 over 5m6s)   kubelet            Pulling image "k8s.gcr.io/autoscaling/cluster-autoscaler:v1.24.0"
  Normal   Pulled     3m3s                  kubelet            Successfully pulled image "k8s.gcr.io/autoscaling/cluster-autoscaler:v1.24.0" in 243.792548ms
  Warning  BackOff    3s (x22 over 4m43s)   kubelet            Back-off restarting failed container

See logs below:

I0714 12:50:27.160546       1 leaderelection.go:248] attempting to acquire leader lease kube-system/cluster-autoscaler...
I0714 12:50:27.191594       1 leaderelection.go:258] successfully acquired lease kube-system/cluster-autoscaler
W0714 12:50:27.194577       1 warnings.go:70] policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget
W0714 12:50:27.335922       1 warnings.go:70] policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget
F0714 12:50:27.735698       1 log.go:24] exoscale-provider: failed to initialize manager: no Exoscale zone specified
goroutine 90 [running]:
k8s.io/klog/v2.stacks(0x1)
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/klog/v2/klog.go:860 +0x8a
k8s.io/klog/v2.(*loggingT).output(0x741d2c0, 0x3, 0x0, 0xc000788690, 0x1, {0x5e39603?, 0x1?}, 0x453790?, 0x0)
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/klog/v2/klog.go:825 +0x686
k8s.io/klog/v2.(*loggingT).printfDepth(0x741d2c0, 0x0?, 0x0, {0x0, 0x0}, 0x10?, {0xc0000568c0, 0x33}, {0xc000912f00, 0x1, ...})
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/klog/v2/klog.go:630 +0x1f2
k8s.io/klog/v2.(*loggingT).printf(...)
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/klog/v2/klog.go:612
k8s.io/klog/v2.Fatalf(...)
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/klog/v2/klog.go:1516
k8s.io/autoscaler/cluster-autoscaler/cloudprovider/exoscale.fatalf({0x47264f7?, 0x35a25a5?}, {0xc000912f00, 0x1, 0x1})
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/cloudprovider/exoscale/log.go:24 +0xa6
k8s.io/autoscaler/cluster-autoscaler/cloudprovider/exoscale.BuildExoscale({{0x3fe0000000000000, 0x3fe0000000000000, 0x8bb2c97000, 0x1176592e000}, 0xa, 0x0, 0x4e200, 0x0, 0x186a0000000000, 0x0, ...}, ...)
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/cloudprovider/exoscale/exoscale_cloud_provider.go:190 +0x99
k8s.io/autoscaler/cluster-autoscaler/cloudprovider/builder.buildCloudProvider({{0x3fe0000000000000, 0x3fe0000000000000, 0x8bb2c97000, 0x1176592e000}, 0xa, 0x0, 0x4e200, 0x0, 0x186a0000000000, 0x0, ...}, ...)
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/cloudprovider/builder/builder_all.go:97 +0x60b
k8s.io/autoscaler/cluster-autoscaler/cloudprovider/builder.NewCloudProvider({{0x3fe0000000000000, 0x3fe0000000000000, 0x8bb2c97000, 0x1176592e000}, 0xa, 0x0, 0x4e200, 0x0, 0x186a0000000000, 0x0, ...})
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/cloudprovider/builder/cloud_provider_builder.go:45 +0x20e
k8s.io/autoscaler/cluster-autoscaler/core.initializeDefaultOptions(0xc000173638)
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/core/autoscaler.go:105 +0x2d8
k8s.io/autoscaler/cluster-autoscaler/core.NewAutoscaler({{{0x3fe0000000000000, 0x3fe0000000000000, 0x8bb2c97000, 0x1176592e000}, 0xa, 0x0, 0x4e200, 0x0, 0x186a0000000000, 0x0, ...}, ...})
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/core/autoscaler.go:68 +0x36
main.buildAutoscaler({0x4d50520, 0xc00096ce70})
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/main.go:357 +0x505
main.run(0xc0002da400?, {0x4d50520, 0xc00096ce70})
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/main.go:363 +0x65
main.main.func2({0x0?, 0x0?})
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/main.go:475 +0x25
created by k8s.io/client-go/tools/leaderelection.(*LeaderElector).Run
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/tools/leaderelection/leaderelection.go:211 +0x11b

goroutine 1 [select]:
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc0000ab1a0?, {0x4d20dc0, 0xc0007b3b30}, 0x1, 0xc0000ab1a0)
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:167 +0x135
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000427380?, 0x77359400, 0x0, 0x20?, 0x7ffab965d108?)
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x89
k8s.io/apimachinery/pkg/util/wait.Until(...)
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90
k8s.io/client-go/tools/leaderelection.(*LeaderElector).renew(0xc000776a20, {0x4d48d68?, 0xc000427340?})
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/tools/leaderelection/leaderelection.go:268 +0xd0
k8s.io/client-go/tools/leaderelection.(*LeaderElector).Run(0xc000776a20, {0x4d48da0, 0xc000058048})
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/tools/leaderelection/leaderelection.go:212 +0x12f
k8s.io/client-go/tools/leaderelection.RunOrDie({0x4d48da0, 0xc000058048}, {{0x4d4d810, 0xc000976280}, 0x37e11d600, 0x2540be400, 0x77359400, {0xc000442e00, 0x4868440, 0x0}, ...})
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/tools/leaderelection/leaderelection.go:226 +0x94
main.main()
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/main.go:465 +0x82e

goroutine 60 [runnable]:
sigs.k8s.io/json/internal/golang/encoding/json.stateBeginStringOrEmpty(0xc000cd2508?, 0x22?)
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/sigs.k8s.io/json/internal/golang/encoding/json/scanner.go:251 +0x95
sigs.k8s.io/json/internal/golang/encoding/json.checkValid({0xc001120000, 0x53451, 0xc0004ab630?}, 0xc000cd2508)
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/sigs.k8s.io/json/internal/golang/encoding/json/scanner.go:34 +0xc8
sigs.k8s.io/json/internal/golang/encoding/json.Unmarshal({0xc001120000, 0x53451, 0x56000}, {0x45bfe80, 0xc000766000}, {0xc0004ab620, 0x2, 0xc00048a1f8?})
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/sigs.k8s.io/json/internal/golang/encoding/json/decode.go:106 +0x92
sigs.k8s.io/json.UnmarshalCaseSensitivePreserveInts(...)
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/sigs.k8s.io/json/json.go:62
k8s.io/apimachinery/pkg/runtime/serializer/json.(*Serializer).unmarshal(0x0?, {0x4d2e6a0?, 0xc000766000?}, {0xc001120000?, 0xc00048a1f0?, 0x7?}, {0xc001120000?, 0xc000247110?, 0x4d20c40?})
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/apimachinery/pkg/runtime/serializer/json/json.go:258 +0x3a5
k8s.io/apimachinery/pkg/runtime/serializer/json.(*Serializer).Decode(0xc0000a8c30, {0xc001120000, 0x53451, 0x56000}, 0x0, {0x0, 0x0?})
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/apimachinery/pkg/runtime/serializer/json/json.go:206 +0xa6a
k8s.io/apimachinery/pkg/runtime.WithoutVersionDecoder.Decode({{0x4d20d20?, 0xc0000a8c30?}}, {0xc001120000?, 0xc000058048?, 0x46cd2a5?}, 0x3?, {0x0?, 0x0?})
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/apimachinery/pkg/runtime/helper.go:252 +0x55
k8s.io/client-go/rest.Result.Get({{0xc001120000, 0x53451, 0x56000}, {0x0, 0x0, 0x0}, {0xc000956a80, 0x10}, {0x0, 0x0}, ...})
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/rest/request.go:1189 +0x6f
k8s.io/client-go/tools/cache.NewFilteredListWatchFromClient.func1({{{0x0, 0x0}, {0x0, 0x0}}, {0x0, 0x0}, {0x0, 0x0}, 0x0, 0x0, ...})
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/tools/cache/listwatch.go:88 +0x1f8
k8s.io/client-go/tools/cache.(*ListWatch).List(0xc000494d50?, {{{0x0, 0x0}, {0x0, 0x0}}, {0x0, 0x0}, {0x0, 0x0}, 0x0, ...})
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/tools/cache/listwatch.go:106 +0x56
k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch.func1.1.2({{{0x0, 0x0}, {0x0, 0x0}}, {0x0, 0x0}, {0x0, 0x0}, 0x0, 0x0, ...})
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/tools/cache/reflector.go:277 +0x62
k8s.io/client-go/tools/pager.SimplePageFunc.func1({0xc000494e40?, 0x7ffa9264c228?}, {{{0x0, 0x0}, {0x0, 0x0}}, {0x0, 0x0}, {0x0, 0x0}, ...})
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/tools/pager/pager.go:40 +0x57
k8s.io/client-go/tools/pager.(*ListPager).List(0xc0004abfb0, {0x4d48da0, 0xc000058040}, {{{0x0, 0x0}, {0x0, 0x0}}, {0x0, 0x0}, {0x0, ...}, ...})
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/tools/pager/pager.go:92 +0x16b
k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch.func1.1()
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/tools/cache/reflector.go:302 +0x207
created by k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch.func1
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/tools/cache/reflector.go:268 +0x2fb

goroutine 170 [runnable]:
k8s.io/client-go/informers.(*sharedInformerFactory).Start.func2()
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/informers/factory.go:134
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1571 +0x1
created by k8s.io/client-go/informers.(*sharedInformerFactory).Start
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/informers/factory.go:134 +0x1be

goroutine 14 [select]:
go.opencensus.io/stats/view.(*worker).start(0xc000159e00)
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/go.opencensus.io/stats/view/worker.go:276 +0xad
created by go.opencensus.io/stats/view.init.0
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/go.opencensus.io/stats/view/worker.go:34 +0x8d

goroutine 30 [chan receive]:
k8s.io/apimachinery/pkg/watch.(*Broadcaster).loop(0xc0004e1940)
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/apimachinery/pkg/watch/mux.go:247 +0x57
created by k8s.io/apimachinery/pkg/watch.NewLongQueueBroadcaster
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/apimachinery/pkg/watch/mux.go:89 +0x116

goroutine 29 [IO wait]:
internal/poll.runtime_pollWait(0x7ffa928907a8, 0x72)
	/usr/local/go/src/runtime/netpoll.go:302 +0x89
internal/poll.(*pollDesc).wait(0xc0003a8000?, 0x0?, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:83 +0x32
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:88
internal/poll.(*FD).Accept(0xc0003a8000)
	/usr/local/go/src/internal/poll/fd_unix.go:614 +0x22c
net.(*netFD).accept(0xc0003a8000)
	/usr/local/go/src/net/fd_unix.go:172 +0x35
net.(*TCPListener).accept(0xc000344000)
	/usr/local/go/src/net/tcpsock_posix.go:139 +0x28
net.(*TCPListener).Accept(0xc000344000)
	/usr/local/go/src/net/tcpsock.go:288 +0x3d
net/http.(*Server).Serve(0xc0004ee000, {0x4d43c30, 0xc000344000})
	/usr/local/go/src/net/http/server.go:3039 +0x385
net/http.(*Server).ListenAndServe(0xc0004ee000)
	/usr/local/go/src/net/http/server.go:2968 +0x7d
net/http.ListenAndServe(...)
	/usr/local/go/src/net/http/server.go:3222
main.main.func1()
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/main.go:430 +0x29b
created by main.main
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/main.go:417 +0x310

goroutine 88 [IO wait]:
internal/poll.runtime_pollWait(0x7ffa928906b8, 0x72)
	/usr/local/go/src/runtime/netpoll.go:302 +0x89
internal/poll.(*pollDesc).wait(0xc000158680?, 0xc000812000?, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:83 +0x32
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:88
internal/poll.(*FD).Read(0xc000158680, {0xc000812000, 0xc755, 0xc755})
	/usr/local/go/src/internal/poll/fd_unix.go:167 +0x25a
net.(*netFD).Read(0xc000158680, {0xc000812000?, 0xc0000d0340?, 0xc000812561?})
	/usr/local/go/src/net/fd_posix.go:55 +0x29
net.(*conn).Read(0xc00000e028, {0xc000812000?, 0xe2c?, 0xc0000d0340?})
	/usr/local/go/src/net/net.go:183 +0x45
crypto/tls.(*atLeastReader).Read(0xc000880c30, {0xc000812000?, 0x0?, 0x0?})
	/usr/local/go/src/crypto/tls/conn.go:784 +0x3d
bytes.(*Buffer).ReadFrom(0xc00059e978, {0x4d1cb00, 0xc000880c30})
	/usr/local/go/src/bytes/buffer.go:204 +0x98
crypto/tls.(*Conn).readFromUntil(0xc00059e700, {0x4d26100?, 0xc00000e028}, 0xc1f9?)
	/usr/local/go/src/crypto/tls/conn.go:806 +0xe5
crypto/tls.(*Conn).readRecordOrCCS(0xc00059e700, 0x0)
	/usr/local/go/src/crypto/tls/conn.go:613 +0x116
crypto/tls.(*Conn).readRecord(...)
	/usr/local/go/src/crypto/tls/conn.go:581
crypto/tls.(*Conn).Read(0xc00059e700, {0xc00019f000, 0x1000, 0x8beea0?})
	/usr/local/go/src/crypto/tls/conn.go:1284 +0x16f
bufio.(*Reader).Read(0xc000771020, {0xc0005182e0, 0x9, 0x8cd2a2?})
	/usr/local/go/src/bufio/bufio.go:236 +0x1b4
io.ReadAtLeast({0x4d1c940, 0xc000771020}, {0xc0005182e0, 0x9, 0x9}, 0x9)
	/usr/local/go/src/io/io.go:331 +0x9a
io.ReadFull(...)
	/usr/local/go/src/io/io.go:350
golang.org/x/net/http2.readFrameHeader({0xc0005182e0?, 0x9?, 0xc001b82840?}, {0x4d1c940?, 0xc000771020?})
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/golang.org/x/net/http2/frame.go:237 +0x6e
golang.org/x/net/http2.(*Framer).ReadFrame(0xc0005182a0)
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/golang.org/x/net/http2/frame.go:498 +0x95
golang.org/x/net/http2.(*clientConnReadLoop).run(0xc000c85f98)
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/golang.org/x/net/http2/transport.go:2101 +0x130
golang.org/x/net/http2.(*ClientConn).readLoop(0xc00018db00)
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/golang.org/x/net/http2/transport.go:1997 +0x6f
created by golang.org/x/net/http2.(*Transport).newClientConn
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/golang.org/x/net/http2/transport.go:725 +0xa65

goroutine 31 [chan receive]:
k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher.func1()
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/tools/record/event.go:304 +0x73
created by k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/tools/record/event.go:302 +0x8c

goroutine 167 [sync.Cond.Wait]:
sync.runtime_notifyListWait(0xc0007e01c8, 0x0)
	/usr/local/go/src/runtime/sema.go:513 +0x13d
sync.(*Cond).Wait(0x35?)
	/usr/local/go/src/sync/cond.go:56 +0x8c
golang.org/x/net/http2.(*pipe).Read(0xc0007e01b0, {0xc0009ba600, 0x200, 0x200})
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/golang.org/x/net/http2/pipe.go:76 +0xeb
golang.org/x/net/http2.transportResponseBody.Read({0x0?}, {0xc0009ba600?, 0x0?, 0x0?})
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/golang.org/x/net/http2/transport.go:2384 +0x85
encoding/json.(*Decoder).refill(0xc000452140)
	/usr/local/go/src/encoding/json/stream.go:165 +0x17f
encoding/json.(*Decoder).readValue(0xc000452140)
	/usr/local/go/src/encoding/json/stream.go:140 +0xbb
encoding/json.(*Decoder).Decode(0xc000452140, {0x3dbac80, 0xc000ba0030})
	/usr/local/go/src/encoding/json/stream.go:63 +0x78
k8s.io/apimachinery/pkg/util/framer.(*jsonFrameReader).Read(0xc000c023c0, {0xc000cd8000, 0x400, 0x400})
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/apimachinery/pkg/util/framer/framer.go:152 +0x19c
k8s.io/apimachinery/pkg/runtime/serializer/streaming.(*decoder).Decode(0xc0004a2050, 0x0?, {0x4d2fe88, 0xc000b8a240})
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/apimachinery/pkg/runtime/serializer/streaming/streaming.go:77 +0xa7
k8s.io/client-go/rest/watch.(*Decoder).Decode(0xc0004b2000)
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/rest/watch/decoder.go:49 +0x4f
k8s.io/apimachinery/pkg/watch.(*StreamWatcher).receive(0xc000b8a200)
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/apimachinery/pkg/watch/streamwatcher.go:105 +0xe5
created by k8s.io/apimachinery/pkg/watch.NewStreamWatcher
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/apimachinery/pkg/watch/streamwatcher.go:76 +0x130

goroutine 191 [runnable]:
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:71
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:71 +0x85

goroutine 155 [select]:
k8s.io/client-go/tools/cache.(*Reflector).watchHandler(0xc000196460, {0x0?, 0x0?, 0x741cea0?}, {0x4d300e0?, 0xc000b26fc0}, 0xc000c83d88, 0xc00034e2a0, 0xc0004fe240)
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/tools/cache/reflector.go:469 +0x1b0
k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch(0xc000196460, 0xc0004fe240)
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/tools/cache/reflector.go:429 +0x676
k8s.io/client-go/tools/cache.(*Reflector).Run.func1()
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/tools/cache/reflector.go:221 +0x26
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x405ea9?)
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155 +0x3e
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc000182320?, {0x4d20da0, 0xc000c0b310}, 0x1, 0xc0004fe240)
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156 +0xb6
k8s.io/client-go/tools/cache.(*Reflector).Run(0xc000196460, 0xc0004fe240)
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/tools/cache/reflector.go:220 +0x1c6
created by k8s.io/autoscaler/cluster-autoscaler/utils/kubernetes.NewUnschedulablePodInNamespaceLister
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/utils/kubernetes/listers.go:190 +0x1b3

goroutine 156 [select]:
k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch.func1(0xc000196540, 0xc0007c6510, 0xc0004fe240, 0xc0007c5d88)
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/tools/cache/reflector.go:315 +0x385
k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch(0xc000196540, 0xc0004fe240)
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/tools/cache/reflector.go:361 +0x245
k8s.io/client-go/tools/cache.(*Reflector).Run.func1()
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/tools/cache/reflector.go:221 +0x26
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x0?)
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155 +0x3e
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc000182370?, {0x4d20da0, 0xc000c0b3b0}, 0x1, 0xc0004fe240)
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156 +0xb6
k8s.io/client-go/tools/cache.(*Reflector).Run(0xc000196540, 0xc0004fe240)
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/tools/cache/reflector.go:220 +0x1c6
created by k8s.io/autoscaler/cluster-autoscaler/utils/kubernetes.NewScheduledPodLister
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/utils/kubernetes/listers.go:214 +0x193

goroutine 157 [select]:
k8s.io/client-go/tools/cache.(*Reflector).watchHandler(0xc000196620, {0x0?, 0x0?, 0x741cea0?}, {0x4d300e0?, 0xc000b27080}, 0xc000da9d88, 0xc00034eb40, 0xc0004fe240)
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/tools/cache/reflector.go:469 +0x1b0
k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch(0xc000196620, 0xc0004fe240)
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/tools/cache/reflector.go:429 +0x676
k8s.io/client-go/tools/cache.(*Reflector).Run.func1()
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/tools/cache/reflector.go:221 +0x26
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x0?)
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155 +0x3e
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc0001823c0?, {0x4d20da0, 0xc000c0b450}, 0x1, 0xc0004fe240)
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156 +0xb6
k8s.io/client-go/tools/cache.(*Reflector).Run(0xc000196620, 0xc0004fe240)
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/tools/cache/reflector.go:220 +0x1c6
created by k8s.io/autoscaler/cluster-autoscaler/utils/kubernetes.NewNodeLister
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/utils/kubernetes/listers.go:248 +0x185

goroutine 158 [select]:
k8s.io/client-go/tools/cache.(*Reflector).watchHandler(0xc000196700, {0x0?, 0x0?, 0x741cea0?}, {0x4d300e0?, 0xc0007971c0}, 0xc000b9dd88, 0xc0004592c0, 0xc0004fe240)
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/tools/cache/reflector.go:469 +0x1b0
k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch(0xc000196700, 0xc0004fe240)
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/tools/cache/reflector.go:429 +0x676
k8s.io/client-go/tools/cache.(*Reflector).Run.func1()
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/tools/cache/reflector.go:221 +0x26
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x0?)
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155 +0x3e
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc000182410?, {0x4d20da0, 0xc000c0b4f0}, 0x1, 0xc0004fe240)
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156 +0xb6
k8s.io/client-go/tools/cache.(*Reflector).Run(0xc000196700, 0xc0004fe240)
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/tools/cache/reflector.go:220 +0x1c6
created by k8s.io/autoscaler/cluster-autoscaler/utils/kubernetes.NewNodeLister
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/utils/kubernetes/listers.go:248 +0x185

goroutine 159 [select]:
k8s.io/client-go/tools/cache.(*Reflector).watchHandler(0xc0001967e0, {0x0?, 0x0?, 0x741cea0?}, {0x4d300e0?, 0xc0007867c0}, 0xc000113d88, 0xc000771b60, 0xc0004fe240)
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/tools/cache/reflector.go:469 +0x1b0
k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch(0xc0001967e0, 0xc0004fe240)
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/tools/cache/reflector.go:429 +0x676
k8s.io/client-go/tools/cache.(*Reflector).Run.func1()
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/tools/cache/reflector.go:221 +0x26
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x0?)
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155 +0x3e
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc000182460?, {0x4d20da0, 0xc000c0b590}, 0x1, 0xc0004fe240)
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156 +0xb6
k8s.io/client-go/tools/cache.(*Reflector).Run(0xc0001967e0, 0xc0004fe240)
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/tools/cache/reflector.go:220 +0x1c6
created by k8s.io/autoscaler/cluster-autoscaler/utils/kubernetes.NewPodDisruptionBudgetLister
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/utils/kubernetes/listers.go:311 +0x185

goroutine 160 [select]:
k8s.io/client-go/tools/cache.(*Reflector).watchHandler(0xc0001968c0, {0x0?, 0x0?, 0x741cea0?}, {0x4d300e0?, 0xc000b8a200}, 0xc0004a7d88, 0xc000459c80, 0xc0004fe240)
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/tools/cache/reflector.go:469 +0x1b0
k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch(0xc0001968c0, 0xc0004fe240)
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/tools/cache/reflector.go:429 +0x676
k8s.io/client-go/tools/cache.(*Reflector).Run.func1()
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/tools/cache/reflector.go:221 +0x26
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x0?)
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155 +0x3e
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc0001824b0?, {0x4d20da0, 0xc000c0b630}, 0x1, 0xc0004fe240)
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156 +0xb6
k8s.io/client-go/tools/cache.(*Reflector).Run(0xc0001968c0, 0xc0004fe240)
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/tools/cache/reflector.go:220 +0x1c6
created by k8s.io/autoscaler/cluster-autoscaler/utils/kubernetes.NewDaemonSetLister
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/utils/kubernetes/listers.go:322 +0x175

goroutine 177 [select]:
k8s.io/client-go/tools/cache.(*Reflector).watchHandler(0xc0001969a0, {0x0?, 0x0?, 0x741cea0?}, {0x4d300e0?, 0xc000b26e80}, 0xc000b9bd88, 0xc000458de0, 0xc0004fe240)
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/tools/cache/reflector.go:469 +0x1b0
k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch(0xc0001969a0, 0xc0004fe240)
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/tools/cache/reflector.go:429 +0x676
k8s.io/client-go/tools/cache.(*Reflector).Run.func1()
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/tools/cache/reflector.go:221 +0x26
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x0?)
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155 +0x3e
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc000182500?, {0x4d20da0, 0xc000c0b6d0}, 0x1, 0xc0004fe240)
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156 +0xb6
k8s.io/client-go/tools/cache.(*Reflector).Run(0xc0001969a0, 0xc0004fe240)
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/tools/cache/reflector.go:220 +0x1c6
created by k8s.io/autoscaler/cluster-autoscaler/utils/kubernetes.NewReplicationControllerLister
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/utils/kubernetes/listers.go:331 +0x185

goroutine 178 [select]:
k8s.io/client-go/tools/cache.(*Reflector).watchHandler(0xc000196a80, {0x0?, 0x0?, 0x741cea0?}, {0x4d300e0?, 0xc000b26f40}, 0xc000dadd88, 0xc000330240, 0xc0004fe240)
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/tools/cache/reflector.go:469 +0x1b0
k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch(0xc000196a80, 0xc0004fe240)
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/tools/cache/reflector.go:429 +0x676
k8s.io/client-go/tools/cache.(*Reflector).Run.func1()
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/tools/cache/reflector.go:221 +0x26
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x0?)
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155 +0x3e
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc000182550?, {0x4d20da0, 0xc000c0b770}, 0x1, 0xc0004fe240)
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156 +0xb6
k8s.io/client-go/tools/cache.(*Reflector).Run(0xc000196a80, 0xc0004fe240)
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/tools/cache/reflector.go:220 +0x1c6
created by k8s.io/autoscaler/cluster-autoscaler/utils/kubernetes.NewJobLister
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/utils/kubernetes/listers.go:340 +0x185

goroutine 179 [select]:
k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch.func1(0xc000196b60, 0xc0009fd8c0, 0xc0004fe240, 0xc0004e7d88)
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/tools/cache/reflector.go:315 +0x385
k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch(0xc000196b60, 0xc0004fe240)
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/tools/cache/reflector.go:361 +0x245
k8s.io/client-go/tools/cache.(*Reflector).Run.func1()
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/tools/cache/reflector.go:221 +0x26
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x0?)
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155 +0x3e
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc0001825a0?, {0x4d20da0, 0xc000c0b810}, 0x1, 0xc0004fe240)
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156 +0xb6
k8s.io/client-go/tools/cache.(*Reflector).Run(0xc000196b60, 0xc0004fe240)
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/tools/cache/reflector.go:220 +0x1c6
created by k8s.io/autoscaler/cluster-autoscaler/utils/kubernetes.NewReplicaSetLister
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/utils/kubernetes/listers.go:349 +0x175

goroutine 180 [select]:
k8s.io/client-go/tools/cache.(*Reflector).watchHandler(0xc000196c40, {0x0?, 0x0?, 0x741cea0?}, {0x4d300e0?, 0xc000b8a2c0}, 0xc0004add88, 0xc0004bc660, 0xc0004fe240)
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/tools/cache/reflector.go:469 +0x1b0
k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch(0xc000196c40, 0xc0004fe240)
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/tools/cache/reflector.go:429 +0x676
k8s.io/client-go/tools/cache.(*Reflector).Run.func1()
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/tools/cache/reflector.go:221 +0x26
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xa?)
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155 +0x3e
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc0001825f0?, {0x4d20da0, 0xc000c0b8b0}, 0x1, 0xc0004fe240)
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156 +0xb6
k8s.io/client-go/tools/cache.(*Reflector).Run(0xc000196c40, 0xc0004fe240)
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/tools/cache/reflector.go:220 +0x1c6
created by k8s.io/autoscaler/cluster-autoscaler/utils/kubernetes.NewStatefulSetLister
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/utils/kubernetes/listers.go:358 +0x175

goroutine 181 [chan receive]:
k8s.io/apimachinery/pkg/watch.(*Broadcaster).loop(0xc000786600)
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/apimachinery/pkg/watch/mux.go:247 +0x57
created by k8s.io/apimachinery/pkg/watch.NewLongQueueBroadcaster
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/apimachinery/pkg/watch/mux.go:89 +0x116

goroutine 182 [chan receive]:
k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher.func1()
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/tools/record/event.go:304 +0x73
created by k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/tools/record/event.go:302 +0x8c

goroutine 209 [select]:
k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch.func2()
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/tools/cache/reflector.go:374 +0x12e
created by k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/tools/cache/reflector.go:368 +0x353

goroutine 95 [select]:
k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch.func2()
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/tools/cache/reflector.go:374 +0x12e
created by k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/tools/cache/reflector.go:368 +0x353

goroutine 163 [runnable]:
sync.(*Map).Load(0x741bfa0, {0x4640e00, 0x4558340})
	/usr/local/go/src/sync/map.go:105 +0xb7
sigs.k8s.io/json/internal/golang/encoding/json.cachedTypeFields({0x4d77878?, 0x4558340})
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/sigs.k8s.io/json/internal/golang/encoding/json/encode.go:1414 +0x4c
sigs.k8s.io/json/internal/golang/encoding/json.(*decodeState).object(0xc0007ce0d0, {0x4558340?, 0xc0008dbde0?, 0x41ed580?})
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/sigs.k8s.io/json/internal/golang/encoding/json/decode.go:706 +0x225
sigs.k8s.io/json/internal/golang/encoding/json.(*decodeState).value(0xc0007ce0d0, {0x4558340?, 0xc0008dbde0?, 0x8?})
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/sigs.k8s.io/json/internal/golang/encoding/json/decode.go:400 +0x45
sigs.k8s.io/json/internal/golang/encoding/json.(*decodeState).object(0xc0007ce0d0, {0x41ed580?, 0xc0008dbdc0?, 0xb?})
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/sigs.k8s.io/json/internal/golang/encoding/json/decode.go:866 +0x1112
sigs.k8s.io/json/internal/golang/encoding/json.(*decodeState).value(0xc0007ce0d0, {0x41ed580?, 0xc0008dbdc0?, 0x8?})
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/sigs.k8s.io/json/internal/golang/encoding/json/decode.go:400 +0x45
sigs.k8s.io/json/internal/golang/encoding/json.(*decodeState).array(0xc0007ce0d0, {0x39158e0?, 0xc00015c1a8?, 0x43fda?})
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/sigs.k8s.io/json/internal/golang/encoding/json/decode.go:595 +0x6e5
sigs.k8s.io/json/internal/golang/encoding/json.(*decodeState).value(0xc0007ce0d0, {0x39158e0?, 0xc00015c1a8?, 0x5?})
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/sigs.k8s.io/json/internal/golang/encoding/json/decode.go:390 +0x7e
sigs.k8s.io/json/internal/golang/encoding/json.(*decodeState).object(0xc0007ce0d0, {0x45bda80?, 0xc00015c150?, 0x0?})
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/sigs.k8s.io/json/internal/golang/encoding/json/decode.go:866 +0x1112
sigs.k8s.io/json/internal/golang/encoding/json.(*decodeState).value(0xc0007ce0d0, {0x45bda80?, 0xc00015c150?, 0xd0?})
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/sigs.k8s.io/json/internal/golang/encoding/json/decode.go:400 +0x45
sigs.k8s.io/json/internal/golang/encoding/json.(*decodeState).unmarshal(0xc0007ce0d0, {0x45bda80?, 0xc00015c150?})
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/sigs.k8s.io/json/internal/golang/encoding/json/decode.go:187 +0x1de
sigs.k8s.io/json/internal/golang/encoding/json.Unmarshal({0xc001020000, 0x38f6d, 0x44000}, {0x45bda80, 0xc00015c150}, {0xc000c87620, 0x2, 0xc000c0e078?})
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/sigs.k8s.io/json/internal/golang/encoding/json/decode.go:112 +0x159
sigs.k8s.io/json.UnmarshalCaseSensitivePreserveInts(...)
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/sigs.k8s.io/json/json.go:62
k8s.io/apimachinery/pkg/runtime/serializer/json.(*Serializer).unmarshal(0xc000c0e078?, {0x4d2d908?, 0xc00015c150?}, {0xc001020000?, 0xc000c0e090?, 0xe?}, {0xc001020000?, 0xc000247110?, 0x4d20c40?})
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/apimachinery/pkg/runtime/serializer/json/json.go:258 +0x3a5
k8s.io/apimachinery/pkg/runtime/serializer/json.(*Serializer).Decode(0xc0000a8c30, {0xc001020000, 0x38f6d, 0x44000}, 0x0, {0x0, 0x0?})
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/apimachinery/pkg/runtime/serializer/json/json.go:206 +0xa6a
k8s.io/apimachinery/pkg/runtime.WithoutVersionDecoder.Decode({{0x4d20d20?, 0xc0000a8c30?}}, {0xc001020000?, 0xc000058048?, 0x46cd2a5?}, 0x3?, {0x0?, 0x0?})
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/apimachinery/pkg/runtime/helper.go:252 +0x55
k8s.io/client-go/rest.Result.Get({{0xc001020000, 0x38f6d, 0x44000}, {0x0, 0x0, 0x0}, {0xc000956a80, 0x10}, {0x0, 0x0}, ...})
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/rest/request.go:1189 +0x6f
k8s.io/client-go/tools/cache.NewFilteredListWatchFromClient.func1({{{0x0, 0x0}, {0x0, 0x0}}, {0x0, 0x0}, {0x0, 0x0}, 0x0, 0x0, ...})
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/tools/cache/listwatch.go:88 +0x1f8
k8s.io/client-go/tools/cache.(*ListWatch).List(0xc0000cdd50?, {{{0x0, 0x0}, {0x0, 0x0}}, {0x0, 0x0}, {0x0, 0x0}, 0x0, ...})
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/tools/cache/listwatch.go:106 +0x56
k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch.func1.1.2({{{0x0, 0x0}, {0x0, 0x0}}, {0x0, 0x0}, {0x0, 0x0}, 0x0, 0x0, ...})
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/tools/cache/reflector.go:277 +0x62
k8s.io/client-go/tools/pager.SimplePageFunc.func1({0x0?, 0x7ffa9250d218?}, {{{0x0, 0x0}, {0x0, 0x0}}, {0x0, 0x0}, {0x0, 0x0}, ...})
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/tools/pager/pager.go:40 +0x57
k8s.io/client-go/tools/pager.(*ListPager).List(0xc00070dfb0, {0x4d48da0, 0xc000058040}, {{{0x0, 0x0}, {0x0, 0x0}}, {0x0, 0x0}, {0x0, ...}, ...})
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/tools/pager/pager.go:92 +0x16b
k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch.func1.1()
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/tools/cache/reflector.go:302 +0x207
created by k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch.func1
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/tools/cache/reflector.go:268 +0x2fb

goroutine 96 [select]:
golang.org/x/net/http2.(*clientStream).writeRequest(0xc000a55980, 0xc00015b100)
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/golang.org/x/net/http2/transport.go:1323 +0x9c9
golang.org/x/net/http2.(*clientStream).doRequest(0xc0004c24e0?, 0x0?)
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/golang.org/x/net/http2/transport.go:1185 +0x1e
created by golang.org/x/net/http2.(*ClientConn).RoundTrip
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/golang.org/x/net/http2/transport.go:1114 +0x30a

goroutine 168 [sync.Cond.Wait]:
sync.runtime_notifyListWait(0xc000358ac8, 0x0)
	/usr/local/go/src/runtime/sema.go:513 +0x13d
sync.(*Cond).Wait(0x7ffab965d108?)
	/usr/local/go/src/sync/cond.go:56 +0x8c
golang.org/x/net/http2.(*pipe).Read(0xc000358ab0, {0xc0009ba800, 0x200, 0x200})
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/golang.org/x/net/http2/pipe.go:76 +0xeb
golang.org/x/net/http2.transportResponseBody.Read({0x0?}, {0xc0009ba800?, 0x0?, 0x0?})
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/golang.org/x/net/http2/transport.go:2384 +0x85
encoding/json.(*Decoder).refill(0xc000452280)
	/usr/local/go/src/encoding/json/stream.go:165 +0x17f
encoding/json.(*Decoder).readValue(0xc000452280)
	/usr/local/go/src/encoding/json/stream.go:140 +0xbb
encoding/json.(*Decoder).Decode(0xc000452280, {0x3dbac80, 0xc000ba0060})
	/usr/local/go/src/encoding/json/stream.go:63 +0x78
k8s.io/apimachinery/pkg/util/framer.(*jsonFrameReader).Read(0xc000c02570, {0xc000cd8400, 0x400, 0x400})
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/apimachinery/pkg/util/framer/framer.go:152 +0x19c
k8s.io/apimachinery/pkg/runtime/serializer/streaming.(*decoder).Decode(0xc0004a2140, 0x0?, {0x4d2fe88, 0xc000b8a300})
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/apimachinery/pkg/runtime/serializer/streaming/streaming.go:77 +0xa7
k8s.io/client-go/rest/watch.(*Decoder).Decode(0xc0004b2040)
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/rest/watch/decoder.go:49 +0x4f
k8s.io/apimachinery/pkg/watch.(*StreamWatcher).receive(0xc000b8a2c0)
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/apimachinery/pkg/watch/streamwatcher.go:105 +0xe5
created by k8s.io/apimachinery/pkg/watch.NewStreamWatcher
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/apimachinery/pkg/watch/streamwatcher.go:76 +0x130

goroutine 188 [select]:
k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch.func2()
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/tools/cache/reflector.go:374 +0x12e
created by k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/tools/cache/reflector.go:368 +0x353

goroutine 259 [select]:
k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch.func2()
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/tools/cache/reflector.go:374 +0x12e
created by k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/tools/cache/reflector.go:368 +0x353

goroutine 187 [sync.Cond.Wait]:
sync.runtime_notifyListWait(0xc000a559c8, 0x0)
	/usr/local/go/src/runtime/sema.go:513 +0x13d
sync.(*Cond).Wait(0x7ffab965d108?)
	/usr/local/go/src/sync/cond.go:56 +0x8c
golang.org/x/net/http2.(*pipe).Read(0xc000a559b0, {0xc000650600, 0x200, 0x200})
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/golang.org/x/net/http2/pipe.go:76 +0xeb
golang.org/x/net/http2.transportResponseBody.Read({0xc0006144e0?}, {0xc000650600?, 0x0?, 0x1?})
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/golang.org/x/net/http2/transport.go:2384 +0x85
encoding/json.(*Decoder).refill(0xc000242b40)
	/usr/local/go/src/encoding/json/stream.go:165 +0x17f
encoding/json.(*Decoder).readValue(0xc000242b40)
	/usr/local/go/src/encoding/json/stream.go:140 +0xbb
encoding/json.(*Decoder).Decode(0xc000242b40, {0x3dbac80, 0xc000344fd8})
	/usr/local/go/src/encoding/json/stream.go:63 +0x78
k8s.io/apimachinery/pkg/util/framer.(*jsonFrameReader).Read(0xc00063e390, {0xc000676000, 0x400, 0x400})
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/apimachinery/pkg/util/framer/framer.go:152 +0x19c
k8s.io/apimachinery/pkg/runtime/serializer/streaming.(*decoder).Decode(0xc000c0b9f0, 0x7ffa9269e898?, {0x4d2fe88, 0xc000786800})
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/apimachinery/pkg/runtime/serializer/streaming/streaming.go:77 +0xa7
k8s.io/client-go/rest/watch.(*Decoder).Decode(0xc00084cc60)
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/rest/watch/decoder.go:49 +0x4f
k8s.io/apimachinery/pkg/watch.(*StreamWatcher).receive(0xc0007867c0)
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/apimachinery/pkg/watch/streamwatcher.go:105 +0xe5
created by k8s.io/apimachinery/pkg/watch.NewStreamWatcher
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/apimachinery/pkg/watch/streamwatcher.go:76 +0x130

goroutine 98 [select]:
k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch.func2()
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/tools/cache/reflector.go:374 +0x12e
created by k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/tools/cache/reflector.go:368 +0x353

goroutine 169 [sleep]:
time.Sleep(0x3b9aca00)
	/usr/local/go/src/runtime/time.go:194 +0x12e
k8s.io/kubernetes/pkg/scheduler/framework/runtime.(*metricsRecorder).run(0xc000b0cc00)
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/kubernetes/pkg/scheduler/framework/runtime/metrics_recorder.go:88 +0x33
created by k8s.io/kubernetes/pkg/scheduler/framework/runtime.newMetricsRecorder
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/kubernetes/pkg/scheduler/framework/runtime/metrics_recorder.go:60 +0x118

goroutine 192 [runnable]:
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:71
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:71 +0x85

goroutine 289 [runnable]:
k8s.io/client-go/tools/cache.(*controller).Run.func1()
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/tools/cache/controller.go:130
created by k8s.io/client-go/tools/cache.(*controller).Run
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/tools/cache/controller.go:130 +0xbe

goroutine 171 [runnable]:
k8s.io/client-go/informers.(*sharedInformerFactory).Start.func2()
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/informers/factory.go:134
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1571 +0x1
created by k8s.io/client-go/informers.(*sharedInformerFactory).Start
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/informers/factory.go:134 +0x1be

goroutine 241 [select]:
golang.org/x/net/http2.(*clientStream).writeRequest(0xc000a54180, 0xc00015a500)
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/golang.org/x/net/http2/transport.go:1323 +0x9c9
golang.org/x/net/http2.(*clientStream).doRequest(0x0?, 0x0?)
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/golang.org/x/net/http2/transport.go:1185 +0x1e
created by golang.org/x/net/http2.(*ClientConn).RoundTrip
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/golang.org/x/net/http2/transport.go:1114 +0x30a

goroutine 257 [select]:
k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch.func2()
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/tools/cache/reflector.go:374 +0x12e
created by k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/tools/cache/reflector.go:368 +0x353

goroutine 189 [select]:
golang.org/x/net/http2.(*clientStream).writeRequest(0xc000358a80, 0xc0007ca800)
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/golang.org/x/net/http2/transport.go:1323 +0x9c9
golang.org/x/net/http2.(*clientStream).doRequest(0x0?, 0x0?)
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/golang.org/x/net/http2/transport.go:1185 +0x1e
created by golang.org/x/net/http2.(*ClientConn).RoundTrip
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/golang.org/x/net/http2/transport.go:1114 +0x30a

goroutine 190 [sync.Cond.Wait]:
sync.runtime_notifyListWait(0xc0004dc648, 0x0)
	/usr/local/go/src/runtime/sema.go:513 +0x13d
sync.(*Cond).Wait(0x35?)
	/usr/local/go/src/sync/cond.go:56 +0x8c
golang.org/x/net/http2.(*pipe).Read(0xc0004dc630, {0xc00045e600, 0x200, 0x200})
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/golang.org/x/net/http2/pipe.go:76 +0xeb
golang.org/x/net/http2.transportResponseBody.Read({0x0?}, {0xc00045e600?, 0x0?, 0xb800000000?})
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/golang.org/x/net/http2/transport.go:2384 +0x85
encoding/json.(*Decoder).refill(0xc0000f3e00)
	/usr/local/go/src/encoding/json/stream.go:165 +0x17f
encoding/json.(*Decoder).readValue(0xc0000f3e00)
	/usr/local/go/src/encoding/json/stream.go:140 +0xbb
encoding/json.(*Decoder).Decode(0xc0000f3e00, {0x3dbac80, 0xc000880c60})
	/usr/local/go/src/encoding/json/stream.go:63 +0x78
k8s.io/apimachinery/pkg/util/framer.(*jsonFrameReader).Read(0xc000b828d0, {0xc000676800, 0x400, 0x400})
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/apimachinery/pkg/util/framer/framer.go:152 +0x19c
k8s.io/apimachinery/pkg/runtime/serializer/streaming.(*decoder).Decode(0xc000c0bc70, 0x203?, {0x4d2fe88, 0xc000797200})
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/apimachinery/pkg/runtime/serializer/streaming/streaming.go:77 +0xa7
k8s.io/client-go/rest/watch.(*Decoder).Decode(0xc0000d0820)
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/rest/watch/decoder.go:49 +0x4f
k8s.io/apimachinery/pkg/watch.(*StreamWatcher).receive(0xc0007971c0)
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/apimachinery/pkg/watch/streamwatcher.go:105 +0xe5
created by k8s.io/apimachinery/pkg/watch.NewStreamWatcher
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/apimachinery/pkg/watch/streamwatcher.go:76 +0x130

goroutine 210 [select]:
golang.org/x/net/http2.(*clientStream).writeRequest(0xc000f3a600, 0xc000f44500)
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/golang.org/x/net/http2/transport.go:1323 +0x9c9
golang.org/x/net/http2.(*clientStream).doRequest(0x0?, 0x0?)
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/golang.org/x/net/http2/transport.go:1185 +0x1e
created by golang.org/x/net/http2.(*ClientConn).RoundTrip
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/golang.org/x/net/http2/transport.go:1114 +0x30a

goroutine 225 [select]:
golang.org/x/net/http2.(*clientStream).writeRequest(0xc000dbe000, 0xc000d9e400)
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/golang.org/x/net/http2/transport.go:1323 +0x9c9
golang.org/x/net/http2.(*clientStream).doRequest(0x0?, 0x0?)
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/golang.org/x/net/http2/transport.go:1185 +0x1e
created by golang.org/x/net/http2.(*ClientConn).RoundTrip
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/golang.org/x/net/http2/transport.go:1114 +0x30a

goroutine 211 [select]:
k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch.func2()
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/tools/cache/reflector.go:374 +0x12e
created by k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/tools/cache/reflector.go:368 +0x353

goroutine 63 [select]:
k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch.func2()
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/tools/cache/reflector.go:374 +0x12e
created by k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/tools/cache/reflector.go:368 +0x353

goroutine 64 [select]:
golang.org/x/net/http2.(*clientStream).writeRequest(0xc0004dc000, 0xc000558a00)
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/golang.org/x/net/http2/transport.go:1323 +0x9c9
golang.org/x/net/http2.(*clientStream).doRequest(0x0?, 0x0?)
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/golang.org/x/net/http2/transport.go:1185 +0x1e
created by golang.org/x/net/http2.(*ClientConn).RoundTrip
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/golang.org/x/net/http2/transport.go:1114 +0x30a

goroutine 258 [select]:
golang.org/x/net/http2.(*clientStream).writeRequest(0xc0004dc600, 0xc000558f00)
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/golang.org/x/net/http2/transport.go:1323 +0x9c9
golang.org/x/net/http2.(*clientStream).doRequest(0x0?, 0x0?)
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/golang.org/x/net/http2/transport.go:1185 +0x1e
created by golang.org/x/net/http2.(*ClientConn).RoundTrip
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/golang.org/x/net/http2/transport.go:1114 +0x30a

goroutine 260 [select]:
golang.org/x/net/http2.(*clientStream).writeRequest(0xc0007e0180, 0xc000559500)
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/golang.org/x/net/http2/transport.go:1323 +0x9c9
golang.org/x/net/http2.(*clientStream).doRequest(0x0?, 0x0?)
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/golang.org/x/net/http2/transport.go:1185 +0x1e
created by golang.org/x/net/http2.(*ClientConn).RoundTrip
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/golang.org/x/net/http2/transport.go:1114 +0x30a

goroutine 261 [sync.Cond.Wait]:
sync.runtime_notifyListWait(0xc0004dc048, 0x0)
	/usr/local/go/src/runtime/sema.go:513 +0x13d
sync.(*Cond).Wait(0x35?)
	/usr/local/go/src/sync/cond.go:56 +0x8c
golang.org/x/net/http2.(*pipe).Read(0xc0004dc030, {0xc000650a00, 0x200, 0x200})
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/golang.org/x/net/http2/pipe.go:76 +0xeb
golang.org/x/net/http2.transportResponseBody.Read({0x0?}, {0xc000650a00?, 0x44b925?, 0x8c5b25?})
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/golang.org/x/net/http2/transport.go:2384 +0x85
encoding/json.(*Decoder).refill(0xc000977e00)
	/usr/local/go/src/encoding/json/stream.go:165 +0x17f
encoding/json.(*Decoder).readValue(0xc000977e00)
	/usr/local/go/src/encoding/json/stream.go:140 +0xbb
encoding/json.(*Decoder).Decode(0xc000977e00, {0x3dbac80, 0xc000c17530})
	/usr/local/go/src/encoding/json/stream.go:63 +0x78
k8s.io/apimachinery/pkg/util/framer.(*jsonFrameReader).Read(0xc0007bc3c0, {0xc0007fe000, 0x400, 0x400})
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/apimachinery/pkg/util/framer/framer.go:152 +0x19c
k8s.io/apimachinery/pkg/runtime/serializer/streaming.(*decoder).Decode(0xc000356230, 0xc00018db50?, {0x4d2fe88, 0xc000b26ec0})
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/apimachinery/pkg/runtime/serializer/streaming/streaming.go:77 +0xa7
k8s.io/client-go/rest/watch.(*Decoder).Decode(0xc00084d5e0)
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/rest/watch/decoder.go:49 +0x4f
k8s.io/apimachinery/pkg/watch.(*StreamWatcher).receive(0xc000b26e80)
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/apimachinery/pkg/watch/streamwatcher.go:105 +0xe5
created by k8s.io/apimachinery/pkg/watch.NewStreamWatcher
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/apimachinery/pkg/watch/streamwatcher.go:76 +0x130

goroutine 262 [sync.Cond.Wait]:
sync.runtime_notifyListWait(0xc000dbe048, 0x0)
	/usr/local/go/src/runtime/sema.go:513 +0x13d
sync.(*Cond).Wait(0x74652e1c320b2801?)
	/usr/local/go/src/sync/cond.go:56 +0x8c
golang.org/x/net/http2.(*pipe).Read(0xc000dbe030, {0xc000650c00, 0x200, 0x200})
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/golang.org/x/net/http2/pipe.go:76 +0xeb
golang.org/x/net/http2.transportResponseBody.Read({0x0?}, {0xc000650c00?, 0x44b925?, 0x8c5b25?})
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/golang.org/x/net/http2/transport.go:2384 +0x85
encoding/json.(*Decoder).refill(0xc000376140)
	/usr/local/go/src/encoding/json/stream.go:165 +0x17f
encoding/json.(*Decoder).readValue(0xc000376140)
	/usr/local/go/src/encoding/json/stream.go:140 +0xbb
encoding/json.(*Decoder).Decode(0xc000376140, {0x3dbac80, 0xc000c17560})
	/usr/local/go/src/encoding/json/stream.go:63 +0x78
k8s.io/apimachinery/pkg/util/framer.(*jsonFrameReader).Read(0xc0007bc480, {0xc0007fe400, 0x400, 0x400})
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/apimachinery/pkg/util/framer/framer.go:152 +0x19c
k8s.io/apimachinery/pkg/runtime/serializer/streaming.(*decoder).Decode(0xc000356280, 0xc00018db50?, {0x4d2fe88, 0xc000b26f80})
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/apimachinery/pkg/runtime/serializer/streaming/streaming.go:77 +0xa7
k8s.io/client-go/rest/watch.(*Decoder).Decode(0xc00084d600)
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/rest/watch/decoder.go:49 +0x4f
k8s.io/apimachinery/pkg/watch.(*StreamWatcher).receive(0xc000b26f40)
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/apimachinery/pkg/watch/streamwatcher.go:105 +0xe5
created by k8s.io/apimachinery/pkg/watch.NewStreamWatcher
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/apimachinery/pkg/watch/streamwatcher.go:76 +0x130

goroutine 263 [sync.Cond.Wait]:
sync.runtime_notifyListWait(0xc000f3a648, 0x0)
	/usr/local/go/src/runtime/sema.go:513 +0x13d
sync.(*Cond).Wait(0x7ffab965d108?)
	/usr/local/go/src/sync/cond.go:56 +0x8c
golang.org/x/net/http2.(*pipe).Read(0xc000f3a630, {0xc000650e00, 0x200, 0x200})
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/golang.org/x/net/http2/pipe.go:76 +0xeb
golang.org/x/net/http2.transportResponseBody.Read({0x0?}, {0xc000650e00?, 0x44b925?, 0x8c5b25?})
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/golang.org/x/net/http2/transport.go:2384 +0x85
encoding/json.(*Decoder).refill(0xc000376280)
	/usr/local/go/src/encoding/json/stream.go:165 +0x17f
encoding/json.(*Decoder).readValue(0xc000376280)
	/usr/local/go/src/encoding/json/stream.go:140 +0xbb
encoding/json.(*Decoder).Decode(0xc000376280, {0x3dbac80, 0xc000c17590})
	/usr/local/go/src/encoding/json/stream.go:63 +0x78
k8s.io/apimachinery/pkg/util/framer.(*jsonFrameReader).Read(0xc0007bc570, {0xc0007fe800, 0x400, 0x400})
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/apimachinery/pkg/util/framer/framer.go:152 +0x19c
k8s.io/apimachinery/pkg/runtime/serializer/streaming.(*decoder).Decode(0xc000356320, 0xc00018db50?, {0x4d2fe88, 0xc000b27000})
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/apimachinery/pkg/runtime/serializer/streaming/streaming.go:77 +0xa7
k8s.io/client-go/rest/watch.(*Decoder).Decode(0xc00084d620)
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/rest/watch/decoder.go:49 +0x4f
k8s.io/apimachinery/pkg/watch.(*StreamWatcher).receive(0xc000b26fc0)
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/apimachinery/pkg/watch/streamwatcher.go:105 +0xe5
created by k8s.io/apimachinery/pkg/watch.NewStreamWatcher
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/apimachinery/pkg/watch/streamwatcher.go:76 +0x130

goroutine 264 [sync.Cond.Wait]:
sync.runtime_notifyListWait(0xc000a541c8, 0x0)
	/usr/local/go/src/runtime/sema.go:513 +0x13d
sync.(*Cond).Wait(0x0?)
	/usr/local/go/src/sync/cond.go:56 +0x8c
golang.org/x/net/http2.(*pipe).Read(0xc000a541b0, {0xc000651000, 0x200, 0x200})
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/golang.org/x/net/http2/pipe.go:76 +0xeb
golang.org/x/net/http2.transportResponseBody.Read({0x0?}, {0xc000651000?, 0x44b925?, 0x8c5b25?})
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/golang.org/x/net/http2/transport.go:2384 +0x85
encoding/json.(*Decoder).refill(0xc0003763c0)
	/usr/local/go/src/encoding/json/stream.go:165 +0x17f
encoding/json.(*Decoder).readValue(0xc0003763c0)
	/usr/local/go/src/encoding/json/stream.go:140 +0xbb
encoding/json.(*Decoder).Decode(0xc0003763c0, {0x3dbac80, 0xc000c175c0})
	/usr/local/go/src/encoding/json/stream.go:63 +0x78
k8s.io/apimachinery/pkg/util/framer.(*jsonFrameReader).Read(0xc0007bc630, {0xc0007fec00, 0x400, 0x400})
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/apimachinery/pkg/util/framer/framer.go:152 +0x19c
k8s.io/apimachinery/pkg/runtime/serializer/streaming.(*decoder).Decode(0xc000356370, 0xc00018db50?, {0x4d2fe88, 0xc000b270c0})
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/apimachinery/pkg/runtime/serializer/streaming/streaming.go:77 +0xa7
k8s.io/client-go/rest/watch.(*Decoder).Decode(0xc00084d640)
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/rest/watch/decoder.go:49 +0x4f
k8s.io/apimachinery/pkg/watch.(*StreamWatcher).receive(0xc000b27080)
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/apimachinery/pkg/watch/streamwatcher.go:105 +0xe5
created by k8s.io/apimachinery/pkg/watch.NewStreamWatcher
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/apimachinery/pkg/watch/streamwatcher.go:76 +0x130

goroutine 172 [runnable]:
runtime/debug.Stack()
	/usr/local/go/src/runtime/debug/stack.go:28 +0x4a
k8s.io/apimachinery/pkg/util/naming.extractStackCreator()
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/apimachinery/pkg/util/naming/from_stack.go:83 +0x1d
k8s.io/apimachinery/pkg/util/naming.GetNameFromCallsite({0x73e8e80, 0x1, 0x1})
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/apimachinery/pkg/util/naming/from_stack.go:36 +0x91
k8s.io/client-go/tools/cache.NewReflector({0x4d32ac0, 0xc000ba0900}, {0x467cfa0, 0xc000452f00}, {0x7ffa9259ebe8, 0xc0009b0640}, 0x4432c0?)
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/tools/cache/reflector.go:167 +0x65
k8s.io/client-go/tools/cache.(*controller).Run(0xc000c7f320, 0xc000b886c0)
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/tools/cache/controller.go:134 +0x138
k8s.io/client-go/tools/cache.(*sharedIndexInformer).Run(0xc0004f0be0, 0xc0004fe240?)
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/tools/cache/shared_informer.go:443 +0x47c
created by k8s.io/client-go/informers.(*sharedInformerFactory).Start
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/informers/factory.go:134 +0x1be

goroutine 173 [runnable]:
k8s.io/client-go/informers.(*sharedInformerFactory).Start.func2()
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/informers/factory.go:134
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1571 +0x1
created by k8s.io/client-go/informers.(*sharedInformerFactory).Start
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/informers/factory.go:134 +0x1be

goroutine 174 [runnable]:
k8s.io/client-go/informers.(*sharedInformerFactory).Start.func2()
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/informers/factory.go:134
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1571 +0x1
created by k8s.io/client-go/informers.(*sharedInformerFactory).Start
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/informers/factory.go:134 +0x1be

goroutine 175 [runnable]:
k8s.io/client-go/informers.(*sharedInformerFactory).Start.func2()
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/informers/factory.go:134
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1571 +0x1
created by k8s.io/client-go/informers.(*sharedInformerFactory).Start
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/informers/factory.go:134 +0x1be

goroutine 176 [runnable]:
runtime/debug.Stack()
	/usr/local/go/src/runtime/debug/stack.go:28 +0x4a
k8s.io/apimachinery/pkg/util/naming.extractStackCreator()
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/apimachinery/pkg/util/naming/from_stack.go:83 +0x1d
k8s.io/apimachinery/pkg/util/naming.GetNameFromCallsite({0x73e8e80, 0x1, 0x1})
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/apimachinery/pkg/util/naming/from_stack.go:36 +0x91
k8s.io/client-go/tools/cache.NewReflector({0x4d32ac0, 0xc000ba0720}, {0x4670520, 0xc0001c3b00}, {0x7ffa9259ebe8, 0xc00088e140}, 0x4432c0?)
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/tools/cache/reflector.go:167 +0x65
k8s.io/client-go/tools/cache.(*controller).Run(0xc000683950, 0xc000b886c0)
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/tools/cache/controller.go:134 +0x138
k8s.io/client-go/tools/cache.(*sharedIndexInformer).Run(0xc0004f08c0, 0xc0007867c0?)
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/tools/cache/shared_informer.go:443 +0x47c
created by k8s.io/client-go/informers.(*sharedInformerFactory).Start
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/informers/factory.go:134 +0x1be

goroutine 273 [runnable]:
k8s.io/client-go/informers.(*sharedInformerFactory).Start.func2()
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/informers/factory.go:134
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1571 +0x1
created by k8s.io/client-go/informers.(*sharedInformerFactory).Start
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/informers/factory.go:134 +0x1be

goroutine 274 [runnable]:
k8s.io/client-go/informers.(*sharedInformerFactory).Start.func2()
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/informers/factory.go:134
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1571 +0x1
created by k8s.io/client-go/informers.(*sharedInformerFactory).Start
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/informers/factory.go:134 +0x1be

goroutine 275 [runnable]:
k8s.io/client-go/informers.(*sharedInformerFactory).Start.func2()
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/informers/factory.go:134
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1571 +0x1
created by k8s.io/client-go/informers.(*sharedInformerFactory).Start
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/informers/factory.go:134 +0x1be

goroutine 276 [runnable]:
k8s.io/client-go/informers.(*sharedInformerFactory).Start.func2()
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/informers/factory.go:134
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1571 +0x1
created by k8s.io/client-go/informers.(*sharedInformerFactory).Start
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/informers/factory.go:134 +0x1be

goroutine 277 [runnable]:
k8s.io/client-go/informers.(*sharedInformerFactory).Start.func2()
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/informers/factory.go:134
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1571 +0x1
created by k8s.io/client-go/informers.(*sharedInformerFactory).Start
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/informers/factory.go:134 +0x1be

goroutine 278 [runnable]:
k8s.io/client-go/informers.(*sharedInformerFactory).Start.func2()
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/informers/factory.go:134
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1571 +0x1
created by k8s.io/client-go/informers.(*sharedInformerFactory).Start
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/informers/factory.go:134 +0x1be

goroutine 279 [runnable]:
k8s.io/client-go/informers.(*sharedInformerFactory).Start.func2()
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/informers/factory.go:134
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1571 +0x1
created by k8s.io/client-go/informers.(*sharedInformerFactory).Start
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/informers/factory.go:134 +0x1be

goroutine 242 [runnable]:
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:71
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:71 +0x85

goroutine 243 [runnable]:
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:71
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:71 +0x85

goroutine 244 [runnable]:
k8s.io/client-go/tools/cache.(*controller).Run.func1()
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/tools/cache/controller.go:130
created by k8s.io/client-go/tools/cache.(*controller).Run
	/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/tools/cache/controller.go:130 +0xbe

@PhilippeChepy
Copy link
Contributor

The relevent part of your log is:

F0714 12:50:27.735698       1 log.go:24] exoscale-provider: failed to initialize manager: no Exoscale zone specified

From your Pod description:

    Environment:
      EXOSCALE_API_KEY:                                    <set to the key 'api-key' in secret 'exoscale-api-credentials'>     Optional: false
      EXOSCALE_API_SECRET:                                 <set to the key 'api-secret' in secret 'exoscale-api-credentials'>  Optional: false
      EXOSCALE_ZONE:                                       <set to the key 'api-zone' in secret 'exoscale-api-credentials'>    Optional: false

The error is comming from this potion of code

What is the content of the api-zone key from the 'exoscale-api-credentials' secret ? It must be the zone where your cluster node pools are provisioned. It MUST be something else than an empty string.
The api-key and api-secret must also be defined.

This secret can be set by the generate-secret.sh helper script in our example directory, given a set of environment variables.
The Exoscale community site also outlines this information:

As described there in the README, you need to first create a secret containing an appropriate API key as well as the zone of your cluster:

export EXOSCALE_API_KEY="EXOxxxxxxxxxxxxxxxxxxxxxxxx"
export EXOSCALE_API_SECRET="xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
export EXOSCALE_ZONE="ch-gva-2"

kubectl -n kube-system create secret generic exoscale-api-credentials \
   --from-literal=api-key="$EXOSCALE_API_KEY" \
   --from-literal=api-secret="$EXOSCALE_API_SECRET" \
   --from-literal=api-zone="$EXOSCALE_ZONE"

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/cluster-autoscaler kind/bug Categorizes issue or PR as related to a bug.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

exoscale provider: Update cluster autoscaler documentation
4 participants
@jbartosik @PhilippeChepy @skol101 @rpbaquing-cloudmade