'Insufficient vpc.amazonaws.com/PrivateIPv4Address' for windows workloads when using existing VPC/Subnets #1512

bcmedeiros · 2019-10-31T03:51:53Z

What happened?

I have created a cluster using existing VPCs and with a windows node group, also running eksctl utils install-vpc-controllers accordingly to the documentation. After running the IIS test pod, the result I get is the following:

FailedScheduling ... Insufficient vpc.amazonaws.com/PrivateIPv4Address

full log

21:43 $ kubectl describe pod windows-server-iis-66bf9745b-sjsjq
Name:           windows-server-iis-66bf9745b-sjsjq
Namespace:      default
Priority:       0
Node:           <none>
Labels:         app=windows-server-iis
                pod-template-hash=66bf9745b
                tier=backend
                track=stable
Annotations:    kubernetes.io/psp: eks.privileged
Status:         Pending
IP:
IPs:            <none>
Controlled By:  ReplicaSet/windows-server-iis-66bf9745b
Containers:
  windows-server-iis:
    Image:      mcr.microsoft.com/windows/servercore:1809
    Port:       80/TCP
    Host Port:  0/TCP
    Command:
      powershell.exe
      -command
      Add-WindowsFeature Web-Server; Invoke-WebRequest -UseBasicParsing -Uri 'https://dotnetbinaries.blob.core.windows.net/servicemonitor/2.0.1.6/ServiceMonitor.exe' -OutFile 'C:\ServiceMonitor.exe'; echo '<html><body><br/><br/><marquee><H1>Hello EKS!!!<H1><marquee></body><html>' > C:\inetpub\wwwroot\default.html; C:\ServiceMonitor.exe 'w3svc';
    Limits:
      vpc.amazonaws.com/PrivateIPv4Address:  1
    Requests:
      vpc.amazonaws.com/PrivateIPv4Address:  1
    Environment:                             <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-fwkxs (ro)
Conditions:
  Type           Status
  PodScheduled   False
Volumes:
  default-token-fwkxs:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-fwkxs
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  beta.kubernetes.io/os=windows
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason            Age                  From               Message
  ----     ------            ----                 ----               -------
  Warning  FailedScheduling  18s (x5 over 6m10s)  default-scheduler  0/3 nodes are available: 2 node(s) didn't match node selector, 3 Insufficient vpc.amazonaws.com/PrivateIPv4Address.

Checking the VPC controller logs, found the the following:

failed to find the route table for subnet subnet-12345678

full log

I1028 16:09:23.456376       1 watcher.go:178] Node watcher processing node ip-10-10-92-124.ec2.internal.
I1028 16:09:23.456411       1 manager.go:109] Node manager adding node ip-10-10-92-124.ec2.internal with instanceID i-0c1544862a42d8885.
E1028 16:09:23.686677       1 manager.go:126] Node manager failed to get resource vpc.amazonaws.com/CIDRBlock pool on node ip-10-10-92-124.ec2.internal: failed to find the route table for subnet subnet-12345678
E1028 16:09:23.686702       1 watcher.go:183] Node watcher failed to add node ip-10-10-92-124.ec2.internal: failed to find the route table for subnet subnet-12345678
E1028 16:09:23.686716       1 watcher.go:266] failed to find the route table for subnet subnet-12345678
W1028 16:09:23.686882       1 watcher.go:267] Node watcher dropping key "ip-10-10-92-124.ec2.internal" out of queue: failed to find the route table for subnet subnet-12345678

My guess is that eksctl is not able to find the route table (which is indeed there as I checked in the web console) because it does not create the VPCs and Subnets, or some permission it would otherwise create is not present.

What you expected to happen?
IIS pod should be correctly scheduled.

How to reproduce it?

Create a cluster with the following config:

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
  name: visagio-internal-apps
  region: us-east-1

vpc:
  id: vpc-33f15f56
  subnets:
    public:
      us-east-1a:
        id: subnet-1111
      us-east-1b:
        id: subnet-2222
      us-east-1c:
        id: subnet-3333
      us-east-1d:
        id: subnet-4444

nodeGroups:
- name: linux-default-1
  instanceType: t3.large
  availabilityZones: ["us-east-1b", "us-east-1d"]
  desiredCapacity: 2
  minSize: 1
  maxSize: 4
  volumeSize: 50
  ssh:
    publicKeyName: bruno.medeiros@brunojcm-notebook
  iam:
    withAddonPolicies:
      ebs: true
      efs: true
- name: windows-default-1
  amiFamily: WindowsServer2019CoreContainer
  instanceType: t3.medium
  availabilityZones: ["us-east-1b"]
  desiredCapacity: 1
  minSize: 1
  maxSize: 2

Install the VPC controller with eksctl utils install-vpc-controllers ...
Try to schedule any Windows workload.

Anything else we need to know?
Nothing special, everything setup based on docs, cluster created from scratch in 1.14 already.

Versions
Please paste in the output of these commands:

$ eksctl version
$ kubectl version

22:00 $ eksctl version
[ℹ]  version.Info{BuiltAt:"", GitCommit:"", GitTag:"0.7.0"}

11:46 $ kubectl version
Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.2", GitCommit:"c97fe5036ef3df2967d086711e6c0c405941e14b", GitTreeState:"clean", BuildDate:"2019-10-15T23:42:50Z", GoVersion:"go1.12.10", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"14+", GitVersion:"v1.14.6-eks-5047ed", GitCommit:"5047edce664593832e9b889e447ac75ab104f527", GitTreeState:"clean", BuildDate:"2019-08-21T22:32:40Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}

Logs
Include the output of the command line when running eksctl. If possible, eksctl should be run with debug logs. For example:
eksctl get clusters -v 4
Make sure you redact any sensitive information before posting.
If the output is long, please consider a Gist.

11:30 $ eksctl get clusters -v 4
2019-10-31T11:47:15+08:00 [▶]  role ARN for the current session is "arn:aws:iam::12345678:user/bruno.medeiros"
NAME			REGION
internal-apps	us-east-1

The text was updated successfully, but these errors were encountered:

sppwf · 2019-11-01T09:00:47Z

Hi,

I have same issue. @brunojcm did you find something? It makes me believe its something wrong with my VPC/Subnets.

Thanks
Sergiu Plotnicu

sppwf · 2019-11-01T10:21:31Z

@brunojcm try to tag subnets/route_tables with "kubetnetes.io/cluster/{cluster_name}" = shared

bcmedeiros · 2019-11-21T03:12:23Z

@sppwf I've just tried that, same problem.

What seems to have worked for me is to create a separated route table and explicitly attach my EKS subnets to it, rather than leaving them in the default routing table. Error changed, I'm recreating my windows worker group and if I don't post here anymore it's because that was probably enough :)

mmsokal · 2020-06-11T11:47:39Z

I am having the same.
Cluster on 1.16

masaeedu · 2020-10-19T17:06:47Z

@brunojcm Could you elaborate on what route table you're referring to? Also, did this end up resolving your problem?

github-actions · 2021-01-18T16:00:18Z

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

github-actions · 2021-01-24T02:58:38Z

This issue was closed because it has been stalled for 5 days with no activity.

bcmedeiros added the kind/bug label Oct 31, 2019

bcmedeiros changed the title ~~Insufficient vpc.amazonaws.com/PrivateIPv4Address for windows workloads when using existing VPC/Subnets~~ 'Insufficient vpc.amazonaws.com/PrivateIPv4Address' for windows workloads when using existing VPC/Subnets Oct 31, 2019

bcmedeiros mentioned this issue Nov 22, 2019

EKS Windows Nodes aws/containers-roadmap#69

Closed

martina-if added the needs-investigation label Apr 14, 2020

github-actions bot added the stale label Jan 18, 2021

michaelbeaumont added awaiting more information and removed awaiting more information labels Jan 18, 2021

github-actions bot closed this as completed Jan 24, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

'Insufficient vpc.amazonaws.com/PrivateIPv4Address' for windows workloads when using existing VPC/Subnets #1512

'Insufficient vpc.amazonaws.com/PrivateIPv4Address' for windows workloads when using existing VPC/Subnets #1512

bcmedeiros commented Oct 31, 2019 •

edited

Loading

sppwf commented Nov 1, 2019

sppwf commented Nov 1, 2019

bcmedeiros commented Nov 21, 2019

mmsokal commented Jun 11, 2020

masaeedu commented Oct 19, 2020

github-actions bot commented Jan 18, 2021

github-actions bot commented Jan 24, 2021

'Insufficient vpc.amazonaws.com/PrivateIPv4Address' for windows workloads when using existing VPC/Subnets #1512

'Insufficient vpc.amazonaws.com/PrivateIPv4Address' for windows workloads when using existing VPC/Subnets #1512

Comments

bcmedeiros commented Oct 31, 2019 • edited Loading

sppwf commented Nov 1, 2019

sppwf commented Nov 1, 2019

bcmedeiros commented Nov 21, 2019

mmsokal commented Jun 11, 2020

masaeedu commented Oct 19, 2020

github-actions bot commented Jan 18, 2021

github-actions bot commented Jan 24, 2021

bcmedeiros commented Oct 31, 2019 •

edited

Loading