handle multiple internal IPs from aws cloud provider

[jlzhao27]

What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.):

• failed to get node address from cloud provider that matches ip
• aws cloud provider multiple ip

---

Kubernetes version
1.5.2

Feature Request

Hi, I am trying configure a cluster on AWS where my kubelets are running on EC2 instances with multiple interfaces.

I want to force the kubelets to report a specific IP so I am using the node-ip flag when starting my kubelets. For example, on a node with 2 interfaces eth0: IP0 and eth1: IP1, I set my kubelet node-ip: IP1. I would expect my kubelet to come up but instead, I get:

failed to get node address from cloud provider that matches ip: <IP1>

running aws ec2 describe-instances ... shows me that the PrivateIP field on the instance object is actually IP0 but the instance has multiple active network interfaces and IP1 is present in one of the interfaces.

Digging through the code, it looks like the node status reporting already handle multiple ip addresses correctly so this seems enhancement in this file: https://github.com/kubernetes/kubernetes/blob/master/pkg/cloudprovider/providers/aws/aws.go#L963

There is already a TODO for supporting multiple IP addresses, would it be a reasonable change to iterate through the network interfaces returned on the instance and report additional active interfaces?

          "NetworkInterfaces": [
                        {
                            "Status": "in-use",
                            "MacAddress": "redacted",
                            "SourceDestCheck": true,
                            "VpcId": "redacted",
                            "Description": "",
                            "NetworkInterfaceId": "redacted",
                            "PrivateIpAddresses": [
                                {
                                    "PrivateDnsName": "ip-10-0-15-49.ec2.internal",
                                    "Primary": true,
                                    "PrivateIpAddress": "10.0.15.49"
                                }
                            ],
                            "PrivateDnsName": "ip-10-0-15-49.ec2.internal",
                            "Attachment": {
                                "Status": "attached",
                                "DeviceIndex": 0,
                                "DeleteOnTermination": true,
                                "AttachmentId": "redacted",
                                "AttachTime": "2017-04-19T19:26:13.000Z"
                            },
                            "Groups": [
                                {
                                    "GroupName": "k8s-master",
                                    "GroupId": "redacted"
                                }
                            ],
                            "SubnetId": "redacted",
                            "OwnerId": "redacted",
                            "PrivateIpAddress": "10.0.15.49"
                        },
...

[jlzhao27]

sample proposed fix: https://github.com/jlz27/kubernetes/pull/1/files

[justinsb]

The change you suggest would certainly be reasonable, but AFAIK kubernetes itself doesn't currently have great support for instances with multiple IPs. What is your reason for multiple IPs?

As an aside, I do think it would be cool to use this functionality of AWS a bit better - it's come up a few times in relation to elastic IPs for services that do IP-based filtering (but there a proxy seems easier). And it would also be nice to build an ingress method using this.

[jlzhao27]

The specific issue I ran into was that we configured separate ENIs for our ec2 instances that were not eth0. Practically as a kubelet, we were still using 1 IP but since it was not the first one reported by the current awsProvider implementation, we were not able to force the kublets to run with the right network settings.

I don't have enough context to comment on the need to support multiple EIP kubelets, but the current bug does not allow kublets to request non eth0 IP addrs in the aws provider.

I'm happy to clean up my CR with unit tests and put it up for review if you'd like.