IPs are allocated to ENIs in an inconsistent way

[nckturner]

When IPAMD attempts to meet pod IP demands by allocating ENIs and IP addresses, it allocates new IP addresses and attaches them to the ENI in an inconsistent way. This behavior is easy to reproduce.

# Using https://github.com/aws/aws-k8s-tester:
aws-k8s-tester eks create config --path ./cni-test.yaml

Check some of the config values:

# cni-test.yaml
kubernetes-version: "1.11"
...
worker-node-asg-desired-capacity: 1
worker-node-asg-max: 1
worker-node-asg-min: 1
worker-node-instance-type: m3.xlarge

aws-k8s-tester eks create cluster --path ./cni-test.yaml

After cluster is ready and KUBECONFIG is set to generated kubeconfig file:

k run nginx --image nginx -r 50

Using the aws cli or console, detach an unused IP address, (determine it is unused by any pod by choosing one and cross referencing with kubectl get pod -o wide --all-namespaces.

Then, scale nginx to 60 replicas. The deployment should never get passed 55 pods in RUNNING state, when it should be 56.

[mogren]

We should verify that the reconcile loop in ipamd detects this and removes the detached IP from the data-store.

[mogren]

This has been added in #1177 where we check with EC2 in the reconcile loop in the case where the datastore and IMDS does not match.