Automatic Instance Group labels for Node Affinity scheduling

[RXminuS]

I love the concept of InstanceGroups because it would allow us to have certain unimportant, stateless, idempotent parts of our infrastructure running on spot instances. However, right now there's no real way for us to only schedule parts of our infrastructure in a specific instance group. For instance I don't want my database pods to end up on the t2.micro or SpotInstance instance groups.

Kubernetes already offers the required constructs by using Node Affinity. However this still requires us to manually label the nodes that Kops has created using kubectl label node. Even though this is possible (but not desirable) in the case where servers are expected to run for a while, it's less than ideal when trying to use SpotInstances. A quick workaround used a simple bash script but it still requires us to maintain a list of nodes and re-run the script every time something changes.

Ideally Kops/Kubernetes would automatically tag nodes with a certain instance-group tag so that it becomes easier to schedule pods to a specific group.

My initial idea was to create a DeamonSet in kubernetes that reads certain node information and assigns it as a label and that Kops makes sure that nodes created in the AutoScaling group get assigned the appropriate data for the DeamonSet to read. But would love some feedback.

[jkemp101test]

I thought there was an issue for this already but I can't find it. I haven't tried it yet but I was going to see if the labelgun project helped by labeling nodes with AWS tags. My hope is it would label with the ASG so I could use node affinity to select them. Might be a temporary workaround until kops label nodes directly.

[justinsb]

I just added the docs on how to do labels (both AWS tags and k8s node labels) at the instance group level. You do have to specify them for each instance group (i.e. there is no "cluster level" labeling yet).

Does this solve the issue? (and if not, what is missing?)

[oO-Zam]

Can the docs be a bit more verbose? where does the nodeLabels value go? top-level? inside the metadata? the spec? lower still?

[chrislovecnm]

@oO-Zam if you are able to contribute please let us know!

[chrislovecnm]

@oO-Zam With the new yaml docs, and the labels docs, I am going to close. Please re-open if you need to.