Wrong Internal-IP for Bottlerocket node in IPv6 kubernetes cluster

[zalejus]

Version

Karpenter Version: v0.27.0

Kubernetes Version: v1.25.0

Expected Behavior

The newly provisioned node based on the Bottlerocket OS-Image should register in the IPv6 EKS kubernetes cluster with Internal-IP family IPv6 (like for Amazon Linux 2). Now, unfortunately, it gets IPv4 IP address.

NAME                                              INTERNAL-IP                    OS-IMAGE                                KERNEL-VERSION         CONTAINER-RUNTIME
ip-xxx-xxx-xxx-xxx.eu-central-1.compute.internal  2a05:d014:xxx:xxxx:0:0:0:xxxx  Amazon Linux 2                          5.10.167-147.601.amzn2.aarch64  containerd://1.6.6
ip-xxx-xxx-xxx-xxx.eu-central-1.compute.internal  xxx.xxx.xxx.xxx                Bottlerocket OS 1.12.0 (aws-k8s-1.25)   5.15.79 
containerd://1.6.15+bottlerocket

Actual Behavior

The Bottlerocket node registers in IPv6 kubernetes cluster with IPv4 as Internal-IP

Steps to Reproduce the Problem

Karpenter deployed from chelm v0.27.0
The newest Kubernetes v1.25.0

Resource Specs and Logs

apiVersion: karpenter.sh/v1alpha5
kind: Provisioner
metadata:
  name: default
spec:
  requirements:
    - key: eks.amazonaws.com/capacityType
      operator: In
      values:
        - SPOT
    - key: karpenter.sh/capacity-type
      operator: In
      values:
        - spot
    - key: karpenter.k8s.aws/instance-family
      operator: In
      values:
        - t3
        - t4g
    - key: karpenter.k8s.aws/instance-size
      operator: In
      values:
        - medium
        - large
        - xlarge
        - 2xlarge
    - key: kubernetes.io/arch
      operator: In
      values:
        - amd64
        - arm64
  limits:
    resources:
      cpu: 20
  providerRef:
    name: default
  consolidation:
    enabled: true

apiVersion: karpenter.k8s.aws/v1alpha1
kind: AWSNodeTemplate
metadata:
  name: default
spec:
  metadataOptions:
    httpEndpoint: enabled
    httpProtocolIPv6: enabled
    httpPutResponseHopLimit: 2
    httpTokens: required
  amiFamily: Bottlerocket
#  amiFamily: AL2
  subnetSelector:
    karpenter.sh/discovery: "k8s-dev"
  securityGroupSelector:
    aws-ids: "sg-000b744f7bf6b8189,sg-0e663652d3b0e2694,sg-044aa1c3ce2e5b48c,sg-095500f515910c241,sg-0b0cb67224858aca9"
  blockDeviceMappings:
    - deviceName: /dev/xvda
      ebs:
        volumeType: gp3
        volumeSize: 10Gi
        deleteOnTermination: true
    - deviceName: /dev/xvdb
      ebs:
        volumeType: gp3
        volumeSize: 100Gi # replace with your required disk size
        deleteOnTermination: true

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[njtran]

Could you share your AWSNodeTemplate as well?

[zalejus]

Of course, no problem. I put it into ticket.

…

On Tue, Mar 21, 2023 at 6:14 PM Nick Tran ***@***.***> wrote: Could you share your AWSNodeTemplate as well? — Reply to this email directly, view it on GitHub <#3623 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AKRYDMQC7C2ME67YOA5ZNXDW5HOYPANCNFSM6AAAAAAWCPNJFA> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[bwagner5]

I just tested an IPv6 setup with Bottlerocket and it is registering with an IPv6 Internal IP. It looks like the version of Bottlerocket you're launching is a little older than mine. Can you try using Bottlerocket OS 1.13.1?

> kubectl get nodes -o custom-columns="Name":.metadata.name,"Internal IP":'.status.addresses[0].address',"OS-IMAGE":'.status.nodeInfo.osImage',"CONTAINER_RUNTIME":.status.nodeInfo.containerRuntimeVersion
Name                                           Internal IP                              OS-IMAGE                                CONTAINER_RUNTIME
ip-192-168-30-225.us-east-2.compute.internal   2600:1f16:c2d:ba00:4086:6a03:6b8e:1a58   Amazon Linux 2                          containerd://1.6.6
ip-192-168-53-58.us-east-2.compute.internal    2600:1f16:c2d:ba01:2057:421a:d42b:49fc   Amazon Linux 2                          containerd://1.6.6
ip-192-168-54-111.us-east-2.compute.internal   2600:1f16:c2d:ba01:0:0:0:db75            Bottlerocket OS 1.13.1 (aws-k8s-1.24)   containerd://1.6.19+bottlerocket

[zalejus]

Hi everyone,
I updated karpenter to the newest version v0.27.1 and everything is fine now :) Many thanks for your support.