Helm chart has non-default dnsPolicy

[fullykubed]

Description

Observed Behavior:

The Karpenter Helm chart has a default dnsPolicy called Default (link). Interestingly, this is not the default dns policy for Kubernetes which is ClusterFirst (link).

I believe the intent with the chart was to use the default Kubernetes DNS settings as I do not believe there is any benefit in overriding the cluster defaults. I have verified ClusterFirst works as expected.

The current default setup is incompatible with clusters with injected sidecars that need ClusterFirst DNS resolution (e.g., service meshes such as Linkerd).

Expected Behavior:

dnsPolicy in the helm chart should be set to ClusterFirst as the default

Reproduction Steps (Please include YAML):

See the Helm chart.

Versions:

• Chart Version: v0.31
• Kubernetes Version (kubectl version): v1.27.4-eks-2d98532

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[jonathan-innis]

This default policy was changed here: #2199. This was originally recommended from #2186 where a user informed us that there's a bit of an ordering problem if you want to scale up your core-dns pods based on Karpenter-managed capacity (since Karpenter would need DNS to be up first in the ClusterFirst mode).

Are you running into any particular issues with the current default DNS policy? I believe you should be able to change this in the helm values if that default isn't working for your specific cluster.

[fullykubed]

Yes, the current setting can cause issues. The most common use case is anytime you have injected sidecar containers (for example, all popular service meshes). These sidecars usually need access to the cluster dns servers and thus will be unable to launch due to this pod-level DNS setting. I can attest to this being the case with Linkerd and Istio.

I understand the issue presented in #2186 . However, it seems like running karpenter prior to even launching dns is going to be a niche setup that is unlikely to impact most users. I'd propose that the default should stay the kubernetes default in order to follow the principle of least surprise.

If someone has a special case like running karpenter prior to their DNS they have the option to configure it. However, the current setting definitely deviates from normal practice and will impact all service mesh users at minimum.

[github-actions]

This issue has been inactive for 14 days. StaleBot will close this stale issue after 14 more days of inactivity.

[jonathan-innis]

I agree with you that we should probably follow the principle of least surprise here. Since using Default is not the default Kubernetes setup, I could see this being surprising when installing the Karpenter deployment.

I'm personally fine with removing the setting in the deployment. I think that we added this to the chart thinking that there were no downsides to doing this. Since changing this would be a breaking change to existing users, we should call this out in the Upgrade docs alongside providing guidance in the "Getting Started" guide around what you should do if you want Karpenter to manage your CoreDNS pods rather than having the CoreDNS pods up before Karpenter starts.