[ACNS] FQDN Filtering Policies

[chasewilson]

GA ETA November 2024*

*ETA's are estimations and subject to change.

Advanced Container Networking Services supports FQDN Filtering

[EppO]

Hi @chasewilson,
is it related to #3797 by any chance?

[TheKangaroo]

@EppO I dont think so. I think it is FQDN filtering based on the new observability tool https://retina.sh .
It was mentioned on the roadmap for retina on the Azure Day at Kubecon Europe:
https://youtu.be/hb__fHnx11s?feature=shared&t=2013

But I would love to hear from @chasewilson if this assumption is correct.

[EppO]

Retina focuses on observability, while FQDN filtering goes beyond this scope. One might expect FQDN filtering policies to be enforced at the CNI level, but this is not currently supported by the NetworkPolicy resource.
Although Cilium has incorporated this feature via their CiliumNetworkPolicy resource, Azure CNI powered by Cilium does not support it yet.
It's unclear whether creating an additional CRD for this feature at the Azure CNI level is more advantageous than using Cilium directly. While it would cover all non-Cilium clusters, advanced users tend to select their CNI based on specific needs and requirements.

[TheKangaroo]

Right, I just saw "FQDN filtering" on the retina roadmap and think it has something to do with ACNS FQDN policies.
I hope @chasewilson can shed some light on this.

[illrill]

Google opted to create its own FQDNNetworkPolicy CRD for this (probably just a wrapper around CiliumNetworkPolicy) in GKE's "Dataplane V2" network mode (which is just their branding of managed Cilium).

Meanwhile, there is also an upcoming FQDN selector in the official Network Policy API: kubernetes-sigs/network-policy-api#200.

It'll be interesting to see how AKS implements this.

[chasewilson]

@TheKangaroo @illrill Keep an eye out for some announcements coming in the next couple months for FQDN filtering. :)

[tamilmani1989]

update: This feature is getting released for public preview this week as part of ACNS offering ACNS. We will share official docs page once it's out.

[EppO]

You can enable FQDN filtering policies on AKS now! (you need aks-preview extension for az):
az aks update --enable-acns -n myAKS -g myRGP and now you can use CiliumNetworkPolicy with DNS-based rules

apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
  name: "fqdn"
spec:
  endpointSelector:
    matchLabels:
      org: empire
      class: mediabot
  egress:
  - toFQDNs:
    - matchPattern: "*.github.com"
  - toEndpoints:
    - matchLabels:
        "k8s:io.kubernetes.pod.namespace": kube-system
        "k8s:k8s-app": kube-dns
    toPorts:
    - ports:
      - port: "53"
        protocol: ANY
      rules:
        dns:
        - matchPattern: "*"

When you add an egress policy, don't forget to allow traffic to CoreDNS in kube-system, otherwise you won't be able to resolve anything.
Please note that L7 rules such as HTTP path filtering are not implemented yet.

[tamilmani1989]

Its released: https://learn.microsoft.com/en-us/azure/aks/advanced-network-container-services-security-cli

[illrill]

Thanks for this @chasewilson @tamilmani1989 & CO! Good work splitting out DNS proxy to its own component for HA 👍

---

We've enabled ACNS in our clusters and have CiliumNetworkPolicy objects successfully using these L3 rule types for egress:

• toFQDNs: ✅ Works
• toEndpoints: ✅ Works
• toEntities: ✅ Works
• toCIDRSet: ✅ Works
• toServices:❔Not tested

Just beware of #4525 (currently, a regular NetworkPolicy with egress rules to port 53 causes the Cilium agent to crash).

[tamilmani1989]

@illrill yes we are aware of that issue. we found the cause and will be rolling out fix soon.

[EppO]

Is there any plans to make the FQDN filtering policy feature support other networking setup? I'm talking about Azure CNI Powered by Cilium without overlay

[chasewilson]

@EppO thanks for the question here.

Great news is this is available for all Cilium compatible CNIs including Azure CNI Overlay, Azure CNI Dynamic IP Allocation, and Azure CNI Static Block Allocation.

[EppO]

@chasewilson: That's good news! I swear I saw this limitation in the AKS docs when it got released but it's indeed now gone. Can't wait for this to go GA!

EDIT: scratch that, the limitation I was mentioning (Overlay mode required) was for Node autoprovisioning