v2.2.2

Summary

Just a small release that fixes DSR that has been broken on some container runtimes during the v2.X release line. See #1754 for more information.

Changelog

• 5179953 - fix(dsr): change grpc resolver to passthrough <Aaron U'Ren>

v2.2.1

What's Changed

• fix: select ICMP version for common ICMP rules by @qbnit in #1713

New Contributors

• @qbnit made their first contribution in #1713

Full Changelog: v2.2.0...v2.2.1

v2.2.0

Summary

In addition to a whole lot of dependency updates. The major update here is re-adding the missing ip6tables binary that was accidentally removed when we reverted Alpine 3.19 -> Alpine 3.18 as part of the v2.1.X line of releases.

Additionally, this also fixes a number of issues with IPv6 where there was:

• Some hard-coded /32 netmasks still in the code
• Specific IPv4 families hard-coded when looking for local interfaces
• Missing allowances for ICMPv6 neighbor discovery when network policies were applied
• An issue where bogus IPv6 routes were being added when service VIPs were added to the kube-dummy-if interface, causing traffic from the same node to no longer be able to route

Contributions

Thanks @ncopa and @mrueg for contributing to this release!

Also a big thanks to @k6av for reporting some critical issues with IPv6 support.

Changelog

• 5affda2 - feat(goreleaser): update goreleaser-action v5->v6 <Aaron U'Ren>
• a0442e5 - fix: allow basic ICMPv6 neighbor discovery <Aaron U'Ren>
• eac472c - feat(go.mod): update all deps to latest <Aaron U'Ren>
• b1a2fbf - feat(go): update 1.21.7 -> 1.22.3 <Aaron U'Ren>
• 285eb49 - build(deps): bump golang.org/x/net from 0.26.0 to 0.27.0 <dependabot[bot]>
• d091738 - build(deps): bump github.com/aws/aws-sdk-go from 1.55.4 to 1.55.5 <dependabot[bot]>
• 2522b4d - fix(getAllLocalIPs): get IPv6 & IPv4 addresses <Aaron U'Ren>
• 71072c1 - doc(NSC): add extra comments to setupHandlers call <Aaron U'Ren>
• b217e7b - fix(NSC): ensure kube-router owns kube-router-svip <Aaron U'Ren>
• 4f3e706 - build(deps): bump github.com/aws/aws-sdk-go from 1.51.32 to 1.55.4 <dependabot[bot]>
• 1cb9499 - build(deps): bump github.com/docker/docker <dependabot[bot]>
• bab5725 - build(deps): bump golang.org/x/net from 0.25.0 to 0.26.0 <dependabot[bot]>
• e8962dd - fix(linux_networking.go): remove dangling IPv6 routes <Aaron U'Ren>
• 26b539e - fix(feature_request.md): update markdown templating <Aaron U'Ren>
• 84c3549 - fix(bug_report.md): update markdown templating <Aaron U'Ren>
• 94b17b8 - fix(utils.go): static /32 subnet mask reference <Aaron U'Ren>
• 787b906 - build(deps): bump google.golang.org/protobuf from 1.34.0 to 1.34.2 <dependabot[bot]>
• af93779 - build(deps): bump github.com/onsi/gomega from 1.32.0 to 1.33.1 <dependabot[bot]>
• 2ed68ac - build(deps): bump docker/build-push-action from 5 to 6 <dependabot[bot]>
• 4013d21 - fact(Dockerfile): fix linter warning for AS <Aaron U'Ren>
• a2d2e01 - fix(Dockerfile): re-add ip6tables binary <Aaron U'Ren>
• a1125f6 - fix: ensure that ipv6 is not disabled in kernel <Natanael Copa>
• f042d08 - build(deps): bump github.com/hashicorp/go-version from 1.6.0 to 1.7.0 <dependabot[bot]>
• ab8c810 - build(deps): bump golang.org/x/net from 0.24.0 to 0.25.0 <dependabot[bot]>
• a6db642 - build(deps): bump github.com/prometheus/client_golang <dependabot[bot]>

v2.1.3

Summary

Another bugfix release. This one fixes an issue with TCP MSS clamping (e.g. packet MTU's not being properly negotiated during TCP handshake) and an issue where iptables wasn't properly identifying that rules existed in chains causing iptables definitions to increase with every call to sync NetworkPolicy ending up in linear growth of iptables rule definitions.

Contributions

Special thanks to @rkojedzinszky for fixing the TCP MSS clamping on DSR services.

Other Notes

The iptables definition growth was seemingly caused by iptables user-space tooling v1.8.10 which means that this likely only affected users of the kube-router container that was updated to alpine 3.19. User's using kube-router as a daemon outside a container runtime, may want to be wary of updating the iptables user-space to that version.

Changelog

• f6c45f3 - feat(alpine): revert 3.19 -> 3.18 <Aaron U'Ren>
• e980a17 - fix(nsc): remove previous TCPMSS rules during setting up DSR <Richard Kojedzinszky>
• defdf64 - fix(nsc): remove previous TCPMSS rules <Aaron U'Ren>
• b1070f1 - feat(nsc): apply TCPMSS rules on kube-bridge interface only <Richard Kojedzinszky>
• 5fdde06 - fix(nsc): TCPMSS rules are created per-service and for reply packets only <Richard Kojedzinszky>

v2.1.2

Summary

Fixes a problem where ipsets were not getting tracked correctly across different IP families for dual-stack clusters. This caused instances where communication with the pod would no longer work correctly after a pod, that had a NetworkPolicy applied to it, got a new IP address such as during restarts or the like.

Changelog

• 82f7917 - fix(ipset): reset ipset handler before use <Aaron U'Ren>
• d086841 - fact(ipset): simplify cleanup code by reducing family complexity <Aaron U'Ren>
• 28585f6 - fix(ipset.go): make IP families distinct in ipset handler <Aaron U'Ren>
• 2d2850a - feat(ipset.go): add set type and new line to debug msg <Aaron U'Ren>

v2.1.1

Summary

This release fixes a lot of issues that have been present since the release of v2.0.0. v2.0.0 has been gaining adoption slowly and it has only been recently that users have been reporting some of the issues related with dual-stack configurations.

The big additions for this release include:

• Fixing broken compatibility with iptables-legacy systems experienced with the upstream kube-router container since version v2.1.0 (which is when Alpine 3.19 was introduced). There are now tests to ensure that this type of regression doesn't happen again.
• Fixes IPv6 network policy which has been substantially broken since v2.0.0. When IPv6 network policy was introduced, it was missed that iptables statements need to reference these sets via the inet6 prefix in order to use them correctly. As such, most network policies were not correctly applying.
• kube-router no longer activates the hairpin controller which was introduced in v2.1.0, instead relying on user's correctly configuring their CNI with hairpinMode: true (see https://www.kube-router.io/docs/user-guide/#hairpin-mode for more details)
• Adds fallback logic for referencing rt_tables, the path of which changed in versions of iproute2 v6.5.0 and above. Users that have newer systems, enable DSR, and run kube-router within a container should check to ensure this file is mounted correctly within the kube-router container.
• kube-router no longer tries and fails to enter pods to setup DSR for pods that are not scheduled on it's node
• kube-router no longer tries and fails to setup DSR for pods that are part of the host's network
• Fixes --cleanup-config mode which has been broken since v2.0.0 (please see docs for updated examples of how to run this from within a container)

Special Note for Users that Run Hairpin Mode Enabled Services

If you use hairpin mode either as a service annotation or a CLI parameter to kube-router, we recommend that you check your CNI configuration file to ensure that you are setting "hairpinMode":true on the bridge CNI plugin. This is the only way that hairpin mode will work correctly as the previous hairpin controller built into kube-router has now been disabled as it was a tricky implementation and had significant problems with irregular containers.

If you find that you need to add this to your CNI config file, please ensure that kubelet has been restarted, and that any pods that rely on hairpinning have been restarted as well.

See https://www.kube-router.io/docs/user-guide/#hairpin-mode for more information.

Contributions

A big thanks @elchenberg & @xujunjie-cover for contributing fixes for this release!

Changelog

• e42792f - kubeadm-kuberouter-all-features-dsr.yaml: update to include hairpinMode <Aaron U'Ren>
• 317c754 - fix(hairpin): rely on CNI hairpin mode <Aaron U'Ren>
• 9d9b796 - fix(service_endpoints_sync): bail out of DSR when HostNetwork detected <Aaron U'Ren>
• a633849 - feat(NSC.utils): add getPodListForService & getServiceForServiceInfo <Aaron U'Ren>
• b270750 - fact: nsc.getPodObjectForEndpoint -> nsc.getPodObjectForEndpointIP <Aaron U'Ren>
• 567c891 - fix(linux_networking): add more information to errors <Aaron U'Ren>
• e40f46e - fix(user-guide.md): update cleanup example <Aaron U'Ren>
• ecaad2c - fix(cleanup): add missing handlers for cleanup <Aaron U'Ren>
• 7755b4a - fix(node.go): improve logic for GetNodeObject <Aaron U'Ren>
• d12f422 - fix(policy): generate ipv6 names correctly <Aaron U'Ren>
• 2c7151b - fix(policy.go): use new utility method ipSetName <Aaron U'Ren>
• c762eaf - feat(ipset): add more name utilities <Aaron U'Ren>
• ada3179 - fix: wrong ipset name used by ip6tables. <xujunjie-cover>
• b423b1f - feat(NSC): ensure rp_filter is set correctly <Aaron U'Ren>
• af1b07a - fix(service_endpoints_sync.go): error to be indicative of failure type <Aaron U'Ren>
• 421a113 - fix(DSR): setup DSR inside pod on local eps only <Aaron U'Ren>
• 886c1d7 - feat(Dockerfile): use iptables-wrapper go binary <elchenberg>
• 683ef6e - feat(Dockerfile): remove obsolete nsswitch.conf creation <elchenberg>
• c685f2f - feat(Dockerfile): add checks for required binaries <elchenberg>
• b1cc158 - fix(Dockerfile): install iptables-legacy package <elchenberg>
• 4b011db - Fix typo <Jean-Philippe Evrard>
• 5b4975b - build(deps): bump github.com/osrg/gobgp/v3 from 3.23.0 to 3.25.0 <dependabot[bot]>
• f37444f - build(deps): bump golang.org/x/net from 0.22.0 to 0.24.0 <dependabot[bot]>
• 8ab6f49 - build(deps): bump google.golang.org/grpc from 1.62.0 to 1.63.2 <dependabot[bot]>
• bd640c3 - build(deps): bump github.com/aws/aws-sdk-go from 1.51.11 to 1.51.21 <dependabot[bot]>
• 1bae5d5 - build(deps): bump actions/stale from 8 to 9 <dependabot[bot]>
• b1f7b9a - build(deps): bump github.com/stretchr/testify from 1.8.4 to 1.9.0 <dependabot[bot]>
• 58fe139 - build(deps): bump github.com/onsi/gomega from 1.31.1 to 1.32.0 <dependabot[bot]>
• 5c871a5 - build(deps): bump docker/setup-buildx-action from 2 to 3 <dependabot[bot]>
• 5fc2914 - build(deps): bump github.com/aws/aws-sdk-go from 1.51.2 to 1.51.11 <dependabot[bot]>
• a737576 - build(deps): bump docker/login-action from 2 to 3 <dependabot[bot]>
• 260759f - build(deps): bump actions/setup-go from 4 to 5 <dependabot[bot]>
• 1db3438 - fix: rt_tables -> rt-tables in daemonset examples <Aaron U'Ren>
• 7092060 - fix(rt_tables): add path fallback logic <Aaron U'Ren>
• 7f67791 - build(deps): bump github.com/docker/docker <dependabot[bot]>
• 0a2a9d4 - build(deps): bump github/codeql-action from 2 to 3 <dependabot[bot]>
• 614d472 - doc(DSR): add /etc/iproute2/rt_tables caveat <Aaron U'Ren>
• 1909918 - build(deps): bump the k8s-dependencies group with 4 updates <dependabot[bot]>
• f9d1528 - build(deps): bump docker/setup-qemu-action from 2 to 3 <dependabot[bot]>
• 603afce - build(deps): bump google.golang.org/protobuf from 1.32.0 to 1.33.0 <dependabot[bot]>
• 49bde6e - build(deps): bump goreleaser/goreleaser-action from 4 to 5 <dependabot[bot]>
• ef10568 - build(deps): bump docker/build-push-action from 4 to 5 <dependabot[bot]>
• 1811ae8 - build(deps): bump github.com/aws/aws-sdk-go from 1.50.30 to 1.51.2 <dependabot[bot]>
• 4818624 - build(deps): bump golang.org/x/net from 0.21.0 to 0.22.0 <dependabot[bot]>
• 8c5bdbf - build(deps): bump actions/checkout from 3 to 4 <dependabot[bot]>
• 785f814 - dependabot: Group kubernetes dependencies <Manuel Rüger>
• 5bbbd13 - doc(CONTRIBUTING.md): fix relative link <Aaron U'Ren>
• cff45a6 - docs(index.md): improve styling <Aaron U'Ren>

v1.6.1

Just some small fixes for DSR on the v1.6.X release line.

Changelog

• 4d8eecd - feat(ci.yml): don't update latest tag for v1.6 releases <Aaron U'Ren>
• a382475 - fix(rt_tables): add path fallback logic <Aaron U'Ren>

v2.1.0

Summary

The major theme of this kube-router release is to make kube-router fully conformant with the end-to-end upstream sig-network tests. These are tests that the Kubernetes project provides for testing significant networking features and becoming conformant with the tests means that the networking functionality of kube-router is more consistent with other network providers like kube-proxy, Calico, and Cilium.

Major New Features / Fixes

• kube-router's init container now installs CNI plugins when they are missing. The location for these plugins can be controlled by setting the environment variable HOST_BIN_PATH within the init container, but will default to /opt/cni/bin. This is something that all major Kubernetes networking providers do, but kube-router has not done in the past. For more information on CNI plugins, please see: https://github.com/containernetworking/plugins
• kube-router now implements .spec.healthCheckNodePort which has long been a part of the Kubernetes service specification, but kube-router hasn't implemented it until now. This port is meant to give visibility about whether or not an endpoint for a service exists on a node to workloads outside the Kubernetes cluster. If a service endpoint is on the node, then this port returns a 200 HTTP response, otherwise it returns a 503 HTTP response if no endpoint exists on the node. By choosing a node that contains an endpoint, a client can ensure source IP preservation.
• Hairpinning in kube-router is now fixed. kube-router now has a dedicated controller to ensuring that the veth interface inside the container is in promiscuous mode. This used to be handled by kubelet and then it was handled by docker-shim and then it was removed entirely. As far as I can tell, it was never ported to containerd or cri-o. Without this functionality, return traffic ends up getting black-holed before it is routed outside of the container. For more details see: 0f3714b
• Users can now specify --metrics-addr to choose the IP address that kube-router listens on when providing Prometheus metrics
• Users can now specify custom protocol timeouts for IPVS services exposed by kube-router
  • --service-tcp-timeout - (default: 0s preserves system value, typically 900 seconds)
  • --service-tcpfin-timeout - (default: 0s preserves system value, typically 120 seconds)
  • --service-udp-timeout - (default: 0s preserves system value, typically 300 seconds)
• kube-router now abides by the service label service.kubernetes.io/service-proxy-name. Setting this label to something other than kube-router will result in kube-router ignoring the service
• kube-router now honors spec.internalTrafficPolicy and implements spec.externalTrafficPolicy correctly. For more information see: https://kubernetes.io/docs/reference/networking/virtual-ips/#traffic-policies for more information

Backward Compatibility Breaks

kube-router now serves traffic to endpoints that that have a status of either ready or serving. See: https://kubernetes.io/docs/concepts/services-networking/endpoint-slices/#conditions for more information on the differences between these conditions. Before this release, kube-router only served traffic to ready endpoints, which improperly ignored pods that were in Terminating status. This change makes kube-router compliant with the upstream standards.

The release switches from Alpine base 3.18 to 3.19 which changes the iptables backend from iptables to nftables. This is a pretty big change, since the two backends are not mixable. The kube-router project recommends coupling this upgrade with a rolling restart of the nodes in your cluster so that the iptables rules get flushed alongside the upgrade.

Other Notes

• If you're running kube-router in a container and use hairpinning, make sure that hostPID: true is set on your kube-router pod. This is the only way that kube-router is able to properly set the veth interface into promiscuous mode. This attribute has now been added to all of the official kube-router DeaemonSet YAML files

Major Dependency Updates

This release also updates:

• Alpine base container from 3.18 to 3.19
• Go from v1.20 to v1.21
• Kubernetes from v1.28 to v1.29

Contributions

Thanks to:

• @nextsux - For improving overlay tunnel device binding
• @jpiper - For adding the ability to configure an IP to bind metrics to
• @mrueg - For always keeping our dependencies up to date
• @AlexisDucastel - For testing the CNI plugin installation functionality
• @hakman - For helping us improve the upstream e2e tests and providing kOps support and feedback

Changelog (from v2.0.1 - v2.1.0)

• a7d7ac5 - feat(alpine): update 3.18 -> 3.19 <Aaron U'Ren>
• 5b56e6c - feat(goreleaser): update v1.21.2 -> v1.24.0 <Aaron U'Ren>
• a648800 - feat(moq): update v0.3.2 -> v0.3.4 <Aaron U'Ren>
• 46a1b17 - feat(go): upgrade 1.20.13 -> 1.21.7 + dep update <Aaron U'Ren>
• c356289 - feat(lint): update markdownlint v0.37.0 -> 0.39.0 <Aaron U'Ren>
• 47fe189 - feat(lint): update golangci-lint and fix lint errors <Aaron U'Ren>
• 9a136c1 - feat(NSC): implement NodePort Health Check <Aaron U'Ren>
• 7aec8d0 - doc(NSC): add comment for hairpin controller <Aaron U'Ren>
• 959022f - feat(NSC): add endpoint statuses to internal struct <Aaron U'Ren>
• 16daa08 - feat(NSC): add endpoints that are ready or serving <Aaron U'Ren>
• efddb2e - Bump dependencies <Manuel Rüger>
• 3a1efa0 - dual_stack_issue_report.md: remove issue report template <Aaron U'Ren>
• dda7ce6 - doc(cni): CNI requirements and install process <Aaron U'Ren>
• a57a226 - feat(cni): add binaries to host when missing <Aaron U'Ren>
• 98eea79 - doc(user-guide.md): add info about traffic policies <Aaron U'Ren>
• fcd21b4 - feat: fully support service traffic policies <Aaron U'Ren>
• 8404260 - feat: increase unit test coverage <Aaron U'Ren>
• 24505f0 - fact(service_endpoints_sync.go): standardize error handling <Aaron U'Ren>
• d3cf4d1 - feat(NSC): add / clarify log messages <Aaron U'Ren>
• d757f49 - feat(NSC): honor headless label <Aaron U'Ren>
• 8afdee8 - fact(NSC): differentiate headless services <Aaron U'Ren>
• 30d3769 - fact(NSC): update Errorf syntax <Aaron U'Ren>
• a0fe844 - feat(NSC): honor service-proxy-name label <Aaron U'Ren>
• ced5102 - feat(NSC): add IPVS service timeouts <Aaron U'Ren>
• 47290a7 - build(deps): bump k8s.io/client-go from 0.28.3 to 0.28.4 <dependabot[bot]>
• 27f2283 - build(deps): bump k8s.io/cri-api from 0.28.3 to 0.28.4 <dependabot[bot]>
• a6de8cb - build(deps): bump github.com/aws/aws-sdk-go from 1.48.10 to 1.48.16 <dependabot[bot]>
• eb462ba - feat(linux_networking.go): add more error info <Aaron U'Ren>
• eb26288 - fix(manifests): add hostPID for hairpin <Aaron U'Ren>
• aebaa48 - fix(NSC): handle endpoint slice ready nil <Aaron U'Ren>
• 0f3714b - fix(hairpin): set hairpin_mode for veth iface <Aaron U'Ren>
• 3ea293d - build(deps): bump github.com/aws/aws-sdk-go from 1.47.9 to 1.48.10 <dependabot[bot]>
• 4b863e9 - build(deps): bump k8s.io/api from 0.28.3 to 0.28.4 <dependabot[bot]>
• f575d31 - build(deps): bump github.com/osrg/gobgp/v3 from 3.20.0 to 3.21.0 <dependabot[bot]>
• 724588c - build(deps): bump golang.org/x/net from 0.18.0 to 0.19.0 <dependabot[bot]>
• 448f8b6 - build(deps): bump github.com/aws/aws-sdk-go from 1.45.24 to 1.47.9 <dependabot[bot]>
• 0209ac5 - build(deps): bump google.golang.org/grpc from 1.58.3 to 1.59.0 <dependabot[bot]>
• b00a997 - build(deps): bump github.com/onsi/gomega from 1.29.0 to 1.30.0 <dependabot[bot]>
• 595d3e4 - build(deps): bump golang.org/x/net from 0.17.0 to 0.18.0 <dependabot[bot]>
• fcf0ad9 - prometheus metrics: add option to specify listen address <Jason Piper>
• 961c63b - build(deps): bump github.com/onsi/gomega from 1.27.10 to 1.29.0 <dependabot[bot]>
• b13cedc - build(deps): bump k8s.io/client-go from 0.28.2 to 0.28.3 <dependabot[bot]>
• 49314ab - build(deps): bump k8s.io/cri-api from 0.28.2 to 0.28.3 <dependabot[bot]>
• e0861fc - build(deps): bump github.com/osrg/gobgp/v3 from 3.19.0 to 3.20.0 <dependabot[bot]>
• 44d8108 - build(deps): bump github.com/docker/docker <dependabot[bot]>
• 66890d5 - feat: Disable binding overlay tunnels to specific device <Martin -nexus- Mlynář>
• f41defc - build(deps): bump google.golang.org/grpc from 1.58.2 to 1.58.3 <dependabot[bot]>
• 9d6b647 - doc(user-guide): remove bootkube, add k3s <Aaron U'Ren>
• e8be179 - doc(RELEASE.md): add more information <Aaron U'Ren>
• a00778e - feat(.goreleaser.yml): add prerelease and header to release <Aaron U'Ren>
• eb30de9 - lint: add markdownlint to linting <Aaron U'Ren>
• dad4991 - doc: update and format all documentation <Aaron U'Ren>
• 6442689 - doc(user-guide.md): update formatting <Aaron U'Ren>
• dbd741a - doc(introduction.md): update and format <Aaron U'Ren>
• 4379104 - doc(generic.md): update and format <Aaron U'Ren>
• 1957907 - doc(kubeadm.md): update and format <Aaron U'Ren>

v2.0.1

Summary

Fixes a major bug for single stack systems in the service proxy controller. Additionally adds a warning between release v2.0.0 - v2.1.0 about breaking changes.

Changelog

• b36775a - feat(version): add version message mechanism <Aaron U'Ren>
• 503f9fe - fix(NSC): only run for enabled families <Aaron U'Ren>
• 9b0c575 - build(deps): bump golang.org/x/net from 0.16.0 to 0.17.0 <dependabot[bot]>
• 8ccd687 - fix(Makefile): version and build date for v2 <Aaron U'Ren>

v2.0.0

Summary

kube-router v2.0.0 is a major release that the project has been working on for almost a year. We're very excited to finally enable some long-requested features.

Its worth noting, that almost 50% of the existing code base had to be modified in order to bring this release to fruition. We've done testing via the several release candidates that we've released over the last few months (huge thanks to all of the users that engaged that by submitting fixes or reporting bugs), but I'm sure that there are still a few out there that we haven't caught yet.

The project recommends that users approach deploying this release to their production critical clusters with caution. If you do find a bug, please open a new GitHub Issue with the Report a v2.0.0 Release Issue issue type so that we can resolve it as soon as possible.

Major New Features

• Dual-Stack implemented in all kube-router controllers (e.g. --run-firewall, --run-router, --run-loadbalancer, & --run-service-proxy)
• IPAM management for LoadBalancer service types (previously kube-router would only perform BGP, firewall, and proxy functionality and users would need to supplement kube-router with something like MetalLB to get IPs from a pool for their LoadBalancer services)
• FoU tunneling capability added for pod networking (previously only ipip was supported)

Backward Compatibility Breaks

You'll also notice that this has been tagged as a major release version. This is due to the fact that there are breaking changes in this version of kube-router that are not backwards compatible with previous versions of kube-router.

Tunnel names have changed, so if you run kube-router with an overlay network (ipip tunnels), you'll want to deploy this update carefully. At this point, the project recommends doing a rolling-reboot of nodes after the application (or as part of) of this release to ensure that unused tunnels are properly cleaned up and don't have a negative impact on traffic flows.

As part of this work, kube-router now needs to be able to get, list, and watch EndpointSlice objects. Please make sure to update your ClusterRole RBAC before deploying this release if you use kube-router with the --run-service-proxy functionality enabled or you'll get an error. For examples of this RBAC please see the daemonset examples.

Other Notes

Dual-stack handling is fairly complex and it comes with a few gotchas. Users that want to enable dual-stack features should carefully read the IPv6 / Dual-Stack Support in kube-router documentation to ensure that they fully understand the feature set, requirements, and limitations.

If you want to run the new load balancer controller (e.g. --run-loadbalancer) kube-router will need some additional RBAC and requires a few environment variables set. See load-balancer allocation docs for more information.

For more information on enabling or migrating to FoU tunneling and any caveats on doing so, please see the Tunnels in kube-router documentation.

Now that kube-router has dual-stack capability, it doesn't make sense to have an annotation that can only represent a single pod CIDR any longer. As such, with this release we are announcing the deprecation of the kube-router.io/pod-cidr annotation in favor of the new kube-router.io/pod-cidrs annotation see the IPv6 / Dual-Stack Support in kube-router documentation for more information.

Major Dependency Updates

This release also updates:

• Alpine base container from 3.17 to 3.18
• Go from v1.19 to v1.20
• Kubernetes from v1.25 to v1.27
• GoBGP from v3.11.0 to v3.17.0.

Contributions

A release this big would not be possible without an awesome community. Thanks so much to all of the following people that helped play a part in this release: @brandond, @iggy, @whooo, @BoleynSu, @k-raval, @rwagoner, @vadorovsky, @thomasferrandiz, and @mrueg

Changelog (from v1.6.0 - v2.0.0)

1c89be8 - feat(go): upgrade 1.20.7 -> 1.20.9 <Aaron U'Ren>
3db03cc - feat(dep): upgrade all dependencies <Aaron U'Ren>
d2c3858 - feat(goreleaser): upgrade v1.14.1 -> v1.21.2 <Aaron U'Ren>
1a4896f - feat(lint): upgrade golangci-lint v1.50.1 -> v1.54.2 <Aaron U'Ren>
678b712 - fix(ecmp_vip.go): non-local service advertisement <Aaron U'Ren>
1a891c3 - fix(dsr): add family specific link inside pod <Aaron U'Ren>
514a8af - fix(dsr): add family for fwmark <Aaron U'Ren>
c92f76a - fix(service_endpoints_sync.go): use save command <Aaron U'Ren>
9abe20d - fix(NSC): compare all pod IPs for endpoint check <Aaron U'Ren>
9f23cf5 - fix(linux_networking.go): add better error messages <Aaron U'Ren>
7ce09a6 - fix(linux_networking.go): don't return err on warn <Aaron U'Ren>
9d63cc6 - feat(debug): add some extra debug at level 3 <Aaron U'Ren>
4c6e19f - feat(ipset): consolidate ipset usage across controllers <Aaron U'Ren>
c62e1b7 - feat(linux_networking.go): add more logging info <Aaron U'Ren>
da73dea - feat(NSC): use EndpointSlice instead of Endpoints <Aaron U'Ren>
15cd4eb - feat(nsc): add more insight into sync steps <Aaron U'Ren>
81bc9e2 - fix(nsc): don't modify netmask during flag setup <Aaron U'Ren>
903466b - fix(nsc): fail fast during init <Aaron U'Ren>
25ecb09 - feat(nsc): add dualstack capabilities <Aaron U'Ren>
f397a1f - feat: increase log level for save/restore msgs <Aaron U'Ren>
a6cd5b0 - lint(metrics_controller.go): reduce line length <Aaron U'Ren>
68a7d03 - fix: take family metrics out of defer <Aaron U'Ren>
301e856 - fix(NPC): remove redundant assign <Aaron U'Ren>
b06b4f0 - Move ipset restore outside policy loop <Brad Davidson>
e34ef29 - Add additional save/restore metrics <Brad Davidson>
aa107d6 - Make metrics registerer/gathererer replacable <Brad Davidson>
e6f668c - fix: syntax updates for Go 1.20.X and k8s 1.27 <Aaron U'Ren>
5cf1265 - fix(NRC): prevent adding routes with mixed families <Aaron U'Ren>
bab0d4f - feat(bgp_policies.go): don't override-nexthop for internal peers <Aaron U'Ren>
afdf553 - add loadbalancer address allocator <Erik Larsson>
7699d16 - doc(tunnels.md): add info about changing live clusters <Aaron U'Ren>
ddf857d - doc(tunnel): add information about tunnels <Aaron U'Ren>
944ab91 - fix(FoU): make more robust <Aaron U'Ren>
bac4ae6 - fix(FoU): add docs, sanity checking, and logic reduction <Aaron U'Ren>
2a57d6c - Adding FoU encapsulation over IPIP tunnel : added checks for restart and multi-node cases <Kartik Raval>
6ce37e6 - Support for FoU encapsulation for IPIP tunnel <Kartik Raval>
4861021 - fix(NPC): update IPBlocks to be ipFamily specific <Aaron U'Ren>
f0d7f1e - netpol: Fix ipset only containing one IP when port name is used. <Boleyn Su>
384ed97 - fix(bgp_policy): allow for statement add / remove <Aaron U'Ren>
1d5c9ce - fix(ecmp_vip): update VIPs based on svc change <Aaron U'Ren>
f5ac980 - fix(bgp_policies.go): return -> continue on family set evaluation <Aaron U'Ren>
3387f5c - use JoinHostPort for GRPC listen address <Erik Larsson>
76ffcbd - add generation of router id based on hash of primary IP <Erik Larsson>
57c9b08 - fix(ecmp_vip.go): ClusterIP -> ClusterIPs <Aaron U'Ren>
fe93978 - feat(bgp_policies_test.go): use different IP ranges <Aaron U'Ren>
31c22ff - fix(bgp_policies.go): don't get BGP peers twice <Aaron U'Ren>
06f5f8b - feat(go): update package version to /v2 <Aaron U'Ren>
e51ee3a - fix(NPC): add warning for unsupported family <Aaron U'Ren>
4e1679f - fix(NPC): don't add chains for missing family <Aaron U'Ren>
a2bb2ba - doc(bgp.md): clean up grammar and syntax <Aaron U'Ren>
0ecb51d - fix(NPC/pod): check drop policy on ipv4 & ipv6 <Aaron U'Ren>
367aedf - fix(bgp_policies): add empty DS set checking <Aaron U'Ren>
aeb51ba - fact(bgp_policies): rename clusterIPPrefixSet -> serviceVIPIPPrefixSet <Aaron U'Ren>
6e03836 - fact(bgp_policies): abstract get DS for GoBGP <Aaron U'Ren>
67254ad - fix(ecmp_vip): handle ipv4 & ipv6 protocols <Aaron U'Ren>
5f952e0 - test(bgp_policies_test): add local address <Aaron U'Ren>
ec12fda - fix(node): do nil checking on FindBestIP util funcs <Aaron U'Ren>
5d7f62c - fix(NRC): ensure local addr IP is bindable early <Aaron U'Ren>
67abc4b - fix(bgp_peers): adv. AfiSafi based on capabability <Aaron U'Ren>
c491bcb - fix(bgp_peers): do peer only if IP protos match <Aaron U'Ren>
0023ded - fix(NRC): error when nec. host IP not found <Aaron U'Ren>
4f284be - fix(NRC): add IPv6 logic to bgp-local-addresses <Aaron U'Ren>
51f4696 - feat(ci): run CI on version prep branches and MRs <Aaron U'Ren>
b3e0768 - fix(options): make clusterIP specification similar to other options <Aaron U'Ren>
a31511d - fix(NPC): actually separate chain indices for ipv4 / ipv6 <Aaron U'Ren>
096da81 - fact(NPC): pluralize newIPTablesHandler <Aaron U'Ren>
ddb0e63 - feat(NRC): make NRC dual stack <Aaron U'Ren>
01f2ff2 - fact(NRC): convert BGP set names to const <Aaron U'Ren>
85cecb6 - feat(pod_cidr): handle multiple pod CIDRs <Aaron U'Ren>
c18d811 - fix(kube-router.go): metric message -> not error <Aaron U'Ren>
3db482b - fix(NPC): separate chain indices for ipv4 / ipv6 <Aaron U'Ren>
9d2b3c7 - fix(node.go): make node address errors more helpful <Aaron U'Ren>
d7e2a14 - fix golangci issues <Thomas Ferrandiz>
12561f9 - fix test co...