Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: kubeapi watch updates, allow configurable cidr #1075

Merged
merged 17 commits into from
Dec 5, 2024
Merged
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 23 additions & 0 deletions docs/reference/configuration/uds-networking-configuration.md
Original file line number Diff line number Diff line change
@@ -2,6 +2,29 @@
title: Networking Configuration
---

## KubeAPI Egress

The UDS operator is responsible for dynamically updating network policies that use `remoteGenerated: KubeAPI` network policies, in response to changes in the Kubernetes API server’s IP address. This ensures that policies remain accurate as cluster configurations evolve. However, in environments where the API server IP(s) frequently change, this behavior can lead to unnecessary overhead or instability.
mjnagel marked this conversation as resolved.
Show resolved Hide resolved

To address this, the UDS operator provides an option to configure a static CIDR range. This approach eliminates the need for continuous updates by using a predefined range of IP addresses for network policies. To configure a specific CIDR range, set an override to `operator.KUBEAPI_CIDR` in your bundle as a value or variable. For example:

```yaml
packages:
- name: uds-core
repository: ghcr.io/defenseunicorns/packages/uds/core
ref: x.x.x
overrides:
uds-operator-config:
uds-operator-config:
values:
- path: operator.KUBEAPI_CIDR
value: "172.0.0.0/24"
```

This configuration directs the operator to use the specified CIDR range (`172.0.0.0/24` in this case) for KubeAPI network policies instead of dynamically tracking the API server’s IP(s).

When configuring a static CIDR range, it is important to make the range as restrictive as possible to limit the potential for unexpected networking access. An overly broad range could inadvertently allow egress traffic to destinations beyond the intended scope. Additionally, careful alignment with the actual IP addresses used by the Kubernetes API server is essential. A mismatch between the specified CIDR range and the cluster's configuration can result in network policy enforcement issues or disrupted connectivity.

## Additional Network Allowances

Applications deployed in UDS Core utilize [Network Policies](https://kubernetes.io/docs/concepts/services-networking/network-policies/) with a "Deny by Default" configuration to ensure network traffic is restricted to only what is necessary. Some applications in UDS Core allow for overrides to accommodate environment-specific requirements.
3 changes: 3 additions & 0 deletions src/pepr/config.ts
Original file line number Diff line number Diff line change
@@ -31,6 +31,9 @@ export const UDSConfig = {
// Redis URI for Authservice
authserviceRedisUri,

// Static CIDR range to use for KubeAPI instead of k8s watch
kubeApiCidr: process.env.KUBEAPI_CIDR,

// Track if UDS Core identity-authorization layer is deployed
isIdentityDeployed: false,
};
128 changes: 84 additions & 44 deletions src/pepr/operator/controllers/network/generators/kubeAPI.ts
Original file line number Diff line number Diff line change
@@ -6,6 +6,7 @@
import { V1NetworkPolicyPeer } from "@kubernetes/client-node";
import { K8s, kind, R } from "pepr";

import { UDSConfig } from "../../../../config";
import { Component, setupLogger } from "../../../../logger";
import { RemoteGenerated } from "../../../crd";
import { anywhere } from "./anywhere";
@@ -17,19 +18,33 @@ const log = setupLogger(Component.OPERATOR_GENERATORS);
let apiServerPeers: V1NetworkPolicyPeer[];

/**
* Initialize the API server CIDR by getting the EndpointSlice and Service for the API server
* Initialize the API server CIDR.
*
* This function checks if a static CIDR is defined in the configuration.
* If a static CIDR exists, it skips the EndpointSlice lookup and uses the static value.
* Otherwise, it fetches the EndpointSlice and updates the CIDR dynamically.
*/
export async function initAPIServerCIDR() {
const slice = await K8s(kind.EndpointSlice).InNamespace("default").Get("kubernetes");
const svc = await K8s(kind.Service).InNamespace("default").Get("kubernetes");
await updateAPIServerCIDR(slice, svc);

// If static CIDR is defined, pass it directly
if (UDSConfig.kubeApiCidr) {
log.info(
`Static CIDR (${UDSConfig.kubeApiCidr}) is defined for KubeAPI, skipping EndpointSlice lookup.`,
);
await updateAPIServerCIDR(svc, UDSConfig.kubeApiCidr); // Pass static CIDR
} else {
const slice = await K8s(kind.EndpointSlice).InNamespace("default").Get("kubernetes");
await updateAPIServerCIDR(svc, slice);
}
}

/**
* Get the API server CIDR
* @returns The API server CIDR
* Get the API server CIDR.
*
* @returns {V1NetworkPolicyPeer[]} The cached API server CIDR if available; otherwise, defaults to `0.0.0.0/0`.
*/
export function kubeAPI() {
export function kubeAPI(): V1NetworkPolicyPeer[] {
// If the API server peers are already cached, return them
if (apiServerPeers) {
return apiServerPeers;
@@ -41,82 +56,107 @@ export function kubeAPI() {
}

/**
* When the kubernetes EndpointSlice is created or updated, update the API server CIDR
* @param slice The EndpointSlice for the API server
* When the Kubernetes EndpointSlice is created or updated, update the API server CIDR.
*
* @param {kind.EndpointSlice} slice - The EndpointSlice object for the API server.
*/
export async function updateAPIServerCIDRFromEndpointSlice(slice: kind.EndpointSlice) {
try {
log.debug(
"Processing watch for endpointslices, getting k8s service for updating API server CIDR",
);
const svc = await K8s(kind.Service).InNamespace("default").Get("kubernetes");
await updateAPIServerCIDR(slice, svc);
await updateAPIServerCIDR(svc, slice);
} catch (err) {
const msg = "Failed to update network policies from endpoint slice watch";
log.error({ err }, msg);
}
}

/**
* When the kubernetes Service is created or updated, update the API server CIDR
* @param svc The Service for the API server
* When the Kubernetes Service is created or updated, update the API server CIDR.
*
* If a static CIDR is defined, it skips fetching the EndpointSlice and uses the static value.
*
* @param {kind.Service} svc - The Service object for the API server.
*/
export async function updateAPIServerCIDRFromService(svc: kind.Service) {
try {
log.debug(
"Processing watch for api service, getting endpoint slices for updating API server CIDR",
);
const slice = await K8s(kind.EndpointSlice).InNamespace("default").Get("kubernetes");
await updateAPIServerCIDR(slice, svc);
if (UDSConfig.kubeApiCidr) {
log.debug("Processing watch for api service, using configured API CIDR for endpoints");
await updateAPIServerCIDR(svc, UDSConfig.kubeApiCidr);
} else {
log.debug(
"Processing watch for api service, getting endpoint slices for updating API server CIDR",
);
const slice = await K8s(kind.EndpointSlice).InNamespace("default").Get("kubernetes");
await updateAPIServerCIDR(svc, slice);
}
} catch (err) {
const msg = "Failed to update network policies from api service watch";
const msg = "Failed to update network policies from API service watch";
log.error({ err }, msg);
}
}

/**
* Update the API server CIDR and update the NetworkPolicies
* Update the API server CIDR and apply it to the NetworkPolicies.
*
* @param slice The EndpointSlice for the API server
* @param svc The Service for the API server
* @param {kind.Service} svc - The Service object representing the Kubernetes API server.
* @param {kind.EndpointSlice | string} slice - Either the EndpointSlice for dynamic CIDR generation or a static CIDR string.
*/
export async function updateAPIServerCIDR(slice: kind.EndpointSlice, svc: kind.Service) {
const { endpoints } = slice;
export async function updateAPIServerCIDR(svc: kind.Service, slice: kind.EndpointSlice | string) {
const k8sApiIP = svc.spec?.clusterIP;

// Flatten the endpoints into a list of IPs
const peers = endpoints?.flatMap(e => e.addresses);
let peers: string[] = [];

// Handle static CIDR or dynamic EndpointSlice
if (typeof slice === "string") {
peers.push(slice);
} else {
const { endpoints } = slice;
peers = endpoints?.flatMap(e => `${e.addresses}/32`) || [];
mjnagel marked this conversation as resolved.
Show resolved Hide resolved
}

// Add the clusterIP from the service
if (k8sApiIP) {
peers?.push(k8sApiIP);
peers.push(`${k8sApiIP}/32`);
}

// If the peers are found, cache and process them
if (peers?.length) {
apiServerPeers = peers.flatMap(ip => ({
// Convert peers into NetworkPolicyPeer objects
if (peers.length) {
apiServerPeers = peers.flatMap(cidr => ({
ipBlock: {
cidr: `${ip}/32`,
cidr,
noahpb marked this conversation as resolved.
Show resolved Hide resolved
},
}));

// Get all the KubeAPI NetworkPolicies
const netPols = await K8s(kind.NetworkPolicy)
.WithLabel("uds.dev/generated", RemoteGenerated.KubeAPI)
.Get();
// Update NetworkPolicies
await updateNetworkPolicies(apiServerPeers);
} else {
log.warn("No peers found for the API server CIDR update.");
}
}

/**
* Update NetworkPolicies with new API server peers.
*
* @param {V1NetworkPolicyPeer[]} newPeers - The updated list of peers to apply to the NetworkPolicies.
*/
async function updateNetworkPolicies(newPeers: V1NetworkPolicyPeer[]) {
const netPols = await K8s(kind.NetworkPolicy)
.WithLabel("uds.dev/generated", RemoteGenerated.KubeAPI)
.Get();

for (const netPol of netPols.items) {
// Get the old peers
const oldPeers = netPol.spec?.egress?.[0].to;
for (const netPol of netPols.items) {
const oldPeers = netPol.spec?.egress?.[0].to;

// Update the NetworkPolicy if the peers have changed
if (!R.equals(oldPeers, apiServerPeers)) {
// Note using the apiServerPeers variable here instead of the oldPeers variable
// in case another EndpointSlice is updated before this one
netPol.spec!.egress![0].to = apiServerPeers;
if (!R.equals(oldPeers, newPeers)) {
netPol.spec!.egress![0].to = newPeers;

log.debug(`Updating ${netPol.metadata!.namespace}/${netPol.metadata!.name}`);
await K8s(kind.NetworkPolicy).Apply(netPol);
}
log.debug(
`Updating KubeAPI NetworkPolicy ${netPol.metadata!.namespace}/${netPol.metadata!.name} with new CIDRs.`,
);
await K8s(kind.NetworkPolicy).Apply(netPol);
}
}
}
13 changes: 8 additions & 5 deletions src/pepr/operator/index.ts
Original file line number Diff line number Diff line change
@@ -34,11 +34,14 @@ const log = setupLogger(Component.OPERATOR);
void initAPIServerCIDR();

// Watch for changes to the API server EndpointSlice and update the API server CIDR
When(a.EndpointSlice)
.IsCreatedOrUpdated()
.InNamespace("default")
.WithName("kubernetes")
.Reconcile(updateAPIServerCIDRFromEndpointSlice);
// Skip if a CIDR is defined in the UDS Config
if (!UDSConfig.kubeApiCidr) {
When(a.EndpointSlice)
.IsCreatedOrUpdated()
.InNamespace("default")
.WithName("kubernetes")
.Reconcile(updateAPIServerCIDRFromEndpointSlice);
}

// Watch for changes to the API server Service and update the API server CIDR
When(a.Service)
1 change: 1 addition & 0 deletions src/pepr/uds-operator-config/values.yaml
Original file line number Diff line number Diff line change
@@ -7,6 +7,7 @@ operator:
UDS_ALLOW_ALL_NS_EXEMPTIONS: "###ZARF_VAR_ALLOW_ALL_NS_EXEMPTIONS###"
UDS_LOG_LEVEL: "###ZARF_VAR_UDS_LOG_LEVEL###"
AUTHSERVICE_REDIS_URI: "###ZARF_VAR_AUTHSERVICE_REDIS_URI###"
KUBEAPI_CIDR: ""
# Allow Pepr watch to be configurable to react to dropped connections faster
PEPR_LAST_SEEN_LIMIT_SECONDS: "300"
# Allow Pepr to re-list resources more frequently to avoid missing resources
Loading