Unable to change cluster CIDR #93

sahlex · 2019-02-28T15:40:14Z

When using k3s server it uses 10.42.0.0/16 as cluster cidr by default. This clashes with our network setup. So I tried to change this by giving a --cluster-cidr 10.10.0.0/16 parameter (as indicated by k3s help). Still, the startup output tells me that k3s is ignoring the parameter:

INFO[2019-02-28T16:19:44.756164411+01:00] Running kube-controller-manager --kubeconfig /var/lib/rancher/k3s/server/cred/kubeconfig-system.yaml --service-account-private-key-file /var/lib/rancher/k3s/server/tls/service.key --allocate-node-cidrs --cluster-cidr 10.42.0.0/16 --root-ca-file /var/lib/rancher/k3s/server/tls/token-ca.crt --port 0 --secure-port 0 --leader-elect=false

To Reproduce

run k3s server --cluster-cidr 10.10.0.0/16
Still see 10.42.0.0/16 in the output

Expected behavior
When calling kube-controller-manager --cluster-cidr parameter should be passed the value given with the call to k3s

The text was updated successfully, but these errors were encountered:

saphoooo · 2019-03-03T14:48:55Z

+1
Also a better documentation on how to run a different network addon, because I still don't know if I have to use k3s server --no-deploy flannel --disable-agentand run each agent with --no-flannel flag or not.

ibuildthecloud · 2019-03-04T05:44:53Z

I'll improve the docs. To disable flannel you just need to pass --no-flannel to the agent. When running k3s server it will automatically run the agent and due to #73 there is no way to disable flannel. So you have to run the agent separately (pass --disable-agent to server command) and each agent needs --no-flannel set.

ibuildthecloud · 2019-03-04T20:41:13Z

@saphoooo @sahlex This should work now in v0.2.0-rc3. Could you please test it out? Thanks.

saphoooo · 2019-03-04T21:40:06Z

works fine with canal:

$ k3s kubectl describe po nginx-7cdbd8cdc9-ghtm8
Name:               nginx-7cdbd8cdc9-ghtm8
Namespace:          default
Priority:           0
PriorityClassName:  <none>
Node:               oidc2/192.168.99.132
Start Time:         Mon, 04 Mar 2019 16:33:37 -0500
Labels:             pod-template-hash=7cdbd8cdc9
                    run=nginx
Annotations:        <none>
Status:             Running
IP:                 10.244.1.4

I don't know if it's linked, but I have an error message on the agents:

I0304 16:39:16.540478     941 log.go:172] suppressing panic for copyResponse error in test; copy error: context canceled

saphoooo · 2019-03-04T22:23:48Z

Apparently I've an error with traefik too:

E0304 17:22:17.259487    5375 pod_workers.go:190] Error syncing pod d8233647-3ecb-11e9-a887-7e9185699db4 ("svclb-traefik-785fb97589-dglld_kube-system(d8233647-3ecb-11e9-a887-7e9185699db4)"), skipping: [failed to "StartContainer" for "http" with CrashLoopBackOff: "Back-off 20s restarting failed container=http pod=svclb-traefik-785fb97589-dglld_kube-system(d8233647-3ecb-11e9-a887-7e9185699db4)"
, failed to "StartContainer" for "https" with CrashLoopBackOff: "Back-off 20s restarting failed container=https pod=svclb-traefik-785fb97589-dglld_kube-system(d8233647-3ecb-11e9-a887-7e9185699db4)"

bjd · 2019-03-04T22:29:01Z

@ibuildthecloud Not the original reporter, but I happened to run into the same issue with conflict on 10.42.0.0 network.
I've installed the rc-3 release and added --cluster-cidr=10.99.0.0/16 to k3s server command line.
It now crashes on startup with the following:

-- Logs begin at ma 2019-03-04 23:14:34 CET, end at ma 2019-03-04 23:17:54 CET. --
mrt 04 23:14:46 master systemd[1]: Starting Lightweight Kubernetes...
mrt 04 23:14:49 master k3s[4972]: time="2019-03-04T23:14:49.248561882+01:00" level=info msg="Starting k3s v0.2.0-rc3 (6de915d)"
mrt 04 23:14:49 master k3s[4972]: time="2019-03-04T23:14:49.259088385+01:00" level=info msg="Running kube-apiserver --watch-cache=false --cert-dir /var/lib/rancher/k3s/server/tls/temporary-certs --allow-privileged=true --authorization-mode Node,RBAC --service-account-signing-key-file /var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range 10.43.0.0/16 --advertise-port 6445 --advertise-address 127.0.0.1 --insecure-port 0 --secure-port 6444 --bind-address 127.0.0.1 --tls-cert-file /var/lib/rancher/k3s/server/tls/localhost.crt --tls-private-key-file /var/lib/rancher/k3s/server/tls/localhost.key --service-account-key-file /var/lib/rancher/k3s/server/tls/service.key --service-account-issuer k3s --api-audiences unknown --basic-auth-file /var/lib/rancher/k3s/server/cred/passwd --kubelet-client-certificate /var/lib/rancher/k3s/server/tls/token-node.crt --kubelet-client-key /var/lib/rancher/k3s/server/tls/token-node.key"
mrt 04 23:14:49 master systemd[1]: Started Lightweight Kubernetes.
mrt 04 23:14:49 master k3s[4972]: time="2019-03-04T23:14:49.501520055+01:00" level=info msg="Running kube-controller-manager --kubeconfig /var/lib/rancher/k3s/server/cred/kubeconfig-system.yaml --service-account-private-key-file /var/lib/rancher/k3s/server/tls/service.key --allocate-node-cidrs --cluster-cidr 10.99.0.0/16 --root-ca-file /var/lib/rancher/k3s/server/tls/token-ca.crt --port 10252 --address 127.0.0.1 --secure-port 0 --leader-elect=false"
mrt 04 23:14:49 master k3s[4972]: Flag --address has been deprecated, see --bind-address instead.
mrt 04 23:14:49 master k3s[4972]: time="2019-03-04T23:14:49.520724757+01:00" level=info msg="Running kube-scheduler --kubeconfig /var/lib/rancher/k3s/server/cred/kubeconfig-system.yaml --port 10251 --address 127.0.0.1 --secure-port 0 --leader-elect=false"
mrt 04 23:14:49 master k3s[4972]: time="2019-03-04T23:14:49.801230868+01:00" level=info msg="Listening on :6443"
mrt 04 23:14:49 master k3s[4972]: time="2019-03-04T23:14:49.803913104+01:00" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/coredns.yaml"
mrt 04 23:14:49 master k3s[4972]: time="2019-03-04T23:14:49.806556460+01:00" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/traefik.yaml"
mrt 04 23:14:50 master k3s[4972]: time="2019-03-04T23:14:50.010316963+01:00" level=info msg="Node token is available at /var/lib/rancher/k3s/server/node-token"
mrt 04 23:14:50 master k3s[4972]: time="2019-03-04T23:14:50.010354933+01:00" level=info msg="To join node to cluster: k3s agent -s https://10.42.0.10:6443 -t ${NODE_TOKEN}"
mrt 04 23:14:50 master k3s[4972]: time="2019-03-04T23:14:50.089738860+01:00" level=info msg="Wrote kubeconfig /etc/rancher/k3s/k3s.yaml"
mrt 04 23:14:50 master k3s[4972]: time="2019-03-04T23:14:50.089769922+01:00" level=info msg="Run: k3s kubectl"
mrt 04 23:14:50 master k3s[4972]: time="2019-03-04T23:14:50.089780387+01:00" level=info msg="k3s is up and running"
mrt 04 23:15:00 master k3s[4972]: F0304 23:15:00.809441    4972 controllermanager.go:203] error starting controllers: failed to mark cidr as occupied: cidr 10.42.0.0/24 is out the range of cluster cidr 10.99.0.0/16
mrt 04 23:15:00 master k3s[4972]: goroutine 229 [running]:
mrt 04 23:15:00 master k3s[4972]: github.com/rancher/k3s/vendor/k8s.io/klog.stacks(0xc0001a4900, 0xc0004c68c0, 0xb5, 0x13a)
mrt 04 23:15:00 master k3s[4972]: /go/src/github.com/rancher/k3s/vendor/k8s.io/klog/klog.go:828 +0xd4
mrt 04 23:15:00 master k3s[4972]: github.com/rancher/k3s/vendor/k8s.io/klog.(*loggingT).output(0x57e6f00, 0xc000000003, 0xc000604a50, 0x53471e5, 0x14, 0xcb, 0x0)
mrt 04 23:15:00 master k3s[4972]: /go/src/github.com/rancher/k3s/vendor/k8s.io/klog/klog.go:779 +0x306
mrt 04 23:15:00 master k3s[4972]: github.com/rancher/k3s/vendor/k8s.io/klog.(*loggingT).printf(0x57e6f00, 0x3, 0x308ee17, 0x1e, 0xc00453f1b8, 0x1, 0x1)
mrt 04 23:15:00 master k3s[4972]: /go/src/github.com/rancher/k3s/vendor/k8s.io/klog/klog.go:678 +0x14b
mrt 04 23:15:00 master k3s[4972]: github.com/rancher/k3s/vendor/k8s.io/klog.Fatalf(0x308ee17, 0x1e, 0xc00453f1b8, 0x1, 0x1)
mrt 04 23:15:00 master k3s[4972]: /go/src/github.com/rancher/k3s/vendor/k8s.io/klog/klog.go:1207 +0x67
mrt 04 23:15:00 master k3s[4972]: github.com/rancher/k3s/vendor/k8s.io/kubernetes/cmd/kube-controller-manager/app.Run.func1(0x35d2640, 0xc0000b6018)
mrt 04 23:15:00 master k3s[4972]: /go/src/github.com/rancher/k3s/vendor/k8s.io/kubernetes/cmd/kube-controller-manager/app/controllermanager.go:203 +0x590
mrt 04 23:15:00 master k3s[4972]: github.com/rancher/k3s/vendor/k8s.io/kubernetes/cmd/kube-controller-manager/app.Run(0xc00187b3d8, 0xc00009c360, 0xc00189299c, 0xc001892950)
mrt 04 23:15:00 master k3s[4972]: /go/src/github.com/rancher/k3s/vendor/k8s.io/kubernetes/cmd/kube-controller-manager/app/controllermanager.go:213 +0x90a
mrt 04 23:15:00 master k3s[4972]: github.com/rancher/k3s/vendor/k8s.io/kubernetes/cmd/kube-controller-manager/app.NewControllerManagerCommand.func1(0xc0019be280, 0xc0019eda00, 0x0, 0x10)
mrt 04 23:15:00 master k3s[4972]: /go/src/github.com/rancher/k3s/vendor/k8s.io/kubernetes/cmd/kube-controller-manager/app/controllermanager.go:99 +0x1e5
mrt 04 23:15:00 master k3s[4972]: github.com/rancher/k3s/vendor/github.com/spf13/cobra.(*Command).execute(0xc0019be280, 0xc0019d8000, 0x10, 0x1e, 0xc0019be280, 0xc0019d8000)
mrt 04 23:15:00 master k3s[4972]: /go/src/github.com/rancher/k3s/vendor/github.com/spf13/cobra/command.go:760 +0x2cc
mrt 04 23:15:00 master k3s[4972]: github.com/rancher/k3s/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0xc0019be280, 0xc000e8ef88, 0x1, 0x1)
mrt 04 23:15:00 master k3s[4972]: /go/src/github.com/rancher/k3s/vendor/github.com/spf13/cobra/command.go:846 +0x2fd
mrt 04 23:15:00 master k3s[4972]: github.com/rancher/k3s/vendor/github.com/spf13/cobra.(*Command).Execute(0xc0019be280, 0x22, 0xc000e8ef88)
mrt 04 23:15:00 master k3s[4972]: /go/src/github.com/rancher/k3s/vendor/github.com/spf13/cobra/command.go:794 +0x2b
mrt 04 23:15:00 master k3s[4972]: github.com/rancher/k3s/pkg/daemons/control.controllerManager.func1(0xc0019d8000, 0x10, 0x1e, 0xc0019be280)
mrt 04 23:15:00 master k3s[4972]: /go/src/github.com/rancher/k3s/pkg/daemons/control/server.go:117 +0xbe
mrt 04 23:15:00 master k3s[4972]: created by github.com/rancher/k3s/pkg/daemons/control.controllerManager
mrt 04 23:15:00 master k3s[4972]: /go/src/github.com/rancher/k3s/pkg/daemons/control/server.go:115 +0x23e
mrt 04 23:15:00 master systemd[1]: k3s.service: main process exited, code=exited, status=255/n/a
mrt 04 23:15:01 master systemd[1]: Unit k3s.service entered failed state.
mrt 04 23:15:01 master systemd[1]: k3s.service failed.

The k3s server host is 10.42.0.10 (and two agents on 10.42.0.11 and 10.42.0.12 but they're not active at the moment)

epicfilemcnulty · 2019-03-05T23:13:05Z

@ibuildthecloud I was able to use cilium as CNI with v0.2.0-rc5, running server with /usr/local/bin/k3s server --cluster-cidr=192.168.0.0/16 --no-flannel --no-deploy=traefik. Is there an option to alse customize --service-cluster-ip-range 10.43.0.0/16 passed to kube-apiserver?

epicfilemcnulty · 2019-03-06T10:44:29Z

well, turns out there was an option, but not initialized, same story as with cluster-cidr:
#171

bjd · 2019-03-06T16:44:57Z

As far as my crash is concerned, it appears that it does not like for cidr-cluster to be changed if k3s server has been run at least once. It does work when set on a fresh, never run config.

sahlex · 2019-03-12T11:09:03Z

Works for me now! Although I was not able to add a agent node yet (but this my be due to another issue like firewalling)

HOSTED-POWER · 2023-05-17T19:37:16Z

Changing my CIDR to support ipv6 results in a panic to

it crashes completely and it logs:

Creating the node lease for IPv4. This is the n.Spec.PodCIDRs: [10.42.0.0/24]
Creating the node lease for IPv6. This is the n.Spec.PodCIDRs: [10.42.0.0/24]

instead of

Creating the node lease for IPv4. This is the n.Spec.PodCIDRs: [10.42.0.0/24 fc00:a0::/64]
Creating the node lease for IPv6. This is the n.Spec.PodCIDRs: [10.42.0.0/24 fc00:a0::/64]

How can we enable ipv6 on an existing cluster?

brandond · 2023-05-17T19:56:03Z

You cannot change the CIDRs or add a new address family to an existing cluster. Cluster networking should be established when the cluster is initially started.

HOSTED-POWER · 2023-05-17T19:57:00Z

So there is no way to ever enable ipv6 without losing ALL pods and data? WOW :(

Xinayder · 2024-05-28T00:19:06Z

You cannot change the CIDRs or add a new address family to an existing cluster. Cluster networking should be established when the cluster is initially started.

I just had the same issue. I'd like to suggest that, if you define a cluster-cidr value other than what's already configured in your node, k3s should throw an error message informing the user that you cannot alter an existing cluster, or just disregard the new values for Cluster CIDR and use the old ones, defined in the Node definition.

ibuildthecloud mentioned this issue Mar 4, 2019

Actually pass cluster-cidr on to server #134

Merged

ibuildthecloud added the status/to-test label Mar 4, 2019

cjellick added this to the v0.2.0 milestone Mar 8, 2019

cjellick added the kind/bug Something isn't working label Mar 8, 2019

ibuildthecloud closed this as completed Mar 9, 2019

jamescun mentioned this issue Nov 30, 2020

Changing Cluster CIDR after initialization causes panic #2603

Closed

kenlasko mentioned this issue Sep 21, 2023

One master node continually spamming logs with "request cluster ID mismatch" #8399

Closed

lorenzo-w mentioned this issue Sep 26, 2023

k3s versions >v1.27.3 ignore Tailscale IPv6 addresses on IPv4-only hosts, causing new fatal errors #8459

Closed

infroger mentioned this issue Oct 6, 2023

Port 80 and 443 are in a filtered state for any node beyond master-1 #8348

Closed

AlbertoSoutullo mentioned this issue Oct 16, 2023

Multicloud K3s not working #8652

Closed

infroger mentioned this issue Nov 16, 2023

Traefik redeployed and configuration lost #8883

Closed

k3s-io locked and limited conversation to collaborators May 28, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Unable to change cluster CIDR #93

Unable to change cluster CIDR #93

sahlex commented Feb 28, 2019

saphoooo commented Mar 3, 2019

ibuildthecloud commented Mar 4, 2019

ibuildthecloud commented Mar 4, 2019

saphoooo commented Mar 4, 2019

saphoooo commented Mar 4, 2019

bjd commented Mar 4, 2019

epicfilemcnulty commented Mar 5, 2019

epicfilemcnulty commented Mar 6, 2019

bjd commented Mar 6, 2019 •

edited

Loading

sahlex commented Mar 12, 2019 •

edited

Loading

HOSTED-POWER commented May 17, 2023

brandond commented May 17, 2023

HOSTED-POWER commented May 17, 2023

Xinayder commented May 28, 2024 •

edited

Loading

Unable to change cluster CIDR #93

Unable to change cluster CIDR #93

Comments

sahlex commented Feb 28, 2019

saphoooo commented Mar 3, 2019

ibuildthecloud commented Mar 4, 2019

ibuildthecloud commented Mar 4, 2019

saphoooo commented Mar 4, 2019

saphoooo commented Mar 4, 2019

bjd commented Mar 4, 2019

epicfilemcnulty commented Mar 5, 2019

epicfilemcnulty commented Mar 6, 2019

bjd commented Mar 6, 2019 • edited Loading

sahlex commented Mar 12, 2019 • edited Loading

HOSTED-POWER commented May 17, 2023

brandond commented May 17, 2023

HOSTED-POWER commented May 17, 2023

Xinayder commented May 28, 2024 • edited Loading

bjd commented Mar 6, 2019 •

edited

Loading

sahlex commented Mar 12, 2019 •

edited

Loading

Xinayder commented May 28, 2024 •

edited

Loading