Host reboot, connectivity lost

[caseydavenport]

From #33

Furthermore, i've just done the test of rebooting my host and all the network interfaces disapeared :(
Calico still seems to be configured as previously

calicoctl profile show
+------------------------------------+
| Name |
+------------------------------------+
| busybox |
| elasticsearch_logging_v1_1jzvs |
| elasticsearch_logging_v1_az6c8 |
| fluentd_elasticsearch_10.115.77.87 |
| fluentd_elasticsearch_10.115.77.88 |
| fluentd_elasticsearch_10.115.77.89 |
| kibana_logging_v1_ifztn |
| kube_dns_v8_l6rtn |
| kube_dns_v8_tq8xf |
| kube_ui_v1_ym5ni |
| monitoring_heapster_v8_9cy06 |
| monitoring_heapster_v8_pt4ey |
| monitoring_influx_grafana_v1_obu8h |
+------------------------------------+
But there'is no calixxxx interface on the host anymore.
The pods on this host are unreachable

The docker daemon is still configured on cbr0

/usr/bin/docker -d -H fd:// --bridge=cbr0 --iptables=false --ip-masq=false
Maybe i forgot something ?

[Smana]

Same behaviour with 2 hosts. I have an unknown "tunl0" interface:

3: cbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
    inet 10.233.2.1/24 scope global cbr0
       valid_lft forever preferred_lft forever
    inet6 fe80::54ac:f8ff:fe54:314d/64 scope link 
       valid_lft forever preferred_lft forever
4: tunl0@NONE: <NOARP,UP,LOWER_UP> mtu 1480 qdisc noqueue state UNKNOWN group default 
    link/ipip 0.0.0.0 brd 0.0.0.0

[caseydavenport]

Hi Smana,

I'm trying to reproduce this behavior locally.

Can you show the output of the following commands?

sudo systemctl status calico-node.service

sudo systemctl status kube-kubelet.service

sudo docker ps

Regarding tunl0, that is a tunnel interface that is created by Calico but will not be used unless ipip is enabled on your IP pool. I think we can safely say it isn't related to the issue you're hitting.

[caseydavenport]

I've managed to reproduce this issue locally - It looks to me like the Kubelet isn't informing the Calico plugin that the pods have been recreated, and so Calico doesn't know to re-create the caliXXX veths.

What I would expect to happen is this:

1. Node reboots and Kubelet is started (There are no running containers at this point)
2. Kubelet sync's each pod's state - creates "/pause" infra containers for each.
3. For each pod, Kubelet notifies the Calico plugin
4. The Calico plugin sets up networking (creates caliXXX veth, security profile, advertises route)
5. The Kubelet creates the pod's remaining containers.

I'm not seeing step 3 occur, which smells like a bug in the Kubelet to me. We're going to do some more digging on this to see what codepaths are getting hit in the Kubelet, and raise a bug on the Kubernetes GitHub if appropriate.

[Smana]

thank you, i stay tuned

[Smana]

In the meantime, as a workaround is there a way to force the step 3 ?

For each pod, Kubelet notifies the Calico plugin

[caseydavenport]

Sorry for the delay. To workaround this, you can simply delete the pods using kubectl. If they are managed by a ReplicationController, they will be re-created automatically (though possibly on a different host) and will have networking once they are created.

If they are not managed by a ReplicationController, then re-create the pods after deleting them.

[Smana]

Hi, any news ?
The problem is really serious, even when i restart the docker service my ips are lost :(
Please let me know if you need more info.

Regards,

[caseydavenport]

Hi Smana,

I believe @Symmetric was looking into this, but he's on vacation now. I'll pick up the investigation of this problem today and post my findings to the issue.

Casey

[caseydavenport]

Hi Smana,

I've made some progress on this.

1. Upgrading to the latest master build of the Kubelet fixes step Calico Policy in Kubernetes Annotations #3 from above - I now see the Kubelet calling the network plugin when it restarts the pods.
2. It then seemed Calico plugin was hitting an error, since Docker was assigning the same IP address to a different container, and Calico hadn't been notified that the pod had been torn down, and thus the IP had been unassigned. I've updated to use a master build of the calico-kubernetes plugin and enabled Calico IPAM in order to sidestep this issue, and I'm now seeing networking come back up after rebooting nodes, and the docker service.

Could you try following these steps and let me know if they work? This is what I had to do to fix this issue.

• Build and install the master version of the kubelet on your nodes.
• Build and install the master version of calico-kubernetes on your nodes.
• Open /etc/systemd/calico-node.service and remove the --kubernetes flag from the start command.
• Open /etc/network-environment and add the CALICO_IPAM=true variable.
• Reload systemd - sudo systemctl daemon-reload
• Restart the kubelet process.

To get this working with Docker IPAM likely requires changes to the Kubelet. I'll have to do some more thinking on what the correct behavior should be there.

[Smana]

Hi Casey,
I'll try that asap and let you know.

[Smana]

Well, first step i've upgraded kubernetes(kubelet included) to the master version.

kubectl version
Client Version: version.Info{Major:"1", Minor:"1+", GitVersion:"v1.1.0-alpha.1.877+bf990acefa97a6", GitCommit:"bf990acefa97a69f430767d964d6fadfec1d9e93", GitTreeState:"clean"}
Server Version: version.Info{Major:"1", Minor:"1+", GitVersion:"v1.1.0-alpha.1.877+bf990acefa97a6", GitCommit:"bf990acefa97a69f430767d964d6fadfec1d9e93", GitTreeState:"clean"}

It didn't helped. Indeed when i rebooted my host, the pods were created but without ip address.

Now i need to figure out how to upgrade calico-kubernetes.
Currently i just downloaded the version 0.6.0 of calicoctl and run the command
calicoctl node --ip=${DEFAULT_IPV4} --kubernetes

[caseydavenport]

Hi Smana -

I suspect we will be releasing a new version of calico-kubernetes in the next 1-2 days. So, if you can wait that long before testing, that might be the easiest option.

If you'd like to test sooner than that, you can follow these steps to build and install calico-kubernetes.

To build from source.

# Clone the repository
git clone [email protected]:projectcalico/calico-kubernetes.git

# Build the plugin
cd calico-kubernetes
make binary

To manually install the binary, copy it from calico_kubernetes/dist/calico_kubernetes, and install it on your Kubernetes node(s).

# Install the binary on the node in this location
sudo mv calico_kubernetes /usr/libexec/kubernetes/kubelet-plugins/net/exec/calico/calico

You'll want to remove the --kubernetes flag when running calicoctl node, otherwise your manually installed binary will be overwritten.

Let me know if you have any trouble with these steps and I'll be glad to help.

[Smana]

Thanks Casey,
I'll wait, in the meantime i have some tests/benchmarks to do.
I stay tuned,

Regards,
Smana

[Smana]

By the way could you please tell me what version of calico-docker is supposed to be used with the latest version of calico-kubernetes plugins ?
Thank you

[caseydavenport]

I'd recommend using any version of calico-docker version v0.5.5 or later.

[caseydavenport]

Hi @Smana - we've released calico-kubernetes v0.2.0 here.

You can download and install the plugin like so:

wget https://github.com/projectcalico/calico-kubernetes/releases/download/v0.2.0/calico_kubernetes
sudo mv calico_kubernetes /usr/libexec/kubernetes/kubelet-plugins/net/exec/calico/calico
sudo systemctl restart kube-kubelet

To enable calico IPAM, edit /etc/network-environment and add this line:

CALICO_IPAM=true

Then, restart the kubelet

sudo systemctl retstart kube-kubelet

A version of calico-docker which includes this binary is expected to be released later this week. In the meantime, calico-docker v0.5.5 or later will work fine.

[Smana]

Thanks Casey,
I'm trying it right now !

:)

[Smana]

Well my first tests using thecalico-kubernetes v0.2.0 is not satisfactory. The networking of a container is not properly configured though i didn't identified exactly the root cause.
On my node i run the calico-docker v0.5.5

./calicoctl node --ip=10.115.77.92

i download the calico-kubernetes plugin

 wget https://github.com/projectcalico/calico-kubernetes/releases/download/v0.2.0/calico_kubernetes -O \
/usr/libexec/kubernetes/kubelet-plugins/net/exec/calico/calico

The network-environment has been updated

grep CALICO_IPAM /etc/network-environment 
CALICO_IPAM=true

finally kubelet is restarted.

Then to test it out, i create a pod busybox and try to ping an ip address on another node and name resolution.

docker exec b8022bcb2bc7 ping -c 1 10.233.0.1
...100% packet loss
docker exec b8022bcb2bc7 ping -c 1 www.google.fr
ping: bad address 'www.google.fr'

Note: the ip address 10.233.0.1 is on another node

Using the --kubernetes which downloads another version on calico-kubernetes plugin it works as expected

./calicoctl node --ip=10.115.77.92 --kubernetes
 systemctl restart kubelet.service
docker exec c7a445f4185d ping -c 1 10.233.0.1
1 packets transmitted, 1 packets received, 0% packet loss
docker exec c7a445f4185d ping -c 1 www.google.fr
1 packets transmitted, 1 packets received, 0% packet loss

I still have the same issue on host reboot.

Regards,
Smana

[Smana]

I forgot to mention that i was using kubernetes v1.6.0.
With the binaries built from master branch i had several other issues.
Please let me know what is the commmit needed in order to benefit from your fix. Maybe i should wait for a kubernetes release which will include that.

[caseydavenport]

Hi Smana,

I've so far been unable to reproduce your issue on my cluster using the following:

• v0.2.0 of the calico-kubernetes plugin
• calicoctl version v0.5.5
• A build of Kubernetes at commit 0902f80f8b94fd23c66398118fcdf9e845c165b1

Could you run the following commands to generate a Calico diagnostics bundle on both the source compute host as well as the destination compute host? This will gather all the logs and routing diagnostics to help me debug.

sudo apt-get install ipset
sudo ETCD_AUTHORITY=<etcdIP:etcdPort> calicoctl diags

Once you've generated the bundles, could you please send them to me at [email protected] so I can take a look?

One thing I did not mention above is that when enabling Calico IPAM, you should make sure that your Docker bridge (cbr0) is not configured with an IP address within the Calico IP pool. This prevents conflicts between the docker bridge (which is still required for running docker) and the IPs assigned by Calico.

# So, assuming your pods are in the 10.233.0.0/16 network.
sudo ifconfig cbr0 172.0.0.1/24

[Smana]

That was exactly what i needed, the containers networking are now well configured after a reboot.
I needed to remove the default pool subnet before adding mine :

calicoctl pool remove 192.168.0.0/16
calicoctl pool add 10.233.0.0/16

But now i can't control the subnet assigned to each node. it is randomly set.
Is it possible to configure manually the subnet assigned to each calico node ?

Regards,
Smana

[caseydavenport]

Glad to hear it's working!

Currently, it is not possible to manually assign subnets to each node when using Calico's IPAM. Is this a required feature for you? Calico IPAM does currently assign address blocks to nodes in order to aggregate routes.

[Smana]

Actually that would give us more control on ip assignement.
For instance currently on my lab i have to add static routes on my router (no bgp) in order to get Internet on the containers.
And i use a dns server (skydns) which have a fixed cluster ip, let's say 10.233.0.10.

My second concern is that i prefer to use a released version of kubernetes for stability reasons.
So i'll wait a moment before upgrading my cluster. For the moment we've decided to perform manual tasks if a node fails. The kubelet daemon is not enabled at startup therefore after 5 minutes, the pods are rescheduled on the remaining nodes.

Anyway i'll open a new ticket if needed.

Thank you.

[Smana]

I can't close this ticket because i'm not the issuer.

[Smana]

Actually, considering i'm using dns (managed by kubernetes) for service name resolution i need to solve this issue, the workaround i've described above is not a solution because if the dns server can't be reached my apps won't work properly.

More generally, how do we deal with kubernetes services addresses please ?
if i have the service address (defined randomly by kubernetes within the pool) 10.233.0.10 and the subnet 10.233.0.0/24 is routed to the node3 what will happen ?

Again, thank you @caseydavenport

[Symmetric]

@Smana the service VIPs should be in a different subnet than the IP pools used for pods. This is because the kube-proxy on each node replaces the service VIP with the IP of one of the pods that is backing the service, and so that Service VIP could resolve to an IP in any pool in the cluster.

Automatically assigned Service ClusterIPs are controlled by the following kube-apiserver flag:

      --service-cluster-ip-range=<nil>: A CIDR notation IP range from which to assign service cluster IPs. This must not overlap with any IP ranges assigned to nodes for pods.

I'm not sure if you're allowed to manually configure a ClusterIP outside of that range; can you check what you've configured for the service-cluster-ip-range on your API server?

[caseydavenport]

For instance currently on my lab i have to add static routes on my router (no bgp) in order to get Internet on the containers.

Are you referring to routes back to your containers here? Do you mean your router does not support BGP and thus you had to manually install routes? Or that BGP distributed routes were not sufficient in some way?

And i use a dns server (skydns) which have a fixed cluster ip, let's say 10.233.0.10.

But that clusterIP shouldn't actually be routable in your DC because of the proxying behavior @Symmetric mentioned above.

My second concern is that i prefer to use a released version of kubernetes for stability reasons.

Understood - Kubernetes v1.1 has just entered code-freeze and will contain the code you need. It should be released in about a month, and then you'll be all set :)

[Smana]

the service VIPs should be in a different subnet than the IP pools used for pods.
Thank you, i thought that the subnet for the kubernetes services had to be the same as the calico pool.

I've change the kubernetes services addresses sunbet
--service-cluster-ip-range=10.233.0.0/18
and the calico pool

calicoctl pool show --ipv4
+----------------+---------+
|   IPv4 CIDR    | Options |
+----------------+---------+
| 10.233.64.0/18 |         |
+----------------+---------+

Are you referring to routes back to your containers here? Do you mean your router does not support BGP and thus you had to manually install routes? Or that BGP distributed routes were not sufficient in some way?

Yes i configured the routes back. Anyway it is specific to my lab and i will have the target plateform very soon (with physical nodes and bgp)

Understood - Kubernetes v1.1 has just entered code-freeze and will contain the code you need. It should be released in about a month, and then you'll be all set :)

Cool! i look forward to it.

Well that's fine as i said now after a reboot the pods ips are configured

• Before reboot

kubectl describe --namespace=test pod backend-bydxe | grep ^IP
IP:             10.233.65.5

calicoctl endpoint show --d | grep backend-bydxe
| kubenode-1 |      docker     | 8086aad975066fa4668c8ef88dce7b6f911f01859d28eb8ff628dc46ade33ec3 | 1c3c07da636711e5830f525400725bce | 10.233.65.5/32 | 36:04:f0:5f:96:35 |                test_backend-bydxe_8086aad97506               | active

• After reboot

kubectl describe --namespace=test pod backend-bydxe | grep ^IP
IP:             10.233.65.8

calicoctl endpoint show --d | grep backend-bydxe
| kubenode-1 |      docker     | 67bc43715405e9b5d92b840180087bb7486d3ed1ea0f58166018edb800e09b67 | e494e4ea636711e588f3525400725bce |  10.233.65.8/32 | 82:e5:0f:0b:a2:c0 |                test_backend-bydxe_67bc43715405               | active |
| kubenode-1 |      docker     | 8086aad975066fa4668c8ef88dce7b6f911f01859d28eb8ff628dc46ade33ec3 | 1c3c07da636711e5830f525400725bce |  10.233.65.5/32 | 36:04:f0:5f:96:35 |                test_backend-bydxe_8086aad97506               | active |

So the ticket can be closed

There are 2 remaining problems but they're not directly related the the reboot issue :

• Now that the docker bridge is configured on a distinct network (172.16.0.0), i can't run docker containers for network debugging purpose. Indeed the pods are configured in the subnet 10.233.64.0/18.
  Do you have a way to run a container into the calico pool network ?
• For an unknown reason the kubelet daemons don't update their status.
  For instance each time I create a pod, the pod is actually created but when i request the status with kubectl it shows that the pod is not running. I have to restart kubelet in order to update the status... strange. Maybe it will be fixed in the version 1.1 of kubernetes, or i've missed a new mandatory option.

Thank you two :)

I will soon share my work regarding how to build a full cluster (kubernetes+calico) on debian jessie with Ansible.

[caseydavenport]

I've raised #49 to discuss your first bullet. The second one doesn't sound like a Calico issue to me, but a Kubernetes issue.

I will soon share my work regarding how to build a full cluster (kubernetes+calico) on debian jessie with Ansible.

Sounds great, looking forward to it!

[Symmetric]

For an unknown reason the kubelet daemons don't update their status.

Can you please open a separate issue for this? It could be a Calico issue, so report to us first. Start kubelet with the --v=5 flag, and then see what logs are emitted on the node when the kubectl get pod command is run. It's possible this is a problem with our handling of the Status hook.