Ingress controller cannot reach itself (Azure-VNet, ACS)

[carlpett]

Hi,
We recently stood up a new Kubernetes cluster with ACS engine, after previously using the ACS service in the azure cli. The new cluster has an issue where ingress controllers (we're using the stock nginx ingress controller) cannot make requests to "themselves", ie cannot reach its' own public IP.

The only difference I've found between the old clusters and the new one is that this automatically got Azure-VNet CNI, whereas the previous ones had no CNI configuration at all (which I guess means kubenet?).

This is a problem because external auth causes the controller to make a request to its' own domain. This traffic seems to be dropped, and the requests time out.

It is clearly related to pod networking, since the container host can reach the ingress properly:

# On the node itself
k8s-agentpool1-28163751-2:~$ curl -vIsm 1 13.95.23.109
* Rebuilt URL to: 13.95.23.109/
*   Trying 13.95.23.109...
* Connected to 13.95.23.109 (13.95.23.109) port 80 (#0)
> HEAD / HTTP/1.1
> Host: 13.95.23.109
> User-Agent: curl/7.47.0
> Accept: */*
> 
< HTTP/1.1 [...]
# Exec into the container
k8s-agentpool1-28163751-2:~$ docker exec 43b4450a0f8c curl -vIsm 1 13.95.23.109
* Rebuilt URL to: 13.95.23.109/
*   Trying 13.95.23.109...
* TCP_NODELAY set
* Connection timed out after 1001 milliseconds
* Curl_http_done: called premature == 1
* stopped the pause stream!
* Closing connection 0

Is this a known limitation of azure-vnet, or a misconfiguration?

[sharmasushant]

@carlpett Is it still happening for you? Can you please share your cluster config. We can try to repro this.

[carlpett]

@sharmasushant Yes, still happens. After some more investigation, it does not only happen for ingresses, but all services with type LoadBalancer. Here is a repro case:

apiVersion: v1
kind: Pod
metadata:
  name: nginx-reachability-testing
  labels:
    app: nginx-reachability-testing
spec:
  containers:
  - name: nginx
    image: nginx:1.7.9
    ports:
    - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: nginx-reachability-testing
spec:
  ports:
  - port: 80
    name: http
  type: LoadBalancer
  selector:
    app: nginx-reachability-testing

The public IP is reachable from anywhere, except from within the pod that is backing the service.

I reported this while waiting for a response on an Azure support ticket, which has now gotten started, so there is an ongoing case there (118011617470683)

[malachma]

The reason for the this reported behaviour is that the interface attached to the bridge (azure0) is not well configured. Meaning a default setting for the hairpin mode is used. The man for bridge states the following

hairpin on or hairpin off
Controls whether traffic may be send back out of the port on which it was received.
By default, this flag is turned off and the bridge will not forward traffic back
out of the receiving port.

As a workaround one can perform the following changes to identify the right interface

Howto find out the interface for a given POD to alter the hairpin flag?

First identify the node on which the POD is running

kubectl get pod nginx-reachability-testing -o wide
NAME READY STATUS RESTARTS AGE IP NODE
nginx-reachability-testing 1/1 Running 0 1h 10.240.0.6 k8s-agentpool1-21713571-0

Second get the interface detail of the POD/Container

kubectl exec -it nginx-reachability-testing bash
root@nginx-reachability-testing:/# ip add
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
22: eth0@if21: <BROADCAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue state UP qlen 1000
link/ether ce:b7:4d:b5:a8:cb brd ff:ff:ff:ff:ff:ff
inet 10.240.0.6/12 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::ccb7:4dff:feb5:a8cb/64 scope link
valid_lft forever preferred_lft forever

Note down the detail:

--> 22: eth0@if21

Third, get to the agent on which the POD/Container is running

As seen previously the network interface has the number 22 and is connected to interface 21. The reason is a veth pair which is in use.
Go to the agent and search for interface the docker container is connected to. In our case it is interface 22--> ip add |grep @if22

ip add | grep @if22
21: azvethc9e7b2c@if22: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master azure0 state UP group default qlen 1000

The interface name we get returned is the one which is connected on the bridge (azure0) This is the name we have to use to get its hairpin mode changed

Howto change the hairpin mode?

The hairpin mode for a given interface can easily be altered by performing this step

--> sudo bridge link set dev azvethc9e7b2c hairpin on

Verification

Get the external IP of the POD

kubectl get service nginx-reachability-testing
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
nginx-reachability-testing LoadBalancer 10.0.157.109 13.74.43.81 80:32594/TCP 1h

connect to your POD:
kubectl exec -it nginx-reachability-testing bash

Connect from inside the POD against the LB

root@nginx-reachability-testing:/# curl 13.74.43.81

<title>Welcome to nginx!</title> <style> body { width: 35em; margin: 0 auto; font-family: Tahoma, Verdana, Arial, sans-serif; } </style>

Welcome to nginx!

If you see this page, the nginx web server is successfully installed and working. Further configuration is required.

For online documentation and support please refer to nginx.org.
Commercial support is available at nginx.com.

Thank you for using nginx.

Change the hairpin mode to off again
sudo bridge link set dev azvethc9e7b2c hairpin off

and try to connect against the LB. This will show that the connection is not established again.
So revert the modification
sudo bridge link set dev azvethc9e7b2c hairpin on

[malachma]

Found the following issue reports which are related to the hairpin mode

kubernetes/kubernetes#53269
containernetworking/cni#476

The main context is this one
kubernetes/kubernetes#45790

[malachma]

To allow some kind of automation I have created this simple script which can be use as a starting base: https://gist.github.com/malachma/8603d1d6daedceb4320d20b70527f0ac
It uses the kubernetes python-client. So install it first. Get also sure that the nodes are able to establish
ssh connections via pub-key only. To listen for a ceratin POD which get added to the cluster replace the variable "nameOfPod"

[kovszilard]

We have the exact same use case and we experience the same issue. The workaround from @malachma worked for us.

[rrudduck]

Any updates on this issue being fixed natively in the CNI plugin? While we could use the script above and automate it, this seems like something the plugin should do natively and is probably a simple change. I'm happy to look at creating a PR if needed.

Thanks.

[tamilmani1989]

This issue is fixed #248