Container is unable to resolve IPv6 localhost address

[ramgdev]

What happened:
We uncovered an issue in our application that runs inside a Docker container and makes url requests to services that are exposed on localhost ports. When trying to connect to the urls using IPv6 address for localhost [::1], all the requests are failing with connection refused error. But the same requests were successful when using the IPv4 address 127.0.0.1.

What you expected to happen:
We expect the both IPv6 and IPv4 formats of localhost to work seamlessly.

How to reproduce it (as minimally and precisely as possible):
I have a sample nodejs app dns-sample.zip attached below that reproduces the same issue. This app listens on IPv6 address ::. When a request comes in, it gets the response by making a localhost request using IPv6 address format. As you can see, the request is failing in this case.

• Extract the zip file and cd dns-sample
• Build the docker image docker build -t ramji/node-web-app .
• Deploy the pod kubectl create -f pod.yaml
• Get the pod IP from kubectl get pods -o wide
• Make a request curl http://<pod ip>:8080

Anything else we need to know?:
We noticed that the problem does not occur when we downgraded Docker version to 17.03. It was a blind attempt after noticing this bug report: containernetworking/cni#531

Environment:

• Kubernetes version (use kubectl version):
  Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.3", GitCommit:"5e53fd6bc17c0dec8434817e69b04a25d8ae0ff0", GitTreeState:"clean", BuildDate:"2019-06-06T01:44:30Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}
  Server Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.4", GitCommit:"a87e9a978f65a8303aa9467537aa59c18122cbf9", GitTreeState:"clean", BuildDate:"2019-07-08T08:43:10Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}

• Cloud provider or hardware configuration:

• OS (e.g: cat /etc/os-release):
  NAME="Ubuntu"
  VERSION="18.04.2 LTS (Bionic Beaver)"
  ID=ubuntu
  ID_LIKE=debian
  PRETTY_NAME="Ubuntu 18.04.2 LTS"
  VERSION_ID="18.04"
  HOME_URL="https://www.ubuntu.com/"
  SUPPORT_URL="https://help.ubuntu.com/"
  BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
  PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
  VERSION_CODENAME=bionic
  UBUNTU_CODENAME=bionic

• Kernel (e.g. uname -a):
  Linux ab2-k8s 4.15.0-54-generic Change flag.Duration descriptions to be unit-free. #58-Ubuntu SMP Mon Jun 24 10:55:24 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

• Install tools:
  docker version:
  Client:
  Version: 18.06.3-ce
  API version: 1.38
  Go version: go1.10.3
  Git commit: d7080c1
  Built: Wed Feb 20 02:28:10 2019
  OS/Arch: linux/amd64
  Experimental: false

Server:
Engine:
Version: 18.06.3-ce
API version: 1.38 (minimum version 1.12)
Go version: go1.10.3
Git commit: d7080c1
Built: Wed Feb 20 02:26:34 2019
OS/Arch: linux/amd64
Experimental: false

• Network plugin and version (if this is a network-related bug):
  calico 3.10

• Others:
  dns-sample.zip

[ramgdev]

@kubernetes/sig-network

[athenabot]

/sig network

These SIGs are my best guesses for this issue. Please comment /remove-sig <name> if I am incorrect about one.

🤖 I am a bot run by vllry. 👩‍🔬

[athenabot]

/triage unresolved

Comment /remove-triage unresolved when the issue is assessed and confirmed.

🤖 I am a bot run by vllry. 👩‍🔬

[vllry]

/assign @caseydavenport

[athenabot]

@caseydavenport
If this issue has been triaged, please comment /remove-triage unresolved.

If you aren't able to handle this issue, consider unassigning yourself and/or adding the help-wanted label.

🤖 I am a bot run by vllry. 👩‍🔬

[athenabot]

@caseydavenport
If this issue has been triaged, please comment /remove-triage unresolved.

If you aren't able to handle this issue, consider unassigning yourself and/or adding the help-wanted label.

🤖 I am a bot run by vllry. 👩‍🔬

[caseydavenport]

@ramgdev looks like the linked issue suggests different versions of Docker set IPv6 sysctls differently.

On the broken system, could you check the following?

net.ipv6.conf.all.disable_ipv6 = 0
net.ipv6.conf.default.disable_ipv6 = 0
net.ipv6.conf.lo.disable_ipv6 = 0

It sounds to me like this is probably an issue with Docker, rather than Kubernetes.

You might also want to try using the latest Calico v3.10.2 patch release, which also includes an IPv6 fix: https://docs.projectcalico.org/v3.10/release-notes/

[athenabot]

@caseydavenport
If this issue has been triaged, please comment /remove-triage unresolved.

If you aren't able to handle this issue, consider unassigning yourself and/or adding the help-wanted label.

🤖 I am a bot run by vllry. 👩‍🔬

[caseydavenport]

/triage support
/remove-triage unresolved

[caseydavenport]

/remove-kind bug

[caseydavenport]

/close

[k8s-ci-robot]

@caseydavenport: Closing this issue.

In response to this:

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[abdennour]

isn't it related to service.spec.ipFamilies ?!