Connection refused for all URLs - TLS and non-TLS - but works as expected when running in kaniko debug

[axsaucedo]

Summary

Github repo (context) gets cloned without issues + docker images are pulled without issues when ran SSHing into a kaniko-project/executor:debug but does not work otherwise.

Actual behavior

Kaniko fails to pull context from github. This can be (and was) circumvented by adding an initcontainer as per #121 (where alpine/git pulls the git repo and makes it available through a shared volume). However, in the next stage the Kaniko container still fails to pull any images with connection refused. This still fails if the --insecure-pull flag is added. This is not due to an issue related to credentials in azure (#425) and it doesn't seem to be (at least directly) related to TLS issues (#168 + #200) because I can confirm it works perfectly when running it directly SSH'd in a kaniko-project/executor:debug container.

More specifically, if I run a kaniko-project/executor:debug container with tail -f /dev/null as command, and then SSH into it with kubectl exec -it kanikodebug sh, if I run the exact same command /kaniko/executor --context git://github.com/user/repo --no-push, it would work perfectly (including the cloning of the repo and the pulling of the images).

It's also worth mentioning that if I use run the kaniko-project/executor:debug container, and try to run it with the command same build-image command /docker/executor --context ... (instead of tail -f /dev/null) then it shows the same strange connection refused behaviour.

This was tested in an Azure AKS cluster.

Expected behavior

To see the same behaviour in the Kaniko executor than the one I see when SSHing and running the exact same commands.

To Reproduce

Connection failed when cloning repo

Below is the config file without init container. This will show a connection refused to the git repo. This works if actually running it with Kaniko:debug container (as per the example below). This also loads a config secret from file containing the auth required for kaniko.

apiVersion: v1
kind: Pod
metadata:
  name: kaniko
spec:
  containers:
  - name: kanikobuilder
    image: gcr.io/kaniko-project/executor:debug
    command:
    - /kaniko/executor 
    args:
    - --context 
    - git://github.com/user/repo
    - --no-push
    volumeMounts:
      - name: config
        mountPath: /kaniko/.docker
  restartPolicy: Never
  volumes:
    - name: config
      secret:
        secretName: config

Connection failed when pulling image

This is the version that clones the repo and shares it through a volume. This allows for the next step to be reached, but then the docker pull fails with a connection refused. The example below is ran with the debug container using the command: /kaniko/executor to show that it still fails, but it can be replaced with the latest kaniko tag and would show the same behaviour.

apiVersion: v1
kind: Pod
metadata:
  name: kanikoinit
spec:
  initContainers:
  - name: init-clone-repo
    image: alpine/git
    args:
        - clone
        - --single-branch
        - --
        - https://github.com/user/repo.git
        - /context
    volumeMounts:
    - name: context
      mountPath: /context
  containers:
  - name: kaniko
    image: gcr.io/kaniko-project/executor:debug
    command:
    - /kaniko/executor 
    args:
    - --dockerfile 
    - core-builder/Dockerfile 
    - --context 
    - /context
    - --no-push
    - --insecure-pull
    volumeMounts:
      - name: context
        mountPath: /context
      - name: config
        mountPath: /kaniko/.docker
  restartPolicy: Never
  volumes:
    - name: config
      secret:
        secretName: config
    - name: context
      emptyDir: {}

Debug Container

Now for the strangest part, is to reproduce how the exact same commands actually run perfectly in the debug container. The pod below would create a debug container which runs with tail -f /dev/null, so it's possible to connect/ssh into the container using kubectl exec -it kanikodebug sh. When connected to the Kaniko image, if the same command /kaniko/executor --context git://github.com/user/repo --no-push would work as expected without any issues (context will be cloned from repo, and images will be pulled and built without issues).

apiVersion: v1
kind: Pod
metadata:
  name: kanikodebug
spec:
  containers:
  - name: kaniko
    image: gcr.io/kaniko-project/executor:debug
    command:
    - tail
    args:
    - -f
    - /dev/null

[Techn0logic]

I observe the exact same issue.

                image="gcr.io/kaniko-project/executor:debug<or latest>",
                args=[
                     "--dockerfile=/workspace/Dockerfile",
                      "--no-push"
                ],

Getting error on:

Error while retrieving image from cache: ubuntu:16.04 Get https://index.docker.io/v2/:

However, running debug image with ["tail", "-f", "/dev/null"], running same args inside pulls ubuntu image no problem.

running on k3s version v1.17.3+k3s1 (5b17a175)

kubectl version
Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.1", GitCommit:"4485c6f18cee9a5d3c3b4e523bd27972b1b53892", GitTreeState:"clean", BuildDate:"2019-07-18T09:18:22Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.3+k3s1", GitCommit:"5b17a175ce333dfb98cb8391afeb1f34219d9275", GitTreeState:"clean", BuildDate:"2020-02-27T07:28:53Z", GoVersion:"go1.13.8", Compiler:"gc", Platform:"linux/amd64"}

would appreciate if someone could help to troubleshoot

[Techn0logic]

Found what is the issue.

Namespace was configured to initialize pods with istio-proxy, which takes some time to come up. Therefore, on the immediate launch pod was unable to reach out and got Connection Refused

Setting pods restartPolicy: OnFailure solved the issue and kaniko worked and ended up in Completed

[ffais]

i have the same problem, but the workaround doesn't work for me.

Kubernetes version: 1.16.7
istio version: 1.5.0