k3d + tekton. No such host

[marniks7]

Expected Behavior

TaskRun should NOT fail when docker image executed without command with private artifactory

Actual Behavior

TaskRun fails

failed to create task run pod "test1-run-zgnhb": translating TaskSpec to Pod: Get "https://artifactory.company.com:17064/v2/": dial tcp: lookup artifactory.company.com on 10.43.0.10:53: no such host. Maybe missing or invalid Task default/test1'

Steps to Reproduce the Problem

1. Install k3d https://k3d.io/v5.4.4/#installation
2. See script below. Change artifactory.company.com:17064 to your company proxy
3. Run

k3d cluster create
kubectl apply --filename https://storage.googleapis.com/tekton-releases/pipeline/latest/release.yaml
kubectl rollout status deployment -n tekton-pipelines tekton-pipelines-controller
kubectl rollout status deployment -n tekton-pipelines tekton-pipelines-webhook

echo '
apiVersion: tekton.dev/v1beta1
kind: Task
metadata:
  name: test
spec:
  steps:
    - name: step
      image: artifactory.company.com:17064/hashicorp/http-echo:alpine
      args: [ "-text=test"]
' | kubectl apply -f -
echo '
apiVersion: tekton.dev/v1beta1
kind: TaskRun
metadata:
  generateName: test-run-
spec:
  taskRef:
    name: test
' | kubectl create -f -

Additional Info

• Kubernetes version:

Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.5", GitCommit:"aea7bbadd2fc0cd689de94a54e5b7b758869d691", GitTreeState:"clean", BuildDate:"2021-09-15T21:10:45Z", GoVersion:"go1.16.8", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.8+k3s1", GitCommit:"53f2d4e7d80c09a7db1858e3f4e7ddfa13256c45", GitTreeState:"clean", BuildDate:"2022-06-27T21:48:01Z", GoVersion:"go1.17.5", Compiler:"gc", Platform:"linux/amd64"}
WARNING: version difference between client (1.21) and server (1.23) exceeds the supported minor version skew of +/-1

• Tekton Pipeline version:
  v0.38.0

Note, with command it works fine

echo '
apiVersion: tekton.dev/v1beta1
kind: Task
metadata:
  name: test2
spec:
  steps:
    - name: step
      image: artifactory.company.com:17064/hashicorp/http-echo:alpine
      command: [ "echo" ]
      args: [ "-text=test"]
' | kubectl apply -f -
echo '
apiVersion: tekton.dev/v1beta1
kind: TaskRun
metadata:
  generateName: test-run-
spec:
  taskRef:
    name: test2
' | kubectl create -f -

[marniks7]

Other notes:

1. The same problem with previous tekton release
2. Works fine with kind

[vdemeester]

@marniks7 👋🏼 Thanks for the issue.
When you don't specify a command, the tekton controller will try to fetch the image configuration, to know what is the configured image entrypoint (aka command). An error like this means that, for some odd reason, the controller cannot resolve artifactory.company.com, and thus fails to continue.
If you create an intercative pod in k3d and try to ping/curl the artifactory instance, does it work ?

$ kubectl run my-shell --rm -i --tty --image ubuntu -- bash
# […]
> ping artifactory.company.com
> curl https://artifactory.company.com:17064

If that doesn't work (aka if from a "random" pod you can't access artifcatory), I would think this is a network configuration problem in the k3d instance.

[marniks7]

Yes, you are right. curl: (6) Could not resolve host

k3d (kubelet) is able to pull the image from private artifactory, but not from inside pod (eg tekton-controller)
Issue: k3d-io/k3d#209 (with workarounds, but they didn't help me so far)

[marniks7]

Is it possible to fetch the images by kubelet?

---

Another variation of this issue (image fetch by tekton controller), but this time for kind with kind load docker-image

# docker build
docker build -t test:0.1 -f - . <<EOF
FROM busybox:latest
CMD echo test
EOF

# load local image to kind:
kind load docker-image test:0.1

# simple test, works fine
kubectl run test --restart Never --rm -it --image=test:0.1

# test with tekton - fails
echo '
apiVersion: tekton.dev/v1beta1
kind: Task
metadata:
  name: test
spec:
  steps:
    - name: step
      image: test:0.1
' | kubectl apply -f -
echo '
apiVersion: tekton.dev/v1beta1
kind: TaskRun
metadata:
  generateName: test-run-
spec:
  taskRef:
    name: test
' | kubectl create -f -

Error:
failed to create task run pod "test-run-bflvb": translating TaskSpec to Pod: GET https://index.docker.io/v2/library/test/manifests/0.1: UNAUTHORIZED: authentication required; [map[Action:pull Class: Name:library/test Type:repository]]. Maybe missing or invalid Task default/test

[tekton-robot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale with a justification.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close with a justification.
If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/lifecycle stale

Send feedback to tektoncd/plumbing.

[tekton-robot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten with a justification.
Rotten issues close after an additional 30d of inactivity.
If this issue is safe to close now please do so with /close with a justification.
If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/lifecycle rotten

Send feedback to tektoncd/plumbing.

[tekton-robot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen with a justification.
Mark the issue as fresh with /remove-lifecycle rotten with a justification.
If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/close

Send feedback to tektoncd/plumbing.

[tekton-robot]

@tekton-robot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen with a justification.
Mark the issue as fresh with /remove-lifecycle rotten with a justification.
If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/close

Send feedback to tektoncd/plumbing.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.