How to use the kind local registry?

[slinkydeveloper]

Just a question: how does the kind local image registry works with knative? Perhaps could we have a very little section in the kn quickstart kind docs explaining how to go from "I have an image built in my local docker/podman daemon" to "kn service create"?

[GeorgGroenendaal]

I found that kn quickstart --registry creates a container with the name kind-registry with its ports forwarded to 0.0.0.0:5001 on the host machine.

However, deploying a knative function using kn func deploy --registry localhost:5001 fails with the following message.

Note: building a function the first time will take longer than subsequent builds
Building function image
🙌 Function built: localhost:5001/randomnu:latest
Pushing function image to the registry "localhost:5001" using the "" user credentials
Error: PUT http://localhost:5001/v2/randomnu/manifests/latest: multiple errors returned: MANIFEST_BLOB_UNKNOWN: blob unknown to registry; sha256:e0a9f5911802534ba097660206feabeb0247a81e409029167b30e2e1f2803b57; MANIFEST_BLOB_UNKNOWN: blob unknown to registry; sha256:2d5a94741de167f82b09ada3b9488619d2efce89be9e6dc9b81653aa70151ffa; MANIFEST_BLOB_UNKNOWN: blob unknown to registry; sha256:5a30d6101dd319eee27c5eb30e923a9fbc9a72f616b6d1900543cd42e74b4893; MANIFEST_BLOB_UNKNOWN: blob unknown to registry; sha256:63f9ee1b72a7d3644fb4a0932b6a5c5157aee6952a3f8a086f80bd3fbd15f4fb; MANIFEST_BLOB_UNKNOWN: blob unknown to registry; sha256:bbc40855e57596d1d12ae58255495ab890f5110aff366c3b41c2f1abbe761311; MANIFEST_BLOB_UNKNOWN: blob unknown to registry; sha256:e21456b047dadc0fcb26aa66b029aae0301acbfd8c80a01a92c346d07171ca22; MANIFEST_BLOB_UNKNOWN: blob unknown to registry; sha256:3f51c6473a1118af9efaf0237fa8dc3b8d830da6a16b1203711754e3f969bc53; MANIFEST_BLOB_UNKNOWN: blob unknown to registry; sha256:6557704db81e9c11c1b86ff51338d77ca017a9137a068f8ab665acf677ae6759; MANIFEST_BLOB_UNKNOWN: blob unknown to registry; sha256:727fe0b5d90151ed152db6c912ed552065e0a5b1e1040dd17b0637699d7fffe7; MANIFEST_BLOB_UNKNOWN: blob unknown to registry; sha256:92f1d3712a46e11688a8a106436d12fcb67b8382dade5ed6a9123f74b20f3369; MANIFEST_BLOB_UNKNOWN: blob unknown to registry; sha256:1fd3e70221b1cbc0f059a5bfd26c095e7e7de5c060f2eb61652eb24df9df2df8; MANIFEST_BLOB_UNKNOWN: blob unknown to registry; sha256:83d85471d9f8a3834b4e27cf701e3f0aef220cc816d9c173c7d32cd73909a590

I can push to the local kind registry using docker directly though.

[psschwei]

There might be some config needed for func to push to a local kind registry ... going to tag in @lkingland and @salaboy who would know more about the func side of the house

[matzew]

@slinkydeveloper 👋
I was now (after #525) able to run kind mode w/ local registry. NOTE: I am using podman, not the docker CLI.

I had issued:

func deploy --registry localhost:5001

and it worked. See a snippet of the deployment behind the ksvc:

...
    spec:
      containers:
      - env:
        - name: BUILT
          value: 20240618T103140
        - name: ADDRESS
          value: 0.0.0.0
        - name: PORT
          value: "8080"
        - name: K_REVISION
          value: s3-logger-00001
        - name: K_CONFIGURATION
          value: s3-logger
        - name: K_SERVICE
          value: s3-logger
        image: localhost:5001/s3-logger@sha256:<.........>
        imagePullPolicy: IfNotPresent

...

Also, I have an entry below in my /etc/containers/registries.conf file:

 [[registry-insecure-local]]
 location = "localhost:THE_ACTUAL_PORT"
 insecure = true

See here for more: https://www.redhat.com/sysadmin/manage-container-registries

[matzew]

UPDATE: for the quickstart registry I have none of these registry entries in that file. Was for a different one. sorry

[matzew]

@GeorgGroenendaal I noticed a similar, but yet different issue. See here:
knative/func#2371

[psschwei]

kind docs on using the registry we may want to make use of: https://kind.sigs.k8s.io/docs/user/local-registry/#using-the-registry

[slinkydeveloper]

@matzew Maybe unrelated issue here, but now i'm struggling to create the kind cluster :)

!10281 ➜ kn quickstart kind --registry  

Running Knative Quickstart using Kind
✅ Checking dependencies...
    Kind version is: 0.23.0
💽 Installing local registry...
☸ Creating Kind cluster...
enabling experimental podman provider
Creating cluster "knative" ...
 ✓ Ensuring node image (kindest/node:v1.30.0) 🖼
 ✓ Preparing nodes 📦  
 ✓ Writing configuration 📜 
 ✓ Starting control-plane 🕹️ 
 ✓ Installing CNI 🔌 
 ✓ Installing StorageClass 💾 
 ✓ Waiting ≤ 2m0s for control-plane = Ready ⏳ 
 • Ready after 15s 💚
Set kubectl context to "kind-knative"
You can now use your cluster with:

kubectl cluster-info --context kind-knative

Thanks for using kind! 😊

Error: creating cluster: existing cluster: failed to create local registry container: exit status 125
Usage:
  kn-quickstart kind [flags]

Flags:
      --extraMountContainerPath string   set the extraMount containerPath on Kind quickstart cluster
      --extraMountHostPath string        set the extraMount hostPath on Kind quickstart cluster
  -h, --help                             help for kind
      --install-eventing                 install Eventing on quickstart cluster
      --install-serving                  install Serving on quickstart cluster
  -k, --kubernetes-version string        kubernetes version to use (1.x.y) or (kindest/node:v1.x.y)
  -n, --name string                      kind cluster name to be used by kn-quickstart (default "knative")
      --registry                         install registry for Kind quickstart cluster

creating cluster: existing cluster: failed to create local registry container: exit status 125
Error: exit status 1

Versions:

!10282 ➜ podman version 
Client:       Podman Engine
Version:      4.9.4
API Version:  4.9.4
Go Version:   go1.21.8
Built:        Tue Mar 26 10:39:52 2024
OS/Arch:      linux/amd64
!10284 ➜ kn version                  
Version:      v1.15.0
Build Date:   2024-07-24 15:17:26
Git Revision: 59dd72a2
Supported APIs:
* Serving
  - serving.knative.dev/v1 (knative-serving v1.15.0)
* Eventing
  - sources.knative.dev/v1 (knative-eventing v1.15.0)
  - eventing.knative.dev/v1 (knative-eventing v1.15.0)
!10285 ➜ kn quickstart version                   
Version:      v1.15.0
Build Date:   2024-07-24 18:08:21
Git Revision: 4ce080d

[psschwei]

I think the command that's failing here is: podman run -d --restart=always -p 0.0.0.0:5001:5000 --network bridge --name kind-registry registry:2

What output do you get from that command?

(when I ran that on my machine, I got a bind error due to the registry also running on docker)

[slinkydeveloper]

What output do you get from that command?

Yes you're right, that command is wrong. The thing is that, with a clean env, when I try to run it, it just works fine. After some googling, I figured that podman exit status code 125 means that the container already exists, so maybe the tool tries to create the container two times or something like that?