Image Update Automation - Can't clone git repo

[dzirkler]

So, starting here because I feel like I may be missing something simple, but can move to an Issue if needed.

Basically, my image updater cannot clone the same git repo that created the image updater in the first place! It always has some sort of connection issue reaching github.com.

You can see below that my ImageRepository and ImagePolicy seem to be processing correctly, and I can provide those definitions if it's helpful.

I tried adding the EXPERIMENTAL_GIT_TRANSPORT flag, and it had no effect.

The most puzzling part is the same repo is what's running Flux and it's obviously working because that's what's applying these CRDs, so I'm hoping this is just something dumb on my part. Is there something obvious I'm missing?

TIA for any assistance!

---

Additional Details:

NAME                    LAST SCAN                       SUSPENDED       READY   MESSAGE
imagerepository/esphome 2023-11-02T05:30:26-07:00       False           True    successful scan: found 1683 tags

NAME                    LATEST IMAGE                    READY   MESSAGE
imagepolicy/esphome     esphome/esphome:2023.10.5       True    Latest image tag for 'esphome/esphome' resolved to 2023.10.5

NAME                                            LAST RUN                        SUSPENDED       READY   MESSAGE

imageupdateautomation/esphome-imageupdater      2023-11-02T09:06:08-07:00       False           False   unable to clone 'ssh://[email protected]/my/fluxcd': ssh: handshake failed: read tcp 10.10.120.11:58530->20.29.134.23:22: read: connection timed out

Describing the image updater has these lines in the Events:

Events:
  Type     Reason  Age                     From                         Message
  ----     ------  ----                    ----                         -------
  Warning  error   36m                     image-automation-controller  unable to clone 'ssh://[email protected]/my/fluxcd': dial tcp 140.82.113.4:22: connect: connection timed out
  Warning  error   5m49s (x18 over 7h43m)  image-automation-controller  unable to clone 'ssh://[email protected]/my/fluxcd': dial tcp 20.29.134.23:22: connect: connection timed out
  Warning  error   3m16s                   image-automation-controller  unable to clone 'ssh://[email protected]/my/fluxcd': ssh: handshake failed: read tcp 10.10.120.11:58530->20.29.134.23:22: read: connection timed out
  Warning  error   62s (x2 over 38m)       image-automation-controller  unable to clone 'ssh://[email protected]/my/fluxcd': dial tcp 140.82.112.3:22: connect: connection timed out

Logs from the image-automation-controller pod look similar:

{"level":"error","ts":"2023-11-02T16:23:44.989Z","msg":"Reconcilererror","controller":"imageupdateautomation","controllerGroup":"image.toolkit.fluxcd.io","controllerKind":"ImageUpdateAutomation","ImageUpdateAutomation":{"name":"flux-system","namespace":"flux-system"},"namespace":"flux-system","name":"flux-system","reconcileID":"e4bc2dad-238a-4cd3-b865-38c309646ab0","error":"unable to clone 'ssh://[email protected]/my/fluxcd': dial tcp 20.29.134.23:22: connect: connection timed out"}
{"level":"info","ts":"2023-11-02T16:23:46.842Z","msg":"no changes made in working directory; no commit","controller":"imageupdateautomation","controllerGroup":"image.toolkit.fluxcd.io","controllerKind":"ImageUpdateAutomation","ImageUpdateAutomation":{"name":"flux-system","namespace":"flux-system"},"namespace":"flux-system","name":"flux-system","reconcileID":"d741bd6a-0352-4a40-943a-5beca476de82"}
{"level":"info","ts":"2023-11-02T16:26:02.161Z","msg":"no changes made in working directory; no commit","controller":"imageupdateautomation","controllerGroup":"image.toolkit.fluxcd.io","controllerKind":"ImageUpdateAutomation","ImageUpdateAutomation":{"name":"flux-system","namespace":"flux-system"},"namespace":"flux-system","name":"flux-system","reconcileID":"4dacd010-cffa-448e-8934-f95e175036fa"}
{"level":"info","ts":"2023-11-02T16:26:03.653Z","msg":"no changes made in working directory; no commit","controller":"imageupdateautomation","controllerGroup":"image.toolkit.fluxcd.io","controllerKind":"ImageUpdateAutomation","ImageUpdateAutomation":{"name":"flux-system","namespace":"flux-system"},"namespace":"flux-system","name":"flux-system","reconcileID":"b0547cfd-f8a7-4c6b-a832-a612ab3e0a55"}
{"level":"info","ts":"2023-11-02T16:26:22.472Z","msg":"no changes made in working directory; no commit","controller":"imageupdateautomation","controllerGroup":"image.toolkit.fluxcd.io","controllerKind":"ImageUpdateAutomation","ImageUpdateAutomation":{"name":"esphome-imageupdater","namespace":"esphome"},"namespace":"esphome","name":"esphome-imageupdater","reconcileID":"72f031fe-2a18-4d62-96ec-c3969680818b"}
{"level":"error","ts":"2023-11-02T16:28:56.340Z","msg":"Reconciler error","controller":"imageupdateautomation","controllerGroup":"image.toolkit.fluxcd.io","controllerKind":"ImageUpdateAutomation","ImageUpdateAutomation":{"name":"flux-system","namespace":"flux-system"},"namespace":"flux-system","name":"flux-system","reconcileID":"83782a82-603e-4b09-854f-073e805eae4e","error":"unable to clone 'ssh://[email protected]/my/fluxcd': dial tcp 140.82.113.4:22: connect: connection timed out"}
{"level":"info","ts":"2023-11-02T16:28:58.330Z","msg":"no changes made in working directory; no commit","controller":"imageupdateautomation","controllerGroup":"image.toolkit.fluxcd.io","controllerKind":"ImageUpdateAutomation","ImageUpdateAutomation":{"name":"flux-system","namespace":"flux-system"},"namespace":"flux-system","name":"flux-system","reconcileID":"a7f4f03 ││ 1-82d2-4ac2-a25f-34824b0bac95"}
{"level":"info","ts":"2023-11-02T16:29:15.990Z","msg":"no changes made in working directory; no commit","controller":"imageupdateautomation","controllerGroup":"image.toolkit.fluxcd.io","controllerKind":"ImageUpdateAutomation","ImageUpdateAutomation":{"name":"flux-system","namespace":"flux-system"},"namespace":"flux-system","name":"flux-system","reconcileID":"cc249f9 ││ a-9d4a-477c-a13c-949574083ac9"}
{"level":"info","ts":"2023-11-02T16:29:17.444Z","msg":"no changes made in working directory; no commit","controller":"imageupdateautomation","controllerGroup":"image.toolkit.fluxcd.io" ││ ,"controllerKind":"ImageUpdateAutomation","ImageUpdateAutomation":{"name":"flux-system","namespace":"flux-system"},"namespace":"flux-system","name":"flux-system","reconcileID":"055cb01 ││ 9-4637-41e1-bcbf-769f616f45f5"}

ImageUpdateAutomation.yml:

apiVersion: image.toolkit.fluxcd.io/v1beta1
kind: ImageUpdateAutomation
metadata:
  name: esphome-imageupdater
  namespace: esphome
spec:
  interval: 3m # 30m
  sourceRef:
    kind: GitRepository
    name: flux-system
    namespace: flux-system
  git:
    checkout:
      ref:
        branch: main
    commit:
      author:
        email: [email protected]
        name: fluxcdbot
      messageTemplate: '{{range .Updated.Images}}{{println .}}{{end}}'
    push:
      branch: main
  update:
    path: clusters/home-k3s
    strategy: Setters
  suspend: false

[makkes]

Can you run the following command and paste its output here, please? I have a suspicion that the image-automation-controller Pod is running on a node from where it can't reach github.com.

kubectl get po -n flux-system -o wide

[dzirkler]

I thought the same early on! But I was able to shell into that pod and was able to ping github.com. Without getting too creative, I wasn't sure how to check if i could open an SSH connection, but at least it's resolving and can ping.

$  kubectl get po -n flux-system -o wide
NAME                                           READY   STATUS    RESTARTS       AGE    IP             NODE                    NOMINATED NODE   READINESS GATES
helm-controller-7b96cdbc75-md7tl               1/1     Running   19 (14h ago)   5d2h   10.42.7.198    kube-work-11-28b5474b   <none>           <none>
image-automation-controller-754ff45ff8-hr6wg   1/1     Running   3 (14h ago)    22h    10.10.120.11   kube-work-11-28b5474b   <none>           <none>
image-reflector-controller-7f8dccbfb7-wdhpd    2/2     Running   3 (14h ago)    24h    10.42.7.232    kube-work-11-28b5474b   <none>           <none>
kustomize-controller-55f55dd7d7-l4xgg          1/1     Running   19 (14h ago)   5d2h   10.42.7.197    kube-work-11-28b5474b   <none>           <none>
notification-controller-7d7747dd84-8vpmk       1/1     Running   16 (14h ago)   5d2h   10.42.7.199    kube-work-11-28b5474b   <none>           <none>
source-controller-6ffb4bb87d-dhw5t             1/1     Running   6 (14h ago)    45h    10.42.7.228    kube-work-11-28b5474b   <none>           <none>

For reference, here's the nslookup and ping from inside that container.

~ $ nslookup github.com
Server:         10.43.0.10
Address:        10.43.0.10:53

Non-authoritative answer:

Non-authoritative answer:
Name:   github.com
Address: 140.82.114.4

~ $ ping github.com
PING github.com (140.82.114.4): 56 data bytes
64 bytes from 140.82.114.4: seq=0 ttl=42 time=69.355 ms
64 bytes from 140.82.114.4: seq=1 ttl=42 time=69.576 ms
64 bytes from 140.82.114.4: seq=2 ttl=42 time=69.454 ms
64 bytes from 140.82.114.4: seq=3 ttl=42 time=70.134 ms
^C
--- github.com ping statistics ---
4 packets transmitted, 4 packets received, 0% packet loss
round-trip min/avg/max = 69.355/69.629/70.134 ms
~ $

[makkes]

Try this:

nc github.com 22

Port 22 might be blocked.

[dzirkler]

Looks like it's able to open the connection and get some text back:

<<K9s-Shell>> Pod: flux-system/image-automation-controller-754ff45ff8-hr6wg | Container: manager
~ $ nc github.com 22
SSH-2.0-babeld-f8b1fc6c
login
^Cpunt!

[GIG0i]

@dzirkler I'm aware this is a relatively old post however I wonder if you found a solution at the end?
I'm seeing the same exact issue with the added variable that when I kill the pod sometimes it starts working! 🤔