Source controller fails to correctly fetch the git repositories #350

Diaoul · 2021-04-23T07:55:30Z

Describe the bug

The bug manifests with errors on reconciliations such as apply failed: error: no objects passed to apply

❯ flux get kustomizations -A
NAMESPACE      NAME                                        READY    MESSAGE                                                            REVISION                                                                 SUSPENDED
flux-system    kube-prometheus-stack-crds                  False    apply failed: error: no objects passed to apply                    kube-prometheus-stack-15.1.3/8e4a0fed3c1ea36293f0e5af4001ccc8c4110293    False

or

kustomization path not found: stat /tmp/system-flux132511846/cluster/base: no such file or directory

It is due to the source controller not fetching the repository correctly (32 bytes tarball)

/data/gitrepository/flux-system/kube-prometheus-stack-crds $ ls -al
total 16
drwxr-sr-x    2 controll 1337          4096 Apr 22 22:47 .
drwxr-sr-x    6 controll 1337          4096 Apr 22 20:18 ..
-rw-r--r--    1 controll 1337            32 Apr 22 22:33 531ce5c7d64da93901b643a2b737d36117a2776a.tar.gz
lrwxrwxrwx    1 controll 1337           106 Apr 22 22:33 latest.tar.gz -> /data/gitrepository/flux-system/kube-prometheus-stack-crds/531ce5c7d64da93901b643a2b737d36117a2776a.tar.gz

Despite reporting success

❯ flux get source git -A
NAMESPACE      NAME                                        READY    MESSAGE                                                                                    REVISION                                                                 SUSPENDED
flux-system    kube-prometheus-stack-crds                  True     Fetched revision: kube-prometheus-stack-15.2.0/531ce5c7d64da93901b643a2b737d36117a2776a    kube-prometheus-stack-15.2.0/531ce5c7d64da93901b643a2b737d36117a2776a    False

To Reproduce

Have a GitRepository
Reconcile it
Check the source controller data

Expected behavior

Repository not empty

Additional context

This bug was introduces with v0.13.0, previous version worked fine.

$ flux --version
flux version 0.13.0
$ flux check
► checking prerequisites
✔ kubectl 1.21.0 >=1.18.0-0
✔ Kubernetes 1.20.6+k3s1 >=1.16.0-0
► checking controllers
✔ kustomize-controller: deployment ready
► ghcr.io/fluxcd/kustomize-controller:v0.11.1
✔ helm-controller: deployment ready
► ghcr.io/fluxcd/helm-controller:v0.10.0
✔ notification-controller: deployment ready
► ghcr.io/fluxcd/notification-controller:v0.13.0
✔ source-controller: deployment ready
► ghcr.io/fluxcd/source-controller:v0.12.0
✔ all checks passed
$ kubectl -n <namespace> get all
NAME                                           READY   STATUS    RESTARTS   AGE
pod/kustomize-controller-78bdd99b66-d2nqm      1/1     Running   0          11h
pod/helm-controller-69667f94bc-jk9bj           1/1     Running   0          11h
pod/notification-controller-5c4d48f476-7k9rh   1/1     Running   0          11h
pod/source-controller-6c458cdb8d-jlzdn         1/1     Running   0          11h

NAME                              TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
service/notification-controller   ClusterIP   10.1.123.23   <none>        80/TCP    28h
service/source-controller         ClusterIP   10.1.234.45    <none>        80/TCP    28h
service/webhook-receiver          ClusterIP   10.1.45.56    <none>        80/TCP    28h

NAME                                      READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/kustomize-controller      1/1     1            1           28h
deployment.apps/helm-controller           1/1     1            1           28h
deployment.apps/notification-controller   1/1     1            1           28h
deployment.apps/source-controller         1/1     1            1           28h

NAME                                                 DESIRED   CURRENT   READY   AGE
replicaset.apps/source-controller-785dd8f8b9         0         0         0       28h
replicaset.apps/kustomize-controller-78bdd99b66      1         1         1       11h
replicaset.apps/kustomize-controller-97dbdcb99       0         0         0       28h
replicaset.apps/helm-controller-69667f94bc           1         1         1       11h
replicaset.apps/helm-controller-5bbd94c75            0         0         0       28h
replicaset.apps/notification-controller-5c4d48f476   1         1         1       11h
replicaset.apps/notification-controller-d66bdc787    0         0         0       28h
replicaset.apps/source-controller-6c458cdb8d         1         1         1       11h
$ kubectl -n flux-system logs deploy/source-controller
{"level":"info","ts":"2021-04-23T06:48:06.075Z","logger":"controller.gitrepository","msg":"Reconciliation finished in 690.693629ms, next run in 30m0s","reconciler group":"source.toolkit.fluxcd.io","reconciler kind":"GitRepository","name":"kube-prometheus-stack-crds","namespace":"flux-system"}
kubectl -n flux-system logs deploy/kustomize-controller
{"level":"error","ts":"2021-04-23T07:48:01.553Z","logger":"controller.kustomization","msg":"Reconciliation failed after 422.112691ms, next try in 30m0s","reconciler group":"kustomize.toolkit.fluxcd.io","reconciler kind":"Kustomization","name":"kube-prometheus-stack-crds","namespace":"flux-system","revision":"kube-prometheus-stack-15.2.0/531ce5c7d64da93901b643a2b737d36117a2776a","error":"apply failed: error: no objects passed to apply\n"}

The text was updated successfully, but these errors were encountered:

stefanprodan · 2021-04-23T07:57:56Z

You have to set the path to where the CRDs are.

Diaoul · 2021-04-23T07:59:43Z

Should I modify this?

---
apiVersion: source.toolkit.fluxcd.io/v1beta1
kind: GitRepository
metadata:
  name: kube-prometheus-stack-crds
  namespace: flux-system
spec:
  interval: 30m
  url: https://github.com/prometheus-community/helm-charts.git
  ref:
    # renovate: registryUrl=https://prometheus-community.github.io/helm-charts
    tag: kube-prometheus-stack-15.2.0
  ignore: |
    # exclude all
    /*
    # include crds directory
    !/charts/kube-prometheus-stack/crds
---
apiVersion: kustomize.toolkit.fluxcd.io/v1beta1
kind: Kustomization
metadata:
  name: kube-prometheus-stack-crds
  namespace: flux-system
spec:
  interval: 30m
  prune: false
  sourceRef:
    kind: GitRepository
    name: kube-prometheus-stack-crds

bjw-s · 2021-04-23T07:59:47Z

I am seeing the same issue, even on my root-level system-flux Kustomization, which looks similar to @Diaoul's manifest. For completeness sake, here are my manifests for that:
https://github.com/bjw-s/k8s-gitops/blob/main/cluster/base/system-flux/gotk-sync.yaml

I didn't see anything in the 0.13 release notes that indicates I would need to change anything?

stefanprodan · 2021-04-23T08:02:28Z

@Diaoul you have to set the path:

apiVersion: kustomize.toolkit.fluxcd.io/v1beta1
kind: Kustomization
metadata:
  name: kube-prometheus-stack-crds
  namespace: flux-system
spec:
  interval: 30m
  prune: false
  sourceRef:
    kind: GitRepository
    name: kube-prometheus-stack-crds
  path: ./charts/kube-prometheus-stack/crds

Diaoul · 2021-04-23T08:03:44Z

Then I imagine the ignore isn't needed anymore in the GitRepository?

Diaoul · 2021-04-23T08:05:42Z

Now I have this:

flux-system	kube-prometheus-stack-crds              	False	kustomization path not found: stat /tmp/kube-prometheus-stack-crds105344257/charts/kube-prometheus-stack/crds: no such file or directory	kube-prometheus-stack-15.1.3/8e4a0fed3c1ea36293f0e5af4001ccc8c4110293	False

I don't see how this solves the repository being empty though

stefanprodan · 2021-04-23T08:06:11Z

Then I imagine the ignore isn't needed anymore in the GitRepository?

Ignore has been refactored and I think we have a bug where all files are removed.

To answer your question, you should use ignore to trim down the artifact, no point in storing files in-cluster that you don't need.

stefanprodan · 2021-04-23T08:11:45Z

@Diaoul as a workaround until we fix this, you can now tell helm-controller to do the CRDs upgrade, no need to use a Kustomization anymore, see https://toolkit.fluxcd.io/components/helm/helmreleases/#crds

hiddeco · 2021-04-23T08:48:37Z

Have identified the core of the issue, will work on a patch that should be released later today.

onedr0p · 2021-04-23T09:02:09Z

@Diaoul as a workaround until we fix this, you can now tell helm-controller to do the CRDs upgrade, no need to use a Kustomization anymore, see https://toolkit.fluxcd.io/components/helm/helmreleases/#crds

That works for helm charts that package CRDs, however in kube-prometheus-stack they were removed from the packaged chart... you have to apply them manually.

hiddeco · 2021-04-23T11:02:44Z

Pulling in v0.12.1 of the source-controller should solve this issue, flux release is building.

bjw-s · 2021-04-23T11:05:05Z

Awesome, thanks for the quick fix!

lrstanley · 2021-08-05T18:19:21Z

Is it possible this bug could have re-surfaced? With just a test k8s cluster (docker desktop), I am trying to install flux2, and point to folder in my repository, ./tooling-eks/clusters/staging (based off https://github.com/fluxcd/flux2-kustomize-helm-example, but throwing everything under an addl. parent folder)

Repo structure (from the root):

liam:flux2-testing$ tree tooling-eks/
tooling-eks/
├── apps
│   ├── base
│   │   └── podinfo
│   │       ├── kustomization.yaml
│   │       ├── namespace.yaml
│   │       └── release.yaml
│   ├── production
│   │   ├── kustomization.yaml
│   │   └── podinfo-values.yaml
│   └── staging
│       ├── kustomization.yaml
│       └── podinfo-values.yaml
├── clusters
│   ├── production
│   │   ├── apps.yaml
│   │   └── infrastructure.yaml
│   └── staging
│       ├── apps.yaml
│       └── infrastructure.yaml
└── infrastructure
    ├── kustomization.yaml
    ├── nginx
    │   ├── kustomization.yaml
    │   ├── namespace.yaml
    │   └── release.yaml
    ├── redis
    │   ├── kustomization.yaml
    │   ├── kustomizeconfig.yaml
    │   ├── namespace.yaml
    │   ├── release.yaml
    │   └── values.yaml
    └── sources
        ├── bitnami.yaml
        ├── kustomization.yaml
        └── podinfo.yaml

Creating the source with:

$ flux create source git staging --url ssh://[email protected]/lrstanley/flux2-testing.git --branch main --interval 30s

$ flux get sources git -A
NAMESPACE       NAME    READY   MESSAGE                                                         REVISION                                        SUSPENDED 
flux-system     staging True    Fetched revision: main/76154b7474b7ed488a23fd0ef57899f8ff0dbb1d main/76154b7474b7ed488a23fd0ef57899f8ff0dbb1d   False

It successfully pulls the right branch and commit. Looking at the files on the pod, under /data/gitrepository/flux-system/staging/, it almost looks as though the tar files are empty. trying to cp the files using kubectl onto my local machine, when I tar -l, nothing is returned...

/data/gitrepository/flux-system/staging $ ls -lah
total 16K
drwxr-sr-x    2 controll 1337        4.0K Aug  5 16:32 .
drwxr-sr-x    3 controll 1337        4.0K Aug  5 16:31 ..
-rw-r--r--    1 controll 1337          32 Aug  5 16:31 76154b7474b7ed488a23fd0ef57899f8ff0dbb1d.tar.gz
lrwxrwxrwx    1 controll 1337          87 Aug  5 16:31 latest.tar.gz -> /data/gitrepository/flux-system/staging/76154b7474b7ed488a23fd0ef57899f8ff0dbb1d.tar.gz

Tried similar with source-controller version v0.15.3, and similar issues, 32 byte file, but I am still unable to read the tar files or get anything form them. Trying to extract them, and it returns successfully, without extracting any files.

And ofc when I try and setup a kustomization based off the source, it says files don't exist (this is on source-controller v0.12.1):

$ flux create kustomization staging --source=GitRepository/staging --path="./tooling-eks/clusters/staging/" --prune=true
✚ generating Kustomization
► applying Kustomization
✔ Kustomization created
◎ waiting for Kustomization reconciliation
✗ kustomization path not found: stat /tmp/staging242136554/tooling-eks/clusters/staging: no such file or directory

Maybe I'm missing something simple here. I tried running v0.15.3 of the source-controller as well.

Any addl. info I can provide?

hiddeco · 2021-08-05T18:28:56Z

Any chance the https://github.com/fluxcd/flux2-kustomize-helm-example/blob/main/.sourceignore is still the same, and excluding your directory?

lrstanley · 2021-08-05T18:43:51Z

Any chance the https://github.com/fluxcd/flux2-kustomize-helm-example/blob/main/.sourceignore is still the same, and excluding your directory?

Fuck. I didn't realize this was a thing. I have been troubleshooting this the entire WEEK, I am not sure how I didn't see this...

Thanks, looks like it's just me being dumb. 🤦🏻

stefanprodan transferred this issue from fluxcd/flux2 Apr 23, 2021

stefanprodan added the bug Something isn't working label Apr 23, 2021

stefanprodan assigned hiddeco Apr 23, 2021

hiddeco mentioned this issue Apr 23, 2021

Configure ignore domain for GitRepository rules #351

Merged

hiddeco closed this as completed in #351 Apr 23, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Source controller fails to correctly fetch the git repositories #350

Source controller fails to correctly fetch the git repositories #350

Diaoul commented Apr 23, 2021

stefanprodan commented Apr 23, 2021

Diaoul commented Apr 23, 2021

bjw-s commented Apr 23, 2021 •

edited

Loading

stefanprodan commented Apr 23, 2021 •

edited

Loading

Diaoul commented Apr 23, 2021 •

edited

Loading

Diaoul commented Apr 23, 2021

stefanprodan commented Apr 23, 2021 •

edited

Loading

stefanprodan commented Apr 23, 2021

hiddeco commented Apr 23, 2021

onedr0p commented Apr 23, 2021 •

edited

Loading

hiddeco commented Apr 23, 2021

bjw-s commented Apr 23, 2021

lrstanley commented Aug 5, 2021 •

edited

Loading

hiddeco commented Aug 5, 2021

lrstanley commented Aug 5, 2021

Source controller fails to correctly fetch the git repositories #350

Source controller fails to correctly fetch the git repositories #350

Comments

Diaoul commented Apr 23, 2021

Describe the bug

To Reproduce

Expected behavior

Additional context

stefanprodan commented Apr 23, 2021

Diaoul commented Apr 23, 2021

bjw-s commented Apr 23, 2021 • edited Loading

stefanprodan commented Apr 23, 2021 • edited Loading

Diaoul commented Apr 23, 2021 • edited Loading

Diaoul commented Apr 23, 2021

stefanprodan commented Apr 23, 2021 • edited Loading

stefanprodan commented Apr 23, 2021

hiddeco commented Apr 23, 2021

onedr0p commented Apr 23, 2021 • edited Loading

hiddeco commented Apr 23, 2021

bjw-s commented Apr 23, 2021

lrstanley commented Aug 5, 2021 • edited Loading

hiddeco commented Aug 5, 2021

lrstanley commented Aug 5, 2021

bjw-s commented Apr 23, 2021 •

edited

Loading

stefanprodan commented Apr 23, 2021 •

edited

Loading

Diaoul commented Apr 23, 2021 •

edited

Loading

stefanprodan commented Apr 23, 2021 •

edited

Loading

onedr0p commented Apr 23, 2021 •

edited

Loading

lrstanley commented Aug 5, 2021 •

edited

Loading