feat!: central cache with symlinks

[Zebradil]

This PR is built on top of #257 and changes it to use symlinks for maintaining the central cache. Tests pass but I might have missed something (also, there are no new tests for new functions, it's TBD for later).

TODO:

• cleanup old links from the vendor directory
• exclude content's path from cache name generation

Create tasks to:

• cleanup cache storage from old downloads (only possible when running myks sync ALL) (Command for cleaning cache #281)
• fix race condition when writing to cache (look for a comment in the code) (Rewrite syncing to avoid race conditions #283)

[fritzduchardt]

Hi @Zebradil, I would like to talk about this one in a call. To give you a head start:

Does not render fully:

apiVersion: vendir.k14s.io/v1alpha1
kind: Config
directories:
- path: charts/
  contents:
  - path: cert-manager
    lazy: true
    helmChart:
      name: cert-manager
      version: v1.14.2
      repository:
        url: https://charts.jetstack.io
- path: charts/
  contents:
  - path: multus-cni
    lazy: true
    helmChart:
      name: multus-cni
      version: 1.1.10
      repository:
        url: https://charts.bitnami.com/bitnami

Does not work at all:

apiVersion: vendir.k14s.io/v1alpha1
kind: Config
directories:
- path: charts/
  contents:
  - path: cert-manager
    lazy: true
    helmChart:
      name: cert-manager
      version: v1.14.2
      repository:
        url: https://charts.jetstack.io
  - path: multus-cni
    lazy: true
    helmChart:
      name: multus-cni
      version: 1.1.10
      repository:
        url: https://charts.bitnami.com/bitnami

[Zebradil]

Thank you for the examples. I'll take a closer look later today. But right now I can tell that the second example doesn't work because I explicitly restrict vendir configs to have only a single content per directory.

[Zebradil]

@fritzduchardt The first example is not valid vendir config:

$ cat <<EOF | vendir sync -f-
apiVersion: vendir.k14s.io/v1alpha1
kind: Config
directories:
  - path: charts/
    contents:
      - path: cert-manager
        lazy: true
        helmChart:
          name: cert-manager
          version: v1.14.2
          repository:
            url: https://charts.jetstack.io
  - path: charts/
    contents:
      - path: multus-cni
        lazy: true
        helmChart:
          name: multus-cni
          version: 1.1.10
          repository:
            url: https://charts.bitnami.com/bitnami
EOF
vendir: Error: Parsing resource config '-':
  Unmarshaling config:
    Validating config:
      Expected to not manage overlapping paths: 'charts/' and 'charts/'

If I change the config in the following way, sync and render work correctly:

--- orig.yaml	2024-05-01 15:36:33.513465042 +0200
+++ changed.yaml	2024-05-01 15:36:07.393463391 +0200
@@ -1,18 +1,18 @@
 apiVersion: vendir.k14s.io/v1alpha1
 kind: Config
 directories:
-  - path: charts/
+  - path: charts/cert-manager
     contents:
-      - path: cert-manager
+      - path: .
         lazy: true
         helmChart:
           name: cert-manager
           version: v1.14.2
           repository:
             url: https://charts.jetstack.io
-  - path: charts/
+  - path: charts/multus-cni
     contents:
-      - path: multus-cni
+      - path: .
         lazy: true
         helmChart:
           name: multus-cni

We can implement support for that, but I'd approach this as a separate topic. What we should definitely have is a proper error message in this case.

[fritzduchardt]

Thanks for fixing the second example. However, rendering with this only leads to the first chart being rendered for me.

[Zebradil]

Thanks for fixing the second example

But I didn't fix it. The second example won't work and it was my deliberate decision. I mentioned this in a comment earlier: #274 (comment)

I also briefly touched on that during our last call, but that probably was missed among lots of other details we discussed.

The reason for that was some logical complications of that state of the code, it could be already not an issue, I'll double-check that.

However, rendering with this only leads to the first chart being rendered for me.

Sorry, I didn't get. With wich one? Could you post the config which doesn't work for you?

[fritzduchardt]

My bad - I meant the first example, the one you fixed. That one only renders the first workload.

[Zebradil]

@fritzduchardt sorry, but I can't reproduce.

With this vendir config (your first example, fixed): https://github.com/Zebradil/vigilant-bassoon/blob/a058bd96be8dd1ff4269ec62ddb6ca7e285a5340/prototypes/two-charts/vendir/all.yaml
Cert-manager is rendered: https://github.com/Zebradil/vigilant-bassoon/blob/a058bd96be8dd1ff4269ec62ddb6ca7e285a5340/rendered/envs/mykso-dev/two-charts/deployment-cert-manager.yaml
Multus-cni is rendered: https://github.com/Zebradil/vigilant-bassoon/blob/a058bd96be8dd1ff4269ec62ddb6ca7e285a5340/rendered/envs/mykso-dev/two-charts/daemonset-multus-cni.yaml

Do I miss something?

[fritzduchardt]

Damn, I keep testing vendir.yamls that have overlapping paths (I put the dot at directory path level), which is now properly validated by vendir. However, this is not validated by myks, so it tries to sync and render those invalid vendir.yamls, without feedback to the user leading to unexpected results. I overlooked your comment regarding this a while back, sorry. Let's get this one merged, but put in a follow-up to ensure that we don't lose vendir validation.

[fritzduchardt]

This could fail, if the dir is not empty, e.g. if the user shortens a path in the vendir yaml, e.g. going from "ytt/grafana-dashboards" to just "ytt". Also, old directories are not cleaned up on path changes. How about simply deleting all previously created dirs before syncing?

[Zebradil]

Good catch! I now delete the whole vendor directory before creating any links.

[fritzduchardt]

One more thing I noticed while using this branch for my home automation: We should consider removing the created cache dir on failure. Failure will usually be linked to wrong vendir.yaml config and thus the prior created entries are obsolete and mess up the cache. Using the cache to look at helm charts is quite useful during development, therefore keeping it tidy is worthwhile.

[Zebradil]

This makes sense, implemented.