[Resolved] Git Cache Behavior Isn't Great

[jeff-mccoy]

Since we've added the caching feature for git repos, we've seen a lot of issues. Initially, I didn't think caching for git was valuable due to its complexity under the hood (unlike files, for example), and we've seen this born out with git-isms and quirks of go-git. I propose we change this behavior to be opt-in or even consider removing it altogether and suggest those that need to use something with actions to manage their very large repos instead.

[Racer159]

See this for another example: https://kubernetes.slack.com/archives/C03B6BJAUJ3/p1676641285886259

[Racer159]

As a note, if we do remove this we could have people explicitly use file://path/to/my/git/repo now that that is supported instead to allow people to control a repo from a local folder.

[kirankt]

I think changes such as this should be well documented/communicated. I found this problem by reading through the code after encountering the git repo failure (I had upgraded from 0.23 to 0.24). Also, I'm not a fan of forcing git creds on end users who just want to consume a public repo.

Edit:
Having said that, I do appreciate the speed with which re-builds happen due to caching.

[jeff-mccoy]

I don't think there's currently a git creds issue anymore--but I do agree on being mindful of communicating any changes in existing behavior, even if we believe there are no side-effects.

[wirewc]

Opt-in sounds like the way to go especially since Zarf is not air gaped only and seems like the caching issue causes issues even in disconnected (but not air gapped) environments if that makes sense.

[Racer159]

Here is an example PR of what a simple removal would look like: #1398

For anyone using git caching for particularly large repos, thoughts on this pattern?

kind: ZarfPackageConfig
metadata:
  name: git-data
  version: v1.0.0

components:
  - name: full-repo
    required: true
    repos:
      - file:///path/to/my/really/big/repo

From what I can see:

Pros:

• Simpler Zarf code maintenance without needing to handle issues that may occur from the way different git hosts and histories may exist
• More control in the hands of the package author to pull the exact point in history they want

Cons:

• The file:///path/repo path is specific to the build machine and there is the potential to create more git repos than desired if that path changes

[jeff-mccoy]

I like this a lot....the only issue I see is for the zarf agent, how would it be able to handle url mutation as it wouldn't know about the upstream?

[jeff-mccoy]

This is resolved now with #1398. Thanks @Racer159 🥳