Add git branch support, non-linear tag support and simplify git repo …

…logic (#1404)

This refactors our git logic to be more in line with the goals of #1405

Fixes #1405

- [x] Bug fix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Other (security config, docs update, etc)

- [ ] Test, docs, adr added or updated as needed
- [ ] [Contributor Guide
Steps](https://github.com/defenseunicorns/zarf/blob/main/CONTRIBUTING.md#developer-workflow)
followed

---------

Co-authored-by: Wayne Starr <[email protected]>
Co-authored-by: Wayne Starr <[email protected]>

examples/git-data/README.md

@@ -1,36 +1,34 @@
 # Git Data
 
-This example shows how to package `git` repositories to be bundled and pushed across the air gap.  This package does not deploy anything itself but pushes assets to the specified `git` service to be consumed as desired.  Within Zarf, there are two main ways to include `git` repositories as described below.
+This example shows how to package `git` repositories within a Zarf package.  This package does not deploy anything itself but pushes assets to the specified `git` service to be consumed as desired.  Within Zarf, there are a few ways to include `git` repositories (as described below).
 
 :::info
 
 To view the example source code, select the `Edit this page` link below the article and select the parent folder.
 
 :::
 
-## Tag-Provided Git Repository Clone
+## Tag-Based Git Repository Clone
 
-Tag-provided `git` repository cloning is the recommended way of cloning a `git` repository for air-gapped deployments because it wraps meaning around a specific point in git history that can easily be traced back to the online world. Tag-provided clones are defined using the `scheme://host/repo@tag` format as seen in the example of the `defenseunicorns/zarf` repository (`https://github.com/defenseunicorns/[email protected]`).
+Tag-based `git` repository cloning is the **recommended** way of cloning a `git` repository for air-gapped deployments because it wraps meaning around a specific point in git history that can easily be traced back to the online world. Tag-based clones are defined using the `scheme://host/repo@tag` format as seen in the example of the `defenseunicorns/zarf` repository (`https://github.com/defenseunicorns/[email protected]`).
 
-A tag-provided clone only mirrors the tag defined in the Zarf definition. The tag will be applied on the `git` mirror to the default trunk branch of the repo (i.e. `master`, `main`, or the default when the repo is cloned).
+A tag-based clone only mirrors the tag defined in the Zarf definition. The tag will be applied on the `git` mirror to a zarf-specific branch name based on the tag name (e.g. the tag `v0.1.0` will be pushed to the `zarf-ref-v0.1.0` branch).  This ensures that this tag will be pushed and received properly by the airgap `git` server.
 
 :::note
 
 If you would like to use a protocol scheme other than http/https, you can do so with something like the following: `ssh://[email protected]/defenseunicorns/[email protected]`.  Using this you can also clone from a local repo to help you manage larger git repositories: `file:///home/zarf/workspace/[email protected]`.
 
 :::
 
-## SHA-Provided Git Repository Clone
+## SHA-Based Git Repository Clone
 
-SHA-provided `git` repository cloning is another supported way of cloning repos in Zarf but is not recommended as it is less readable/understandable than tag cloning.  Commit SHAs are defined using the same `scheme://host/repo@shasum` format as seen in the example of the `defenseunicorns/zarf` repository (`https://github.com/defenseunicorns/zarf.git@c74e2e9626da0400e0a41e78319b3054c53a5d4e`).
+In addition to tags, Zarf also supports cloning and pushing a specific SHA hash from a `git` repository, but this is **not recommended** as it is less readable/understandable than tag cloning.  Commit SHAs are defined using the same `scheme://host/repo@shasum` format as seen in the example of the `defenseunicorns/zarf` repository (`https://github.com/defenseunicorns/zarf.git@c74e2e9626da0400e0a41e78319b3054c53a5d4e`).
 
-A SHA-provided clone only mirrors the SHA hash defined in the Zarf definition. The SHA will be applied on the `git` mirror to the default trunk branch of the repo (i.e. `master`, `main`, or the default when the repo is cloned) as Zarf does with tagging.
+A SHA-based clone only mirrors the SHA hash defined in the Zarf definition. The SHA will be applied on the `git` mirror to a zarf-specific branch name based on the SHA hash (e.g. the SHA `c74e2e9626da0400e0a41e78319b3054c53a5d4e` will be pushed to the `zarf-ref-c74e2e9626da0400e0a41e78319b3054c53a5d4e` branch).  This ensures that this tag will be pushed and received properly by the airgap `git` server.
 
-:::note
-
-If you use a SHA hash or a tag that is on a separate branch this will be placed on the default trunk branch on the offline mirror (i.e. `master`, `main`, etc).  This may result in conflicts upon updates if this SHA or tag is not in the update branch's history.
+## Git Reference-Based Git Repository Clone
 
-:::
+If you need even more control, Zarf also supports providing full `git` [refspecs](https://git-scm.com/book/en/v2/Git-Internals-The-Refspec), as seen in `https://repo1.dso.mil/big-bang/bigbang.git@refs/heads/release-1.53.x`.  This allows you to pull specific tags or branches by using this standard.  The branch name used by zarf on deploy will depend on the kind of ref specified, branches will use the upstream branch name, whereas other refs (namely tags) will use the `zarf-ref-*` branch name.
 
 ## Git Repository Full Clone
 

examples/git-data/zarf.yaml

@@ -5,26 +5,38 @@ metadata:
   description: "Demo Zarf loading resources into a gitops service"
 
 components:
-  - name: full-repo
+  - name: flux-demo
     required: true
     images:
       - ghcr.io/stefanprodan/podinfo:6.0.0
     repos:
-        # Do a full Git Repo Mirror
+      # Do a full Git Repo Mirror of a flux repo
       - https://github.com/stefanprodan/podinfo.git
-        # Clone an azure repo that breaks in go-git and has to fall back to the host git
-      - https://[email protected]/me0515/zarf-public-test/_git/zarf-public-test
+
+  - name: full-repo
+    required: true
+    repos:
+      # Do a full Git Repo Mirror
+      - https://github.com/kelseyhightower/nocode.git
 
   - name: specific-tag
     required: true
     repos:
-        # Do a tag-provided Git Repo mirror
+      # Do a tag-provided Git Repo mirror
       - https://github.com/defenseunicorns/[email protected]
+      # Use the git refspec pattern to get a tag
+      - https://github.com/defenseunicorns/zarf.git@refs/tags/v0.16.0
+
+  - name: specific-branch
+    required: true
+    repos:
+      # Do a branch-provided Git Repo mirror
+      - "https://repo1.dso.mil/big-bang/bigbang.git@refs/heads/release-1.53.x"
 
   - name: specific-hash
     required: true
     repos:
-        # Do a commit hash Git Repo mirror
+      # Do a commit hash Git Repo mirror
       - https://github.com/defenseunicorns/zarf.git@c74e2e9626da0400e0a41e78319b3054c53a5d4e
-        # Clone an azure repo (w/SHA) that breaks in go-git and has to fall back to the host git
+      # Clone an azure repo (w/SHA) that breaks in go-git and has to fall back to the host git
       - https://[email protected]/me0515/zarf-public-test/_git/zarf-public-test@524980951ff16e19dc25232e9aea8fd693989ba6

src/internal/cluster/tunnel.go

@@ -99,39 +99,39 @@ func (c *Cluster) PrintConnectTable() error {
 
 // ServiceInfoFromNodePortURL takes a nodePortURL and parses it to find the service info for connecting to the cluster. The string is expected to follow the following format:
 // Example nodePortURL: 127.0.0.1:{PORT}.
-func ServiceInfoFromNodePortURL(nodePortURL string) *ServiceInfo {
+func ServiceInfoFromNodePortURL(nodePortURL string) (*ServiceInfo, error) {
 	// Attempt to parse as normal, if this fails add a scheme to the URL (docker registries don't use schemes)
 	parsedURL, err := url.Parse(nodePortURL)
 	if err != nil {
 		parsedURL, err = url.Parse("scheme://" + nodePortURL)
 		if err != nil {
-			return nil
+			return nil, err
 		}
 	}
 
 	// Match hostname against localhost ip/hostnames
 	hostname := parsedURL.Hostname()
 	if hostname != config.IPV4Localhost && hostname != "localhost" {
-		return nil
+		return nil, fmt.Errorf("node port services should be on localhost")
 	}
 
 	// Get the node port from the nodeportURL.
 	nodePort, err := strconv.Atoi(parsedURL.Port())
 	if err != nil {
-		return nil
+		return nil, err
 	}
 	if nodePort < 30000 || nodePort > 32767 {
-		return nil
+		return nil, fmt.Errorf("node port services should use the port range 30000-32767")
 	}
 
 	kube, err := k8s.NewWithWait(message.Debugf, labels, defaultTimeout)
 	if err != nil {
-		return nil
+		return nil, err
 	}
 
 	services, err := kube.GetServices("")
 	if err != nil {
-		return nil
+		return nil, err
 	}
 
 	for _, svc := range services.Items {
@@ -142,43 +142,43 @@ func ServiceInfoFromNodePortURL(nodePortURL string) *ServiceInfo {
 						Namespace: svc.Namespace,
 						Name:      svc.Name,
 						Port:      int(port.Port),
-					}
+					}, nil
 				}
 			}
 		}
 	}
 
-	return nil
+	return nil, fmt.Errorf("no matching node port services found")
 }
 
 // ServiceInfoFromServiceURL takes a serviceURL and parses it to find the service info for connecting to the cluster. The string is expected to follow the following format:
 // Example serviceURL: http://{SERVICE_NAME}.{NAMESPACE}.svc.cluster.local:{PORT}.
-func ServiceInfoFromServiceURL(serviceURL string) *ServiceInfo {
+func ServiceInfoFromServiceURL(serviceURL string) (*ServiceInfo, error) {
 	parsedURL, err := url.Parse(serviceURL)
 	if err != nil {
-		return nil
+		return nil, err
 	}
 
 	// Get the remote port from the serviceURL.
 	remotePort, err := strconv.Atoi(parsedURL.Port())
 	if err != nil {
-		return nil
+		return nil, err
 	}
 
 	// Match hostname against local cluster service format.
 	pattern := regexp.MustCompile(serviceURLPattern)
-	matches := pattern.FindStringSubmatch(parsedURL.Hostname())
+	get, err := utils.MatchRegex(pattern, parsedURL.Hostname())
 
 	// If incomplete match, return an error.
-	if len(matches) != 3 {
-		return nil
+	if err != nil {
+		return nil, err
 	}
 
 	return &ServiceInfo{
-		Namespace: matches[pattern.SubexpIndex("namespace")],
-		Name:      matches[pattern.SubexpIndex("name")],
+		Namespace: get("namespace"),
+		Name:      get("name"),
 		Port:      remotePort,
-	}
+	}, nil
 }
 
 // NewTunnel will create a new Tunnel struct.

src/internal/packager/git/checkout.go

@@ -14,25 +14,19 @@ import (
 )
 
 // CheckoutTag performs a `git checkout` of the provided tag to a detached HEAD.
-func (g *Git) CheckoutTag(tag string) {
-	message.Debugf("git.CheckoutTag(%s)", tag)
-
+func (g *Git) CheckoutTag(tag string) error {
 	options := &git.CheckoutOptions{
-		Branch: plumbing.ReferenceName("refs/tags/" + tag),
+		Branch: g.parseRef(tag),
 	}
-	g.checkout(options)
+	return g.checkout(options)
 }
 
 func (g *Git) checkoutRefAsBranch(ref string, branch plumbing.ReferenceName) error {
-	var err error
-
-	if isHash(ref) {
-		err = g.checkoutHashAsBranch(plumbing.NewHash(ref), branch)
-	} else {
-		err = g.checkoutTagAsBranch(ref, branch)
+	if plumbing.IsHash(ref) {
+		return g.checkoutHashAsBranch(plumbing.NewHash(ref), branch)
 	}
 
-	return err
+	return g.checkoutTagAsBranch(ref, branch)
 }
 
 // checkoutTagAsBranch performs a `git checkout` of the provided tag but rather
@@ -60,8 +54,6 @@ func (g *Git) checkoutTagAsBranch(tag string, branch plumbing.ReferenceName) err
 func (g *Git) checkoutHashAsBranch(hash plumbing.Hash, branch plumbing.ReferenceName) error {
 	message.Debugf("git.checkoutHasAsBranch(%s,%s)", hash.String(), branch.String())
 
-	_ = g.deleteBranchIfExists(branch)
-
 	repo, err := git.PlainOpen(g.GitPath)
 	if err != nil {
 		message.Fatal(err, "Not a valid git repo or unable to open")

src/internal/packager/git/clone.go

@@ -6,67 +6,91 @@ package git
 
 import (
 	"context"
-	"errors"
 
 	"github.com/defenseunicorns/zarf/src/pkg/message"
 	"github.com/defenseunicorns/zarf/src/pkg/utils"
 	"github.com/defenseunicorns/zarf/src/pkg/utils/exec"
 	"github.com/go-git/go-git/v5"
+	goConfig "github.com/go-git/go-git/v5/config"
+	"github.com/go-git/go-git/v5/plumbing"
 )
 
 // clone performs a `git clone` of a given repo.
-func (g *Git) clone(gitDirectory string, gitURL string, onlyFetchRef bool) (*git.Repository, error) {
+func (g *Git) clone(gitURL string, ref plumbing.ReferenceName) error {
 	cloneOptions := &git.CloneOptions{
 		URL:        gitURL,
 		Progress:   g.Spinner,
 		RemoteName: onlineRemoteName,
 	}
 
-	if onlyFetchRef {
+	// Don't clone all tags if we're cloning a specific tag.
+	if ref.IsTag() {
 		cloneOptions.Tags = git.NoTags
+		cloneOptions.ReferenceName = ref
 	}
 
-	gitCred := utils.FindAuthForHost(gitURL)
+	// Use a single branch if we're cloning a specific branch.
+	if ref.IsBranch() {
+		cloneOptions.SingleBranch = true
+		cloneOptions.ReferenceName = ref
+	}
 
-	// Gracefully handle no git creds on the system (like our CI/CD)
+	// Setup git credentials if we have them, ignore if we don't.
+	gitCred := utils.FindAuthForHost(gitURL)
 	if gitCred.Auth.Username != "" {
 		cloneOptions.Auth = &gitCred.Auth
 	}
 
-	// Clone the given repo
-	repo, err := git.PlainClone(gitDirectory, false, cloneOptions)
-
-	if errors.Is(err, git.ErrRepositoryAlreadyExists) {
-		repo, err = git.PlainOpen(gitDirectory)
+	// Clone the given repo.
+	repo, err := git.PlainClone(g.GitPath, false, cloneOptions)
+	if err != nil {
+		message.Debugf("Failed to clone repo %s: %s", gitURL, err.Error())
+		return g.gitCloneFallback(gitURL, ref)
+	}
 
-		if err != nil {
-			return nil, err
+	// If we're cloning the whole repo or a commit hash, we need to also fetch the other branches besides the default.
+	if ref == emptyRef {
+		fetchOpts := &git.FetchOptions{
+			RemoteName: onlineRemoteName,
+			Progress:   g.Spinner,
+			RefSpecs:   []goConfig.RefSpec{"refs/*:refs/*", "HEAD:refs/heads/HEAD"},
+			Tags:       git.AllTags,
+		}
+		if err := repo.Fetch(fetchOpts); err != nil {
+			return err
 		}
+	}
 
-		return repo, git.ErrRepositoryAlreadyExists
-	} else if err != nil {
-		message.Debugf("Failed to clone repo %s: %s", gitURL, err.Error())
-		g.Spinner.Updatef("Falling back to host git for %s", gitURL)
+	return nil
+}
 
-		// If we can't clone with go-git, fallback to the host clone
-		// Only support "all tags" due to the azure clone url format including a username
-		cmdArgs := []string{"clone", "--origin", onlineRemoteName, gitURL, gitDirectory}
+// gitCloneFallback is a fallback if go-git fails to clone a repo.
+func (g *Git) gitCloneFallback(gitURL string, ref plumbing.ReferenceName) error {
+	g.Spinner.Updatef("Falling back to host git for %s", gitURL)
 
-		if onlyFetchRef {
-			cmdArgs = append(cmdArgs, "--no-tags")
-		}
+	// If we can't clone with go-git, fallback to the host clone
+	// Only support "all tags" due to the azure clone url format including a username
+	cmdArgs := []string{"clone", "--origin", onlineRemoteName, gitURL, g.GitPath}
 
-		execConfig := exec.Config{
-			Stdout: g.Spinner,
-			Stderr: g.Spinner,
-		}
-		_, _, err := exec.CmdWithContext(context.TODO(), execConfig, "git", cmdArgs...)
-		if err != nil {
-			return nil, err
-		}
+	// Don't clone all tags if we're cloning a specific tag.
+	if ref.IsTag() {
+		cmdArgs = append(cmdArgs, "--no-tags")
+	}
 
-		return git.PlainOpen(gitDirectory)
-	} else {
-		return repo, nil
+	// Use a single branch if we're cloning a specific branch.
+	if ref.IsBranch() {
+		cmdArgs = append(cmdArgs, "-b", ref.String())
+		cmdArgs = append(cmdArgs, "--single-branch")
 	}
+
+	execConfig := exec.Config{
+		Stdout: g.Spinner,
+		Stderr: g.Spinner,
+	}
+	_, _, err := exec.CmdWithContext(context.TODO(), execConfig, "git", cmdArgs...)
+	if err != nil {
+		return err
+	}
+
+	return nil
 }