Clone fails with "reference not found" error

[dave]

Cloning the repo https://gopkg.in/src-d/go-billy.v4 works in command line git:

$ git clone https://gopkg.in/src-d/go-billy.v4
Cloning into 'go-billy.v4'...
remote: Counting objects: 786, done.
remote: Total 786 (delta 0), reused 0 (delta 0), pack-reused 786
Receiving objects: 100% (786/786), 165.29 KiB | 137.00 KiB/s, done.
Resolving deltas: 100% (395/395), done.

... but fails with go-git:

func TestClone2(t *testing.T) {
	fs := memfs.New()

	ctx := context.Background()

	store, err := filesystem.NewStorage(memfs.New())
	if err != nil {
		t.Fatal(err.Error())
	}

	_, err = git.CloneContext(ctx, store, fs, &git.CloneOptions{
		URL:               "https://gopkg.in/src-d/go-billy.v4",
		SingleBranch:      true,
		Depth:             1,
		RecurseSubmodules: git.DefaultSubmoduleRecursionDepth,
		Progress:          os.Stdout,
	})
	if err != nil {
		t.Fatal(err.Error())
	}
}

Output:

Counting objects: 224, done.
Compressing objects: 100% (161/161), done.
Total 224 (delta 95), reused 156 (delta 55), pack-reused 0
	get_get_test.go:73: reference not found

[zkry]

I was looking into this and it seems that it is trying to find the HEAD reference for this repository. The HEAD reference is not getting set because the call to https://gopkg.in/src-d/go-billy.v4/info/refs\?service\=git-upload-pack is not returning the symref capability. I check other repositories on gopkg.in (like the go-git repo) and they don't set the symref capability either, thus giving them the same problem. (see the addSymbolicRefs function in plumbing/protocol/packp/advrefs.go)

❯ curl https://gopkg.in/src-d/go-billy.v4/info/refs\?service\=git-upload-pack
001e# service=git-upload-pack
00000108027dceab1aa836eb310d92c2eb2266e4dc9f4b81 HEADmulti_ack thin-pack side-band side-band-64k ofs-delta shallow deepen-since deepen-not deepen-relative no-progress include-tag multi_ack_detailed no-done oldref=HEAD:refs/heads/master agent=git/github-g8e539663647e
003f027dceab1aa836eb310d92c2eb2266e4dc9f4b81 refs/heads/master
0049dd5fc475f21d51d5637a702363cef39e6e8111dc refs/heads/fix-dir-temporal
003eb041d4fb54825e485e067b2a24e0b96e05581f22 refs/pull/1/head
...

We still do get a HEAD reference though. Should we set the HEAD reference despite not having the symref capability?

[dave]

Would that have any side-effects? This is all rather over my HEAD 😉

[dave]

So instead of symref it's returning oldref... I tried adding a capability for that and inspecting both in that function:

func addSymbolicRefs(s storer.ReferenceStorer, ar *AdvRefs) error {
	for _, c := range []capability.Capability{capability.SymRef, capability.OldRef} {
		if !supports(ar, c) {
			continue
		}
		for _, symref := range ar.Capabilities.Get(c) {
			chunks := strings.Split(symref, ":")
			if len(chunks) != 2 {
				err := fmt.Errorf("bad number of `:` in symref value (%q)", symref)
				return plumbing.NewUnexpectedError(err)
			}
			name := plumbing.ReferenceName(chunks[0])
			target := plumbing.ReferenceName(chunks[1])
			ref := plumbing.NewSymbolicReference(name, target)
			if err := s.SetReference(ref); err != nil {
				return nil
			}
		}
	}

	return nil
}

func supports(ar *AdvRefs, c capability.Capability) bool {
	return ar.Capabilities.Supports(c)
}

... this seems to work, but I have very little knowledge of git internals, so this may well be broken.

[zkry]

Yeah, this is a bit over my head too 😅
Exactly, it was returning oldref which gives the HEAD. Unfortunately I am not that familiar with the git internals either to know of any side effects.

[dave]

Well having a hack like this unblocks my immediate problem, so I'm happy to wait until a git guru floats by.

[dave]

Here's another one that doesn't work (same error):

https://code.googlesource.com/google-api-go-client

$ curl https://code.googlesource.com/google-api-go-client/info/refs\?service\=git-upload-pack
001e# service=git-upload-pack
000000f324928b980e6919be4c72647aacd53ebcbb8c4bab HEAD include-tag multi_ack_detailed multi_ack ofs-delta side-band side-band-64k thin-pack no-progress shallow no-done allow-tip-sha1-in-want allow-reachable-sha1-in-want agent=JGit/4-google filter
004486bc30d7a35e40b2c4aa97fa80d98a8c9586ada8 refs/changes/00/1500/1
00459dc78a731af9c6d4b61b3c545bc76e9039e41c07 refs/changes/00/1500/10
0045c9f3065a1afeff7284af317195125a1a82748980 refs/changes/00/1500/11
...

[dave]

OK I got it working by hardcoding HEAD:refs/heads/master if nothing is found. I'm sure this is wrong, but it works for now.

func addSymbolicRefs(s storer.ReferenceStorer, ar *AdvRefs) error {
	var found bool
	for _, c := range []capability.Capability{capability.SymRef, capability.OldRef} {
		if !supports(ar, c) {
			continue
		}
		for _, symref := range ar.Capabilities.Get(c) {
			chunks := strings.Split(symref, ":")
			if len(chunks) != 2 {
				err := fmt.Errorf("bad number of `:` in symref value (%q)", symref)
				return plumbing.NewUnexpectedError(err)
			}
			name := plumbing.ReferenceName(chunks[0])
			target := plumbing.ReferenceName(chunks[1])
			ref := plumbing.NewSymbolicReference(name, target)
			if err := s.SetReference(ref); err != nil {
				return nil
			}
			found = true
		}
	}
	if !found {
		ref := plumbing.NewSymbolicReference("HEAD", "refs/heads/master")
		if err := s.SetReference(ref); err != nil {
			return nil
		}
	}

	return nil
}

func supports(ar *AdvRefs, c capability.Capability) bool {
	return ar.Capabilities.Supports(c)
}