jj git fetch becomes slow over time

[martinvonz]

Description

After working for a long time on a repo, I now have >23k refs/jj/keep/* refs (which prevent git from GC-ing commits). That makes jj git fetch slow because all those refs are sent as "haves" for git's "negotiation" (what's called "discovery" in hg). We should figure out a way of excluding them from negotiation. I don't know if that's possible with libgit2.

Specifications

• Version: 0.4.0

[arxanas]

git-branchless gets around this with its own git-branchless gc command, which GCs refs which are no longer accessible by a current live head. It's run as part of the pre-auto-gc hook. Is that an option for jj?

At the very least, you could try to run a "compaction" pass over the keep refs to keep only the refs which are heads, since the other refs would be implicit. But I'm not sure if that would actually speed up Git operations or not.

[martinvonz]

git-branchless gets around this with its own git-branchless gc command, which GCs refs which are no longer accessible by a current live head. It's run as part of the pre-auto-gc hook. Is that an option for jj?

The operation log means that pretty much all commits are still accessible. There's #12 about implementing GC. I imagine the user would be able to specify the oldest operation they want to keep and then everything before that operation becomes eligible for GC (whatever is only reachable before, of course).

At the very least, you could try to run a "compaction" pass over the keep refs to keep only the refs which are heads, since the other refs would be implicit. But I'm not sure if that would actually speed up Git operations or not.

That would help a bit, but not enough. jj debug index says I have 17k heads in my repo.

[arxanas]

Instead of keeping commits live by storing a ref to each commit directly, you could store a ref to a single commit, whose first parent is another such commit and second parent is the actual commit you want to keep live. Assuming that "haves" only look at individual refs and don't traverse them, this should speed it up.

You would have to take more care about keeping commits live in an atomic manner, though. (One idea: rather than store directly in the main commit/ref, you could write loose refs and then compact any loose refs in this way when you find them.)

[martinvonz]

Instead of keeping commits live by storing a ref to each commit directly, you could store a ref to a single commit, whose first parent is another such commit and second parent is the actual commit you want to keep live. Assuming that "haves" only look at individual refs and don't traverse them, this should speed it up.

Interesting idea :) I'd considered doing a similar trick with octopus merges, but your idea is simpler. Thanks!

Making this part of GC (#12) makes sense.

[ChrisPenner]

I'm definitely running into this too, have to go for coffee everytime I jj git fetch ; if anyone's interested in taking a look it's easy to reproduce with https://github.com/unisonweb/unison 😄

[ilyagr]

Two things you can try are, assuming you are not dealing with a fresh clone.

• jj util gc now repacks refs, I think.
• You can re-clone.

If you are dealing with a fresh clone, then it might be a different problem.