git-branchless CLI hangs at 100% CPU

[steveluscher]

Description of the bug

When I run git sl the CLI hangs indefinitely.

Expected behavior

The smartlog should appear.

Actual behavior

• the CLI hangs (no output)
• the git-branchless process consumes 100% of one CPU core
• ps aux | grep git looks like this:

  % ps aux | grep git
  sluscher         53485  99.4 20.2 413491888 3381648 s000  R+    2:19PM   1:09.67 /Users/sluscher/.cargo/bin/git-branchless smartlog
  sluscher         53925   0.0  0.0 408104048   1456 s001  S+    2:20PM   0:00.00 grep --color=auto --exclude-dir=.bzr --exclude-dir=CVS --exclude-dir=.git --exclude-dir=.hg --exclude-dir=.svn --exclude-dir=.idea --exclude-dir=.tox git
  sluscher         53484   0.0  0.0 407976192   2144 s000  S+    2:19PM   0:00.01 /Applications/Xcode.app/Contents/Developer/usr/bin/git sl
  sluscher         53483   0.0  0.0 408128800   4944 s000  S+    2:19PM   0:00.01 git-branchless wrap -- sl
  

Version of rustc

rustc 1.63.0-nightly (dc80ca78b 2022-06-21)

Automated bug report

git branchless bug-report also hangs.

Version of git-branchless

git-branchless 0.4.0

Version of git

git version 2.37.0 (Apple Git-136)

[arxanas]

Hi @steveluscher, thanks for reporting. Sorry for the inconvenience.

• Does this happen on all repos on your machine, or only specific ones?
• What operating system are you using?
• Can you try getting the stack trace of the running process to see where it's stuck? On macOS, you can try this: https://apple.stackexchange.com/questions/272508/how-do-you-get-a-stack-trace-of-a-running-process. On Linux, you can try attaching to the process with gdb and printing the backtrace with bt, or using the pstack command if you have it.
• Is there anything special about your repository that's worth noting? For example, is it a shallow or partial clone?

[steveluscher]

• Seems to be just specific repos; others on my computer work just fine.
• Mac OS 12.5.1 (21G83)
• Stack trace of git-branchless while hanging: https://gist.github.com/steveluscher/8ef36881402103eb90a8996ffc0da3db
• That repo is just a regular git clone'd repository. Two remotes (upstream and origin – the source and my fork respectively).

[arxanas]

From the sample, it looks like the smartlog is not obviously stuck in an infinite loop, as it's looking up objects from the object database.

• If the repository is publicly available, could you share it?
• How large is the repository, in commits? What are the results of git rev-list --count HEAD and git rev-list --count --branches?
• What is your main branch configured to, and how many commits would be rendered in the smartlog? What are the results of something likegit rev-list --count main...HEAD? It's possible that we are just trying to render too many commits.
  • I've also had the situation where a repository changed from using master to main, which resulted in trying to render commits which occurred since a very stale branch. In that case, try git branchless init --main-branch main to update the main branch setting.
• Is there any particular highly-nested branch structure of note in the repository?

[arxanas]

I checked just now, and it looks like the profiling instructions at https://github.com/arxanas/git-branchless/wiki/Runbook#with-tracing should work even if you interrupt the process with control-c. (Presumably some events at the end might be lost, but that should be fine.) Could you try generating a profile such as with RUST_PROFILE=1 git-branchless smartlog and exiting after a second or two and uploading it so that I can take a look, or try analyzing it yourself in e.g. https://perfetto.dev, and see if you find anything anomalous? If there's a specific function call that seems to be taking longer than it should, it would be ideal if you could include function arguments in the trace by setting RUST_PROFILE_INCLUDE_ARGS=1.

You could also try the cargo flamegraph instructions, although it might contain less useful information. Usually, the tracing output is much more informative, but both have been useful in the past.

[arxanas]

Here's an example trace that you can open in Perfetto or chrome://tracing for comparison purposes:

trace-1663917397.json-0.zip

For me, the majority of the time is spent in get_events because my event log is large (i.e. I've done a lot of work in this repo while git-branchless was installed), and that hasn't been optimized much.

Also feel free to drop by the Discord server to chat about the performance issues.

[steveluscher]

• It's @solana-labs/wallet-adapter (or steveluscher/wallet-adapter) but I'm guessing if you did a fresh clone it would work fine.
• git rev-list --count HEAD -> 1742
• git rev-list --count --branches -> 1745
• git rev-list --count master...HEAD -> 18
• “Is there any particular highly-nested branch structure of note in the repository?” I honestly don't remember, but I don't think so. I think at the moment I'm on a branch, directly off master
• I tried to create a profile with RUST_PROFILE_INCLUDE_ARGS=true RUST_PROFILE=1 git-branchless smartlog but the trace file was exactly zero bytes. I let it run for minutes before pressing Ctrl-C 🫢

Here's the flamegraph from ~60s of hanging (interactive if downloaded and opened in Mac OS Preview).

[arxanas]

@steveluscher if you are okay with zipping and uploading the entire repo somewhere, I can download it and take a look.

Do you use git rebase -i often, and, if so, do you create/edit commits while doing that? I wonder if this is an instance of #316, where the user reports that a commit has been rewritten into itself, which breaks some invariants. I suppose it's possible that it could cause an infinite loop in smartlog operations. (Do other operations, like maybe git sync or git move, also hang indefinitely?)

You can also try:

• Running git branchless repair.
• Deleting the database at .git/branchless/db.sqlite3. Be aware that you'll lose track of any commits which don't have branches attached to them.

[steveluscher]

Do you use git rebase -i often, and, if so, do you create/edit commits while doing that?

Oh, let me tell you. I use git rebase -i like it's going out of style. There isn't a thing that I haven't done with interactive rebase, so you're probably on the right track with that one.

I'll zip it up and find a place to put it in a second.

[steveluscher]

Here. Didn't touch the repo and tar'd it up: https://drive.google.com/file/d/1NmLIuJWPx_q1_3HowEy365v1MJj9_U2w/view

[steveluscher]

git branchless repair --no-dry-run had no effect.

[steveluscher]

rm .git/branchless/db.sqlite3 also had no effect.

[arxanas]

Thanks for uploading. I was able to resolve the 100% CPU usage by unsetting the remote-tracking branch for master (i.e. editing .git/config and commenting out the merge = refs/heads/master). If you're still using the repository, can you confirm that this fixes the issue for you?

I'm not yet sure why the upstream branch is causing the smartlog to hang indefinitely. As a workaround, you could unset the remote-tracking branch, or you could use an alternative branch like upstream/master as your main branch.

[arxanas]

It looks like you have both refs/heads/origin/master and refs/remotes/origin/master, and they point to different objects, which is suspicious. There might be some code which gets surprised when referring to origin/master for some reason. You probably want to delete the former?

[steveluscher]

Drat, I did git update-ref -d refs/heads/origin/master and that didn't help. Commenting out merge = refs/heads/master in .git/config did bring smartlog back to life though!