[REG 2.46.0 -> 2.47.0] Git hangs on fetch

[orgads]

• I was not able to find an open or closed issue matching what I'm seeing

Setup

• Which version of Git for Windows are you using? Is it 32-bit or 64-bit?

$ git --version --build-options

git version 2.47.0.windows.1
cpu: x86_64
built from commit: 8e380191abeda7a995e87abf4acc9e1702fdc698
sizeof-long: 4
sizeof-size_t: 8
shell-path: F:/git-sdk-64/usr/bin/sh
feature: fsmonitor--daemon
libcurl: 8.10.1
OpenSSL: OpenSSL 3.2.3 3 Sep 2024
zlib: 1.3.1

• Which version of Windows are you running? Vista, 7, 8, 10? Is it 32-bit or 64-bit?

$ cmd.exe /c ver

Microsoft Windows [Version 10.0.19045.4355]

• What options did you set as part of the installation? Or did you choose the
  defaults?

# One of the following:
> type "C:\Program Files\Git\etc\install-options.txt"
> type "C:\Program Files (x86)\Git\etc\install-options.txt"
> type "%USERPROFILE%\AppData\Local\Programs\Git\etc\install-options.txt"
> type "$env:USERPROFILE\AppData\Local\Programs\Git\etc\install-options.txt"
$ cat /etc/install-options.txt

Editor Option: VIM
Custom Editor Path:
Default Branch Option:
Path Option: Cmd
SSH Option: OpenSSH
Tortoise Option: false
CURL Option: WinSSL
CRLF Option: CRLFAlways
Bash Terminal Option: MinTTY
Git Pull Behavior Option: Rebase
Use Credential Manager: Enabled
Performance Tweaks FSCache: Enabled
Enable Symlinks: Enabled
Enable FSMonitor: Disabled

Details

• Which terminal/shell are you running Git from? e.g Bash/CMD/PowerShell/other

Git Bash

• What commands did you run to trigger this issue? If you can provide a
  Minimal, Complete, and Verifiable example
  this will help us understand the issue.

git fetch

• What did you expect to occur after running these commands?

It should be done successfully after a while.

• What actually happened instead?

It sometimes hangs. I see the following spawned process, which doesn't use any CPU, and doesn't terminate:

git index-pack --stdin -v --fix-thin "--keep=fetch-pack 4272 on my-pc" --pack_header=2,291

The output is:

remote: Counting objects: 195, done
remote: Finding sources: 100% (292/292)
<nothing happens>
<when I press Ctrl-C>
fetch-pack: unexpected disconnect while reading sideband packet

[dscho]

• What commands did you run to trigger this issue? If you can provide a
  Minimal, Complete, and Verifiable example
  this will help us understand the issue.

git fetch

To make this example a little more complete, could you describe from where you are fetching? Is it github.com, a self-hosted git daemon, from a network share? Is it a big fetch, i.e. have already dozens of megabytes been transferred, or is it a small fetch? Did you try to disable sideband?

[orgads]

It's a hosted gerrit 3.8 server. The fetch is small, took a few seconds after I downgraded to 2.46.

I can try to disable sideband tomorrow.

[jfcherng]

Latest update: Now it clones okay... without any local env modification... Okay again it just happened.

---

I tried to clone my https://github.com/jfcherng/copilot-node-server .

It seems to alway hang if I clone [email protected]:jfcherng/copilot-node-server.git. But it's okay if I clone https://github.com/jfcherng/copilot-node-server.git.

---

Update: It still hangs with GIT_CURL_VERBOSE=1 GIT_TRACE=1 git -c sendpack.sideband=false clone [email protected]:jfcherng/copilot-node-server.git

[jfcherng@HOME Desktop]$ GIT_CURL_VERBOSE=1 GIT_TRACE=1 git -c sendpack.sideband=false clone [email protected]:jfcherng/copilot-node-server.git
10:14:30.318887 exec-cmd.c:266          trace: resolved executable dir: C:/Program Files/Git/mingw64/bin
10:14:30.329889 git.c:479               trace: built-in: git clone [email protected]:jfcherng/copilot-node-server.git
Cloning into 'copilot-node-server'...
10:14:30.350892 run-command.c:667       trace: run_command: unset GIT_CONFIG_PARAMETERS GIT_DIR; GIT_PROTOCOL=version=2 ssh -o SendEnv=GIT_PROTOCOL [email protected] 'git-upload-pack '\''jfcherng/copilot-node-server.git'\'''
10:14:30.350892 run-command.c:928       trace: start_command: ssh -o SendEnv=GIT_PROTOCOL [email protected] 'git-upload-pack '\''jfcherng/copilot-node-server.git'\'''
remote: Enumerating objects: 542, done.
remote: Counting objects: 100% (100/100), done.
10:14:33.230184 run-command.c:667       trace: run_command: git index-pack --stdin -v --fix-thin '--keep=fetch-pack 32844 on 3bc-f19' --check-self-contained-and-connected
10:14:33.230184 run-command.c:928       trace: start_command: git index-pack --stdin -v --fix-thin '--keep=fetch-pack 32844 on 3bc-f19' --check-self-contained-and-connected
10:14:33.298183 exec-cmd.c:266          trace: resolved executable dir: C:/Program Files/Git/mingw64/libexec/git-core
10:14:33.311182 git.c:479               trace: built-in: git index-pack --stdin -v --fix-thin '--keep=fetch-pack 32844 on 3bc-f19' --check-self-contained-and-connected
remote: Compressing objects: 100% (65/65), done.
Receiving objects:   1% (6/542)

It just hangs on Receiving objects: 1% (6/542). It is always 1% (6/542) everytime I retried. Eventually it failed as the following.

[jfcherng@HOME Desktop]$ GIT_CURL_VERBOSE=1 GIT_TRACE=1 git -c sendpack.sideband=false clone [email protected]:jfcherng/copilot-node-server.git
10:16:32.600368 exec-cmd.c:266          trace: resolved executable dir: C:/Program Files/Git/mingw64/bin
10:16:32.614373 git.c:479               trace: built-in: git clone [email protected]:jfcherng/copilot-node-server.git
Cloning into 'copilot-node-server'...
10:16:32.653368 run-command.c:667       trace: run_command: unset GIT_CONFIG_PARAMETERS GIT_DIR; GIT_PROTOCOL=version=2 ssh -o SendEnv=GIT_PROTOCOL [email protected] 'git-upload-pack '\''jfcherng/copilot-node-server.git'\'''
10:16:32.653368 run-command.c:928       trace: start_command: ssh -o SendEnv=GIT_PROTOCOL [email protected] 'git-upload-pack '\''jfcherng/copilot-node-server.git'\'''
remote: Enumerating objects: 542, done.
remote: Counting objects: 100% (100/100), done.
10:16:35.393876 run-command.c:667       trace: run_command: git index-pack --stdin -v --fix-thin '--keep=fetch-pack 20220 on 3bc-f19' --check-self-contained-and-connected
10:16:35.393876 run-command.c:928       trace: start_command: git index-pack --stdin -v --fix-thin '--keep=fetch-pack 20220 on 3bc-f19' --check-self-contained-and-connected
10:16:35.455879 exec-cmd.c:266          trace: resolved executable dir: C:/Program Files/Git/mingw64/libexec/git-core
10:16:35.469968 git.c:479               trace: built-in: git index-pack --stdin -v --fix-thin '--keep=fetch-pack 20220 on 3bc-f19' --check-self-contained-and-connected
remote: Compressing objects: 100% (65/65), done.
fetch-pack: unexpected disconnect while reading sideband packet
fatal: fetch-pack: invalid index-pack output

[orgads]

Ah right, my case was ssh too.

[EemilAhonen]

We are having this same problem with SSH on multiple machines that upgraded to the latest 2.47.0. All clones over 1mb from Gitlab are failing, the Git window just freezes at random points.
Uninstalling 2.47.0 and reinstalling 2.44.0 solved the issue.

[dscho]

Does it work if you copy the usr\bin\ssh.exe over from v2.46.2?

[jfcherng]

Does it work if you copy the usr\bin\ssh.exe over from v2.46.2?

In my case, no.

[dscho]

Does it work if you copy the usr\bin\ssh.exe over from v2.46.2?

In my case, no.

How about copying over usr\bin\msys-2.0.dll?

[jfcherng]

How about copying over usr\bin\msys-2.0.dll?

Yes, that's the culprit. I tried in a fresh portable build. No need to replace ssh.exe.

[dscho]

I feared as much. There is a rather big jump in the MSYS2 runtime upgrade between Git for Windows v2.46.2 and v2.47.0. And MSYS2 runtime issues are notoriously hard to fix.

You wouldn't happen to have a good reproducer that other people might be able to use, would you? I am afraid that git fetch as an MCVE does not pass muster.

[elindoorn]

A git pull from a azure repo consistently hung for me today until i reverted to 2.46.2

git pull
remote: Azure Repos
remote: Found 308 objects to send. (757 ms)

the trace suggests it can collect branches/tags and such before it hangs

`> git pull
13:28:53.108594 exec-cmd.c:266 trace: resolved executable dir: C:/s/git/mingw64/bin
13:28:53.118594 git.c:479 trace: built-in: git pull
13:28:53.157446 run-command.c:667 trace: run_command: git merge-base --fork-point refs/remotes/origin/redacted redacted
13:28:53.157446 run-command.c:928 trace: start_command: git merge-base --fork-point refs/remotes/origin/redacted redacted
13:28:53.197744 run-command.c:667 trace: run_command: git fetch --update-head-ok
13:28:53.197744 run-command.c:928 trace: start_command: git fetch --update-head-ok
13:28:53.209496 exec-cmd.c:266 trace: resolved executable dir: C:/s/git/mingw64/libexec/git-core
13:28:53.219721 git.c:479 trace: built-in: git fetch --update-head-ok
13:28:53.223755 run-command.c:667 trace: run_command: unset GIT_PREFIX; GIT_PROTOCOL=version=2 ssh -o SendEnv=GIT_PROTOCOL [email protected] 'git-upload-pack '''v3/redacted''''
13:28:53.224873 run-command.c:928 trace: start_command: ssh -o SendEnv=GIT_PROTOCOL [email protected] 'git-upload-pack '''v3/redacted''''
13:28:53.987805 pkt-line.c:86 packet: fetch< 191d8833b443043e99cd2d3184f4b66c154d3847 HEAD\0 multi_ack thin-pack side-band side-band-64k no-progress multi_ack_detailed no-done shallow allow-tip-sha1-in-want filter symref=HEAD:refs/heads/develop
...
13:28:54.115251 pkt-line.c:86 packet: fetch< NAK
13:28:54.136540 pkt-line.c:86 packet: fetch< ACK eb10cf30356d9de23600c64368d94d664db97888
13:28:54.138227 pkt-line.c:86 packet: sideband< \2Azure Repos
remote: Azure Repos
13:28:54.139224 pkt-line.c:86 packet: sideband< \2\15Found 303 objects to send. (12 ms)
remote: Found 303 objects to send. (12 ms)

nothing after this`

[ryanmkurtz]

I'm having the same issue too with the move from 2.46.0 to 2.47.0. Both clone and fetch now hang:

$ git fetch
remote: Enumerating objects: 8510, done.
remote: Counting objects: 100% (5831/5831), done.
remote: Compressing objects: 100% (782/782), done.
<hang>

I am working with my organization's GitLab repo over ssh.

[derrickstolee]

I was able to reproduce the failing clones and have a workaround:

git -c core.sshCommand="'C:\Windows\System32\OpenSSH\ssh.exe'" clone ...

or

git config --global core.sshCommand "'C:\Windows\System32\OpenSSH\ssh.exe'"
git clone ...

This configures Git to use the SSH agent built into Windows instead of the OpenSSL-based one provided by MSYS2.

[b12k]

Same here.
Hangs on clone with 2.47.0

@derrickstolee Solution helped 👆

[Nbc66]

Can Confirm git clone & git push hang when trying to call em this only happens on version 2.47.0 the other version like 2.46.2 work just fine

this might be related to the MSYS2 upgrade on the latest version

[goostleek]

I can confirm, I thought it was specific to my repo only, but using an example provided at https://github-debug.com/

GIT_TRACE=1 GIT_TRANSFER_TRACE=1 GIT_CURL_VERBOSE=1 git clone [email protected]:github/debug-repo /tmp/debug-repo-ssh

causes the SSH connection to freeze

Tip

The possible workaround is to temporarily switch from ssh:// to https:// when feasible

git remote set-url origin 'https://<repo-url>'

[mm65de]

After installing Git-2.47.0-64-bit.exe the git commands clone and fetch also hang on my machine.
My previous installation without that problem was based on this setup file: Git-2.46.2-64-bit.exe

Here is an example of this problem:

2024-10-10 11:24:35  c:\Tmp\git_test_folder> git clone [email protected]:git-for-windows/git.git
Cloning into 'git'...
remote: Enumerating objects: 622752, done.
remote: Counting objects: 100% (7374/7374), done.
remote: Compressing objects: 100% (2904/2904), done.

Now nothing happens anymore.
If I press Ctrl-C then I get this:

fetch-pack: unexpected disconnect while reading sideband packet
fatal: fetch-pack: invalid index-pack output

2024-10-10 11:26:13  c:\Tmp\git_test_folder>

Here comes the System Info section of the text file generated by the command git bugreport:

[System Info]
git version:
git version 2.47.0.windows.1
cpu: x86_64
built from commit: d53e4648cb65eb75dd8d8a093d17400a18a9a15d
sizeof-long: 4
sizeof-size_t: 8
shell-path: D:/git-sdk-64-build-installers/usr/bin/sh
feature: fsmonitor--daemon
libcurl: 8.10.1
OpenSSL: OpenSSL 3.2.3 3 Sep 2024
zlib: 1.3.1
uname: Windows 10.0 19045 
compiler info: gnuc: 14.2
libc info: no libc information available
$SHELL (typically, interactive shell): <unset>

After executing the setup file Git-2.46.2-64-bit.exe and accepting the downgrade warning the command git clone worked well again:

2024-10-10 11:51:53  c:\Tmp\git_test_folder> git clone [email protected]:git-for-windows/git.git
Cloning into 'git'...
remote: Enumerating objects: 622752, done.
remote: Counting objects: 100% (7379/7379), done.
remote: Compressing objects: 100% (2905/2905), done.
remote: Total 622752 (delta 6166), reused 4919 (delta 4472), pack-reused 615373 (from 1)
Receiving objects: 100% (622752/622752), 310.19 MiB | 5.87 MiB/s, done.
Resolving deltas: 100% (450376/450376), done.
Updating files: 100% (4589/4589), done.

2024-10-10 11:53:16  c:\Tmp\git_test_folder>

[Nbc66]

After installing Git-2.47.0-64-bit.exe the git commands clone and fetch also hang on my machine. My previous installation without that problem was based on this setup file: Git-2.46.2-64-bit.exe

Here is an example of this problem:

2024-10-10 11:24:35  c:\Tmp\git_test_folder> git clone [email protected]:git-for-windows/git.git
Cloning into 'git'...
remote: Enumerating objects: 622752, done.
remote: Counting objects: 100% (7374/7374), done.
remote: Compressing objects: 100% (2904/2904), done.

Now nothing happens anymore. If I press Ctrl-C then I get this:

fetch-pack: unexpected disconnect while reading sideband packet
fatal: fetch-pack: invalid index-pack output

2024-10-10 11:26:13  c:\Tmp\git_test_folder>

Here comes the System Info section of the text file generated by the command git bugreport:

[System Info]
git version:
git version 2.47.0.windows.1
cpu: x86_64
built from commit: d53e4648cb65eb75dd8d8a093d17400a18a9a15d
sizeof-long: 4
sizeof-size_t: 8
shell-path: D:/git-sdk-64-build-installers/usr/bin/sh
feature: fsmonitor--daemon
libcurl: 8.10.1
OpenSSL: OpenSSL 3.2.3 3 Sep 2024
zlib: 1.3.1
uname: Windows 10.0 19045 
compiler info: gnuc: 14.2
libc info: no libc information available
$SHELL (typically, interactive shell): <unset>

After executing the setup file Git-2.46.2-64-bit.exe and accepting the downgrade warning the command git clone worked well again:

2024-10-10 11:51:53  c:\Tmp\git_test_folder> git clone [email protected]:git-for-windows/git.git
Cloning into 'git'...
remote: Enumerating objects: 622752, done.
remote: Counting objects: 100% (7379/7379), done.
remote: Compressing objects: 100% (2905/2905), done.
remote: Total 622752 (delta 6166), reused 4919 (delta 4472), pack-reused 615373 (from 1)
Receiving objects: 100% (622752/622752), 310.19 MiB | 5.87 MiB/s, done.
Resolving deltas: 100% (450376/450376), done.
Updating files: 100% (4589/4589), done.

2024-10-10 11:53:16  c:\Tmp\git_test_folder>

Its an issue with the latest MYSYS2 update they did

[mm65de]

If it helps, here are more configuration details of my machine:

2024-10-10 12:10:46  c:\Tmp\git_test_folder> git config --list --system
diff.astextplain.textconv=astextplain
filter.lfs.clean=git-lfs clean -- %f
filter.lfs.smudge=git-lfs smudge -- %f
filter.lfs.process=git-lfs filter-process
filter.lfs.required=true
http.sslbackend=openssl
http.sslcainfo=C:/Program Files/Git/mingw64/etc/ssl/certs/ca-bundle.crt
core.autocrlf=false
core.fscache=true
core.symlinks=false
core.editor="C:\\Program Files\\Notepad++\\notepad++.exe" -multiInst -notabbar -nosession -noPlugin
pull.rebase=true
credential.helper=manager
credential.https://dev.azure.com.usehttppath=true
init.defaultbranch=master

2024-10-10 12:11:18  c:\Tmp\git_test_folder>

If I use the HTTPS URL instead of the SSH URL to clone the repository, then the command git clone even works with the new version 2.47.0.windows.1.

[david-siegert]

After update to 2.47 I get the same issue when trying to use git clone using ssh. It hangs indefinatelly and i have to terminate the command using ctrl+c resulting in "unexpected disconnect while reading sideband packet". After rolling back to 2.46 it works fine.

[jeremyVignelles]

You wouldn't happen to have a good reproducer that other people might be able to use, would you? I am afraid that git fetch as an MCVE does not pass muster.

For reference, I came across this issue from mutagen-io/mutagen#511 :
Mutagen detects the ssh from git-for-windows and tries to use it to connect to the remote server. The update in git-for-windows broke the mutagen detection as well, and we had to revert msys-2.0.dll.

I don't know what kind of reproduction project you need, but our use case is just a virtualbox VM and trying to sync something with mutagen mutagen sync create...

[dscho]

I can reproduce the issue, and am bisecting now. Experience suggests that it might take a couple of days to get to the bottom of this, so please have patience.

[dscho]

Okay, I have a fix in hand: It seems to be a regression in Cygwin v3.5.3, and while I am not completely happy about the extent to which I understand what is going wrong, it seems safe enough to revert to the working behavior (even if it should not completely reflect how Linux would behave in the same situation).

Y'all, could I ask you to download the install.zip artifact produced by the PR build, and extract its usr/bin/msys-2.0.dll over your existing one, then verify that it fixes the issue not only for me, but also for all of you gentle people?

[jfcherng]

Y'all, could I ask you to download the install.zip artifact produced by the PR build, and extract its usr/bin/msys-2.0.dll over your existing one, then verify that it fixes the issue not only for me, but also for all of you gentle people?

That /usr/bin/msys-2.0.dll works on my end. Thanks.

[orgads]

Works for me. Thank you very much!

Did you report the issue to cygwin?

[dscho]

Works for me. Thank you very much!

Awesome!

Did you report the issue to cygwin?

Not yet, I first wanted to be sure that I have something that works not only for me, but for all others, too.

[inflamously]

Oh my god, had I just found this. Nearly lost a day since I've recently reinstalled windows 11 (24H2) latest build whatsoever and git v2.47.0.windows.1. Couldn't for life clone a repository with using git clone when using a ssh url. Now I've downgraded for now, but the tips above are also helpful. Thanks.

[dscho]

/add relnote bug A regression where clones, fetches and pushes via SSH would dead-lock was fixed.

The workflow run was started

[nga888]

FYI: I also ran into this issue and fixed it with this patch:

--- a/winsup/cygwin/select.cc
+++ b/winsup/cygwin/select.cc
@@ -674,7 +674,8 @@ pipe_data_available (int fd, fhandler_base *fh, HANDLE h, int flags)
          paranoid_printf ("fd %d, %s, write: size %u, avail %u", fd,
                           fh->get_name (), fpli.InboundQuota,
                           fpli.WriteQuotaAvailable);
-         return fpli.WriteQuotaAvailable;
+         return (flags & PDA_SELECT) ? MAX (PIPE_BUF, fpli.WriteQuotaAvailable)
+                                     : fpli.WriteQuotaAvailable;
        }
       /* TODO: Buffer really full or non-Cygwin reader? */
     }

I also emailed Corinna Vinschen about the issue along with this patch.

[david-siegert]

Okay, I have a fix in hand: It seems to be a regression in Cygwin v3.5.3, and while I am not completely happy about the extent to which I understand what is going wrong, it seems safe enough to revert to the working behavior (even if it should not completely reflect how Linux would behave in the same situation).

Y'all, could I ask you to download the install.zip artifact produced by the PR build, and extract its usr/bin/msys-2.0.dll over your existing one, then verify that it fixes the issue not only for me, but also for all of you gentle people?

Maybe redundant but I also confirm that the fix works on my end too. Thank you.

[nga888]

I also emailed Corinna Vinschen about the issue along with this patch.

I'm not particularly familiar with Cygwin development, so I'm not sure that I have reported the issue "correctly", i.e. I haven't had any response to my email from over 2 weeks ago, so it's probably worth reporting this issue "properly".

Sorry, I didn't mention this issue earlier on GFW. I don't use the "official" GFW release but have my own UCRT64 based build with "mainline" MSYS2, so hit this issue a bit earlier after the MSYS2 runtime updated to 3.5.4.

[dscho]

I'm not sure that I have reported the issue "correctly", i.e. I haven't had any response to my email from over 2 weeks ago

Given that I cannot find the patch in https://inbox.sourceware.org/cygwin-patches/?q=WriteQuotaAvailable nor in https://inbox.sourceware.org/cygwin/?q=WriteQuotaAvailable, I suspect that you sent this as a private mail (which is a pretty big no-no in open source)?

The lack of response might be due to a vacation, which is even more reason not to send patches a private emails to maintainers.

[dscho]

Please note that there is now a snapshot with the fix, as I am planning on holding off from releasing a new Git for Windows version because I anticipate a Git v2.47.1 to be released soon, with even more bug fixes.

[rescenic]

In my machine, I fixed it using msys-2.0.dll from GitKraken then copy & replace it to Git usr\bin folder.

[kun2001github]

I'm running into this issue as well, I downgraded it to version 2.45.2

[dscho]

I'm running into this issue as well, I downgraded it to version 2.45.2

@kun2001github I would strongly suggest to upgrade to the latest snapshot instead.

[t-b]

Please note that there is now a snapshot with the fix, as I am planning on holding off from releasing a new Git for Windows version because I anticipate a Git v2.47.1 to be released soon, with even more bug fixes.

I can confirm that the snapshot from friday fixes the issue.

[radnvlad]

I'm running into this issue as well, I downgraded it to version 2.45.2

@kun2001github I would strongly suggest to upgrade to the latest snapshot instead.

FYI, I upgraded to the snapshot and the original issue is gone, but it still didn't get to work fully, it failed at the end and got back

fetch-pack: unexpected disconnect while reading sideband packet
fatal: early EOF
fatal: unpack-objects failed

Downgrading to 2.46.0 fixed the issue.

It's possible that this is something from my end though.

[dscho]

@radnvlad since yours is the only report where the issue is not fixed by the snapshot, you are likely hitting a different bug. Please open a separate issue for that, providing ample amounts of information about your specific scenario. It would also be good if you could bisect through the snapshots (always replacing the msys-2.0.dll with the version from v2.46.0) to see when your particular bug was introduced.

For all other readers stumbling over this here issue: please upgrade to the latest snapshot.