"git ls-remote" fails with error about failing to load library libcurl-4.dll

[nikobockerman]

• I was not able to find an open or closed issue matching what I'm seeing

Setup

• Which version of Git for Windows are you using? Is it 32-bit or 64-bit?

64-bit

Non-working 2.42.0 version:

$ git --version --build-options
git version 2.42.0.windows.1
cpu: x86_64
built from commit: a2e49ec355ae22e74288fa74f50821f3cc1a1551
sizeof-long: 4
sizeof-size_t: 8
shell-path: /bin/sh
feature: fsmonitor--daemon

Working older 2.39.2 version:

$ git --version --build-options
git version 2.39.2.windows.1
cpu: x86_64
built from commit: a82fa99b36ddfd643e61ed45e52abe314687df67
sizeof-long: 4
sizeof-size_t: 8
shell-path: /bin/sh
feature: fsmonitor--daemon

• Which version of Windows are you running? Vista, 7, 8, 10? Is it 32-bit or 64-bit?
  Windows 11 22H2, 64-bit

$ cmd.exe /c ver

Microsoft Windows [Version 10.0.22621.2134]

• What options did you set as part of the installation? Or did you choose the
  defaults?

$ cat /etc/install-options.txt
Editor Option: VisualStudioCode
Custom Editor Path:
Default Branch Option: main
Path Option: Cmd
SSH Option: OpenSSH
Tortoise Option: false
CURL Option: WinSSL
CRLF Option: CRLFAlways
Bash Terminal Option: MinTTY
Git Pull Behavior Option: FFOnly
Use Credential Manager: Enabled
Performance Tweaks FSCache: Enabled
Enable Symlinks: Disabled
Enable Pseudo Console Support: Disabled
Enable FSMonitor: Enabled

• Any other interesting things about your environment that might be related
  to the issue you're seeing?

Somewhat hardened and restricted corporate setup. But nothing that I would imagine affecting GfW. Especially as the older 2.39.2 version doesn't have same issue.

Details

• Which terminal/shell are you running Git from? e.g Bash/CMD/PowerShell/other

All of them fail in the same way.

• What commands did you run to trigger this issue? If you can provide a
  Minimal, Complete, and Verifiable example
  this will help us understand the issue.

In Git Bash:

$ git ls-remote https://github.com
fatal: failed to load library 'libcurl-4.dll'

$ GIT_TRACE=2 git ls-remote https://github.com
13:32:08.133684 exec-cmd.c:244          trace: resolved executable dir: C:/Users/<redacted>/AppData/Local/Programs/Git/mingw64/bin
13:32:08.149319 git.c:463               trace: built-in: git ls-remote https://github.com
13:32:08.149319 run-command.c:659       trace: run_command: git remote-https https://github.com https://github.com
13:32:08.196196 exec-cmd.c:244          trace: resolved executable dir: C:/Users/<redacted>/AppData/Local/Programs/Git/mingw64/libexec/git-core
13:32:08.196196 git.c:749               trace: exec: git-remote-https https://github.com https://github.com
13:32:08.196196 run-command.c:659       trace: run_command: git-remote-https https://github.com https://github.com
13:32:08.243622 exec-cmd.c:244          trace: resolved executable dir: C:/Users/<redacted>/AppData/Local/Programs/Git/mingw64/libexec/git-core
fatal: failed to load library 'libcurl-4.dll'

$ GIT_TRACE2=2 /mingw64/libexec/git-core/git-remote-https.exe https://github.com
13:34:44.363642 common-main.c:55                  version 2.42.0.windows.1
13:34:44.363642 common-main.c:56                  start 'C:\Users\<redacted>\AppData\Local\Programs\Git\mingw64\libexec\git-core\git-remote-https.exe' https://github.com
13:34:44.363642 remote-curl.c:1544                cmd_name remote-curl (remote-curl)
13:34:44.363642 usage.c:61                        error failed to load library 'libcurl-4.dll'
fatal: failed to load library 'libcurl-4.dll'
13:34:44.363642 usage.c:75                        exit elapsed:0.020978 code:128
13:34:44.363642 trace2/tr2_tgt_normal.c:127       atexit elapsed:0.020994 code:128

• What did you expect to occur after running these commands?

The command to be able to load libcurl successfully and complain that https://github.com is not a repository.

With 2.39.2 of GfW, the same command worked properly. Therefore, this feels like a regression.

2.41.0.3 of GfW fails in the same way as 2.42.0.

• What actually happened instead?

The command fails with an error about failing to load libcurl-4.dll.

• If the problem was occurring with a specific repository, can you provide the
  URL to that repository to help us with testing?

Not related to specific repository.

Extra information

• Can extra items in $PATH cause problems?

No:

$ bash test-git-with-clean-path.sh
+ export PATH=/bin:/usr/bin
+ PATH=/bin:/usr/bin
+ echo /bin:/usr/bin
/bin:/usr/bin
+ export GIT_TRACE2=2
+ GIT_TRACE2=2
+ /mingw64/libexec/git-core/git-remote-https.exe https://github.com
13:45:54.153599 common-main.c:55                  version 2.42.0.windows.1
13:45:54.153599 common-main.c:56                  start 'C:\Users\<redacted>\AppData\Local\Programs\Git\mingw64\libexec\git-core\git-remote-https.exe' https://github.com
13:45:54.153599 remote-curl.c:1544                cmd_name remote-curl (remote-curl)
13:45:54.153599 usage.c:61                        error failed to load library 'libcurl-4.dll'
fatal: failed to load library 'libcurl-4.dll'
13:45:54.153599 usage.c:75                        exit elapsed:0.020037 code:128
13:45:54.153599 trace2/tr2_tgt_normal.c:127       atexit elapsed:0.020054 code:128

• Does libcurl-4.dll exist in the installation?

In two places:

$ find /mingw64/ -name libcurl-4.dll
/mingw64/bin/libcurl-4.dll
/mingw64/libexec/git-core/libcurl-4.dll

[dscho]

Is it possible that in your restricted environment, regular programs are prohibited from loading .dll files via LoadLibraryExW()?

[nikobockerman]

I doubt that, but I don't know for sure. Is there a way I could verify that?

And has there been a change in the use of that LoadLibraryExW() since GfW 2.39.2 where the same command works?

[dscho]

I doubt that, but I don't know for sure. Is there a way I could verify that?

If you can compile C programs, you could try to compile and run this:

#include <windows.h>
#include <stdio.h>

int main(int argc, char **argv)
{
    const char *dll = argc > 1 ? argv[1] : "C:\\Program Files\\Git\\mingw64\\bin\\libcurl-4.dll";

    printf("LoadLibrary(\"%s\") returned %p\n", dll, LoadLibraryA(dll));
    return 0;
}

And has there been a change in the use of that LoadLibraryExW() since GfW 2.39.2 where the same command works?

Yes, there has been: #4410. It was necessary to go forward with the OpenSSL upgrade from the version whose support will end in a few weeks from now.

[nikobockerman]

I compiled that program and got the following output (I adjusted the path used to the correct installation location which is under by user AppData\Local):

LoadLibrary("C:\Users\<redacted>\AppData\Local\Programs\Git\mingw64\bin\libcurl-4.dll") returned 0000000000000000

[nikobockerman]

When I try loading with that program other libraries from under mingw64\libexec\git-core I don't get NULL result. For example libbz2-1.dll and Avalonia.Base.dll load with non null result. I didn't try it for all dlls there, but libcurl-4.dll is the only one that fails to load.

[nikobockerman]

GetLastError() returns 126 after the LoadLibraryA has returned NULL for libcurl-4.dll.

According to Win32 System Error Codes 0-499 126 means this:

ERROR_MOD_NOT_FOUND
126 (0x7E)
The specified module could not be found.

I don't really know what that could possibly mean.

[dscho]

Most likely it's because libcurl-4.dll depends on other files in that same directory. Could you try appending that directory to the PATH and then trying again?

[nikobockerman]

That doesn't help either:

$ bash test-git-with-clean-path.sh
+ export PATH=/bin:/usr/bin:/mingw64/libexec/git-core:/c/Users/<redacted>/AppData/Local/Programs/Git/mingw64/libexec/git-core
+ PATH=/bin:/usr/bin:/mingw64/libexec/git-core:/c/Users/<redacted>/AppData/Local/Programs/Git/mingw64/libexec/git-core
+ echo /bin:/usr/bin:/mingw64/libexec/git-core:/c/Users/<redacted>/AppData/Local/Programs/Git/mingw64/libexec/git-core
/bin:/usr/bin:/mingw64/libexec/git-core:/c/Users/<redacted>/AppData/Local/Programs/Git/mingw64/libexec/git-core
+ export GIT_TRACE2=2
+ GIT_TRACE2=2
+ /mingw64/libexec/git-core/git-remote-https.exe https://github.com
11:04:04.260789 common-main.c:55                  version 2.42.0.windows.1
11:04:04.260789 common-main.c:56                  start 'C:\Users\<redacted>\AppData\Local\Programs\Git\mingw64\libexec\git-core\git-remote-https.exe' https://github.com
11:04:04.271010 remote-curl.c:1544                cmd_name remote-curl (remote-curl)
11:04:04.271010 usage.c:61                        error failed to load library 'libcurl-4.dll'
fatal: failed to load library 'libcurl-4.dll'
11:04:04.271010 usage.c:75                        exit elapsed:0.019523 code:128
11:04:04.271010 trace2/tr2_tgt_normal.c:127       atexit elapsed:0.019541 code:128

[nikobockerman]

Hmm.. When I append that path to PATH and attempt to use the test program you gave sources for to load libcurl-4.dll, then loading the library succeeds. At least when executed from PowerShell.

I'll try few more things to narrow down the conditions and have more detailed information of what works and what doesn't.

[nikobockerman]

It seems that whatever I do with PATH, git-remote-https.exe fails to load libcurl-4.dll.

But when the /mingw64/libexec/git-core directory is added to PATH the test program is able to load libcurl-4.dll.
This is consistent through PowerShell and Git Bash prompts.

One thought that just came to my mind. My username contains non-ascii character ö. That would be in the <redacted> part of all outputs I've sent above.

Could that non-ascii character cause the load to fail when attempted by the GfW code?

[dscho]

One thought that just came to my mind. My username contains non-ascii character ö. That would be in the <redacted> part of all outputs I've sent above.

Could that non-ascii character cause the load to fail when attempted by the GfW code?

You can find out by installing, say, Portable Git into a location that has only ASCII characters in its path. For example, you could create C:\test-git (by default any user can create directories in the top-level directory of the C: drive), and unpack the portable Git there.

Another tool you could use to investigate further would be Process Monitor.

But your hunch about non-ASCII characters does have a point, as the code uses the *A() variant of LoadLibraryEx() instead of the *W() variant: https://github.com/git-for-windows/git/blob/v2.42.0.windows.1/compat/lazyload-curl.c#L60

[nikobockerman]

Verified with Portable Git:

• Installed to path with only ASCII characters: git.exe ls-remote https://github.com successfully loads libcurl-4.dll
• Installed to a path with non-ASCII character: same command fails to load libcurl-4.dll

[dscho]

Verified with Portable Git:

Cool!

I have a PR up to fix this: #4575

[nikobockerman]

I can confirm that version 2.42.0.windows.2 has fixed this problem I faced. Thank you very much!