Can't re-build git 2.10.0 the same as the official one

[YueLinHo]

• [v] I was not able to find an open or closed issue matching what I'm seeing

Setup

• Which version of Git for Windows are you using? Is it 32-bit or 64-bit?

Shift + Right click in C:\Program Files\Git\cmd folder, then invoking the Windows Command Prompt.
Then,

$ git --version --build-options
git version 2.10.0.windows.1
sizeof-long: 4
machine: x86_64

• Which version of Windows are you running? Vista, 7, 8, 10? Is it 32-bit or 64-bit?

Win 7 Pro 64 bit

$ cmd.exe /c ver
Microsoft Windows [版本 6.1.7601]

• What options did you set as part of the installation? Or did you choose the defaults?

> type "C:\Program Files\Git\etc\install-options.txt"
Path Option: Cmd
SSH Option: OpenSSH
CRLF Option: CRLFAlways
Bash Terminal Option: MinTTY
Performance Tweaks FSCache: Enabled

• Any other interesting things about your environment that might be related
  to the issue you're seeing?

TortoiseGit installed

Details

• Which terminal/shell are you running Git from? e.g Bash/CMD/PowerShell/other

CMD

• What commands did you run to trigger this issue? If you can provide a Minimal, Complete, and Verifiable example this will help us understand the issue.

1. Un-install Git for Windows
2. Install Git for Windows 2.10.0
3. Clone a remote repository while its URL contains a Chinese word(s).

   C:\Program Files\Git\cmd>git clone "D:\Temp\i2859\Test中文.git" "D:\Temp\i2859\Test中文"
   Cloning into 'D:\Temp\i2859\Test中文'...
   done.
   

4. Download Git for Windows SDK - git-sdk-installer-1.0.3-64.7z.exe from the page
5. Run that SDK
6. The path is D:\git-sdk-64
   
7. the running, as you can see, the msys2-runtime-2.6.0
   
8. Start to clone...
   
9. Start to build on commit c6703c2
   
10. got an error

        GEN gitk-wish
    Generating catalog po/ru.msg
    msgfmt --statistics --tcl po/ru.po -l ru -d po/
    w 307 .
    make[1]: *** [Makefile:76: po/ru.msg] Error 127
    make: *** [Makefile:1704: all] Error 2
    

    
11. Press any key and a Window popup
    
    Not sure if git is built. So...

    $ git --version --build-options
    git version 2.10.2.windows.1
    sizeof-long: 4
    machine: x86_64
    

12. I found there is no files in working tree of build-extra, MINGW-packages and MSYS2-packages under my path D:\git-sdk-64\usr\src, so...

    $ cd /usr/src/build-extra
    $ git fetch
    $ git checkout master
    

    

    $ cd /usr/src/MSYS2-packages
    $ git fetch
    $ git checkout master
    

    

    $ cd /usr/src/MINGW-packages
    $ git fetch
    $ git checkout master
    

    
13. build git again

    $ cd /usr/src/git
    $ make install
    

    
14. result
    
15. try clone inside of SDK with 2.10.2, and failed
    
16. checkout git 2.10.0

    $ git checkout v2.10.0.windows.1 -b 2.10.0
    Switched to a new branch '2.10.0'
    

17. build git 2.10.0

    $ make install
    

    
18. finish build
    
19. try clone with my git 2.10.0, and failed. (The same as step 15)
    

• What did you expect to occur after running these commands?

my git 2.10.0 can clone while the remote URL contains Chinese word(s) as the official one does.

• What actually happened instead?

Can't clone

---

Try msys2-runtime 2.4.1?

[dscho]

The most likely reason for LC_ALL=C fixing the clone is that one of the components provided by MSYS2 all of a sudden does not default to C any longer, but to US-ASCII or worse, ISO-8859-1 or something like that.

Now, when you initialize a Git for Windows SDK, it updates all of the MSYS2 packages to their current versions. Those versions may very well be different from the ones current at the time any past Git for Windows release was built by me. You can inspect the exact versions in any Git for Windows' installation's /etc/package-versions.txt.

Theoretically, it would be possible to ask Pacman to downgrade every package to the version given in /etc/package-version.txt. But that would most likely not be enough: there are packages that are only installed for building Git for Windows, but their files are not necessarily part of the final installer. For instance, msys2-runtime is required both for the SDK as well as for the end-user installation, but msys2-runtime-devel is only required in the SDK. Naturally, the /etc/package-versions.txt file would list only the former.

The 127 error when building gitk's localizations sounds familiar. As far as I recall, I fixed the way it worked because there was a pretty ugly bug that for some reason only showed its ugly head when I tried building in one of my VMs, but not when I built on my main development machine. My memory on this may be flawed, though. It is just as likely that you have to install something like mingw-w64-x86_64-gettext (or gettext) manually (via something like pacman -S gettext) to make things work. It should be more obvious what needs to be done by running make V=1 in /usr/src/git/.

Please also note that after building Git in /usr/src/git/, git version still calls /mingw64/bin/git.exe. You need to prefix the executable with a dot-slash, i.e. ./git version.

I am still not quite sure where things go wrong when LC_ALL is not set, and given that I have a workaround now, I will have to leave a proper analysis to others (you, maybe?).

i would very much welcome if you could trace this further, e.g. by tracing the execution in gdb and single-stepping through mingw_startup() to see whether at least the wide-character versions of the command-line parameters (called wargv in mingw_startup()) get the correct characters, or whether they are already wrong.

[YueLinHo]

can inspect the exact versions in any Git for Windows' installation's /etc/package-versions.txt.

There are 131 packages.
21 of them are different.
Here is the diff. of /etc/package-versions.txt between 2.10.0 and 2.10.1:

-msys2-runtime 2.4.1.34115.1ca92fa-1
+msys2-runtime 2.6.0.34694.8626feb-1

-mingw-w64-x86_64-git 2.10.0.1.1f448f5-1
+mingw-w64-x86_64-git 2.10.1.1.3374789-1

-git-extra 1.1.120.6a8c857-1
+git-extra 1.1.126.824cd7f-1

-git-flow 1.9.1-1
+git-flow 1.10.1-1

-curl 7.50.1-1
+curl 7.50.3-1
-libcurl 7.50.1-1
+libcurl 7.50.3-1
-mingw-w64-x86_64-curl 7.50.1-1
+mingw-w64-x86_64-curl 7.50.3-1

-expat 2.1.1-1
+expat 2.2.0-1
-libexpat 2.1.1-1
+libexpat 2.2.0-1
-mingw-w64-x86_64-expat 2.1.1-1
+mingw-w64-x86_64-expat 2.2.0-1

-openssl 1.0.2.h-1
+openssl 1.0.2.j-1
-libopenssl 1.0.2.h-1
+libopenssl 1.0.2.j-1
-mingw-w64-x86_64-openssl 1.0.2.h-1
+mingw-w64-x86_64-openssl 1.0.2.j-1


-ncurses 6.0.20160220-1
+ncurses 6.0.20160917-1


-mingw-w64-x86_64-gcc-libgfortran 6.1.0-1
+mingw-w64-x86_64-gcc-libgfortran 6.2.0-2

-mingw-w64-x86_64-gcc-libs 6.1.0-1
+mingw-w64-x86_64-gcc-libs 6.2.0-2

-mingw-w64-x86_64-gnutls 3.5.2-1
+mingw-w64-x86_64-gnutls 3.5.4-1

-mingw-w64-x86_64-nghttp2 1.13.0-1
+mingw-w64-x86_64-nghttp2 1.14.1-1


-mintty 1~2.4.2-1
+mintty 1~2.5.0-1

-mingw-w64-x86_64-git-credential-manager 1.6.1-1
+mingw-w64-x86_64-git-credential-manager 1.7.0-1

-mingw-w64-x86_64-git-doc-html 2.10.0.1.1f448f5-1
+mingw-w64-x86_64-git-doc-html 2.10.1.1.3374789-1

[YueLinHo]

the /etc/package-versions.txt file would list only the former.

(former is those packages only for Git for Windows, not for development.)

So, how to find the others which is excluded? (msys2-runtime-devel is one.)
Or, how to get the list of all packages?

[dscho]

Or, how to get the list of all packages?

I think that the other packages are either irrelevant (for example, it does not matter which make version was used to build, as the result depends on the compiler and the options, not which program called the compiler) or correlated with packages that are partially bundled in Git for windows (e.g. all of the GCC compiler suite shares the same version as libgcc, parts of which are included in our installer).

So I guess that you should be able to concoct a script that generates a pacman.exe command-line to downgrade to the versions given by /etc/package-versions.txt; Pacman will then most likely complain about certain packages having certain requirements not met, e.g. msys2-runtime-devel explicitly requires the same version as msys2-runtime.

However, I am fairly certain that all of this is overkill. If you want to find out what changed with regards to LC_ALL, it has to be something that affects the compiled git.exe, as you experienced the issue of the incorrect encoding even from CMD, bypassing all of MSYS2. So my guess would lie with mingw-w64-x86_64-libgcc.

[YueLinHo]

However, I am fairly certain that all of this is overkill.

Yep, I think so. :-D
I am new for those stuff, so I ask many questions. Just want to know those things around this issue.

by tracing the execution in gdb and single-stepping through mingw_startup() to see whether at least the wide-character versions of the command-line parameters (called wargv in mingw_startup()) get the correct characters, or whether they are already wrong.

So, I am going to trace it. (never use gdb) :-)
(I guess the 127 error when building gitk's localizations can be skipped. Right?)

[dscho]

So, I am going to trace it. (never use gdb) :-)

;-)

FWIW I just added a little bit more information to the page about debugging with gdb: https://github.com/git-for-windows/git/wiki/Debugging-Git#debugging-git-using-the-gnu-debugger-gdb

(I guess the 127 error when building gitk's localizations can be skipped. Right?)

It is probably something completely unrelated, yes. Nothing to worry about in your context.

[YueLinHo]

debug git

1. edit /usr/src/git/Makefile to remove the -O2 from the CFLAGS = -g -O2 -Wall line
2. make
   
3. the version
   
4. try clone
   

Clone is OK in bash. Now, my question is how can I use gdb in Windows Command Prompt? :-/

[YueLinHo]

1. I copy the following binary file to my /usr/src/git folder (those files are from my C:\Program Files\Git\mingw64\bin(GfW 2.10.2))

libiconv-2.dll
libintl-8.dll
libpcre-1.dll
zlib1.dll
libssp-0.dll

2. Shift + Right Click in /usr/src/git folder and invoke Windows Command Prompt, and test
   

('gdb' 不是內部或外部命令、可執行的程式或批次檔。)

[dscho]

yow can I use gdb in Windows Command Prompt?

C:\git-sdk-64\usr\bin\gdb.exe -args C:\git-sdk-64\usr\src\git\git.exe --exec-path C:\git-sdk-64\usr\src\git ...

Shift + Right Click in /usr/src/git folder and invoke Windows Command Prompt, and test

Please try to past text not as image. I cannot quote it nor easily copy-paste it into my own cmd.

The error you see ("cannot spawn sh") means that C:\git-sdk-64\usr\bin is not in your PATH (and you would want C:\git-sdk-64\mingw64\bin in your PATH, too.

[dscho]

What's the status on this? Can we close this ticket?

[YueLinHo]

@dscho Sorry for the late. I am busy these days. I wish I could dig/debug more.
I am closing it, because still don't have much time on it.
Really really thanks for your help~~~