Windows arm64 support

[dennisameling]

Builds upon the work that was started in #2507

I was able to build a Git executable for arm64 locally by doing some hacks/manual fixes. I can create artifacts as well, just can't run git.exe yet due to an error that iconv-2.dll is missing. To be continued 🚀

Related to #2346

[dscho]

I was able to build a Git executable for arm64 locally by doing some hacks/manual fixes.

Wow, nice progress!

[dennisameling]

There seems to be an issue with libiconv.dll which is currently blocking me: microsoft/vcpkg#14780 (comment)

This clearly shows up in the CI logs as well: https://github.com/git-for-windows/git/pull/2915/checks?check_run_id=1468481105

The reason the x64 build in GH Actions doesn't fail is that it's downloading vcpkg artifacts from https://dev.azure.com/git/git/_build which were built on October 19 (before the issue occured). When installing locally for x64 the same issue occurs, so it's unrelated to arm64 specifically.

I'll wait for the issue with libiconv.dll to be fixed; confident that we're pretty close to a working arm64 build now 🚀

[dennisameling]

@dscho Good news: ARM64 artifacts were successfully generated in CI and are available at https://github.com/git-for-windows/git/actions/runs/389986817 🎉

I can run Git natively on ARM64 now, and commands like git --version work correctly:

PS C:\Users\denni\Downloads\git4> .\git --version
git version 2.29.0.GIT
PS C:\Users\denni\Downloads\git4> echo $env:PROCESSOR_ARCHITECTURE
ARM64

However, things like git clone don't work yet because of the following (it also doesn't on x64, so I guess some packaging needs to be done):

PS C:\Users\denni\Downloads\git4> .\git clone https://github.com/git-for-windows/git.git
Cloning into 'git'...
warning: templates not found in /mingw64/share/git-core/templates
git: 'remote-https' is not a git command. See 'git --help'.

The remote-https executable is present though:

What are the next steps now? 😊

[dennisameling]

When I follow the steps from .github/workflows/main.yml (vs-build) I can get the Git executables. However, when I run make MSVC=1, it still defaults to the x64 target, leading to errors like module machine type 'ARM64' conflicts with target machine type 'x64'.

I'm not finding any references to msbuild in the source code, so not sure how to make make install target ARM64 as well. In CI I solved it with CMAKE_GENERATOR_PLATFORM=arm64; is there any way I can make that happen for make install as well?

[dscho]

PS C:\Users\denni\Downloads\git4> .\git clone https://github.com/git-for-windows/git.git
Cloning into 'git'...
warning: templates not found in /mingw64/share/git-core/templates
git: 'remote-https' is not a git command. See 'git --help'.

Yep, this is a known issue: running Git in-place does not work. You can work around that either by calling bin-wrappers/git (but this requires to go through Bash), or you can set GIT_EXEC_PATH to the absolute path of the top-level directory of your Git worktree.

The reason the x64 build in GH Actions doesn't fail is that it's downloading vcpkg artifacts from https://dev.azure.com/git/git/_build which were built on October 19 (before the issue occured). When installing locally for x64 the same issue occurs, so it's unrelated to arm64 specifically.

Good point. Should be start building arm64 artifacts in https://dev.azure.com/git/git/_build/results?buildId=2098&view=results, too?

[dscho]

Should be start building arm64 artifacts in https://dev.azure.com/git/git/_build/results?buildId=2098&view=results, too?

This is how the relevant step looks currently:

& compat/vcbuild/vcpkg_install.bat
if (!$?) { exit(1); }

git init vcbuild-artifacts\compat\vcbuild\vcpkg
if (!$?) { exit(1); }
Copy-Item compat\vcbuild\vcpkg\vcpkg.exe vcbuild-artifacts\compat\vcbuild\vcpkg -Force
if (!$?) { exit(1); }
Copy-Item compat\vcbuild\vcpkg\installed vcbuild-artifacts\compat\vcbuild\vcpkg -Force -Recurse
if (!$?) { exit(1); }
Copy-Item compat\vcbuild\vcpkg\packages vcbuild-artifacts\compat\vcbuild\vcpkg -Force -Recurse
if (!$?) { exit(1); }

# Remove manual pages (they are not used, and only bloat the artifact)
Get-ChildItem vcbuild-artifacts\compat\vcbuild\vcpkg html -recurse -directory | ForEach-Object {Remove-Item $_.FullName -Recurse -Force}
if (!$?) { exit(1); }

I'm not finding any references to msbuild in the source code, so not sure how to make make install target ARM64 as well

Hrm. I don't know, either. There are mentions of install in CMakeLists.txt, but I don't know whether the Visual Studio backend uses them at all...

[dscho]

There are mentions of install in CMakeLists.txt, but I don't know whether the Visual Studio backend uses them at all...

Oh, this looks interesting: CMake generates an INSTALL.vcxproj file in contrib\buildsystems\out. Maybe you can play around with that for a bit?

[dennisameling]

Should be start building arm64 artifacts in https://dev.azure.com/git/git/_build/results?buildId=2098&view=results, too?

Sounds good. You'd just have to change compat/vcbuild/vcpkg_install.bat to compat/vcbuild/vcpkg_install.bat arm64-windows (and x64-windows) (which were added in this PR) and then it should install those packages for the given architecture.

Oh, this looks interesting: CMake generates an INSTALL.vcxproj file in contrib\buildsystems\out. Maybe you can play around with that for a bit?

Oh great! I'll check that tomorrow towards the end of the day. Thanks! 👍

[dennisameling]

Note to self: this might help: https://docs.microsoft.com/en-us/cpp/build/cmake-projects-in-visual-studio?view=msvc-160#ide-integration

Visual Studio runs cmake.exe and generates the CMake cache file (CMakeCache.txt) for the default (x64 Debug) configuration. The CMake command line is displayed in the Output Window, along with additional output from CMake.

[dscho]

this might help: https://docs.microsoft.com/en-us/cpp/build/cmake-projects-in-visual-studio?view=msvc-160#ide-integration

Visual Studio runs cmake.exe and generates the CMake cache file (CMakeCache.txt) for the default (x64 Debug) configuration. The CMake command line is displayed in the Output Window, along with additional output from CMake.

If I understand correctly, this does not use the same technique as the msbuild step, but it internally uses ninja instead.

[dennisameling]

Apparently make should only be used on Linux,BSD,MacOS:

git/contrib/buildsystems/CMakeLists.txt

Lines 7 to 40 in ffcedc7

	Instructions how to use this in Visual Studio:
	Open the worktree as a folder. Visual Studio 2019 and later will detect
	the CMake configuration automatically and set everything up for you,
	ready to build. You can then run the tests in `t/` via a regular Git Bash.
	Note: Visual Studio also has the option of opening `CMakeLists.txt`
	directly; Using this option, Visual Studio will not find the source code,
	though, therefore the `File>Open>Folder...` option is preferred.
	Instructions to run CMake manually:
	mkdir -p contrib/buildsystems/out
	cd contrib/buildsystems/out
	cmake ../ -DCMAKE_BUILD_TYPE=Release
	This will build the git binaries in contrib/buildsystems/out
	directory (our top-level .gitignore file knows to ignore contents of
	this directory).
	Possible build configurations(-DCMAKE_BUILD_TYPE) with corresponding
	compiler flags
	Debug : -g
	Release: -O3
	RelWithDebInfo : -O2 -g
	MinSizeRel : -Os
	empty(default) :
	NOTE: -DCMAKE_BUILD_TYPE is optional. For multi-config generators like Visual Studio
	this option is ignored
	This process generates a Makefile(Linux/*BSD/MacOS) , Visual Studio solution(Windows) by default.
	Run `make` to build Git on Linux/*BSD/MacOS.
	Open git.sln on Windows and build Git.

... meaning that cmake needs to be called explicitly on Windows if you want to build on that platform with MSVC, like what is currently done in the CI config. What do you think?

git/.github/workflows/main.yml

Lines 208 to 214 in ffcedc7

	- name: generate Visual Studio solution
	shell: bash
	run: |
	cmake `pwd`/contrib/buildsystems/ -DCMAKE_PREFIX_PATH=`pwd`/compat/vcbuild/vcpkg/installed/${{ matrix.arch }}-windows \
	-DMSGFMT_EXE=`pwd`/git-sdk-64-minimal/mingw64/bin/msgfmt.exe -DPERL_TESTS=OFF -DPYTHON_TESTS=OFF -DCURL_NO_CURL_CMAKE=ON -DCMAKE_GENERATOR_PLATFORM=${{ matrix.arch }} -DVCPKG_ARCH=${{ matrix.arch }}-windows
	- name: MSBuild
	run: msbuild git.sln -property:Configuration=Release -property:Platform=${{ matrix.arch }} -maxCpuCount:4 -property:PlatformToolset=v142

[dscho]

cmake needs to be called explicitly on Windows if you want to build on that platform with MSVC, like what is currently done in the CI config.

Yes, that's the idea. If you do this, you should be able to run msbuild using the INSTALL.vcxproj file, probably specifying the target directory via a property, to install the build artifacts.

[dscho]

@dennisameling since I cherry-picked your iconv fix, it was my fault that there were merge conflicts. So I figured I'd better resolve them, and so I did (by rebasing the patches, squashing a few fixups here and there and avoiding white-space issues). I force-pushed to your branch (thank you for permitting to do that!), could you have a look?

[dennisameling]

@dscho Good news: I was able to get native Git arm64 working on my Surface Pro X. The results:

git clone https://github.com/git-for-windows/git.git git-temp
Native arm64: 31 seconds
x86 emulation: 1 minute 2 seconds

So it's twice as fast cloning this specific repo as the x86 emulated version of Git 🚀

These were the steps I had to take:

1. Open PowerShell

cd C:\git-sdk-64\usr\src\git
compat\vcbuild\vcpkg_install.bat arm64-windows

2. Open Git SDK Bash

cd /usr/src/git
mkdir -p contrib/buildsystems/out
cd contrib/buildsystems/out
cmake ../ -DCMAKE_GENERATOR_PLATFORM=arm64 -DVCPKG_ARCH=arm64-windows -DCMAKE_INSTALL_PREFIX="C:\git-sdk-32\arm64"

# Build the Git executables
msbuild git.sln -property:Configuration=Release

# Link the Git executables
msbuild INSTALL.vcxproj -property:Configuration=Release

Contents of C:\gitbuild:

The steps below take place on the arm64 device

3. Copy the contents of C:\gitbuild to C:\Program Files (x86)\Git\mingw32 (had to copy iconv-2.dll manually as well because of Fix the vs-build job so that CI builds pass again #2923)

4. Remove C:\Program Files (x86)\Git\mingw32\libexec\git-core\git-remote-https.exe. There should be a git-remote-https.exe.exe in that folder; rename it to git-remote-https.exe

Then I got the following error while trying a git clone:

Error: SSL certificate problem: unable to get local issuer certificate

So I did git config --global --global http.sslbackend schannel and things started working 🚀

So I basically installed 32-bit Git on my arm64 device and then overwrote the Git binaries with the native arm64 ones.

What is the next step now to release an "arm64 version" of Git which is basically mingw32 + native arm64 Git binaries? 😊

[dscho]

Excellent news!

I would recommend installing into <Git>\arm64\bin, and adjust the PATH to include that before mingw64\bin and usr\bin. This will need to be done (somehow ✋👋) in /etc/profile and in the Git wrapper (probably by inventing MSYSTEM=ARM64 and by setting msystem_bin to arm64/bin and then appending .../mingw32/bin to the PATH, which also needs to increment the 3 in this line).

BTW if you set -DCMAKE_INSTALL_PREFIX="C:\gitbuild" dynamically, you could potentially transmogrify a mingw-w64-i686-git-<version>.pkg.tar.zstd built via https://github.com/git-for-windows/MINGW-packages/blob/master/mingw-w64-git/PKGBUILD into a mingw-w64-arm64-git-<version.pkg.tar.zstd by simply adding the respective build products from the MSBuild into the latter.

Unfortunately, we cannot simply include the ARM64 build products in the -i686- package because git-bash.exe and git-cmd.exe are both installed to the top-level directory, and therefore the i686 and ARM64 artifacts would clash.

[dscho]

Oh, and should we maybe merge this PR before doing anything else?