Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

enable windows arm64 #1940

Open
13 of 18 tasks
beckermr opened this issue Apr 19, 2023 · 43 comments
Open
13 of 18 tasks

enable windows arm64 #1940

beckermr opened this issue Apr 19, 2023 · 43 comments

Comments

@beckermr
Copy link
Member

beckermr commented Apr 19, 2023

This is a tracking issue to enable windows arm64.

We think the path forward is to cross-compile and do out of band testing on native hardware.

Let's put todo items here:

cc @zooba @isuruf @jakirkham @jezdez @chenghlee

@chenghlee
Copy link

x-refs for related ecosystem issues:

@isuruf
Copy link
Member

isuruf commented Apr 20, 2023

@zooba I added a few TODOs above if you or someone else want to help.

@zooba
Copy link

zooba commented Apr 24, 2023

@finnagin is looking at the conda-build changes, and will probably have some Qs about any user visible changes (such as how best to inform conda-build it should be doing a cross-compile). Hopefully that will unblock us to start looking at recipe/feedstock changes and figuring out the best way to automatically migrate existing ones (if needed).

@isuruf
Copy link
Member

isuruf commented Apr 24, 2023

@zooba, @finnagin, task 4 needs to be done to build some packages. Task 5 is needed only after python itself is built.

@zooba
Copy link

zooba commented Apr 25, 2023

Task 4 is definitely first, yeah, and part of that seems to be plugging in the arch and host_arch configs to allow running an ARM64 targeted build on AMD64 machine. That plus a few recipe updates should enable building Python - we don't actually need conda/conda-build itself to run natively (it'll be nice, but it's not essential).

@isuruf
Copy link
Member

isuruf commented Apr 25, 2023

msvc_env_cmd is a deprecated function, so we need not update it to actually work. We just need it to not error out. So, a check for bits to see if it is arm64 and returning without doing anything should be enough.

@zooba
Copy link

zooba commented May 3, 2023

@isuruf You mentioned in the call that you got the zlib build working. Can you post how you did that for @finnagin's benefit?

@isuruf
Copy link
Member

isuruf commented May 3, 2023

See conda-forge/zlib-feedstock#72

conda build recipe -m .ci_support/win_arm64_.yaml

@zooba
Copy link

zooba commented May 3, 2023

Ah neat, it's using CMake's support. I guess that means the env variables needed were already being set? Or did those get slipped in recently without me noticing?

@isuruf
Copy link
Member

isuruf commented May 3, 2023

No, we should add support for an env variable like CMAKE_ARGS which we use in Linux/macOS. The two systems are so similar that even if cmake is not told that it is cross compiling, things just work because the correct compiler is used.

@zooba
Copy link

zooba commented May 3, 2023

Oh cool, I didn't see where the compiler was being picked. Just a vcvarsall call? That should work for anything with MSBuild or setuptools (if relatively up to date) too.

@isuruf
Copy link
Member

isuruf commented May 3, 2023

Just a vcvarsall call?

Yes.

C:\Program Files\Microsoft Visual Studio\2022\Enterprise>CALL "VC\Auxiliary\Build\vcvarsamd64_arm64.bat" -vcvars_ver=14.35 10.0.22621.0 

@finnagin
Copy link

finnagin commented May 4, 2023

Thanks @isuruf! This is very helpful!

@isuruf isuruf pinned this issue Jul 5, 2023
@isuruf
Copy link
Member

isuruf commented Jul 6, 2023

There should be a zlib package for win-arm64 now. It's untested though.

@isuruf
Copy link
Member

isuruf commented Jul 11, 2023

Can someone with hardware access check the zstd package?
Download https://anaconda.org/conda-forge/zstd/1.5.2/download/win-arm64/zstd-1.5.2-h45ec737_7.conda
and run

conda build -t /path/to/zstd-1.5.2-h45ec737_7.conda

@finnagin
Copy link

Hey @isuruf I can try running that for you! However, I'm currently waiting for access to the network that the machine I would use is on so I likely won't be able to do it until tomorrow.

@shijunz
Copy link

shijunz commented Jul 12, 2023

Can someone with hardware access check the zstd package? Download https://anaconda.org/conda-forge/zstd/1.5.2/download/win-arm64/zstd-1.5.2-h45ec737_7.conda and run

conda build -t /path/to/zstd-1.5.2-h45ec737_7.conda

yes, I download it, but how to run it? I get the error ">conda build -t zstd-1.5.2-h45ec737_7.conda
'conda' is not recognized as an internal or external command,
operable program or batch file."

@h-vetinari
Copy link
Member

yes, I download it, but how to run it?

You need to have conda installed and conda build available. Both of these don't exist for win-arm64 yet, so I think it will be tricky to test these artefacts natively until we've built conda & conda-build...

@h-vetinari
Copy link
Member

h-vetinari commented Jul 12, 2023

Looking at conda install -n test conda conda-build and filtering out the noarch/ packages, that would mean building at least:

The following NEW packages will be INSTALLED:

  brotli-python      conda-forge/win-64::brotli-python-1.0.9-py311h12c1d0e_9
  bzip2              conda-forge/win-64::bzip2-1.0.8-h8ffe710_4
  ca-certificates    conda-forge/win-64::ca-certificates-2023.5.7-h56e8100_0
  cffi               conda-forge/win-64::cffi-1.15.1-py311h7d9ee11_3
  chardet            conda-forge/win-64::chardet-5.1.0-py311h1ea47a8_0
  conda              conda-forge/win-64::conda-23.5.0-py311h1ea47a8_1
  conda-build        conda-forge/win-64::conda-build-3.25.0-py311h1ea47a8_0
  cryptography       conda-forge/win-64::cryptography-41.0.2-py311h28e9c30_0
  libarchive         conda-forge/win-64::libarchive-3.6.2-h6f8411a_1
  libexpat           conda-forge/win-64::libexpat-2.5.0-h63175ca_1
  libffi             conda-forge/win-64::libffi-3.4.2-h8ffe710_5
  libiconv           conda-forge/win-64::libiconv-1.17-h8ffe710_0
  liblief            conda-forge/win-64::liblief-0.12.3-h63175ca_0
  libsqlite          conda-forge/win-64::libsqlite-3.42.0-hcfcfb64_0
  libxml2            conda-forge/win-64::libxml2-2.11.4-hc3477c8_0
  libzlib            conda-forge/win-64::libzlib-1.2.13-hcfcfb64_5
  lz4-c              conda-forge/win-64::lz4-c-1.9.4-hcfcfb64_0
  lzo                conda-forge/win-64::lzo-2.10-he774522_1000
  m2-msys2-runtime   conda-forge/win-64::m2-msys2-runtime-2.5.0.17080.65c939c-3
  m2-patch           conda-forge/win-64::m2-patch-2.7.5-2
  m2w64-gcc-libgfor~ conda-forge/win-64::m2w64-gcc-libgfortran-5.3.0-6
  m2w64-gcc-libs     conda-forge/win-64::m2w64-gcc-libs-5.3.0-7
  m2w64-gcc-libs-co~ conda-forge/win-64::m2w64-gcc-libs-core-5.3.0-7
  m2w64-gmp          conda-forge/win-64::m2w64-gmp-6.1.0-2
  m2w64-libwinpthre~ conda-forge/win-64::m2w64-libwinpthread-git-5.0.0.4634.697f757-2
  markupsafe         conda-forge/win-64::markupsafe-2.1.3-py311ha68e1ae_0
  menuinst           conda-forge/win-64::menuinst-1.4.19-py311h1ea47a8_1
  msys2-conda-epoch  conda-forge/win-64::msys2-conda-epoch-20160418-1
  openssl            conda-forge/win-64::openssl-3.1.1-hcfcfb64_1
  psutil             conda-forge/win-64::psutil-5.9.5-py311ha68e1ae_0
  py-lief            conda-forge/win-64::py-lief-0.12.3-py311h12c1d0e_0
  pycosat            conda-forge/win-64::pycosat-0.6.4-py311ha68e1ae_1
  python             conda-forge/win-64::python-3.11.4-h2628c8c_0_cpython
  python-libarchive~ conda-forge/win-64::python-libarchive-c-4.0-py311h1ea47a8_2
  python_abi         conda-forge/win-64::python_abi-3.11-3_cp311
  pyyaml             conda-forge/win-64::pyyaml-6.0-py311ha68e1ae_5
  ripgrep            conda-forge/win-64::ripgrep-13.0.0-h7f3b576_2
  ruamel.yaml        conda-forge/win-64::ruamel.yaml-0.17.32-py311ha68e1ae_0
  ruamel.yaml.clib   conda-forge/win-64::ruamel.yaml.clib-0.2.7-py311ha68e1ae_1
  tk                 conda-forge/win-64::tk-8.6.12-h8ffe710_0
  ucrt               conda-forge/win-64::ucrt-10.0.22621.0-h57928b3_0
  vc                 conda-forge/win-64::vc-14.3-h64f974e_17                           ✅ exists for win-arm64
  vc14_runtime       conda-forge/win-64::vc14_runtime-14.36.32532-hfdfe4a8_17          ✅ exists for win-arm64
  vs2015_runtime     conda-forge/win-64::vs2015_runtime-14.36.32532-h05e6639_17        ✅ exists for win-arm64
  xz                 conda-forge/win-64::xz-5.2.6-h8d14728_0
  yaml               conda-forge/win-64::yaml-0.2.5-h8ffe710_2
  zstandard          conda-forge/win-64::zstandard-0.19.0-py311he5d195f_2
  zstd               conda-forge/win-64::zstd-1.5.2-h12be248_7                         ✅ exists for win-arm64

I think it should be possible to stub out m2-patch (at least as long as we're not applying patches, which isn't necessary for testing artefacts), so we avoid having to build the msys2-stack

@isuruf
Copy link
Member

isuruf commented Jul 12, 2023

Just install an x86_64 version of conda and conda-build (and windows 11 for x86_64 emulation) and run the command.

@finnagin
Copy link

@isuruf when I run conda build -t /path/to/zstd-1.5.2-h45ec737_7.conda I get the following error:

Solving environment: ...working... failed
failed to get install actions, retrying.  exception was: Unsatisfiable dependencies for platform win-arm64: {'cmake', 'pkg-config'}
WARNING:conda_build.build:failed to get install actions, retrying.  exception was: Unsatisfiable dependencies for platform win-arm64: {'cmake', 'pkg-config'}

looks like cmake and pkg-config need arm64 versions first?

@shijunz
Copy link

shijunz commented Jul 13, 2023

@isuruf when I run conda build -t /path/to/zstd-1.5.2-h45ec737_7.conda I get the following error:

Solving environment: ...working... failed
failed to get install actions, retrying.  exception was: Unsatisfiable dependencies for platform win-arm64: {'cmake', 'pkg-config'}
WARNING:conda_build.build:failed to get install actions, retrying.  exception was: Unsatisfiable dependencies for platform win-arm64: {'cmake', 'pkg-config'}

looks like cmake and pkg-config need arm64 versions first?

(base) C:\Users\shijunz\Downloads>conda build -t zstd-1.5.2-h45ec737_7.conda
TEST START: zstd-1.5.2-h45ec737_7.conda
Copying package to conda-build croot. No packages otherwise alongside yours will be available unless you specify -c local.
To avoid this warning, your package must reside in a channel structure with platform-subfolders. See more info on what a
valid channel is at https://conda.io/docs/user-guide/tasks/create-custom-channels.html
WARNING:conda_build.build:Copying package to conda-build croot. No packages otherwise alongside yours will be available un
less you specify -c local. To avoid this warning, your package must reside in a channel structure with platform-subfolders
. See more info on what a valid channel is at https://conda.io/docs/user-guide/tasks/create-custom-channels.html
INFO:conda_index.index.convert_cache:Migrate database
INFO:conda_index.index.convert_cache:CONVERT .cache
INFO:conda_index.index.convert_cache:Migrate database
INFO:conda_index.index:noarch cached 0 B from 0 packages at 0 B/second
INFO:conda_index.index.convert_cache:Migrate database
INFO:conda_index.index.convert_cache:CONVERT .cache
INFO:conda_index.index.convert_cache:Migrate database
INFO:conda_index.index:win-arm64 cached 253 KB from 1 packages at 22.5 MB/second
INFO:conda_index.index:Subdir: noarch Gathering repodata
INFO:conda_index.index:noarch Writing pre-patch repodata
INFO:conda_index.index:noarch Applying patch instructions
INFO:conda_index.index:noarch Writing patched repodata
INFO:conda_index.index:noarch Building current_repodata subset
INFO:conda_index.index:noarch Writing current_repodata subset
INFO:conda_index.index:noarch Writing index HTML
INFO:conda_index.index:Completed noarch
INFO:conda_index.index:Subdir: win-arm64 Gathering repodata
INFO:conda_index.index:win-arm64 Writing pre-patch repodata
INFO:conda_index.index:win-arm64 Applying patch instructions
INFO:conda_index.index:win-arm64 Writing patched repodata
INFO:conda_index.index:win-arm64 Building current_repodata subset
INFO:conda_index.index:win-arm64 Writing current_repodata subset
INFO:conda_index.index:win-arm64 Writing index HTML
INFO:conda_index.index:Completed win-arm64
INFO:conda_index.index:Channeldata subdir: noarch
INFO:conda_index.index:Channeldata subdir: win-arm64
Multiple meta files found. The meta.yaml file in the base directory (C:\Users\shijunz\AppData\Local\Temp\tmpgcv9dcgo\info\r
ecipe) will be used.
WARNING:conda_build.utils:Multiple meta files found. The meta.yaml file in the base directory (C:\Users\shijunz\AppData\Loc
al\Temp\tmpgcv9dcgo\info\recipe) will be used.
No numpy version specified in conda_build_config.yaml. Falling back to default numpy value of 1.22
WARNING:conda_build.metadata:No numpy version specified in conda_build_config.yaml. Falling back to default numpy value of
1.22
Collecting package metadata (repodata.json): ...working... done
Solving environment: ...working... failed
failed to get install actions, retrying. exception was: Unsatisfiable dependencies for platform win-arm64: {'cmake', "libz
lib[version='>=1.2.13,<1.3.0a0']", 'pkg-config', "vc14_runtime[version='>=14.36.32532']", "vc[version='>=14.3,<15']", 'vs20
22_win-arm64'}
WARNING:conda_build.build:failed to get install actions, retrying. exception was: Unsatisfiable dependencies for platform
win-arm64: {'cmake', "libzlib[version='>=1.2.13,<1.3.0a0']", 'pkg-config', "vc14_runtime[version='>=14.36.32532']", "vc[ver
sion='>=14.3,<15']", 'vs2022_win-arm64'}

Leaving build/test directories:
Work:
C:\ProgramData\miniconda3\conda-bld\work
Test:
C:\ProgramData\miniconda3\conda-bld\test_tmp
Leaving build/test environments:
Test:
source activate C:\ProgramData\miniconda3\conda-bld_test_env
Build:
source activate C:\ProgramData\miniconda3\conda-bld_build_env

TESTS FAILED: zstd-1.5.2-h45ec737_7.tar.bz2

(base) C:\Users\shijunz\Downloads>

@shijunz
Copy link

shijunz commented Jul 13, 2023

@isuruf when I run conda build -t /path/to/zstd-1.5.2-h45ec737_7.conda I get the following error:

Solving environment: ...working... failed
failed to get install actions, retrying.  exception was: Unsatisfiable dependencies for platform win-arm64: {'cmake', 'pkg-config'}
WARNING:conda_build.build:failed to get install actions, retrying.  exception was: Unsatisfiable dependencies for platform win-arm64: {'cmake', 'pkg-config'}

looks like cmake and pkg-config need arm64 versions first?

cmake should be ok, I build a lot windows on arm64 app using the x64 version cmake. it support crossing build. maybe we should not check the it? or just tell the cmake -A arm64.

but the pkg-config I have no idea.

@shijunz
Copy link

shijunz commented Jul 13, 2023

Just install an x86_64 version of conda and conda-build (and windows 11 for x86_64 emulation) and run the command.

llecting package metadata (repodata.json): ...working... done
Solving environment: ...working... failed
failed to get install actions, retrying. exception was: Unsatisfiable dependencies for platform win-arm64: {'pkg-config', "vc[version='>=14.3,<15']", "vc14_runtime[version='>=14.36.32532']", 'cmake', "libzlib[version='>=1.2.13,<1.3.0a0']", 'vs2022_win-arm64'}
WARNING:conda_build.build:failed to get install actions, retrying. exception was: Unsatisfiable dependencies for platform win-arm64: {'pkg-config', "vc[version='>=14.3,<15']", "vc14_runtime[version='>=14.36.32532']", 'cmake', "libzlib[version='>=1.2.13,<1.3.0a0']", 'vs2022_win-arm64'}

yes, I run it on the windows 11 for x86-64 emulation mode) and get the similar error.

@finnagin
Copy link

cmake should be ok, I build a lot windows on arm64 app using the x64 version cmake. it support crossing build.

If it's relevant, in addition to cross-compiling with the x64 version cmake also has arm64 Windows releases.

@isuruf
Copy link
Member

isuruf commented Jul 13, 2023

We'll need to disable those tests for now. In the meantime, can you do,

set CONDA_SUBDIR=win-arm64
conda create -n arm64 zstd
set CONDA_SUBDIR=
conda activate arm64
conda config --env --set subdir win-arm64

and then try zstd --version?

@finnagin
Copy link

Sure!

>zstd --version
*** Zstandard CLI (64-bit) v1.5.5, by Yann Collet ***

@shijunz
Copy link

shijunz commented Jul 13, 2023

(arm64) C:\Windows\System32>conda install zstd
Collecting package metadata (current_repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Collecting package metadata (repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.

PackagesNotFoundError: The following packages are not available from current channels:

  • zstd

Current channels:

To search for alternate channels that may provide the conda package you're
looking for, navigate to

https://anaconda.org

and use the search bar at the top of the page.

@isuruf
Copy link
Member

isuruf commented Jul 13, 2023

@shijunz, you need to add conda-forge to the list of channels.
@finnagin, thanks.

@shijunz
Copy link

shijunz commented Jul 14, 2023

@shijunz, you need to add conda-forge to the list of channels. @finnagin, thanks.

(env_win-arm64) C:>conda install -c conda-forge zstd
Collecting package metadata (current_repodata.json): done
Solving environment: done

Package Plan

environment location: C:\Users\shijunz\AppData\Local\miniconda3\envs\env_win-arm64

added / updated specs:
- zstd

The following packages will be downloaded:

package                    |            build
---------------------------|-----------------
libzlib-1.2.13             |       h50ba5a6_5          52 KB  conda-forge
vc-14.3                    |      h6ed9334_17          16 KB  conda-forge
vc14_runtime-14.36.32532   |      he64ce00_17        14.4 MB  conda-forge
zstd-1.5.2                 |       h45ec737_7         253 KB  conda-forge
------------------------------------------------------------
                                       Total:        14.7 MB

The following NEW packages will be INSTALLED:

libzlib conda-forge/win-arm64::libzlib-1.2.13-h50ba5a6_5
vc conda-forge/win-arm64::vc-14.3-h6ed9334_17
vc14_runtime conda-forge/win-arm64::vc14_runtime-14.36.32532-he64ce00_17
zstd conda-forge/win-arm64::zstd-1.5.2-h45ec737_7

Proceed ([y]/n)? y

Downloading and Extracting Packages

Preparing transaction: done
Verifying transaction: done
Executing transaction: done

@shijunz
Copy link

shijunz commented Jul 14, 2023

but, in window on arm, I can NOT run ' conda build -t /path/to/zstd-1.5.2-h45ec737_7.conda', because I try to

(env_win-arm64) C:\Users\shijunz\Downloads>conda install -c conda-forge conda-build
Collecting package metadata (current_repodata.json): done
Solving environment: unsuccessful initial attempt using frozen solve. Retrying with flexible solve.
Collecting package metadata (repodata.json): done
Solving environment: unsuccessful initial attempt using frozen solve. Retrying with flexible solve.

PackagesNotFoundError: The following packages are not available from current channels:

  • conda-build

Current channels:

To search for alternate channels that may provide the conda package you're
looking for, navigate to

https://anaconda.org

and use the search bar at the top of the page.

@shijunz
Copy link

shijunz commented Jul 14, 2023

I run conda install conda-build at the x64 env, and conda info show the conda-build is ok, but run : conda build -t zstd-1.5.2-h45ec737_7.tar.bz2n and get the same error.
Collecting package metadata (repodata.json): ...working... done
Solving environment: ...working... failed
failed to get install actions, retrying. exception was: Unsatisfiable dependencies for platform win-arm64: {'cmake', "vc14_runtime[version='>=14.36.32532']", 'pkg-config', "libzlib[version='>=1.2.13,<1.3.0a0']", "vc[version='>=14.3,<15']", 'vs2022_win-arm64'}
WARNING:conda_build.build:failed to get install actions, retrying. exception was: Unsatisfiable dependencies for platform win-arm64: {'cmake', "vc14_runtime[version='>=14.36.32532']", 'pkg-config', "libzlib[version='>=1.2.13,<1.3.0a0']", "vc[version='>=14.3,<15']", 'vs2022_win-arm64'}

@ytausch
Copy link
Contributor

ytausch commented Feb 16, 2024

win_arm64 should be added to the provider section of the docs

@h-vetinari
Copy link
Member

I'm going to unpin this for now to make space for the numpy migration in the top bar, which is going to hit a lot of feedstocks now. We can repin it of course once things calm down there.

@h-vetinari
Copy link
Member

Repinning this now that a slot opened up again, after the stdlib migration has been unpinned

@h-vetinari h-vetinari pinned this issue Jul 9, 2024
@jakirkham
Copy link
Member

cc @arnabanimesh (as this context may also be useful in your efforts to rebuild for Windows ARM 🙂)

@jaimergp
Copy link
Member

For the item "Build a python entrypoint launcher (cli-arm64.exe) for conda-build", would it be sufficient to copy the current launcher cli-64.exe as cli-arm64.exe and assume that Windows will "Rosetta" it just fine?

@jakirkham
Copy link
Member

How was the current entrypoint built? Presumably we could compile a new one given the source code

@jaimergp
Copy link
Member

It has a tricky backstory. It was compiled locally and committed directly to the repo(s) (conda and conda-build), then signed years later by Anaconda and resubmitted. I tried to clear up the story in https://github.com/conda/conda-launchers but there has been little follow up. I'm not sure how to sign the executables within the conda-forge pipelines either, because it would require having the Anaconda certificates somewhere in the repo and that's probably a no-go.

@jaimergp
Copy link
Member

See conda/conda-launchers#4 for more details.

@jakirkham
Copy link
Member

Should we add this to next week's Conda community meeting?

@jaimergp
Copy link
Member

Added the PR for the notes next week and included this item in the agenda. I might not make it, though, so if someone is around, please feel free to discuss it!

@baszalmstra
Copy link
Member

I took the liberty to whip up a migrator. Not sure if I did it right, feel free to comment! regro/cf-scripts#3194

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

No branches or pull requests