Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Specifying _x86_64-microarch-level=something doesn't solve properly. #13

Closed
1 task done
ikrommyd opened this issue Dec 18, 2024 · 12 comments · Fixed by #14
Closed
1 task done

Specifying _x86_64-microarch-level=something doesn't solve properly. #13

ikrommyd opened this issue Dec 18, 2024 · 12 comments · Fixed by #14
Labels
bug Something isn't working

Comments

@ikrommyd
Copy link

ikrommyd commented Dec 18, 2024

Solution to issue cannot be found in the documentation.

  • I checked the documentation.

Issue

Example: conda create -n bug awkward=2.7.2 python=3.11 _x86_64-microarch-level=1 is giving

Channels:
 - conda-forge
Platform: linux-64
Collecting package metadata (repodata.json): done
Solving environment: failed

LibMambaUnsatisfiableError: Encountered problems while solving:
  - nothing provides __archspec 1 core2 needed by _x86_64-microarch-level-1-0_core2

Could not solve for environment specs
The following packages are incompatible
└─ _x86_64-microarch-level =1 * is not installable because there are no viable options
   ├─ _x86_64-microarch-level 1 would require
   │  └─ __archspec ==1 core2, which is missing on the system;
   ├─ _x86_64-microarch-level 1 would require
   │  └─ __archspec ==1 k10, which is missing on the system;
   ├─ _x86_64-microarch-level 1 would require
   │  └─ __archspec ==1 nocona, which is missing on the system;
   └─ _x86_64-microarch-level 1 would require
      └─ __archspec ==1 x86_64, which is missing on the system.

The way I read it, this is a problem with __archspec=1=broadwell which you can see in the conda info because broadwell is not a viable microarch 1/2 version. However the newer ones are extensions, so older instructions are supported by newer cpus so it shouldn't care right?

On the other hand, pixi is able to do this fine

$ pixi add 'awkward 2.7.2' 'python 3.11' '_x86_64-microarch-level 1'
✔ Added awkward 2.7.2
✔ Added python 3.11
✔ Added _x86_64-microarch-level 1

cc @matthewfeickert

Installed packages

Well this is creating a new environment but the `conda list` of `base` is

iason@lima-x86-ubuntu:/Users/iason$ conda list
# packages in environment at /home/iason.linux/miniforge3:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       2_gnu    conda-forge
archspec                  0.2.3              pyhd8ed1ab_0    conda-forge
boltons                   24.0.0             pyhd8ed1ab_1    conda-forge
brotli-python             1.1.0           py312h2ec8cdc_2    conda-forge
bzip2                     1.0.8                h4bc722e_7    conda-forge
c-ares                    1.34.3               hb9d3cd8_1    conda-forge
ca-certificates           2024.8.30            hbcca054_0    conda-forge
certifi                   2024.8.30          pyhd8ed1ab_0    conda-forge
cffi                      1.17.1          py312h06ac9bb_0    conda-forge
charset-normalizer        3.4.0              pyhd8ed1ab_1    conda-forge
colorama                  0.4.6              pyhd8ed1ab_1    conda-forge
conda                     24.11.0         py312h7900ff3_0    conda-forge
conda-libmamba-solver     24.11.1            pyhd8ed1ab_0    conda-forge
conda-package-handling    2.4.0              pyha770c72_1    conda-forge
conda-package-streaming   0.11.0             pyhd8ed1ab_0    conda-forge
cpp-expected              1.1.0                hf52228f_0    conda-forge
distro                    1.9.0              pyhd8ed1ab_0    conda-forge
fmt                       11.0.2               h434a139_0    conda-forge
frozendict                2.4.6           py312h66e93f0_0    conda-forge
h2                        4.1.0              pyhd8ed1ab_1    conda-forge
hpack                     4.0.0              pyhd8ed1ab_1    conda-forge
hyperframe                6.0.1              pyhd8ed1ab_1    conda-forge
idna                      3.10               pyhd8ed1ab_1    conda-forge
jsonpatch                 1.33               pyhd8ed1ab_1    conda-forge
jsonpointer               3.0.0           py312h7900ff3_1    conda-forge
keyutils                  1.6.1                h166bdaf_0    conda-forge
krb5                      1.21.3               h659f571_0    conda-forge
ld_impl_linux-64          2.43                 h712a8e2_2    conda-forge
libarchive                3.7.7                h4585015_3    conda-forge
libcurl                   8.11.1               h332b0f4_0    conda-forge
libedit                   3.1.20191231         he28a2e2_2    conda-forge
libev                     4.33                 hd590300_2    conda-forge
libexpat                  2.6.4                h5888daf_0    conda-forge
libffi                    3.4.2                h7f98852_5    conda-forge
libgcc                    14.2.0               h77fa898_1    conda-forge
libgcc-ng                 14.2.0               h69a702a_1    conda-forge
libgomp                   14.2.0               h77fa898_1    conda-forge
libiconv                  1.17                 hd590300_2    conda-forge
liblzma                   5.6.3                hb9d3cd8_1    conda-forge
libmamba                  2.0.5                hf72d635_0    conda-forge
libmambapy                2.0.5           py312hf3f0a4e_0    conda-forge
libnghttp2                1.64.0               h161d5f1_0    conda-forge
libnsl                    2.0.1                hd590300_0    conda-forge
libsolv                   0.7.30               h3509ff9_0    conda-forge
libsqlite                 3.47.2               hee588c1_0    conda-forge
libssh2                   1.11.1               hf672d98_0    conda-forge
libstdcxx                 14.2.0               hc0a3c3a_1    conda-forge
libstdcxx-ng              14.2.0               h4852527_1    conda-forge
libuuid                   2.38.1               h0b41bf4_0    conda-forge
libxcrypt                 4.4.36               hd590300_1    conda-forge
libxml2                   2.13.5               h0d44e9d_1    conda-forge
libzlib                   1.3.1                hb9d3cd8_2    conda-forge
lz4-c                     1.10.0               h5888daf_1    conda-forge
lzo                       2.10              hd590300_1001    conda-forge
mamba                     2.0.5                hfdd0a45_0    conda-forge
menuinst                  2.2.0           py312h7900ff3_0    conda-forge
ncurses                   6.5                  he02047a_1    conda-forge
nlohmann_json             3.11.3               he02047a_1    conda-forge
openssl                   3.4.0                hb9d3cd8_0    conda-forge
packaging                 24.2               pyhd8ed1ab_2    conda-forge
pip                       24.3.1             pyh8b19718_0    conda-forge
platformdirs              4.3.6              pyhd8ed1ab_1    conda-forge
pluggy                    1.5.0              pyhd8ed1ab_1    conda-forge
pybind11-abi              4                    hd8ed1ab_3    conda-forge
pycosat                   0.6.6           py312h66e93f0_2    conda-forge
pycparser                 2.22               pyh29332c3_1    conda-forge
pysocks                   1.7.1              pyha55dd90_7    conda-forge
python                    3.12.8          h9e4cc4f_1_cpython    conda-forge
python_abi                3.12                    5_cp312    conda-forge
readline                  8.2                  h8228510_1    conda-forge
reproc                    14.2.5.post0         hb9d3cd8_0    conda-forge
reproc-cpp                14.2.5.post0         h5888daf_0    conda-forge
requests                  2.32.3             pyhd8ed1ab_1    conda-forge
ruamel.yaml               0.18.6          py312h66e93f0_1    conda-forge
ruamel.yaml.clib          0.2.8           py312h66e93f0_1    conda-forge
setuptools                75.6.0             pyhff2d567_1    conda-forge
simdjson                  3.11.3               h84d6215_0    conda-forge
spdlog                    1.14.1               hed91bc2_1    conda-forge
tk                        8.6.13          noxft_h4845f30_101    conda-forge
tqdm                      4.67.1             pyhd8ed1ab_0    conda-forge
truststore                0.10.0             pyhd8ed1ab_0    conda-forge
tzdata                    2024b                hc8b5060_0    conda-forge
urllib3                   2.2.3              pyhd8ed1ab_1    conda-forge
wheel                     0.45.1             pyhd8ed1ab_1    conda-forge
yaml-cpp                  0.8.0                h59595ed_0    conda-forge
zstandard                 0.23.0          py312hef9b889_1    conda-forge
zstd                      1.5.6                ha6fb4c9_0    conda-forge

Environment info

iason@lima-x86-ubuntu:/Users/iason/work/coffea_dev$ conda info

     active environment : None
            shell level : 0
       user config file : /home/iason.linux/.condarc
 populated config files : /home/iason.linux/miniforge3/.condarc
                          /home/iason.linux/.condarc
          conda version : 24.11.0
    conda-build version : not installed
         python version : 3.12.8.final.0
                 solver : libmamba (default)
       virtual packages : __archspec=1=broadwell
                          __conda=24.11.0=0
                          __glibc=2.35=0
                          __linux=5.15.0=0
                          __unix=0=0
       base environment : /home/iason.linux/miniforge3  (writable)
      conda av data dir : /home/iason.linux/miniforge3/etc/conda
  conda av metadata url : None
           channel URLs : https://conda.anaconda.org/conda-forge/linux-64
                          https://conda.anaconda.org/conda-forge/noarch
          package cache : /home/iason.linux/miniforge3/pkgs
                          /home/iason.linux/.conda/pkgs
       envs directories : /home/iason.linux/miniforge3/envs
                          /home/iason.linux/.conda/envs
               platform : linux-64
             user-agent : conda/24.11.0 requests/2.32.3 CPython/3.12.8 Linux/5.15.0-100-generic ubuntu/22.04.4 glibc/2.35 solver/libmamba conda-libmamba-solver/24.11.1 libmambapy/2.0.5
                UID:GID : 501:1000
             netrc file : None
           offline mode : False
@ikrommyd ikrommyd added the bug Something isn't working label Dec 18, 2024
@ikrommyd
Copy link
Author

ikrommyd commented Dec 18, 2024

Also reported on mamba: mamba-org/mamba#3698 as there is a closed somewhat similar issue but not exactly the same mamba-org/mamba#3222

@isuruf
Copy link
Member

isuruf commented Dec 18, 2024

What's your use-case for doing such a convoluted solve?

@ikrommyd
Copy link
Author

What's your use-case for doing such a convoluted solve?

We make docker containers that are unpacked into apptainers and are then used by workers of a condor clusters. Sometimes those workers may have old CPUs that don't support newer instructions. Therefore, if libraries provide packages with different microarch builds (like awkward does), we would like to install the microarch=1 build.

@isuruf
Copy link
Member

isuruf commented Dec 18, 2024

In that case, the correct invocation is

CONDA_OVERRIDE_ARCHSPEC=x86_64 conda create -n bug awkward=2.7.2 python=3.11

or

CONDA_OVERRIDE_ARCHSPEC=x86_64_v2 conda create -n bug awkward=2.7.2 python=3.11

or similar

@ikrommyd
Copy link
Author

ikrommyd commented Dec 18, 2024

In that case, the correct invocation is

CONDA_OVERRIDE_ARCHSPEC=x86_64 conda create -n bug awkward=2.7.2 python=3.11

or

CONDA_OVERRIDE_ARCHSPEC=x86_64_v2 conda create -n bug awkward=2.7.2 python=3.11

or similar

Ah yes, I also read about this CONDA_OVERRIDE_ARCHSPEC in the docs. Thanks for this! It works.
One question though, is this safe? In the sense If I do, CONDA_OVERRIDE_ARCHSPEC=x86_64_v2, will it check that my system actually supports x86_64_v2 or will it just install things anyways?

@ikrommyd
Copy link
Author

ikrommyd commented Dec 18, 2024

@jjerphan, ccing from the other issue on mamba. The suggested solution from @isuruf is good. Intuitively, I would expect _x86_64-microarch-level=1 to work as well though. Just ccing you in case you wanna leave it as is and only allow CONDA_OVERRIDE_ARCHSPEC as the solution. In either case, it should be a little more clear in the docs how to pick your microarch.

@isuruf
Copy link
Member

isuruf commented Dec 18, 2024

Intuitively, I would expect _x86_64-microarch-level=1 to work as well though.

It starts with a underscore, so it's not a public thing that a user should be dealing with.

@isuruf
Copy link
Member

isuruf commented Dec 18, 2024

will it check that my system actually supports x86_64_v2 or will it just install things anyways?

No checks

@ikrommyd
Copy link
Author

will it check that my system actually supports x86_64_v2 or will it just install things anyways?

No checks

Isn't that a little dangerous?

@isuruf
Copy link
Member

isuruf commented Dec 18, 2024

Isn't that a little dangerous?

It is, but it also allows an upgrade too. i.e. install a better optimized package to be run in a compute node (eg: level=4) than what is there in a login node (eg: level = 2)

@ikrommyd
Copy link
Author

Alrighty, if that is the only recommended solution and you don't wanna make any changes to _x86_64-microarch-level, I'm happy and you can close this. Already used in CoffeaTeam/af-images#60.
I'd like to request some documentation of this though as I couldn't find it anywhere. Related issue already exists: conda-forge/conda-forge.github.io#2105

@jjerphan
Copy link
Member

Feel free to add documentation for it if you have time, @ikrommyd.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants