Issues when building both wheel and conda package

[srilman]

Checklist

• I added a descriptive title
• I searched open reports and couldn't find a duplicate

What happened?

For my package, I would like to build both a Conda package and a wheel together using the same infrastructure. I saw that conda-build has some support for wheels and wanted to try it out. To test it out, I took the existing conda-forge recipe for the toolz python package and modified it to output both a wheel and conda package:

{% set name = "toolz" %}
{% set version = "0.12.0" %}

package:
  name: {{ name|lower }}
  version: {{ version }}

source:
  url: https://pypi.io/packages/source/{{ name[0] }}/{{ name }}/toolz-{{ version }}.tar.gz
  sha256: 88c570861c440ee3f2f6037c4654613228ff40c93a6c25e0eba70d17282c6194

build:
  number: 0
  noarch: python
  script: {{ PYTHON }} -m pip install . -vv

requirements:
  host:
    - pip
    - python >=3.5
  run:
    - python >=3.5

test:
  imports:
    - tlz
    - toolz
    - toolz.curried
    - toolz.functoolz
    - toolz.sandbox
  commands:
    - pip check
  requires:
    - pip

outputs:
  - type: wheel
    name: {{ name|lower }}
  - type: conda
    name: {{ name|lower }}

First, I noticed that I had to specify a name for both packages. If I only specify 1 output type (either wheel or conda), I don't need to. Is that because conda-build thinks I'm defining multiple subpackages?

Second, I noticed that conda-build was only attempting to build a wheel, not a conda package. Here is the output when I triggered the build:

❯ conda build recipe -c conda-forge                                                    
WARNING: No numpy version specified in conda_build_config.yaml.  Falling back to default numpy value of 1.22
Adding in variants from internal_defaults
Attempting to finalize metadata for toolz
Reloading output folder (local): ...working... done
Getting pinned dependencies: ...working... done
Reloading output folder (local): ...working... done
Getting pinned dependencies: ...working... done
BUILD START: ['wheel file for toolz in: <HOME-RETRACTED>/mambaforge/conda-bld/noarch']
Reloading output folder (local): ...working... done
Solving environment (_h_env): ...working... done

Is this the correct way to specify multiple output types? Is it possible? Or do I have to run conda build twice, somehow switching between the 2 outputs? Do you have any recommendations? Thanks!

Conda Info

active environment : build-test
    active env location : <HOME-RETRACTED>/mambaforge/envs/build-bodosql
            shell level : 2
       user config file : <HOME-RETRACTED>/.condarc
 populated config files : <HOME-RETRACTED>/mambaforge/.condarc
                          <HOME-RETRACTED>/.condarc
          conda version : 23.11.0
    conda-build version : 3.27.0
         python version : 3.11.6.final.0
                 solver : libmamba (default)
       virtual packages : __archspec=1=m1
                          __conda=23.11.0=0
                          __osx=14.2.1=0
                          __unix=0=0
       base environment : <HOME-RETRACTED>/mambaforge  (writable)
      conda av data dir : <HOME-RETRACTED>/mambaforge/etc/conda
  conda av metadata url : None
           channel URLs : https://conda.anaconda.org/conda-forge/osx-arm64
                          https://conda.anaconda.org/conda-forge/noarch
          package cache : <HOME-RETRACTED>/mambaforge/pkgs
                          <HOME-RETRACTED>/.conda/pkgs
       envs directories : <HOME-RETRACTED>/mambaforge/envs
                          <HOME-RETRACTED>/.conda/envs
               platform : osx-arm64
             user-agent : conda/23.11.0 requests/2.28.2 CPython/3.11.6 Darwin/23.2.0 OSX/14.2.1 solver/libmamba conda-libmamba-solver/23.12.0 libmambapy/1.5.6
                UID:GID : 501:20
             netrc file : None
           offline mode : False

Conda Config

==> <HOME-RETRACTED>/mambaforge/.condarc <==
channels:
  - conda-forge

==> <HOME-RETRACTED>/.condarc <==
auto_activate_base: False
changeps1: False
aggressive_update_packages:
  - nodefaults
remote_max_retries: 5
channels:
  - conda-forge
report_errors: False

Conda list

# Name                    Version                   Build  Channel
anaconda-client           1.12.1             pyhd8ed1ab_1    conda-forge
anaconda-project          0.11.1             pyhd8ed1ab_0    conda-forge
annotated-types           0.6.0              pyhd8ed1ab_0    conda-forge
anyio                     3.7.1              pyhd8ed1ab_0    conda-forge
archspec                  0.2.2              pyhd8ed1ab_0    conda-forge
attrs                     23.1.0             pyh71513ae_1    conda-forge
beautifulsoup4            4.12.2             pyha770c72_0    conda-forge
boa                       0.16.0             pyhd8ed1ab_1    conda-forge
boltons                   23.0.0             pyhd8ed1ab_0    conda-forge
brotli-python             1.1.0           py311ha891d26_1    conda-forge
bzip2                     1.0.8                h93a5062_5    conda-forge
c-ares                    1.22.1               h93a5062_0    conda-forge
ca-certificates           2023.11.17           hf0a4a13_0    conda-forge
cctools                   973.0.1             hd1ac623_15    conda-forge
cctools_osx-arm64         973.0.1             h2a25c60_15    conda-forge
certifi                   2023.11.17         pyhd8ed1ab_0    conda-forge
cffi                      1.16.0          py311h4a08483_0    conda-forge
chardet                   5.2.0           py311h267d04e_1    conda-forge
charset-normalizer        3.3.2              pyhd8ed1ab_0    conda-forge
check-wheel-contents      0.6.0              pyhd8ed1ab_0    conda-forge
click                     8.1.7           unix_pyh707e725_0    conda-forge
clyent                    1.2.2              pyhd8ed1ab_2    conda-forge
colorama                  0.4.6              pyhd8ed1ab_0    conda-forge
conda                     23.10.0         py311h267d04e_1    conda-forge
conda-build               3.28.3          py311h267d04e_0    conda-forge
conda-index               0.3.0              pyhd8ed1ab_1    conda-forge
conda-libmamba-solver     23.11.1            pyhd8ed1ab_0    conda-forge
conda-pack                0.7.1              pyhd8ed1ab_0    conda-forge
conda-package-handling    2.2.0              pyh38be061_0    conda-forge
conda-package-streaming   0.9.0              pyhd8ed1ab_0    conda-forge
conda-verify              3.1.1           py311h267d04e_1006    conda-forge
curl                      8.4.0                h2d989ff_0    conda-forge
defusedxml                0.7.1              pyhd8ed1ab_0    conda-forge
exceptiongroup            1.2.0              pyhd8ed1ab_0    conda-forge
filelock                  3.13.1             pyhd8ed1ab_0    conda-forge
fmt                       10.1.1               h6aa02a4_1    conda-forge
freetype                  2.12.1               hadb7bae_2    conda-forge
future                    0.18.3             pyhd8ed1ab_0    conda-forge
icu                       73.2                 hc8870d7_0    conda-forge
idna                      3.6                pyhd8ed1ab_0    conda-forge
importlib_resources       6.1.1              pyhd8ed1ab_0    conda-forge
jinja2                    3.1.2              pyhd8ed1ab_1    conda-forge
joblib                    1.3.2              pyhd8ed1ab_0    conda-forge
json5                     0.9.14             pyhd8ed1ab_0    conda-forge
jsonpatch                 1.33               pyhd8ed1ab_0    conda-forge
jsonpointer               2.4             py311h267d04e_3    conda-forge
jsonschema                4.20.0             pyhd8ed1ab_0    conda-forge
jsonschema-specifications 2023.11.1          pyhd8ed1ab_0    conda-forge
jupyter_core              5.5.0           py311h267d04e_0    conda-forge
krb5                      1.21.2               h92f50d5_0    conda-forge
lcms2                     2.15                 hf2736f0_3    conda-forge
ld64                      609                 h89fa09d_15    conda-forge
ld64_osx-arm64            609                 hc4dc95b_15    conda-forge
lerc                      4.0.0                h9a09cb3_0    conda-forge
libarchive                3.7.2                h82b9b87_0    conda-forge
libcurl                   8.4.0                h2d989ff_0    conda-forge
libcxx                    16.0.6               h4653b0c_0    conda-forge
libdeflate                1.19                 hb547adb_0    conda-forge
libedit                   3.1.20191231         hc8eb9b7_2    conda-forge
libev                     4.33                 h642e427_1    conda-forge
libexpat                  2.5.0                hb7217d7_1    conda-forge
libffi                    3.4.2                h3422bc3_5    conda-forge
libiconv                  1.17                 he4db4b2_0    conda-forge
libjpeg-turbo             3.0.0                hb547adb_1    conda-forge
liblief                   0.12.3               hb7217d7_0    conda-forge
libllvm15                 15.0.7               h504e6bf_3    conda-forge
libmamba                  1.5.3                h0a6dc31_2    conda-forge
libmambapy                1.5.3           py311hab8398d_2    conda-forge
libnghttp2                1.58.0               ha4dd798_0    conda-forge
libpng                    1.6.39               h76d750c_0    conda-forge
libsolv                   0.7.27               h9e231a4_0    conda-forge
libsqlite                 3.44.2               h091b4b1_0    conda-forge
libssh2                   1.11.0               h7a5bd25_0    conda-forge
libtiff                   4.6.0                ha8a6c65_2    conda-forge
libwebp-base              1.3.2                hb547adb_0    conda-forge
libxcb                    1.15                 hf346824_0    conda-forge
libxml2                   2.11.6               h0d0cfa8_0    conda-forge
libzlib                   1.2.13               h53f4e23_5    conda-forge
lz4-c                     1.9.4                hb7217d7_0    conda-forge
lzo                       2.10              h642e427_1000    conda-forge
markdown-it-py            3.0.0              pyhd8ed1ab_0    conda-forge
markupsafe                2.1.3           py311heffc1b2_1    conda-forge
mdurl                     0.1.0              pyhd8ed1ab_0    conda-forge
menuinst                  2.0.2           py311h267d04e_0    conda-forge
more-itertools            10.1.0             pyhd8ed1ab_0    conda-forge
nbformat                  5.9.2              pyhd8ed1ab_0    conda-forge
ncurses                   6.4                  h463b476_2    conda-forge
openjpeg                  2.5.0                h4c1507b_3    conda-forge
openssl                   3.2.0                h0d3ecfb_1    conda-forge
packaging                 23.2               pyhd8ed1ab_0    conda-forge
patch                     2.7.6             h27ca646_1002    conda-forge
pillow                    10.1.0          py311hb9c5795_0    conda-forge
pip                       23.3.1             pyhd8ed1ab_0    conda-forge
pkginfo                   1.9.6              pyhd8ed1ab_0    conda-forge
pkgutil-resolve-name      1.3.10             pyhd8ed1ab_1    conda-forge
platformdirs              4.0.0              pyhd8ed1ab_0    conda-forge
pluggy                    1.3.0              pyhd8ed1ab_0    conda-forge
prompt-toolkit            3.0.41             pyha770c72_0    conda-forge
prompt_toolkit            3.0.41               hd8ed1ab_0    conda-forge
psutil                    5.9.5           py311heffc1b2_1    conda-forge
pthread-stubs             0.4               h27ca646_1001    conda-forge
py-lief                   0.12.3          py311ha397e9f_0    conda-forge
pybind11-abi              4                    hd8ed1ab_3    conda-forge
pycosat                   0.6.6           py311heffc1b2_0    conda-forge
pycparser                 2.21               pyhd8ed1ab_0    conda-forge
pydantic                  2.5.3              pyhd8ed1ab_0    conda-forge
pydantic-core             2.14.6          py311h94f323b_1    conda-forge
pygments                  2.17.2             pyhd8ed1ab_0    conda-forge
pysocks                   1.7.1              pyha2e5f31_6    conda-forge
python                    3.11.6          h47c9636_0_cpython    conda-forge
python-dateutil           2.8.2              pyhd8ed1ab_0    conda-forge
python-fastjsonschema     2.19.0             pyhd8ed1ab_0    conda-forge
python-libarchive-c       4.0             py311h267d04e_2    conda-forge
python_abi                3.11                    4_cp311    conda-forge
pytz                      2023.3.post1       pyhd8ed1ab_0    conda-forge
pyyaml                    6.0.1           py311heffc1b2_1    conda-forge
readline                  8.2                  h92ec313_1    conda-forge
referencing               0.31.0             pyhd8ed1ab_0    conda-forge
reproc                    14.2.4.post0         h93a5062_1    conda-forge
reproc-cpp                14.2.4.post0         h965bd2d_1    conda-forge
requests                  2.31.0             pyhd8ed1ab_0    conda-forge
requests-toolbelt         1.0.0              pyhd8ed1ab_0    conda-forge
rich                      13.7.0             pyhd8ed1ab_0    conda-forge
ripgrep                   13.0.0               h5ef7bb8_3    conda-forge
rpds-py                   0.13.1          py311h94f323b_0    conda-forge
ruamel.yaml               0.18.5          py311h05b510d_0    conda-forge
ruamel.yaml.clib          0.2.7           py311heffc1b2_2    conda-forge
ruamel_yaml               0.15.80         py311heffc1b2_1009    conda-forge
setuptools                68.2.2             pyhd8ed1ab_0    conda-forge
setuptools-scm            8.0.4              pyhd8ed1ab_0    conda-forge
setuptools_scm            8.0.4                hd8ed1ab_0    conda-forge
sigtool                   0.1.3                h44b9a77_0    conda-forge
six                       1.16.0             pyh6c4a22f_0    conda-forge
sniffio                   1.3.0              pyhd8ed1ab_0    conda-forge
soupsieve                 2.5                pyhd8ed1ab_1    conda-forge
tapi                      1100.0.11            he4954df_0    conda-forge
tk                        8.6.13               h5083fa2_1    conda-forge
tomli                     2.0.1              pyhd8ed1ab_0    conda-forge
tornado                   6.3.3           py311heffc1b2_1    conda-forge
tqdm                      4.66.1             pyhd8ed1ab_0    conda-forge
traitlets                 5.14.0             pyhd8ed1ab_0    conda-forge
truststore                0.8.0              pyhd8ed1ab_0    conda-forge
typing-extensions         4.8.0                hd8ed1ab_0    conda-forge
typing_extensions         4.8.0              pyha770c72_0    conda-forge
tzdata                    2023c                h71feb2d_0    conda-forge
urllib3                   2.1.0              pyhd8ed1ab_0    conda-forge
watchgod                  0.8.2              pyhd8ed1ab_0    conda-forge
wcwidth                   0.2.12             pyhd8ed1ab_0    conda-forge
wheel                     0.42.0             pyhd8ed1ab_0    conda-forge
wheel-filename            1.4.1              pyhd8ed1ab_0    conda-forge
xorg-libxau               1.0.11               hb547adb_0    conda-forge
xorg-libxdmcp             1.1.3                h27ca646_0    conda-forge
xz                        5.2.6                h57fd34a_0    conda-forge
yaml                      0.2.5                h3422bc3_2    conda-forge
yaml-cpp                  0.8.0                h13dd4ca_0    conda-forge
zipp                      3.17.0             pyhd8ed1ab_0    conda-forge
zstandard                 0.22.0          py311h67b91a1_0    conda-forge
zstd                      1.5.5                h4f39d0f_0    conda-forge

Additional Context

No response

[srilman]

Found a somewhat related issue that mentions some dicey-ness when using the same name for both packages: #2729. Trying some alternatives including the solution mentioned here

[srilman]

Ok so the suggestion works in theory, but I ran into the issue that just including:

outputs:             #[wheel]
  - type: wheel   #[wheel]

And building both variants (where wheel is true and false) at the same time forced only the Conda package to be built, instead of both. So instead, I built them in separate invocations of conda-build like:

conda build recipe -c conda-forge --variants "{wheel: True}"
conda build recipe -c conda-forge --variants "{wheel: False}"

and that worked! Now testing with a more complex example.

[srilman]

Ok a more complex example has an issue for wheels only

{% set name = "toolz" %}
{% set version = "0.12.0" %}

package:
  name: {{ name|lower }}
  version: {{ version }}

source:
  url: https://pypi.io/packages/source/{{ name[0] }}/{{ name }}/toolz-{{ version }}.tar.gz
  sha256: 88c570861c440ee3f2f6037c4654613228ff40c93a6c25e0eba70d17282c6194

build:
  number: 0
  noarch: python
  script: {{ PYTHON }} -m pip install . -vv

requirements:
  build:
    - pip
    - python >=3.5
    - setuptools
    - setuptools_scm
  host:
    - pip
    - python >=3.5
    - setuptools
    - setuptools_scm
  run:
    - python >=3.5

test:
  imports:
    - tlz
    - toolz
    - toolz.curried
    - toolz.functoolz
    - toolz.sandbox
  commands:
    - pip check
  requires:
    - pip

outputs:             #[wheel]
  - type: wheel   #[wheel]

I added setuptools and setuptools_scm as host dependencies. For some reason, setuptools_scm was not in the host environment. From what I understand, wheel packages use build primarily for the building environment instead of host. But it wasn't picked up anywhere. Why?

[srilman]

OK I dug into the source code a little bit, and I think the code that determines what packages to build from the meta.yaml, which is conda_build/metadata.py:MetaData:get_output_metadata_set has a bug in the single outputs: - type: wheel case. It doesn't seem to recognize that the main package should be a wheel and instead thinks that some unnamed subpackage should be one. The function specifically that has an issue is conda_build/metadata.py:get_output_dicts_from_metadata. The main section of code where we can see the problem is:

                elif output_d.get("type") == "wheel":
                    if not output_d.get("requirements", {}).get("build") or not any(
                        "pip" in req for req in output_d["requirements"]["build"]
                    ):
                        build_reqs = output_d.get("requirements", {}).get("build", [])
                        build_reqs.extend(
                            ["pip", "python {}".format(m.config.variant["python"])]
                        )
                        output_d["requirements"] = output_d.get("requirements", {})
                        output_d["requirements"]["build"] = build_reqs
                        m.meta["requirements"] = m.get_section("requirements")
                        m.meta["requirements"]["build"] = build_reqs
                    non_conda_packages.append((output_d, m))

For wheels, we need to include pip for the packaging step, I believe.

[srilman]

Actually, I think what I previous had:

outputs:
  - type: wheel
    name: {{ name|lower }}
  - type: conda
    name: {{ name|lower }}

was also breaking in the same function conda_build/metadata.py:get_output_dicts_from_metadata because it was having difficulty recognizing both outputs as the same package but different types.

[srilman]

In retrospective, I'm not sure if this is a good approach in general. IMO, it seems odd to set:

outputs:
  - type: wheel

to indicate that the top package is a wheel. It seems like outputs was designed for subpackages and dependent packages. I'm not sure if it should support my use case.

I'd like to suggest an alternative syntax. Instead, why can't we set the type of the package as a top-level item in the recipe:

package:
  name: {{ name|lower }}
  version: {{ version }}
type: wheel  #[wheel]                <------------ Change
source:
  url: https://pypi.io/packages/source/{{ name[0] }}/{{ name }}/toolz-{{ version }}.tar.gz
  sha256: 88c570861c440ee3f2f6037c4654613228ff40c93a6c25e0eba70d17282c6194
build:
  number: 0
  noarch: python
  script: {{ PYTHON }} -m pip install . -vv
...

It is more consistent with the outputs syntax and better indicates the intention. We lose the potential benefit to build both package types while reusing the build environment, but I can't do that now anyways.

@msarahan Since you responded in #2729, I wanted to ask you. WDYT?

[srilman]

Anyways for the time being, I can do the following:

outputs:
  - type: wheel                         #[wheel]
    name: {{ name|lower }}       #[wheel]
  - type: conda                        #[wheel]
    name: {{ name|lower }}      #[wheel]

And that at least uses the build dependencies I specified for wheels. Going with that approach for now, but would like any better suggestions.

[srilman]

Ok while it seems to work OK for noarch packages, it has issue with C/C++ extension packages. @kenodegard Since you seem to be the most active contributor right now, let me ask you. What should I do next? It seems to me that wheel support is very unmaintained, since I ran into so many issues. I'm happy to submit PRs for bugfixes, but I don't know if there's any interest to maintain them other than me?

[github-actions]

Hi there, thank you for your contribution!

This issue has been automatically marked as stale because it has not had recent activity. It will be closed automatically if no further activity occurs.

If you would like this issue to remain open please:

1. Verify that you can still reproduce the issue at hand
2. Comment that the issue is still reproducible and include:
   - What OS and version you reproduced the issue on
   - What steps you followed to reproduce the issue

NOTE: If this issue was closed prematurely, please leave a comment.

Thanks!