marimo tries to install cv2 with pixi if cv2 python module is not available

[traversaro]

Describe the bug

First of all, thanks a lot for all the work on marimo!

I saw the feature in https://twitter.com/marimo_io/status/1774168957739827442, and I wanted to try it.

On purpose, I tried a bit tricky use case: a case in which the Python module is not called as the conda package, that is the cv2 method that is provided by the opencv conda-forge package.

In this case, the functionality fails:

traversaro@IITBMP014LW012:~$ pixi init piximarimo
traversaro@IITBMP014LW012:~$ cd piximarimo
traversaro@IITBMP014LW012:~/piximarimo$ pixi add marimo
traversaro@IITBMP014LW012:~/piximarimo$ pixi run marimo edit

        Create a new marimo app in your browser 🛠

        URL: http://localhost:2718

gio: http://127.0.0.1:2718: Operation not supported
  × could not determine any available versions for cv2 on linux-64. Either the package could not be found or version
  │ constraints on other dependencies result in a conflict.
  ╰─▶ Cannot solve the request because of: No candidates were found for cv2 *.

This is the notebook:

import marimo

__generated_with = "0.3.9"
app = marimo.App()


@app.cell
def __():
    import cv2

    return cv2,


if __name__ == "__main__":
    app.run()

Environment

traversaro@IITBMP014LW012:~/piximarimo$ marimo env
{
  "marimo": "0.3.9",
  "OS": "Linux",
  "OS Version": "5.15.146.1-microsoft-standard-WSL2",
  "Processor": "x86_64",
  "Python Version": "3.12.2",
  "Binaries": {
    "Browser": "--",
    "Node": "--"
  },
  "Requirements": {
    "click": "8.1.7",
    "importlib-resources": "missing",
    "jedi": "0.19.1",
    "markdown": "3.6",
    "pymdown-extensions": "10.7.1",
    "pygments": "2.17.2",
    "tomlkit": "0.12.4",
    "uvicorn": "0.29.0",
    "starlette": "0.37.2",
    "websocket": "missing",
    "typing-extensions": "4.10.0",
    "black": "24.3.0"
  }
}
traversaro@IITBMP014LW012:~/piximarimo$ pixi list
Package             Version       Build               Size       Kind   Source
_libgcc_mutex       0.1           conda_forge         2.5 KiB    conda  _libgcc_mutex-0.1-conda_forge.tar.bz2
_openmp_mutex       4.5           2_gnu               23.1 KiB   conda  _openmp_mutex-4.5-2_gnu.tar.bz2
anyio               4.3.0         pyhd8ed1ab_0        99.9 KiB   conda  anyio-4.3.0-pyhd8ed1ab_0.conda
black               24.3.0        py312h7900ff3_0     373.2 KiB  conda  black-24.3.0-py312h7900ff3_0.conda
bzip2               1.0.8         hd590300_5          248.3 KiB  conda  bzip2-1.0.8-hd590300_5.conda
ca-certificates     2024.2.2      hbcca054_0          151.8 KiB  conda  ca-certificates-2024.2.2-hbcca054_0.conda
click               8.1.7         unix_pyh707e725_0   82.5 KiB   conda  click-8.1.7-unix_pyh707e725_0.conda
docutils            0.20.1        py312h7900ff3_3     877.2 KiB  conda  docutils-0.20.1-py312h7900ff3_3.conda
exceptiongroup      1.2.0         pyhd8ed1ab_2        20.1 KiB   conda  exceptiongroup-1.2.0-pyhd8ed1ab_2.conda
h11                 0.14.0        pyhd8ed1ab_0        47.1 KiB   conda  h11-0.14.0-pyhd8ed1ab_0.tar.bz2
idna                3.6           pyhd8ed1ab_0        48.9 KiB   conda  idna-3.6-pyhd8ed1ab_0.conda
importlib-metadata  7.1.0         pyha770c72_0        26.4 KiB   conda  importlib-metadata-7.1.0-pyha770c72_0.conda
jedi                0.19.1        pyhd8ed1ab_0        821.6 KiB  conda  jedi-0.19.1-pyhd8ed1ab_0.conda
ld_impl_linux-64    2.40          h41732ed_0          688.2 KiB  conda  ld_impl_linux-64-2.40-h41732ed_0.conda
libexpat            2.6.2         h59595ed_0          72 KiB     conda  libexpat-2.6.2-h59595ed_0.conda
libffi              3.4.2         h7f98852_5          56.9 KiB   conda  libffi-3.4.2-h7f98852_5.tar.bz2
libgcc-ng           13.2.0        h807b86a_5          752.4 KiB  conda  libgcc-ng-13.2.0-h807b86a_5.conda
libgomp             13.2.0        h807b86a_5          409.9 KiB  conda  libgomp-13.2.0-h807b86a_5.conda
libnsl              2.0.1         hd590300_0          32.6 KiB   conda  libnsl-2.0.1-hd590300_0.conda
libsqlite           3.45.2        h2797004_0          837.4 KiB  conda  libsqlite-3.45.2-h2797004_0.conda
libuuid             2.38.1        h0b41bf4_0          32.8 KiB   conda  libuuid-2.38.1-h0b41bf4_0.conda
libxcrypt           4.4.36        hd590300_1          98 KiB     conda  libxcrypt-4.4.36-hd590300_1.conda
libzlib             1.2.13        hd590300_5          60.1 KiB   conda  libzlib-1.2.13-hd590300_5.conda
marimo              0.3.9         py312h7900ff3_0     5.3 MiB    conda  marimo-0.3.9-py312h7900ff3_0.conda
markdown            3.6           pyhd8ed1ab_0        76.5 KiB   conda  markdown-3.6-pyhd8ed1ab_0.conda
mypy_extensions     1.0.0         pyha770c72_0        10.2 KiB   conda  mypy_extensions-1.0.0-pyha770c72_0.conda
ncurses             6.4.20240210  h59595ed_0          874.7 KiB  conda  ncurses-6.4.20240210-h59595ed_0.conda
openssl             3.2.1         hd590300_1          2.7 MiB    conda  openssl-3.2.1-hd590300_1.conda
packaging           24.0          pyhd8ed1ab_0        48.7 KiB   conda  packaging-24.0-pyhd8ed1ab_0.conda
parso               0.8.3         pyhd8ed1ab_0        69.4 KiB   conda  parso-0.8.3-pyhd8ed1ab_0.tar.bz2
pathspec            0.12.1        pyhd8ed1ab_0        40.2 KiB   conda  pathspec-0.12.1-pyhd8ed1ab_0.conda
platformdirs        4.2.0         pyhd8ed1ab_0        19.7 KiB   conda  platformdirs-4.2.0-pyhd8ed1ab_0.conda
pygments            2.17.2        pyhd8ed1ab_0        840.3 KiB  conda  pygments-2.17.2-pyhd8ed1ab_0.conda
pymdown-extensions  10.7.1        pyhd8ed1ab_0        154.5 KiB  conda  pymdown-extensions-10.7.1-pyhd8ed1ab_0.conda
python              3.12.2        hab00c5b_0_cpython  30.8 MiB   conda  python-3.12.2-hab00c5b_0_cpython.conda
python_abi          3.12          4_cp312             6.2 KiB    conda  python_abi-3.12-4_cp312.conda
pyyaml              6.0.1         py312h98912ed_1     192 KiB    conda  pyyaml-6.0.1-py312h98912ed_1.conda
readline            8.2           h8228510_1          274.9 KiB  conda  readline-8.2-h8228510_1.conda
sniffio             1.3.1         pyhd8ed1ab_0        14.7 KiB   conda  sniffio-1.3.1-pyhd8ed1ab_0.conda
starlette           0.37.2        pyhd8ed1ab_0        56.3 KiB   conda  starlette-0.37.2-pyhd8ed1ab_0.conda
tk                  8.6.13        noxft_h4845f30_101  3.2 MiB    conda  tk-8.6.13-noxft_h4845f30_101.conda
tomlkit             0.12.4        pyha770c72_0        36.3 KiB   conda  tomlkit-0.12.4-pyha770c72_0.conda
tornado             6.4           py312h98912ed_0     820.8 KiB  conda  tornado-6.4-py312h98912ed_0.conda
typing_extensions   4.10.0        pyha770c72_0        36.2 KiB   conda  typing_extensions-4.10.0-pyha770c72_0.conda
tzdata              2024a         h0c530f3_0          117 KiB    conda  tzdata-2024a-h0c530f3_0.conda
uvicorn             0.29.0        py312h7900ff3_0     124 KiB    conda  uvicorn-0.29.0-py312h7900ff3_0.conda
websockets          12.0          py312h98912ed_0     196.7 KiB  conda  websockets-12.0-py312h98912ed_0.conda
xz                  5.2.6         h166bdaf_0          408.6 KiB  conda  xz-5.2.6-h166bdaf_0.tar.bz2
yaml                0.2.5         h7f98852_2          87.1 KiB   conda  yaml-0.2.5-h7f98852_2.tar.bz2
zipp                3.17.0        pyhd8ed1ab_0        18.5 KiB   conda  zipp-3.17.0-pyhd8ed1ab_0.conda

Code to reproduce

See original issue.

[traversaro]

See also prefix-dev/pixi#958 for a related discussion on pixi.

[akshayka]

We have a rudimentary mapping from python module names to names on pypi, and cv2 isn't on there, moreover cv2 has a different name on pypi than on conda forge.

I looked at the discussion and our link to https://raw.githubusercontent.com/regro/cf-graph-countyfair/master/mappings/pypi/name_mapping.yaml -- can we just use that as our mapping source of truth? It looks pretty comprehensive ...

[akshayka]

I looked at the discussion and our link to https://raw.githubusercontent.com/regro/cf-graph-countyfair/master/mappings/pypi/name_mapping.yaml -- can we just use that as our mapping source of truth? It looks pretty comprehensive ...

This also looks incomplete. It's missing scikit-learn, but has several scikit-learn-* packages.

If anyone knows of a canonical open mapping from module names to pypi/conda package names, that would be very helpful.

[akshayka]

I have a PR out that fixes the particular case of cv2. I am going to rename this issue to reflect the broader problem of obtaining a canonical mapping from module name to pypi name/conda (forge?) name.

[traversaro]

If anyone knows of a canonical open mapping from module names to pypi/conda package names, that would be very helpful.

Some tricky ones are added in https://github.com/conda-incubator/grayskull/blob/main/grayskull/strategy/config.yaml, and new one can be added by the community, see also conda/grayskull#168 .

[traversaro]

Thanks a lot for the quick fix!