No module named 'distutils.msvccompiler' in fresh environments

[michaelosthege]

This is a new problem that was already reported twice on discourse:

• https://discourse.pymc.io/t/conda-install-error/10163/3
• https://discourse.pymc.io/t/fresh-install-error-no-module-named-distutils-msvccompiler/10165

The underlying cause is further down the dependency stack, but it's highly likely that we'll need to make some patches in the feedstock until the actual bugfixes become available.

One of the main discussions appears to be:

• ModuleNotFoundError: No module named 'distutils.msvccompiler' in package build CI pyodide/pyodide#2971

At the PyMC level, the problem can be reproduced by creating a new environment:

mamba create -n pmtest -c conda-forge "python=3.9" "pymc==4.1.4" -y
mamba activate pmtest
python -c "import pymc"

It looks like in the pyodide issue they've already identified the problem.

The TODO for us remains to investigate which dependencies we have to pin to which versions.

[maresb]

Does this bug only happen for Windows environments?

Does adding "setuptools !=65.*" to the conda/mamba commands fix the issue? For example:

mamba install -c conda-forge "setuptools !=65.*"

[random-walks]

Failing for me on windows even while using

mamba install -c conda-forge "setuptools !=65.*"

running python 3.10

[jonititan]

So I'm on windows 10 and the following suggested by @maresb worked for me.
I just did it in a clean environment.
conda create -c conda-forge -n pymc_env "setuptools !=65.*" "pymc>=4.0.0"

Edited to add more details
Python 3.10.5 | packaged by conda-forge | (main, Jun 14 2022, 06:57:50) [MSC v.1929 64 bit (AMD64)] on win32
It gave me pymc v 4.1.4
setuptools v 64.0.3
distutils v 3.10.5

[jonititan]

There is now also a more general thread to cover this. Setuptools Issue 3532
This was caused by Setuptools PR 3505

[random-walks]

Thanks @jonititan and @maresb, working now building the fresh env using that command.

fingers crossed on a patch soon looks like this bug is wreaking some havoc on a bunch of pkgs

[jonititan]

It looks like the breaking change in setuptools is now also being reverted so that should flow through in due course
.
pypa/setuptools#3505 (comment)

[maresb]

The new setuptools version 65.0.2 which reverts the breaking change has been merged into conda-forge, and is already available for download.

As far as I can tell, this can be triggered in the following two ways:

1. by PyMC directly via my patch to suppress spurious warnings, which affects PyMC v4.1.2, v4.1.3, and v4.1.4. (Subsequent versions will be fine due to Revert numpy warnings workaround pymc-devs/pymc#6025.)
2. indirectly by aesara >=2.5.0,<=2.7.7.

I intend to fix 2. by retroactively adding a setuptools !=65 to conda-forge/conda-forge-repodata-patches-feedstock#307.

We have the following dependencies of PyMC on Aesara:

• PyMC v4.1.2 → Aesara v2.7.5
• PyMC v4.1.3 → Aesara v2.7.7
• PyMC v4.1.4 → Aesara v2.7.9

Thus by fixing 2, the only broken version will be PyMC v4.1.4. This should be fixable by a repodata patch.

Since repodata patches are annoying and time-consuming, the most practical solution is to extend the setuptools !=65patch to aesara >=2.5.0,<=2.7.7. This is slightly more than technically necessary, but is an elegant solution. (Otherwise I'd have to do another repodata patch just for PyMC v4.1.4.)

[maresb]

To be explicit, this should now be effectively fixed by the upstream release of setuptools v65.0.2.

New environments should default to the latest setuptools v65.0.2. Old broken environments should be fixable via the command above.

The proposed solution in my previous comment takes care of any remaining edge cases by actively preventing the solver from installing a broken version setuptools alongside PyMC or Aesara.

[maresb]

@random-walks, could you please open a new issue of type "Bug report" and fill in the template with the details of your environment so that we can have a closer look at what's going on?

[maresb]

I'm waiting for conda-forge/conda-forge-repodata-patches-feedstock#307 to be merged before closing this. They seem to have a backlog of non-critical PRs. This should be fixed at some point, but I don't see it as urgent, so I'm going to be patient.

[maresb]

Closed via conda-forge/conda-forge-repodata-patches-feedstock#307