Pyo3 seems to ignore python environment in unittests since 0.19

[nfwvogt]

Bug Description

When unittesting pyo3 interfaces in a virtual environment and loading a python module for the unittests, pyo3 aborts with a module not found error, even though the module is installed in the virtual environment.

The issue vanishes when installing the python package/module in the base python install additionally to the virtual environement.

Steps to Reproduce

1. Create python venv (or conda environment)
2. Install python package in venv (e.g. numpy)
3. Use following test function

    #[test]
    fn test() {
        pyo3::prepare_freethreaded_python();
        Python::with_gil(|py| {
            let _numpy = PyModule::import(py, "numpy").unwrap();
        });
    }

The error disappears when manually installing numpy also in the base python install.

The same error can also be produced by using the numpy crate

    #[test]
    fn test_numpy() {
        pyo3::prepare_freethreaded_python();
        Python::with_gil(|py| {
            let _a: &PyArray1<usize> = vec![1, 2, 3].into_pyarray(py);
        });
    }

Backtrace

thread 'tests::test' panicked at 'called `Result::unwrap()` on an `Err` value: PyErr { type: <class 'ModuleNotFoundError'>, value: ModuleNotFoundError("No module named 'pydantic'"), traceback: None }'

Your operating system and version

macos 13.4.1 and debian bullseye

Your Python version (python --version)

3.11.4 (also on 3.10 and 3.9)

Your Rust version (rustc --version)

rustc 1.71.0

Your PyO3 version

0.19.1

How did you install python? Did you use a virtualenv?

homebrew + venv on macos
micromamba using micromamba created conda environments on linux

Additional Info

The same tests on the same machines seemed to work without issues pre-0.19

[davidhewitt]

As far as I'm aware, we've never knowingly had this working, see #1741. We think that there is scope to update the way we initialize python once we drop 3.7 which may fix this.

If you have a way to reproduce what did work before, that would be helpful in giving us a way to design a permanently working solution.

[nfwvogt]

Thank you for the clarification. After revisiting what worked before on the local machine before, it seems it always was a fluke of the installed packages underlying python setup that this worked in the first place.
So unfortunately I don't think I can provide further insights at the moment.

I'm sorry for the wrong bug report.
@davidhewitt Should I just close this issue?

[davidhewitt]

No problem. I think #1741 is a suitable duplicate of this problem, so I'll close this one.