Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ModuleNotFoundError in pypy3.8 created with symlinked binary #2447

Closed
starhel opened this issue Nov 14, 2022 · 5 comments
Closed

ModuleNotFoundError in pypy3.8 created with symlinked binary #2447

starhel opened this issue Nov 14, 2022 · 5 comments
Labels
bug help-wanted

Comments

@starhel
Copy link

starhel commented Nov 14, 2022

Issue
I'm keeping all my python versions in $HOME/bin directory as symlinks. From 20.16.7 pypy3 virtualenvs created this way are not working properly:

> python
ModuleNotFoundError: No module named 'encodings'
debug: OperationError:
debug:  operror-type: ModuleNotFoundError
debug:  operror-value: No module named 'encodings'

I've noticed that the home key in pyvenv.cfg changed from a directory where symlinked python exists to a directory with a symlink. Changing this value to correct one fixed the issue with pypy.

Environment

  • OS: Ubuntu 20
  • pip list of the venv with virtualenv installed:
Package      Version
------------ -------
distlib      0.3.6
filelock     3.8.0
pip          20.2.3
platformdirs 2.5.4
setuptools   49.2.1
virtualenv   20.16.7

Reproduction

wget https://downloads.python.org/pypy/pypy3.8-v7.3.9-linux64.tar.bz2
tar -xf pypy3.8-v7.3.9-linux64.tar.bz2 
mkdir bin
ln -s ../pypy3.8-v7.3.9-linux64/bin/pypy3.8 bin/pypy3.8

python3 -m venv venv
source venv/bin/activate
pip install --upgrade virtualenv

virtualenv --always-copy -p bin/pypy3.8 venv_pypy
source venv_pypy/bin/activate
python --version  # error
# change home in pyvenv.cfg from bin to pypy3.8-v7.3.9-linux64
python --version  # ok

Output of the virtual environment creation

51 setup logging to NOTSET [DEBUG report:35]
55 find interpreter for spec PythonSpec(path=bin/pypy3.8) [INFO builtin:56]
56 Attempting to acquire lock 140514597215584 on /home/astachlewski/.local/share/virtualenv/py_info/1/42185fae4a9759b8ee9b49fdff8b66ca18709e3cfc13aa4daa779c8cf7116d3e.lock [DEBUG _api:172]
56 Lock 140514597215584 acquired on /home/astachlewski/.local/share/virtualenv/py_info/1/42185fae4a9759b8ee9b49fdff8b66ca18709e3cfc13aa4daa779c8cf7116d3e.lock [DEBUG _api:176]
56 got python info of /home/astachlewski/tmp/bin/pypy3.8 from /home/astachlewski/.local/share/virtualenv/py_info/1/42185fae4a9759b8ee9b49fdff8b66ca18709e3cfc13aa4daa779c8cf7116d3e.json [DEBUG via_disk_folder:129]
57 Attempting to release lock 140514597215584 on /home/astachlewski/.local/share/virtualenv/py_info/1/42185fae4a9759b8ee9b49fdff8b66ca18709e3cfc13aa4daa779c8cf7116d3e.lock [DEBUG _api:209]
57 Lock 140514597215584 released on /home/astachlewski/.local/share/virtualenv/py_info/1/42185fae4a9759b8ee9b49fdff8b66ca18709e3cfc13aa4daa779c8cf7116d3e.lock [DEBUG _api:212]
57 proposed PythonInfo(spec=PyPy3.8.13.final.0-64, exe=/home/astachlewski/tmp/bin/pypy3.8, platform=linux, version='3.8.13 (4b1398fe9d76ad762155d03684c2a153d230b2ef, Mar 29 2022, 07:08:24)\n[PyPy 7.3.9 with GCC 10.2.1 20210130 (Red Hat 10.2.1-11)]', encoding_fs_io=utf-8-UTF-8) [INFO builtin:63]
57 accepted PythonInfo(spec=PyPy3.8.13.final.0-64, exe=/home/astachlewski/tmp/bin/pypy3.8, platform=linux, version='3.8.13 (4b1398fe9d76ad762155d03684c2a153d230b2ef, Mar 29 2022, 07:08:24)\n[PyPy 7.3.9 with GCC 10.2.1 20210130 (Red Hat 10.2.1-11)]', encoding_fs_io=utf-8-UTF-8) [DEBUG builtin:65]
58 filesystem is case-sensitive [DEBUG info:24]
79 create virtual environment via PyPy3Posix(dest=/home/astachlewski/tmp/venv_pypy, clear=False, no_vcs_ignore=False, global=False) [INFO session:48]
80 create folder /home/astachlewski/tmp/venv_pypy/bin [DEBUG _sync:9]
80 create folder /home/astachlewski/tmp/venv_pypy/lib/pypy3.8/site-packages [DEBUG _sync:9]
80 write /home/astachlewski/tmp/venv_pypy/pyvenv.cfg [DEBUG pyenv_cfg:30]
80 	home = /home/astachlewski/tmp/bin [DEBUG pyenv_cfg:34]
80 	implementation = PyPy [DEBUG pyenv_cfg:34]
80 	version_info = 3.8.13.final.0 [DEBUG pyenv_cfg:34]
80 	virtualenv = 20.16.7 [DEBUG pyenv_cfg:34]
80 	include-system-site-packages = false [DEBUG pyenv_cfg:34]
80 	base-prefix = /home/astachlewski/tmp/pypy3.8-v7.3.9-linux64 [DEBUG pyenv_cfg:34]
80 	base-exec-prefix = /home/astachlewski/tmp/pypy3.8-v7.3.9-linux64 [DEBUG pyenv_cfg:34]
80 	base-executable = /home/astachlewski/tmp/bin/pypy3.8 [DEBUG pyenv_cfg:34]
80 copy /home/astachlewski/tmp/bin/pypy3.8 to /home/astachlewski/tmp/venv_pypy/bin/pypy [DEBUG _sync:36]
81 copy /home/astachlewski/tmp/bin/pypy3.8 to /home/astachlewski/tmp/venv_pypy/bin/pypy3 [DEBUG _sync:36]
81 copy /home/astachlewski/tmp/bin/pypy3.8 to /home/astachlewski/tmp/venv_pypy/bin/python [DEBUG _sync:36]
81 copy /home/astachlewski/tmp/bin/pypy3.8 to /home/astachlewski/tmp/venv_pypy/bin/python3 [DEBUG _sync:36]
81 copy /home/astachlewski/tmp/bin/pypy3.8 to /home/astachlewski/tmp/venv_pypy/bin/python3.8 [DEBUG _sync:36]
81 copy /home/astachlewski/tmp/pypy3.8-v7.3.9-linux64/bin/libpypy3-c.so.debug to /home/astachlewski/tmp/venv_pypy/bin/libpypy3-c.so.debug [DEBUG _sync:36]
94 copy /home/astachlewski/tmp/pypy3.8-v7.3.9-linux64/bin/libpypy3-c.so to /home/astachlewski/tmp/venv_pypy/bin/libpypy3-c.so [DEBUG _sync:36]
128 copy /home/astachlewski/tmp/pypy3.8-v7.3.9-linux64/lib/libncursesw.so.6 to /home/astachlewski/tmp/venv_pypy/lib/libncursesw.so.6 [DEBUG _sync:36]
128 copy /home/astachlewski/tmp/pypy3.8-v7.3.9-linux64/lib/libtinfow.so.6 to /home/astachlewski/tmp/venv_pypy/lib/libtinfow.so.6 [DEBUG _sync:36]
128 copy /home/astachlewski/tmp/pypy3.8-v7.3.9-linux64/lib/libtk8.5.so to /home/astachlewski/tmp/venv_pypy/lib/libtk8.5.so [DEBUG _sync:36]
129 copy /home/astachlewski/tmp/pypy3.8-v7.3.9-linux64/lib/libpanelw.so.6 to /home/astachlewski/tmp/venv_pypy/lib/libpanelw.so.6 [DEBUG _sync:36]
129 copy /home/astachlewski/tmp/pypy3.8-v7.3.9-linux64/lib/libexpat.so.1 to /home/astachlewski/tmp/venv_pypy/lib/libexpat.so.1 [DEBUG _sync:36]
130 copy /home/astachlewski/tmp/pypy3.8-v7.3.9-linux64/lib/liblzma.so.5 to /home/astachlewski/tmp/venv_pypy/lib/liblzma.so.5 [DEBUG _sync:36]
130 copy /home/astachlewski/tmp/pypy3.8-v7.3.9-linux64/lib/libgdbm.so.4 to /home/astachlewski/tmp/venv_pypy/lib/libgdbm.so.4 [DEBUG _sync:36]
130 copy directory /home/astachlewski/tmp/pypy3.8-v7.3.9-linux64/lib/tcl8.5 to /home/astachlewski/tmp/venv_pypy/lib/tcl8.5 [DEBUG _sync:36]
143 copy directory /home/astachlewski/tmp/pypy3.8-v7.3.9-linux64/lib/tk8.5 to /home/astachlewski/tmp/venv_pypy/lib/tk8.5 [DEBUG _sync:36]
152 copy /home/astachlewski/tmp/pypy3.8-v7.3.9-linux64/lib/libsqlite3.so.0 to /home/astachlewski/tmp/venv_pypy/lib/libsqlite3.so.0 [DEBUG _sync:36]
153 copy /home/astachlewski/tmp/pypy3.8-v7.3.9-linux64/lib/libffi.so.6 to /home/astachlewski/tmp/venv_pypy/lib/libffi.so.6 [DEBUG _sync:36]
153 copy /home/astachlewski/tmp/pypy3.8-v7.3.9-linux64/lib/libtcl8.5.so to /home/astachlewski/tmp/venv_pypy/lib/libtcl8.5.so [DEBUG _sync:36]
154 create virtualenv import hook file /home/astachlewski/tmp/venv_pypy/lib/pypy3.8/site-packages/_virtualenv.pth [DEBUG api:89]
154 create /home/astachlewski/tmp/venv_pypy/lib/pypy3.8/site-packages/_virtualenv.py [DEBUG api:92]
154 ============================== target debug ============================== [DEBUG session:50]
154 debug via /home/astachlewski/tmp/venv_pypy/bin/pypy3 /home/astachlewski/tmp/venv/lib/python3.8/site-packages/virtualenv/create/debug.py [DEBUG creator:197]
154 {
  "out": "",
  "err": "ModuleNotFoundError: No module named 'encodings'\ndebug: OperationError:\ndebug:  operror-type: ModuleNotFoundError\ndebug:  operror-value: No module named 'encodings'\n",
  "returncode": 1,
  "exception": "Exception(\"ModuleNotFoundError: No module named 'encodings'\\ndebug: OperationError:\\ndebug:  operror-type: ModuleNotFoundError\\ndebug:  operror-value: No module named 'encodings'\\n\")"
} [DEBUG session:51]
194 add seed packages via FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/astachlewski/.local/share/virtualenv) [INFO session:55]
198 got embed update of distribution setuptools from /home/astachlewski/.local/share/virtualenv/wheel/3.8/embed/3/setuptools.json [DEBUG via_disk_folder:129]
199 got embed update of distribution wheel from /home/astachlewski/.local/share/virtualenv/wheel/3.8/embed/3/wheel.json [DEBUG via_disk_folder:129]
199 got embed update of distribution pip from /home/astachlewski/.local/share/virtualenv/wheel/3.8/embed/3/pip.json [DEBUG via_disk_folder:129]
206 got embed update of distribution wheel from /home/astachlewski/.local/share/virtualenv/wheel/3.8/embed/3/wheel.json [DEBUG via_disk_folder:129]
206 using periodically updated wheel /home/astachlewski/.local/share/virtualenv/wheel/house/wheel-0.37.1-py2.py3-none-any.whl [DEBUG periodic_update:41]
206 got embed update of distribution setuptools from /home/astachlewski/.local/share/virtualenv/wheel/3.8/embed/3/setuptools.json [DEBUG via_disk_folder:129]
206 got embed update of distribution pip from /home/astachlewski/.local/share/virtualenv/wheel/3.8/embed/3/pip.json [DEBUG via_disk_folder:129]
207 using periodically updated wheel /home/astachlewski/.local/share/virtualenv/wheel/house/setuptools-65.5.0-py3-none-any.whl [DEBUG periodic_update:41]
207 using periodically updated wheel /home/astachlewski/.local/share/virtualenv/wheel/house/pip-22.3-py3-none-any.whl [DEBUG periodic_update:41]
208 install wheel from wheel /home/astachlewski/.local/share/virtualenv/wheel/house/wheel-0.37.1-py2.py3-none-any.whl via CopyPipInstall [DEBUG via_app_data:47]
208 install setuptools from wheel /home/astachlewski/.local/share/virtualenv/wheel/house/setuptools-65.5.0-py3-none-any.whl via CopyPipInstall [DEBUG via_app_data:47]
209 Attempting to acquire lock 140514568567536 on /home/astachlewski/.local/share/virtualenv/wheel/3.8/image/1/CopyPipInstall/wheel-0.37.1-py2.py3-none-any.lock [DEBUG _api:172]
209 Attempting to acquire lock 140514568567296 on /home/astachlewski/.local/share/virtualenv/wheel/3.8/image/1/CopyPipInstall/setuptools-65.5.0-py3-none-any.lock [DEBUG _api:172]
209 install pip from wheel /home/astachlewski/.local/share/virtualenv/wheel/house/pip-22.3-py3-none-any.whl via CopyPipInstall [DEBUG via_app_data:47]
209 Lock 140514568567536 acquired on /home/astachlewski/.local/share/virtualenv/wheel/3.8/image/1/CopyPipInstall/wheel-0.37.1-py2.py3-none-any.lock [DEBUG _api:176]
210 Attempting to acquire lock 140514568567632 on /home/astachlewski/.local/share/virtualenv/wheel/3.8/image/1/CopyPipInstall/pip-22.3-py3-none-any.lock [DEBUG _api:172]
210 Lock 140514568567296 acquired on /home/astachlewski/.local/share/virtualenv/wheel/3.8/image/1/CopyPipInstall/setuptools-65.5.0-py3-none-any.lock [DEBUG _api:176]
210 Lock 140514568567632 acquired on /home/astachlewski/.local/share/virtualenv/wheel/3.8/image/1/CopyPipInstall/pip-22.3-py3-none-any.lock [DEBUG _api:176]
210 Attempting to release lock 140514568567536 on /home/astachlewski/.local/share/virtualenv/wheel/3.8/image/1/CopyPipInstall/wheel-0.37.1-py2.py3-none-any.lock [DEBUG _api:209]
210 Lock 140514568567536 released on /home/astachlewski/.local/share/virtualenv/wheel/3.8/image/1/CopyPipInstall/wheel-0.37.1-py2.py3-none-any.lock [DEBUG _api:212]
210 Attempting to release lock 140514568567632 on /home/astachlewski/.local/share/virtualenv/wheel/3.8/image/1/CopyPipInstall/pip-22.3-py3-none-any.lock [DEBUG _api:209]
210 Lock 140514568567632 released on /home/astachlewski/.local/share/virtualenv/wheel/3.8/image/1/CopyPipInstall/pip-22.3-py3-none-any.lock [DEBUG _api:212]
210 copy directory /home/astachlewski/.local/share/virtualenv/wheel/3.8/image/1/CopyPipInstall/pip-22.3-py3-none-any/pip-22.3.dist-info to /home/astachlewski/tmp/venv_pypy/lib/pypy3.8/site-packages/pip-22.3.dist-info [DEBUG _sync:36]
210 Attempting to release lock 140514568567296 on /home/astachlewski/.local/share/virtualenv/wheel/3.8/image/1/CopyPipInstall/setuptools-65.5.0-py3-none-any.lock [DEBUG _api:209]
211 Lock 140514568567296 released on /home/astachlewski/.local/share/virtualenv/wheel/3.8/image/1/CopyPipInstall/setuptools-65.5.0-py3-none-any.lock [DEBUG _api:212]
211 copy /home/astachlewski/.local/share/virtualenv/wheel/3.8/image/1/CopyPipInstall/wheel-0.37.1-py2.py3-none-any/wheel-0.37.1.virtualenv to /home/astachlewski/tmp/venv_pypy/lib/pypy3.8/site-packages/wheel-0.37.1.virtualenv [DEBUG _sync:36]
212 copy directory /home/astachlewski/.local/share/virtualenv/wheel/3.8/image/1/CopyPipInstall/setuptools-65.5.0-py3-none-any/setuptools-65.5.0.dist-info to /home/astachlewski/tmp/venv_pypy/lib/pypy3.8/site-packages/setuptools-65.5.0.dist-info [DEBUG _sync:36]
212 copy directory /home/astachlewski/.local/share/virtualenv/wheel/3.8/image/1/CopyPipInstall/wheel-0.37.1-py2.py3-none-any/wheel to /home/astachlewski/tmp/venv_pypy/lib/pypy3.8/site-packages/wheel [DEBUG _sync:36]
214 copy directory /home/astachlewski/.local/share/virtualenv/wheel/3.8/image/1/CopyPipInstall/pip-22.3-py3-none-any/pip to /home/astachlewski/tmp/venv_pypy/lib/pypy3.8/site-packages/pip [DEBUG _sync:36]
217 copy directory /home/astachlewski/.local/share/virtualenv/wheel/3.8/image/1/CopyPipInstall/setuptools-65.5.0-py3-none-any/pkg_resources to /home/astachlewski/tmp/venv_pypy/lib/pypy3.8/site-packages/pkg_resources [DEBUG _sync:36]
225 copy directory /home/astachlewski/.local/share/virtualenv/wheel/3.8/image/1/CopyPipInstall/wheel-0.37.1-py2.py3-none-any/wheel-0.37.1.dist-info to /home/astachlewski/tmp/venv_pypy/lib/pypy3.8/site-packages/wheel-0.37.1.dist-info [DEBUG _sync:36]
231 generated console scripts wheel-3.8 wheel3 wheel3.8 wheel [DEBUG base:41]
241 copy directory /home/astachlewski/.local/share/virtualenv/wheel/3.8/image/1/CopyPipInstall/setuptools-65.5.0-py3-none-any/setuptools to /home/astachlewski/tmp/venv_pypy/lib/pypy3.8/site-packages/setuptools [DEBUG _sync:36]
271 copy directory /home/astachlewski/.local/share/virtualenv/wheel/3.8/image/1/CopyPipInstall/setuptools-65.5.0-py3-none-any/_distutils_hack to /home/astachlewski/tmp/venv_pypy/lib/pypy3.8/site-packages/_distutils_hack [DEBUG _sync:36]
271 copy /home/astachlewski/.local/share/virtualenv/wheel/3.8/image/1/CopyPipInstall/setuptools-65.5.0-py3-none-any/distutils-precedence.pth to /home/astachlewski/tmp/venv_pypy/lib/pypy3.8/site-packages/distutils-precedence.pth [DEBUG _sync:36]
271 copy /home/astachlewski/.local/share/virtualenv/wheel/3.8/image/1/CopyPipInstall/setuptools-65.5.0-py3-none-any/setuptools-65.5.0.virtualenv to /home/astachlewski/tmp/venv_pypy/lib/pypy3.8/site-packages/setuptools-65.5.0.virtualenv [DEBUG _sync:36]
272 generated console scripts  [DEBUG base:41]
288 copy /home/astachlewski/.local/share/virtualenv/wheel/3.8/image/1/CopyPipInstall/pip-22.3-py3-none-any/pip-22.3.virtualenv to /home/astachlewski/tmp/venv_pypy/lib/pypy3.8/site-packages/pip-22.3.virtualenv [DEBUG _sync:36]
289 generated console scripts pip3 pip pip3.8 pip-3.8 [DEBUG base:41]
289 add activators for Bash, CShell, Fish, Nushell, PowerShell, Python [INFO session:61]
292 write /home/astachlewski/tmp/venv_pypy/pyvenv.cfg [DEBUG pyenv_cfg:30]
292 	home = /home/astachlewski/tmp/bin [DEBUG pyenv_cfg:34]
292 	implementation = PyPy [DEBUG pyenv_cfg:34]
292 	version_info = 3.8.13.final.0 [DEBUG pyenv_cfg:34]
292 	virtualenv = 20.16.7 [DEBUG pyenv_cfg:34]
292 	include-system-site-packages = false [DEBUG pyenv_cfg:34]
292 	base-prefix = /home/astachlewski/tmp/pypy3.8-v7.3.9-linux64 [DEBUG pyenv_cfg:34]
293 	base-exec-prefix = /home/astachlewski/tmp/pypy3.8-v7.3.9-linux64 [DEBUG pyenv_cfg:34]
293 	base-executable = /home/astachlewski/tmp/bin/pypy3.8 [DEBUG pyenv_cfg:34]
293 created virtual environment PyPy3.8.13.final.0-64 in 243ms
  creator PyPy3Posix(dest=/home/astachlewski/tmp/venv_pypy, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/astachlewski/.local/share/virtualenv)
    added seed packages: pip==22.3, setuptools==65.5.0, wheel==0.37.1
  activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator [WARNING __main__:17]
@starhel starhel added the bug label Nov 14, 2022
@gaborbernat
Copy link
Contributor

This is probably related to the changes introduced by @vfazio?

@vfazio
Copy link
Contributor

vfazio commented Nov 15, 2022

oof. i'm not familiar with pypy at all but the driving force of #2441 was due to PEP405 which explicitly states that the "home" key should be the parent of the python binary: https://peps.python.org/pep-0405/#specification

CPython handles this by recursing up the directories for landmark directories/files: https://github.com/python/cpython/blob/main/Modules/getpath.py#L208

Sorry that my PR broke functionality. I'm open to ideas here.

@vfazio
Copy link
Contributor

vfazio commented Nov 15, 2022

Note that the "correct" value for home is important for 3.11+ since the home key now drives path resolution on posix for the system executable (where prior to 3.11 it did not). See the following:

python-poetry/poetry#6940
python/cpython#99204

https://bugs.python.org/issue46028
python/cpython#30144

@starhel
Copy link
Author

starhel commented Nov 15, 2022

I've checked also venv module in pypy3.8 and it's also not working with symlinked binary and --copies file.

> ./bin/pypy3.8 -m venv venv_pypy --copies
Error: Command '['/home/astachlewski/tmp/venv_pypy/bin/pypy3.8', '-Im', 'ensurepip', '--upgrade', '--default-pip']' returned non-zero exit status 127.

Both virtualenv and venv work if binaries are symlinked instead of copied.

As far as I understand all these linked discussions and PEP405, I think that virtualenv is working correctly. My workaround is probably a not supported hack (I've added this bin directory to PATH to provide all pypy and cpython versions without colissions -> adding original bin paths leads to cpython be shadowed by pypy).

@gaborbernat
Copy link
Contributor

Closing per above, if upstream pypy changes behaviour we can revisit.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug help-wanted
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@gaborbernat @vfazio @starhel