ModuleNotFoundError: No module named 'encodings' when the virtualenv path is different from the project home

[w4x-gh]

Environment:

• OS: debian bullseye
• uwsgi version: 2.0.19.1-debian
• python version: 3.9

I'm getting a ModuleNotFoundError: No module named 'encodings' error when the virtualenv path is different from the project home eg.
virtualenv: /home/venvs/py39
project home: /opt/local/apps/myproject

However, the error does not appear when the project home is in the virtual env ie.
virtualenv: /home/venvs/py39
project home: /home/venvs/py39/apps/myproject

The failing configuration:

[uwsgi]
project-home = /opt/local/apps/MyProject
plugins-dir = /usr/lib/uwsgi/plugins
plugin = python39
pythonpath = %(project-home)
virtualenv = /home/venvs/py39 
master = 1 
chdir = %(project-home)
socket = /var/run/uwsgi/%n.sock
chmod-socket = 666
manage-script-name = True
python-path = %(project-home) 
module = wsgi
callable = app
uid = www-data
gid = www-data
processes = 8
log-date = true

The error message:

Python version: 3.9.2 (default, Feb 28 2021, 17:03:44)  [GCC 10.2.1 20210110]
!!! Python Home is not a directory: /home/venvs/py39 !!!
Set PythonHome to /home/venvs/py39
Python path configuration:
  PYTHONHOME = '/home/venvs/py39'
  PYTHONPATH = (not set)
  program name = '/home/venvs/py39/bin/python'
  isolated = 0
  environment = 1
  user site = 1
  import site = 1
  sys._base_executable = '/home/venvs/py39/bin/python'
  sys.base_prefix = '/home/venvs/py39'
  sys.base_exec_prefix = '/home/venvs/py39'
  sys.platlibdir = 'lib'
  sys.executable = '/home/venvs/py39/bin/python'
  sys.prefix = '/home/venvs/py39'
  sys.exec_prefix = '/home/venvs/py39'
  sys.path = [
    '/home/venvs/py39/lib/python39.zip',
    '/home/venvs/py39/lib/python3.9',
    '/home/venvs/py39/lib/python3.9/lib-dynload',
  ]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'

Working configuration ie. when project home is a directory inside the venv

[uwsgi]
project-home = /home/venvs/py39/apps/MyProject
plugins-dir = /usr/lib/uwsgi/plugins
plugin = python39
pythonpath = %(project-home)
virtualenv = /home/venvs/py39 
master = 1 
chdir = %(project-home)
socket = /var/run/uwsgi/%n.sock
chmod-socket = 666
manage-script-name = True
python-path = %(project-home) 
module = wsgi
callable = app
uid = www-data
gid = www-data
processes = 8
log-date = true

The above conf is successful ans this message is part of the logs:

*** Operational MODE: preforking ***
added /home/venvs/py39/apps/MyProject to pythonpath.
*** uWSGI is running in multiple interpreter mode ***

When the error appears, the project path is not added to the python path ie. the setup process doesn't get to this point or fails at this point or something else.

[jsmera]

I got the same issue, I had to rollback to Uwsgi 2.0.18

[bernd-wechner]

I have the same issue with:

Python 3.10.12
uwsgi 2.0.20-debian
Distro: Ubuntu 22.04.3 LTS (Jammy Jellyfish)

I have no workaround currently.

Exactly the same crash. Only on one of a few uwsgi sites I am migrating to this server.

[chrisjsimpson]

ModuleNotFoundError: No module named 'encodings'

If python can't find the encodings module, which is included in the standard library, that's strong indicator that something is wrong with where python is configured to look for module files (PYTHONPATH - "the default search path for module files")

One way to overcome this, if you're situation warrants it, includes the following.

• Checking / setting your PYTHONPATH to point to the location of where the default python module files are stored.

Or, a more over the top approach:

• I compiled python 3.12 using the official instructions

• Cloned the official uwsgi repo

• Built uwsgi using official docs e.g. <checkout-location>/cpython/python uwsgiconfig.py --build

  • Note you must of course use the recently compiled python (which I cloned to cpython) as a consequence of following official python.org instructions
  • Note uwsgiconfig.py will fail to find libpython3.12d.a (or whichever python version you build) because uwsgiplugin-py assumes it will be located in /usr/local/lib giving you a file not found. To overcome this you can't use UWSGI_INCLUDES as per the uwsgi docs because the way ldline is built does not appear to take UWSGI_INCLUDES into account at all (perhaps internally?). I overcame that by putting a breakpoint there during the build and 'fixing' the path in the ldline string. If I had more time perhaps there could/should be a patch so that the location of /usr/local/lib can be changed* (or perhaps there already is- during the build maybe you can/should set PYTHONPATH- no. Possibly due to how get_gcll works, not sure).

  *The location at which the libs dir gets assumed to be /usr/local/lib/ is somewhere before

  uwsgi/uwsgiconfig.py

  Line 552 in f9e4f0e

  libs += up['LIBS']

The current master commit ( d95638b ) has the same behaviour (assumes /usr/local/lib/ )

Ref #2525