Error running locally built image with pack: Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding

[david-caro]

Not sure what I'm doing wrong, it might be a dummy thing, but when trying to run a locally built image (with pack) using the heroku/builder:22, I get the error:

dcaro@vulcanus$ docker run --rm wm-lol
[uWSGI] getting INI configuration from uwsgi.ini
*** Starting uWSGI 2.0.21 (64bit) on [Tue Mar 14 19:03:17 2023] ***
compiled with version: 11.3.0 on 01 January 1980 00:00:01
os: Linux-6.1.0-2-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.7-1 (2023-01-18)
nodename: afb11a88e4cc
machine: x86_64
clock source: unix
pcre jit disabled
detected number of CPU cores: 8
current working directory: /workspace
detected binary path: /layers/heroku_python/dependencies/bin/uwsgi
your memory page size is 4096 bytes
detected max file descriptor number: 1048576
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
uwsgi socket 0 bound to TCP address :8000 fd 3
Python version: 3.11.2 (main, Feb  8 2023, 12:54:20) [GCC 11.3.0]
Could not find platform independent libraries <prefix>
Could not find platform dependent libraries <exec_prefix>
Python path configuration:
  PYTHONHOME = (not set)
  PYTHONPATH = (not set)
  program name = '/layers/heroku_python/dependencies/bin/uwsgi'
  isolated = 0
  environment = 1
  user site = 1
  safe_path = 0
  import site = 1
  is in build tree = 0
  stdlib dir = '/app/.heroku/python/lib/python3.11'
  sys._base_executable = '/layers/heroku_python/dependencies/bin/uwsgi'
  sys.base_prefix = '/app/.heroku/python'
  sys.base_exec_prefix = '/app/.heroku/python'
  sys.platlibdir = 'lib'
  sys.executable = '/layers/heroku_python/dependencies/bin/uwsgi'
  sys.prefix = '/app/.heroku/python'
  sys.exec_prefix = '/app/.heroku/python'
  sys.path = [
    '/app/.heroku/python/lib/python311.zip',
    '/app/.heroku/python/lib/python3.11',
    '/app/.heroku/python/lib/python3.11/lib-dynload',
  ]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'

Current thread 0x00007f6228908440 (most recent call first):
  <no Python frame>

The build process works well:

dcaro@vulcanus$ pack build --builder heroku/builder:22 wm-lol
22: Pulling from heroku/builder
Digest: sha256:eb9486a5587e666f43695ff794a28056b18033fd2f6566c81137634087d23034
Status: Image is up to date for heroku/builder:22
22-cnb: Pulling from heroku/heroku
Digest: sha256:8ed2415c4e53df324c1af95e7286f1065a9cced71ea7e449574064a2c268d55c
Status: Image is up to date for heroku/heroku:22-cnb
latest: Pulling from buildpacksio/lifecycle
Digest: sha256:f75a04887fced3ae0504a37edb2c0d29d366511cd9ede34dbb90c5282b106e79
Status: Image is up to date for buildpacksio/lifecycle:latest
===> ANALYZING
[analyzer] Restoring data for SBOM from previous image
===> DETECTING
[detector] heroku/python   0.1.0
[detector] heroku/procfile 2.0.0
===> RESTORING
[restorer] Restoring metadata for "heroku/python:python" from app image
[restorer] Restoring metadata for "heroku/python:pip-cache" from cache
[restorer] Restoring metadata for "heroku/python:shim" from cache
[restorer] Restoring data for "heroku/python:pip-cache" from cache
[restorer] Restoring data for "heroku/python:python" from cache
[restorer] Restoring data for "heroku/python:shim" from cache
===> BUILDING
[builder]
[builder] [Determining Python version]
[builder] No Python version specified, using the current default of Python 3.11.2.
[builder] To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes
[builder]
[builder] [Installing Python and packaging tools]
[builder] Using cached Python 3.11.2
[builder] Using cached pip 23.0.1, setuptools 67.5.0 and wheel 0.38.4
[builder]
[builder] [Installing dependencies using Pip]
[builder] Using cached pip download/wheel cache
[builder] Running pip install
[builder] Collecting flask
[builder]   Using cached Flask-2.2.3-py3-none-any.whl (101 kB)
[builder] Collecting werkzeug
[builder]   Using cached Werkzeug-2.2.3-py3-none-any.whl (233 kB)
[builder] Collecting uwsgi
[builder]   Using cached uWSGI-2.0.21-cp311-cp311-linux_x86_64.whl
[builder] Collecting Jinja2>=3.0
[builder]   Using cached Jinja2-3.1.2-py3-none-any.whl (133 kB)
[builder] Collecting itsdangerous>=2.0
[builder]   Using cached itsdangerous-2.1.2-py3-none-any.whl (15 kB)
[builder] Collecting click>=8.0
[builder]   Using cached click-8.1.3-py3-none-any.whl (96 kB)
[builder] Collecting MarkupSafe>=2.1.1
[builder]   Using cached MarkupSafe-2.1.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (27 kB)
[builder] Installing collected packages: uwsgi, MarkupSafe, itsdangerous, click, werkzeug, Jinja2, flask
[builder] Successfully installed Jinja2-3.1.2 MarkupSafe-2.1.2 click-8.1.3 flask-2.2.3 itsdangerous-2.1.2 uwsgi-2.0.21 werkzeug-2.2.3
[builder]
[builder] [Discovering process types]
[builder] Procfile declares types -> web
===> EXPORTING
[exporter] Adding layer 'heroku/python:dependencies'
[exporter] Reusing layer 'heroku/python:python'
[exporter] Reusing layer 'buildpacksio/lifecycle:launch.sbom'
[exporter] Adding 1/1 app layer(s)
[exporter] Reusing layer 'buildpacksio/lifecycle:launcher'
[exporter] Adding layer 'buildpacksio/lifecycle:config'
[exporter] Reusing layer 'buildpacksio/lifecycle:process-types'
[exporter] Adding label 'io.buildpacks.lifecycle.metadata'
[exporter] Adding label 'io.buildpacks.build.metadata'
[exporter] Adding label 'io.buildpacks.project.metadata'
[exporter] Setting default process type 'web'
[exporter] Saving wm-lol...
[exporter] *** Images (f36700bd8dd2):
[exporter]       wm-lol
[exporter] Adding cache layer 'heroku/python:pip-cache'
[exporter] Reusing cache layer 'heroku/python:python'
Successfully built image wm-lol

I can run it though if I set both PYTHONHOME and LD_LIBRARY_PATH to point to the python layer dirs:

docker run --rm --env LD_LIBRARY_PATH=/layers/heroku_python/python/lib/ --env PYTHONHOME=/layers/heroku_python/python/ wm-lol

The code I'm running is https://github.com/david-caro/wm-lol (simple python app uwsgi+flask), but tested with the sample python app too with the same error

Note: moved from heroku/heroku-buildpack-python#1427

[edmorley]

@david-caro Thank you for the great bug report + testcase, that helped a lot with my being able to reproduce.

This looks like a potential bug in uWSGI (or at least a limitation of it), since it's not honouring Python's detection of stdlib location, and instead is using the original build time location of Python, before it was relocated. For example, if you run docker run --rm -it --entrypoint launcher wm-lol python -m sysconfig, you will see that the "Paths:" section right at the top has all the correct locations.

I'm planning on reporting this upstream to uWSGI, as well as having the Python CNB set PYTHONHOME as a workaround in the meantime (which may end up needing to be set long term, given there will be apps in the wild using old uWSGI versions that never get the fix, even if uWSGI do fix this).

[edmorley]

I've reported this upstream to the uWSGI project here:
unbit/uwsgi#2525

[ipmb]

This is more of a workaround than a fix, but I'm curious if using https://pypi.org/project/pyuwsgi/ resolves the issue.

[edmorley]

I just noticed the GCP Python CNB sets PYTHONHOME because of uWSGI too:
https://github.com/GoogleCloudPlatform/buildpacks/blob/99553d0a2051834324d621f20ad5355453f675a1/cmd/python/runtime/main.go#L93-L96