[BUG] No module named 'pkg_resources.extern'

[LuisHenri]

setuptools version

setuptools==70.0.0

Python version

3.8.10

OS

Windows 11

Additional environment information

No response

Description

I use PyInstaller and PyArmor to obfuscate my code and create an Executable out of it.
On a pipeline of mine, an error started to throw today:

Traceback (most recent call last):
  File "Lib\site-packages\PyInstaller\hooks\rthooks\pyi_rth_pkgres.py", line 16, in <module>
  File "PyInstaller\loader\pyimod03_importers.py", line 495, in exec_module
  File "pkg_resources\__init__.py", line 77, in <module>
ModuleNotFoundError: No module named 'pkg_resources.extern'

Inside this Pipeline I run python -m pip install -U setuptools. I.e., it uses the latest version of it.

I tested it locally and it happens with version 70.0.0 but not with version 69.5.1

Expected behavior

Obfuscated Executable from PyArmor and PyInstaller is created.

How to Reproduce

1. Create a virtual environment
2. Install pyinstaller[encryption]==4.* and pyarmor==7.*
3. pip install tendo==0.2.* (just for the matter of having some import on the main.py file) (for some reason, it works with tendo>=0.3.0)
4. Create a test.py

   import tendo.singleton
   
   def main():
       print("Hello World!")
   
   if __name__ == "__main__":
       me = tendo.singleton.SingleInstance()
       main()

5. Run pyarmor pack .\test.py -x "--exclude .venv"
6. Run the test.exe on dist/test/

Output

python -m pip install -U setuptools
pip install -U pyarmor==7.* pyinstaller[encryption]==4.*
pyarmor pack main.py

Traceback (most recent call last):
  File "Lib\site-packages\PyInstaller\hooks\rthooks\pyi_rth_pkgres.py", line 158, in <module>
  File "Lib\site-packages\PyInstaller\hooks\rthooks\pyi_rth_pkgres.py", line 36, in _pyi_rthook
  File "PyInstaller\loader\pyimod02_importers.py", line 419, in exec_module
  File "pkg_resources\__init__.py", line 77, in <module>
ModuleNotFoundError: No module named 'pkg_resources.extern'

[abravalheri]

Hi @LuisHenri, in the Wheel submitted to the PyPI we can see that the pkg_resources/extern/__init__.py file exists: https://inspector.pypi.io/project/setuptools/70.0.0/packages/de/88/70c5767a0e43eb4451c2200f07d042a4bcd7639276003a9c54a68cfcc1f8/setuptools-70.0.0-py3-none-any.whl/pkg_resources/extern/__init__.py. It is a bit weird that we are getting the error ModuleNotFoundError: No module named 'pkg_resources.extern' (considering that the file is there).

So I am not sure why this error is happening. Would it be possible to create a reproducer without packing the main? I am not sure how PyArmor/PyInstaller works internally and if that is somehow related to this problem...

[altendky]

I have the same error this morning at https://github.com/Chia-Network/chia-blockchain/actions/runs/9175546125/job/25229722015?pr=16898. PyInstaller has to pick what files to include in it's output executable and I'm guessing it needs adapted to a change in setuptools v70. But, I'm just starting to look now.

[abravalheri]

@LuisHenri, @altendky, when I try to run the reproducer in a controlled environment, I don't see the error reported:

> docker run --rm -it python:3.8-windowsservercore-1809 powershell

mkdir $env:TEMP\myproj
cd $env:TEMP\myproj
new-item main.py

@"
print('hello world')
"@ | add-content -Path main.py

python -m pip install -U pip
python -m pip install -U setuptools==70.0.0
python -m pip install -U pyarmor==7.* pyinstaller[encryption]==4.*

python -m pip list
# Package                   Version
# ------------------------- --------
# altgraph                  0.17.4
# importlib_metadata        7.1.0
# packaging                 24.0
# pefile                    2023.2.7
# pip                       24.0
# pyarmor                   7.7.4
# pyinstaller               4.10
# pyinstaller-hooks-contrib 2024.6
# pywin32-ctypes            0.2.2
# setuptools                70.0.0
# tinyaes                   1.1.0
# wheel                     0.36.2
# zipp                      3.18.2

pyarmor pack main.py
# ...
# INFO     PyArmor Trial Version 7.7.4
# ...
# 226 INFO: PyInstaller: 4.10
# 226 INFO: Python: 3.8.10
# 241 INFO: Platform: Windows-10-10.0.17763-SP0
# ...
# INFO     Final output path: dist
# INFO     Pack obfuscated scripts successfully.

.\dist\main\main.exe
# hello world

(I don't have access to a machine with Windows 11 to try the reproducer).

[altendky]

I'm using PyInstaller 6.6.0, fwiw, and no encryption etc. I think I'm also only having an issue on Windows (which you tested). I'll let you know when I figure anything useful out.

[abravalheri]

I'm using PyInstaller 6.6.0, fwiw, and no encryption etc.

I have also run the given reproducer on PyInstaller 6.6.0 with no encryption:

> docker run --rm -it python:3.8-windowsservercore-1809 powershell

mkdir $env:TEMP\myproj
cd $env:TEMP\myproj
new-item main.py

@"
import math
print(f'hello world: {math.pi=}')
"@ | add-content -Path main.py

python -m pip install -U pip
python -m pip install -U setuptools==70.0.0
python -m pip install -U pyarmor==7.* pyinstaller==6.6.0

python -m pip list
# Package                   Version
# ------------------------- --------
# altgraph                  0.17.4
# importlib_metadata        7.1.0
# packaging                 24.0
# pefile                    2023.2.7
# pip                       24.0
# pyarmor                   7.7.4
# pyinstaller               6.6.0
# pyinstaller-hooks-contrib 2024.6
# pywin32-ctypes            0.2.2
# setuptools                70.0.0
# wheel                     0.36.2
# zipp                      3.18.2

pyarmor pack main.py
# ...
# INFO     PyArmor Trial Version 7.7.4
# INFO     Python 3.8.10
# INFO     Target platforms: Native
# ...
# 273 INFO: PyInstaller: 6.6.0, contrib hooks: 2024.6
# 273 INFO: Python: 3.8.10
# 273 INFO: Platform: Windows-10-10.0.17763-SP0
# ...
# INFO     Final output path: dist
# INFO     Pack obfuscated scripts successfully.

.\dist\main\main.exe
# hello world

The program seems to work without the exception...

Maybe the description of the reproducer need a bit more detail?

[LuisHenri]

@abravalheri
So, I tested somethings one more time here and adding pkg_resources.extern to PyInstaller's Hidden Imports fixes the issue.

As @altendky said, it might actually be something that PyInstaller needs to fix upon the changes on SetupTools. Although I wonder what actually changed on SetupTools that made it break on PyInstaller side

[altendky]

@abravalheri, I notice you are running Windows in docker. Might affect the outcome. I'm in Linux and haven't run that same example yet, just been trying to explore my own case here. Waiting on CI. *twiddles thumbs*

And...

https://github.com/Chia-Network/chia-blockchain/actions/runs/9178294773/job/25237858145?pr=18049 shows a pass with setuptools 804ccd2 while https://github.com/Chia-Network/chia-blockchain/actions/runs/9178307533/job/25237903410?pr=18050 shows a failure with setuptools e999582. Just to narrow down the change to the latter commit.

[abravalheri]

Although I wonder what actually changed on SetupTools that made it break on PyInstaller side

Difficult to guess...

There were some changes in the imports in pkg_resources: https://github.com/pypa/setuptools/compare/v69.5.1..v70.0.0.

[abravalheri]

@abravalheri, I notice you are running Windows in docker. Might affect the outcome. I'm in Linux and haven't run that same example yet, just been trying to explore my own case here. Waiting on CI. *twiddles thumbs*

And...

https://github.com/Chia-Network/chia-blockchain/actions/runs/9178294773/job/25237858145?pr=18049 shows a pass with setuptools 804ccd2 while https://github.com/Chia-Network/chia-blockchain/actions/runs/9178307533/job/25237903410?pr=18050 shows a failure with setuptools e999582. Just to narrow down the change to the latter commit.

So in that commit there was a change in the way the imports are done:

- from pkg_resources.extern import platformdirs
- from pkg_resources.extern import packaging
- 
- __import__('pkg_resources.extern.packaging.version')
- __import__('pkg_resources.extern.packaging.specifiers')
- __import__('pkg_resources.extern.packaging.requirements')
- __import__('pkg_resources.extern.packaging.markers')
- __import__('pkg_resources.extern.packaging.utils')
+ from pkg_resources.extern.packaging import markers as _packaging_markers
+ from pkg_resources.extern.packaging import requirements as _packaging_requirements
+ from pkg_resources.extern.packaging import utils as _packaging_utils
+ from pkg_resources.extern.packaging import version as _packaging_version
+ from pkg_resources.extern.platformdirs import user_cache_dir

But that is not something that changes the way things work in runtime... Maybe the heuristics PyInstaller uses are too specific?

[altendky]

PyInstaller having to keep up with changes "like this" is indeed common. FWIW, I went and looked for an issue report against PyInstaller before I cam over here. :]

[altendky]

I'll file an issue over there, if nobody beats me to it.

[LuisHenri]

@altendky I was just going to open an Issue there. But you can head it :P
I was just looking at a hook that PyInstaller has specifically for pkg_resources already inside it: https://github.com/pyinstaller/pyinstaller/blob/develop/PyInstaller/hooks/hook-pkg_resources.py

Probably they'll need to update it.

PyInstaller having to keep up with changes "like this" is indeed common

Crazy @_@
I wonder if they'll just fix the most recent version (v6) or also some older ones (I use v4, f. ex.)

[bwoodsend]

Using __import__ instead of import is enough to break PyInstaller's dependency scanning. It can't see dynamic imports (even on constant strings).

[altendky]

The interesting thing about that is that e999582 went the other direction. But that commit also went and switched to as _* imports. Anyways, this is likely trivial for Hartmut to fix once they look at the commit.

[abravalheri]

Thank you very much for having a deep look at this and contacting PyInstaller.

I suppose that this is going to be handled in pyinstaller/pyinstaller#8554, with no further action in setuptools.

I will leave it open for a while so users can easily find it.

[bwoodsend]

Anyways, this is likely trivial for Hartmut to fix once they look at the commit.

Harmut's tolerance ran out a long time ago.

We'll fix this on our end. Feel free to close this off as PyInstaller's problem.

[bwoodsend]

Should be fixed now. pip install -U pyinstaller then use the --clean flag next time you run pyinstaller.

[altendky]

Works for me. Thanks again here too. :]

[mnmt]

Should be fixed now. pip install -U pyinstaller then use the --clean flag next time you run pyinstaller.

For clarity for future readers: the changes introduced in v70.0.0 of setuptools have been accounted for in v6.7.0 of pyinstaller onwards (released May 21st 2024).

https://github.com/pyinstaller/pyinstaller/releases/tag/v6.7.0
https://pyinstaller.org/en/v6.7.0/CHANGES.html#hooks
pyinstaller/pyinstaller#8555