pip install of fasttext package is failing

[SaschaSchwarze0]

This simple project, https://github.com/src2img/e2e/tree/python-pip-fasttext, is containing a Python project with just one library in its requirements.txt, fastText.

The sample also contains a Dockerfile that mimics the steps that Paketo imo performs = it uses Python 3.10, pip is updated so that it is at version 23.1.2, it runs as non-root then a pip install command with the same arguments that Paketo uses. The Dockerfile succeeds, Paketo does not.

Expected Behavior

pip install succeeds

Current Behavior

It fails like this:

[builder]   Executing build process
[builder]     Running 'pip install --requirement requirements.txt --exists-action=w --cache-dir=/layers/paketo-buildpacks_pip-install/cache --compile --user --disable-pip-version-check'
[builder]       Collecting fasttext==0.9.2 (from -r requirements.txt (line 1))
[builder]         Downloading fasttext-0.9.2.tar.gz (68 kB)
[builder]            ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 68.8/68.8 kB 409.4 kB/s eta 0:00:00
[builder]         Installing build dependencies: started
[builder]         Installing build dependencies: finished with status 'done'
[builder]         Getting requirements to build wheel: started
[builder]         Getting requirements to build wheel: finished with status 'error'
[builder]         error: subprocess-exited-with-error
[builder]         
[builder]         × Getting requirements to build wheel did not run successfully.
[builder]         │ exit code: 1
[builder]         ╰─> [25 lines of output]
[builder]             /layers/paketo-buildpacks_cpython/cpython/bin/python: No module named pip
[builder]             Traceback (most recent call last):
[builder]               File "<string>", line 38, in __init__
[builder]             ModuleNotFoundError: No module named 'pybind11'
[builder]             
[builder]             During handling of the above exception, another exception occurred:
[builder]             
[builder]             Traceback (most recent call last):
[builder]               File "/layers/paketo-buildpacks_pip/pip/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
[builder]                 main()
[builder]               File "/layers/paketo-buildpacks_pip/pip/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
[builder]                 json_out['return_val'] = hook(**hook_input['kwargs'])
[builder]               File "/layers/paketo-buildpacks_pip/pip/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 118, in get_requires_for_build_wheel
[builder]                 return hook(config_settings)
[builder]               File "/tmp/pip-build-env-d30ersr5/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 341, in get_requires_for_build_wheel
[builder]                 return self._get_build_requires(config_settings, requirements=['wheel'])
[builder]               File "/tmp/pip-build-env-d30ersr5/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 323, in _get_build_requires
[builder]                 self.run_setup()
[builder]               File "/tmp/pip-build-env-d30ersr5/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 487, in run_setup
[builder]                 super(_BuildMetaLegacyBackend,
[builder]               File "/tmp/pip-build-env-d30ersr5/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 338, in run_setup
[builder]                 exec(code, locals())
[builder]               File "<string>", line 72, in <module>
[builder]               File "<string>", line 41, in __init__
[builder]             RuntimeError: pybind11 install failed.
[builder]             [end of output]

Possible Solution

Steps to Reproduce

1. git clone https://github.com/src2img/e2e
2. cd e2e
3. git checkout python-pip-fasttext
4. pack build --builder paketobuildpacks/builder-jammy-full myimage

Motivations

Additional Information

There is a pull request open against fastText that aims to resolve exactly the error from above (facebookresearch/fastText#1331). Though, I am not a Python/Pip expert to assess if it is applicable. So wondering why it works without that fix in the Dockerfile approach.

[robdimsdale]

I can reproduce both sides of this - the failure to build via buildpacks and the successful build via the Dockerfile. I also validated that using 1001 as the User ID in the Dockerfile also results in a successful build.

I'm not sure what the difference is. I assume that the version of pip that comes with the python:3.10 base image is configured differently - specifically that it is able to download pybind11 during build-time, which the Paketo pip is not.

To explore this further, we could see if pre-downloading pybind11 during compilation of pip helps. The change would look similar to this line. I am a loathe to just make this change, because those lines are to support the offline use-case which isn't the issue we're running into here. The only reason I suggest it is in case the mere presence of the pybind11 library is necessary at compile time if it is to be used later (i.e. pip must be compiled against the pybind11 headers).

I tried to look at pip debug --verbose but I didn't see any references to pybind11 in the python:3.10 base image, so I didn't check the paketo-compiled one.

The next thing I'd like to explore is if it's possible to provide build-time/system dependencies in the requirements.txt file, which would be analogous to the PR linked above.

[SaschaSchwarze0]

Thanks @robdimsdale for spending time.

The next thing I'd like to explore is if it's possible to provide build-time/system dependencies in the requirements.txt file, which would be analogous to the PR linked above.

Do you refer here to the requirements.txt of the project that is being built? I remember that I tried things in that area like adding pybind11 to it - without success.