pkg_resources.DistributionNotFound message after installing SIMD 4.3.0.post0

[bjoernhaeuser]

What did you do?

Execute these commands (Dockerfile) from a CentOS 7 base image:

RUN set -x \
  && pip install --upgrade pip \
  && pip install --upgrade setuptools \
  && pip install pexif==0.15 \
  && pip install -e git+https://github.com/thumbor/thumbor.git@master#egg=thumbor \
  && pip install opencv-engine==1.0.1 \
  && pip install thumbor-s3==0.2.0 \
  && pip install tc-prometheus==0.1.1 \
  && pip uninstall -y pillow || true \
  && pip install -U --force-reinstall pillow-simd \
  && python -c "from PIL import Image; print(Image.PILLOW_VERSION)" \
  && pip list

What did you expect to happen?

That pillow-simd is a replacement package for pillow.

What actually happened?

I get this this error message when running thumbor:

Traceback (most recent call last):
  File "/usr/bin/thumbor", line 6, in <module>
    from pkg_resources import load_entry_point
  File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 3142, in <module>
    @_call_aside
  File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 3126, in _call_aside
    f(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 3155, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 664, in _build_master
    ws.require(__requires__)
  File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 981, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 867, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'Pillow<5.0.0,>=3.0.0' distribution was not found and is required by thumbor

What versions of Pillow and Python are you using?

Pillow: 4.3.0.post0
Python: 2.7.5

[kkopachev]

@bjoernhaeuser did you solve it?

I found that changed package name makes it fails. So now it's not drop-in replacement for Pillow anymore. In order to use it, it has to be required as Pillow-simd?

[bjoernhaeuser]

Nope, did not solve it. But eventually stop trying it for running with thumbor.

[kkopachev]

Ah. I did reference previous commit (which had old package name) and it seems to work.

CC="cc -m$SIMD_LEVEL" pip install --no-cache-dir -U --force-reinstall \
  git+git://github.com/uploadcare/pillow-simd@d87f2d67ea62d03432f76e33d428e0b07768c94a#egg=Pillow==4.3.0

[gingerlime]

@kkopachev also bumped into this. It's a bit weird that a drop-in replacement can't be used as a drop-in replacement because of a name change ;-) Or is there a way to use this in thumbor besides using an older version?

[homm]

Drop-in replacement refers to the same API and same functionality. Any installation issues are only installation issues and should not affect functionality once they solved.

[gingerlime]

Could you give any pointers on how to solve the inatallation issue? I’m just using pip, but am getting the same error that was originally posted. The only workaround seems to be to pin the version to the one before the name change...

[homm]

Why ever thumbor checks dependencies on runtime? I suppose this is the root of the
problem.

In one of my project where Pillow is requred by third-party library I also use egg parameter in requirements.txt

[gingerlime]

Why ever thumbor checks dependencies on runtime? I suppose this is the root of the
problem.

I asked @kkopachev because he's a contributor to thumbor :) Admittedly, this issue is in a different repo. I'm not sure it's purely Thumbor's "fault" as such. I think they specify it under install_requires in setup.py, and to be fair, it used to work just fine when Pillow-SIMD was still named Pillow ...

... btw, If I'm reading this correctly, then #egg= is going to be deprecated

[kkopachev]

@homm How do you use #egg parameter?

I created a local dummy package with following setup.py:

from setuptools import setup

setup(
    name='SomePackage',
    version='0.0.1',
    packages=['SomePackage'],

    install_requires=[
        "Pillow",
    ],

    entry_points={
        'console_scripts': [
            'somepackage=SomePackage:main',
        ],
    },
)

and following in SomePackage/__init__.py:

#!/usr/bin/python
import PIL

def main():
    print (PIL.__version__)

if __name__ == '__main__':
    main()

Now, running pip install . installs my package and pillow. Running somepackage outputs 5.1.1 (pillow version).

Readme for pillow-simd requires to remove pillow first, and then install simd version.

pip uninstall -y pillow
pip install git+git://github.com/uploadcare/[email protected]#egg=Pillow==5.1.1 --global-option="build_ext" --global-option="--disable-zlib" --global-option="build_ext" --global-option="--disable-jpeg"

That second command produces warning:

Running setup.py (path:/tmp/pip-build-OmAN68/Pillow/setup.py) egg_info for package Pillow produced metadata for project name pillow-simd. Fix your #egg=Pillow fragments.

which results in freshly installed package to be named Pillow-SIMD:

# pip freeze
Pillow-SIMD==5.1.1.post0
SomePackage==0.0.1
virtualenv==15.1.0

making my SomePackage unusable:

# somepackage 
Traceback (most recent call last):
  File "/usr/local/bin/somepackage", line 6, in <module>
    from pkg_resources import load_entry_point
  File "/usr/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 3147, in <module>
    @_call_aside
  File "/usr/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 3131, in _call_aside
    f(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 3160, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/usr/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 666, in _build_master
    ws.require(__requires__)
  File "/usr/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 984, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/usr/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 870, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'Pillow' distribution was not found and is required by SomePackage

[kkopachev]

I guess there are 2 options from here.

1. Do not try to run console_script defined in setup.py, but instead run program directly:

python SomePackage/__init__.py

Or make a custom entrypoint script.
2. Make fake pillow.egg-info in site-packages, so pkg_resources would think it is installed

[homm]

Now it's a bit clear to me. I haven't ever used entry_points and it looks like it checks dependencies on runtime instead of just running the code.

I checked my config, there is xhtml2pdf I used, and it depends on Pillow. So I have the following in my requirements.txt:

xhtml2pdf
-e git+ssh://[email protected]/uploadcare/pillow-simd.git@simd/4.3.x-prod#egg=pillow

This prevents installation of Pillow and Pillow-SIMD simultaneously.

Likely, xhtml2pdf also have command line utility (which I never used) through entry_points and it doesn't work, as expected:

$ pisa
Traceback (most recent call last):
  File "/home/homm/env/env_ucare/bin/pisa", line 6, in <module>
    from pkg_resources import load_entry_point
...
pkg_resources.DistributionNotFound: The 'Pillow' distribution was not found and is required by xhtml2pdf

Do not try to run console_script defined in setup.py, but instead run program directly

I believe this is the right way, but only with one correction: you can use -m CLI argument for python, which could be much more convenient:

$ python -m xhtml2pdf.pisa
USAGE: pisa [options] SRC [DEST]

[FichteFoll]

Also running into this issue with deluge now (https://dev.deluge-torrent.org/ticket/3282). It seems that pillow-simd just isn't a drop-in replacement for when entry_points are used in setuptools and pillow is in install_requires.

Should this become a feature request in setuptools or can it be solved on pillow-simd's side?