DistributionNotFound Errors

[irritant]

First of all, great work on Zappa! This is a really interesting tool.

I deployed a small Flask app over the weekend that went very smoothly. Not I'm trying to deploy something a little more sophisticated that makes use of the gcloud lib (via Pyrebase). I'm getting the following trace, which indicates that the gcloud distribution isn't available:

[1473707873959] The 'gcloud' distribution was not found and is required by the application: DistributionNotFound
Traceback (most recent call last):
  File "/var/task/handler.py", line 460, in lambda_handler
    return LambdaHandler.lambda_handler(event, context)
  File "/var/task/handler.py", line 182, in lambda_handler
    handler = cls()
  File "/var/task/handler.py", line 109, in __init__
    self.app_module = importlib.import_module(self.settings.APP_MODULE)
  File "/usr/lib64/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)
  File "/var/task/wfh/__init__.py", line 3, in <module>
    import pyre base
  File "/private/var/folders/qm/1ytmdwmj73jg86ggnqxljf2c0000gp/T/pip-build-9viq6Z/Pyrebase/pyrebase/__init__.py", line 1, in <module>
  File "/private/var/folders/qm/1ytmdwmj73jg86ggnqxljf2c0000gp/T/pip-build-9viq6Z/Pyrebase/pyrebase/pyrebase.py", line 16, in <module>
  File "/private/var/folders/qm/1ytmdwmj73jg86ggnqxljf2c0000gp/T/pip-build-9viq6Z/gcloud/gcloud/__init__.py", line 19, in <module>
  File "/var/task/pkg_resources/__init__.py", line 557, in get_distribution
    dist = get_provider(dict)
  File "/var/task/pkg_resources/__init__.py", line 431, in get_provider
    return working_set.find(moduleOrReq) or require(str(moduleOrReq))[0]
  File "/var/task/pkg_resources/__init__.py", line 968, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/var/task/pkg_resources/__init__.py", line 854, in resolve
    raise DistributionNotFound(req, requirers)
DistributionNotFound: The 'gcloud' distribution was not found and is required by the application

All dependencies are installed in my virtualenv and listed in my requirements.txt. The app runs locally without any issues. Is anyone aware of a way to fix this or am I just trying to do something that Lambda and/or Zappa isn't quite ready for? Thanks for any insight you can provide.

[Miserlou]

I haven't used the 'gcloud' distribution before, but I suspect it's related to this, which I thought was fixed:

#45

Barring that, it could be related to this:

#319

You may have to dig a little bit into your virtualenv to help us crack this nut!

Thanks for reporting!

[irritant]

Thanks for such a quick reply. I'll look into these and report back.

[irritant]

I really don't know enough about how Python loads modules to be certain, but I think #319 may be the more likely culprit. The error seems to originate on line 19 of init.py in the gcloud module:

__version__ = get_distribution('gcloud').version

I tried replacing this line with __version__ = '0.18.1' (the actual version string for this distribution) but to no avail. Similar code appears throughout the gcloud module. Most of the instances are in files at the root level of the module, but it also appears in at least one nested module (bigtable).

[Miserlou]

How did you install gcloud? Through pip, or from source?

[irritant]

From pip as a dependency of Pyrebase. I also tried pip install --upgrade gcloud while looking for a solution, but it looks like Pyrebase already specifies the latest release.

[Miserlou]

You could try setting delete_local_zip to false in your zappa_settings, then opening up the zip file and seeing if it copied the gcloud package into the zip properly.

[irritant]

Thanks for that tip. It appears to have copied gcloud properly, but it doesn't copy the distribution folders that are in my virtualenv/lib/python2.7/site-packages dir (for any module, not just gcloud). Would that break get_distribution('gcloud')? I don't see similar code in any of the other modules.

[Miserlou]

Hm, that's a really interesting theory. I haven't used get_distribution, but that seems like a good theory to me. If you copy the dir into your extracted zip and try to run that command from that location, does it pass?

[Miserlou]

.dist-info is explicitly excluded from packages to trim down the zip weight, but I think it looks like we have to include it for this package to work. Acceptable solution?

[irritant]

I'm not sure. Another import error occurs when I try to run locally from the Zappa archive:

Traceback (most recent call last):
  File "run.py", line 1, in <module>
    from wfh import app
  File "/Users/<user>/Documents/Projects/project/wfh/__init__.py", line 3, in <module>
    import pyre base
  File "/Users/<user>/Documents/Projects/project/venv/lib/python2.7/site-packages/pyrebase/__init__.py", line 1, in <module>
    from .pyrebase import initialize_app
  File "/private/var/folders/qm/1ytmdwmj73jg86ggnqxljf2c0000gp/T/pip-build-9viq6Z/Pyrebase/pyrebase/pyrebase.py", line 21, in <module>
  File "/Users/<user>/Desktop/project-dev-1473778404/Crypto/PublicKey/RSA.py", line 78, in <module>
    from Crypto import Random
  File "/Users/<user>/Desktop/project-dev-1473778404/Crypto/Random/__init__.py", line 28, in <module>
    from Crypto.Random import OSRNG
ImportError: cannot import name OSRNG

The OSRNG module was copied to the archive. This may or may not be related to our current issue.

Is there a way I can test .dist-info locally?

[Miserlou]

Is pycrypto installed in your package? Actually, that actually might be because the crypto library has been replaced with the linux version during packaging (which it's supposed to).

Here, I'm about to publish a new version of Zappa to fix an unrelated bug anyway, but I'll take out the dist-info skip, so maybe you can try upgrading Zappa version to 0.23.1 and then see if that works?

[irritant]

I would happily try that out!

[Miserlou]

Okay, I'll publish once the build passes. Should be about 5 minutes.

[Miserlou]

Feel free to join the Slack as well, if you want to talk about this more directly: https://slack.zappa.io/

[Miserlou]

Published.

[irritant]

That seems to have fixed the issue! Thanks so much for your time.

[Miserlou]

Yesssssssssss 💃

Thanks for reporting and for using Zappa!