When slim_handler is used, there's no way to get binary libraries into the LD_LIBRARY_PATH

[mbox]

Context

Normally, anything in the /lib directory will end up in /var/task/lib which Lambda includes on LD_LIBRARY_PATH which lets code link to it.
However, when slim_handler is in effect, the code lands up at /tmp/<projname>/lib which isn't on LD_LIBRARY_PATH. You can't modify the path when running as the dynamic linker has already read the value, so changes have no effect.

Expected Behavior

Either there should be a way to say "when building the slim_handler, include these libraries in /lib", or there should be a symlink in the slim_handler package to the "real" location in the project files on /tmp

Actual Behavior

You can't get binary libraries to work

Possible Fix

Steps to Reproduce

1. In the project directory, create a lib subdirectory
2. Add a library to lib
3. Set slim_handler to true
4. Package & deploy
5. Code running will not be able to link to the library in lib
6. Set slim_handler to false
7. Deploy
8. Code can link to the library

Your Environment

• Zappa version used: 0.39.1
• Operating System and Python version: Mac OS X 10.11 , Python 2.7.13
• Link to your project (optional):
• Your zappa_settings.py:

[mcrowson]

I think we'd need to do something like this in the handler: https://docs.python.org/2.7/library/ctypes.html

The problem is knowing what to load.

[Miserlou]

I think the general consensus after discussion in Slack is that certain file-types (.so) should be left in the main package so that they're in the right place for the rest.

[srtonz]

Are there any known workarounds for this at the moment?

[srtonz]

So it looks like the include setting can be used to solve this, by specifying a path relative to the handler's base directory. If you have libraries in lib/ you would use "include": ["lib/mylibrary.so"].

Note that order of imports is very important here. When loading a library X that depends on a library Y, make sure to load Y first. The current version of zappa provides very little feedback on failure -- "Failed to find library...right filename?" is shown in all cases and the actual exception is hidden.

Also this won't help if any of your python modules use ctypes.utils.find_library. In that case your best bet is to also set LD_LIBRARY_PATH and use a python3.6 lambda

I think this issue might have been solved by #789

[glassresistor]

This error isn't very helpful either "Failed to find library...right filename?" I'd rather get the original error or atleast a library or filename.

[cscanlin]

I'm experiencing this issue trying to deploy a lambda function that uses pandas. I can't deploy without slim_handler because the unzipped size is too large, but pandas doesn't work when slim_handler is true.

[zgazak]

Similar issue, I think python Babel put my deployment over the limit. Just getting timeouts and "Failed to find library...right filename?". Any advice on how to proceed or even debug?

[themmes]

@zgazak @cscanlin Have you found a solution to this problem or method for debugging since your comment here?

[chrisbrantley]

I am having this same issue. Need to use slim_handler due to file size but shared libraries (.so files) are not being found.

[chelicerae]

It seems I'm having the same problem while deploying chromedriver and headless-chromium, they just crash and have this error "Failed to find library...right filename?" before the crash. Has anyone figured out how to fix this?

[ckyleda]

I am also having this issue.

Surely this is an incredible oversight in the design of slim_handler?

Is there any way around this at all?

[zgazak]

I wasn't able to solve this, @themmes

[ckyleda]

Where do we go from here?

This is clearly a long-standing issue and essentially removes all the usefulness of slim_handler.

@geotob Have you had any more success with this?

[srtonz]

@ckyleda it's been too long since I last looked at it.

From memory using the include setting worked in most cases (see my earlier comment).

I think we also patched the library to return more useful debugging output than Failed to find library...right filename?. That code is here

Zappa/zappa/handler.py

Line 117 in 4cf80b2

print ("Failed to find library...right filename?")

and you can quite easily overwrite it locally to tell you exactly what library it's looking for.

[jtszalay]

print ("Failed to find library {} ...right filename?").format(library)

I set mine to the above locally. Thanks for the tip!

[elcolie]

Any workaround on this?

[SmileSydney]

@geotob Bit new to zappa - question about the include setting.

I'm using slim_handler and getting libpq.so.5: cannot open shared object file: No such file or directory: ImportError. I believe libpq.so.5 is a shared library installed with postgres client.

If my virtualenv is at ~/.virtualenvs//

Where do I need to put the libpq.so.5?
What path do I put in the include setting?

[aralmpa]

Hello guys,

My apollogies if I am overlooking something obvious but I had a similar issue with shared libraries and slim_handler and I would like to share the solution that worked for me - maybe it 'll help you.

AWS Lambda service offers a feature called Layers. Basically it seems to be an option to use when large binary files (like shared libs) need to be available at runtime in the lambda function. A fork of Zappa by @jedestep supports "attaching" layers to your lambda setting a related option in your zappa_settings.json.

I am using Geodjano which requires quite a few geospatial libraries (.so files) to be available to django and uploading them in a layer which made them available through the /opt/lib/ path solved my problem. If you are interested in more info check the related issue #985.

I did not make much testing with slim_handler set to true but a deployment I made few minutes ago seemed to find all libraries. Again sorry if I am missing something and this approach is not suitable for this issue.

[jbonifield3]

I ran into a related issue setting up python3-saml on with the dependent python-xmlsec import failing: ImportError: libxmlsec1.so.: cannot open shared object file: No such file or directory. (I also had slim_handler set to true.)

In order to get lambda to load the dependencies I first set the LD_LIBRARY_PATH to lib:$LD_LIBRARY_PATH as suggested by this set of tips (which as a newbie to lambda I found helpful). Then set the include parameter like @geotob suggests in your zappa_settings file to path of the dependency (e.g. "include": ["/var/task/lib"]), and drop the dependencies in a lib folder at the root from which you are calling zappa.

[deanq]

I just wanted to add something that helped me solve my issues with binary libraries not getting referenced or missing during packaging.

• This guarantees the required prerequisites in the build system before running pip installs:
  apt-get update && apt-get install -y build-essential <insert required libs here> && ldconfig -v
• Setting environment variable STATIC_DEPS=true changes how your pip install links to the binary libraries. Every package builds and retains the required binary libraries locally. This is ideal for zappa packaging and lambda deployment. For example: STATIC_DEPS=true pip install -r requirements.txt
• If you have issues with lxml, stop wasting your time trying to figure out how to install it. The problem is with zappa's lambda-packages portion of the packaging process. It overwrites some libraries that thinks it has a better version of. In this case, it tries to install a broken lxml package. I wasted days before figuring this out. Solution: inside zappa_settings.json, use use_precompiled_packages=false

I hope this helps someone out there. My problem stemmed out of python3-saml where it kept breaking looking for missing *.so files, and then eventually got stuck at a missing lxml.etree module. Also pay attention to the Python runtime version. 3.6, 3.7, and 3.8 all had varying errors that ended up at that lxml issue.