Add support to load library externally

[cmendesce]

As pyzbar is a very usable library to handle images, some applications wants to handle images in a very scalable context like serverless platforms (aws lambda, azure function and so on). These platforms run workloads on top of default environments that is not easy to customize to add a shared lib as pyzbar requires.

So, my suggestions is to add a new try during the load library function in pyzbar/zbar_library.py. The code below is showing only trying get the library in PATH, but a good try is find the library in a specific env var like ZBAR_PATH.

else:
        # Assume a shared library on the path
        path = find_library('zbar')
        if not path:
            path = os.environ.get('ZBAR_PATH')
            if not path: 
                raise ImportError('Unable to find zbar shared library')
        libzbar = cdll.LoadLibrary(path)
        dependencies = []

[ArvindSridhar]

Having the same issue with my app deployed on Heroku. Any word on the status of the PR?

[3Mcolab]

I have the same issues with AWS ec2. Don't know how to assign a path

[quicklizard99]

Apologies for this problem (and for my rather late response).

Thanks very much for the PR. I have merged to feature/40-path-to-zbar, where I will add unit tests and make other changes. I have no experience of AWS nor or Heroku.

Where does libzbar.so get installed to on these platforms?

The behaviour of find_library (the Python standard library function used to find libzbar.so) changed in Python 3.6 - "...the value of the environment variable LD_LIBRARY_PATH is used when searching for libraries, if a library cannot be found by any other means". Do you see the behaviour when using Python 3.6 or 3.7?

[cmendesce]

Hi,

In case of AWS ec2, the library is installed by instance owner, in other hand in heroku and others managed application server, the library get installed in a given docker image or can be load from disk, thus in second case you can embed the lib in application code and load it at run time.

The PR I had pushed, add the option to load the library from any path to handle cases where the installation is not a responsibility of the developer.

[quicklizard99]

OK thanks. It would be good to use the standard mechanisms for finding the library unless there really is no other option.

Do these platforms really not make the shared lib location available on PATH or LD_LIBRARY_PATH? If not, can you not append the correct location to PATH or LD_LIBRARY_PATH, as part of a setup or provisioning step?

[cplankey]

Does anyone have an example of how they got this to work with a Lambda function in AWS? I am installing pyzbar in the packaged deploy of my lambda but still getting issues about the zbar shared library.

[novomotus]

Can't speak for AWS lambda, but deploying to an Amazon Linus 2.8:

Adding the zbar package via an .ebextensions declaration works:

packages:
  yum:
    zbar: []

This is the equivalent of yum install zbar

Deployment was successful after this and desired functionality via pyzbar was working.

[kuznetsov-m]

If anyone still has problems using the library to work with heroku, take a look at this. I described my example setup heroku app.

[sebastiandev]

I'm gonna leave this for posterity. In alpine it seems the libzbar gets installed as usr/lib/linzbar.so.0 so is not found by the python module. We had to create a link with the name /usr/lib/libzbar.so.

Here's the complete line for docker alpine

RUN apk --no-cache add libzbar --repository=http://dl-cdn.alpinelinux.org/alpine/edge/community && \
    ln -s /usr/lib/libzbar.so.0 /usr/lib/libzbar.so

[vincentcox]

@sebastiandev hey there! Looks very interesting. Any idea how to make an AWS lambda layer? The one you use is based on alpine, so it's a bit different for AWS.

I did some research and tried the following:

Creating a file Docerfile:

FROM amazonlinux

RUN yum update -y
RUN yum install -y gcc gcc-c++ make
RUN yum install -y wget
RUN yum install -y tar
RUN yum install -y bzip2
RUN mkdir /lambda-DecodeQR
RUN wget https://jaist.dl.sourceforge.net/project/zbar/zbar/0.10/zbar-0.10.tar.bz2
RUN tar -jxvf zbar-0.10.tar.bz2

RUN cd zbar-0.10 && ./configure --build=x86_64 --prefix=/lambda-DecodeQR CPPFLAGS=-I/usr/include --with-libiconv-prefix=/usr/include --with-python=no --with-libiconv-prefix=/usr/include --without-imagemagick$
RUN ln -s /usr/lib/libzbar.so.0 /usr/lib/libzbar.so
RUN yum install -y zip
RUN yum -y clean all
RUN zip -gr DecodeQR.zip share lib64 lib include bin

Then running the following commands:

docker build -t amzlinuxpy .
docker run -it amzlinuxpy bash # let this run in another tab
docker ps # change the XXX_ID_XXX below and match it with the container ID you see with this command
docker cp XXX_ID_XXX:/DecodeQR.zip DecodeQR.zip

Not proud on the Dockerfile, but i's something.

This zip contains the libraries of zbar, that should work on AWS Lambda inside a Lambda layer. I have no clue what to with this now, because I just learned about AWS Lambda today and I don't understand docker layers completely. I think https://hackernoon.com/how-to-deploy-aws-lambda-with-docker-containers-e51j3141 might be an interesting article to look at for continuing this journey.

Any feedback is welcome, if this is not relevant anymore for you, feel free to yeet it with a 🚀 emoji I guess :)

Might also be interesting for @novomotus, @cmendesce, @3Mcolab because they reference AWS too.

[nickovs]

[EDITED]

In part to answer the question from @vincentcox, but also to re-open this thread, getting pyzbar under AWS Lambda to load a library provided by the user is extremely hard, specifically because this proposed patch did not get used.

The problem is that the way that ctypes.utils.find_library() implements searching LD_LIBRARY_PATH relies on calling the ld program, and that is not present on AWS Lambda Python instances. As a result pyzbar can not find libraries using LD_LIBRARY_PATH when running in a modern Lambda instance. Having support for explicitly setting the location of libzbar.so would therefore still be very helpful.

On older versions of Lambda there was an ugly, sub-optimal work-around to this problem that @vincentcox is facing involving linking libzbar.so into some place that ldconfig could find it and then rebuilding the library cache. This previously worked because the implementation of find_library() tries printing our the ldconfig cache as one of its ways to try to find a matching library. Unfortunately modern Lambda instances mount /etc in a read-only location, so the library cache can not be updated. As a result, without being about to provide an explicit path for the zbar library it does not currently seem possible to use pyzbar in a Lambda instance without building a complete resorting to deploying a complete Docker image, which is complex, time-consuming and causes a major hit on the function start-up time.

All in all it would be hugely better if we could just explicitly provide the file path and avoid having to build a complete Docker image just to deploy a Python function that uses pyzbar.

As a final note, I believe that @vincentcox 's script above for building the zbar library may not work as required because the current version on pyzbar seems not to work with the v0.10 zbar source. I initially tried pretty much the same method as proposed above and the resulting library wouldn't find QR codes in my images. I have created a gist for building an up-to-date libzbar.so from the more recent (and still supported) fork of zbar on GitHub which should help.

[felixb-dev]

Thank you @nickovs for providing the script for building the Zbar library for AWS Lambda. I followed your gist and generated three libraries libzbar.so (0kB) , libzbar.so.0 (0kB) and libzbar.so.0.3.0 (1002kB). Is the file size correct? How do I continue from here? Do I have to package a .zip together or do I have to use a docker image to upload to AWS Lambda? Sorry - i am fairly new to building my own layers for AWS Lambda and I need to use pyzbar for QR Code detection. Thank you for your help.

[nickovs]

@felixb-dev Yes, those sizes are what I get. Note that the first two are just soft links; the third file is the real library.

As for how to use the library, right now you need to apply the patch in #41 to your own copy of pyzbar and include the patched version in the code you deploy in Lambda. Hopefully the patch will get merged onto the master branch and released to PyPI at some point soon, but we need some contributor like @quicklizard99 to do that.

[MarGul]

@nickovs Thanks a lot for all your research around this. I would also love to have this PR merged into master. Would help a lot. Or have something like pdf2image package has where you can set the poppler_path in the function.

I'm still pretty fresh on both python and lambda. I have run the script from your gist and added the .so files as a layer. I have forked the repo and added your #41 patch and pulled that repo into my function and set the ZBAR_PATH env variable to the correct in my layer. How can I now import this package? The file structure is:

app.py
--- pyzbar
          [git files and other from the pyzbar repo]
          --- pyzbar
                    --- pyzbar.py

I have tried in my app.py with from pyzbar.pyzbar import decode but get an error that it could not find the module.

Thanks a lot and really hope that @quicklizard99 or some other contributor look at this again. Would be very helpful to be able to set the zbar path somehow.

[nickovs]

@MarGul What I did was copied the inner pyzbar directory (i.e. the one that directly contains the Python files) into the base directory that I was packaging up for Lambda. I copied the libzbar.so.0.3.0 file into the same base directory and renamed it libzbar.so, so the files I package up look like this:

lambda_bundle
  - myscript.py
  - libzbar.so
  - pyzbar
    - __init__.py
    - ...

If you do this then you should be able to just use from pyzbar import decode from your Lambda script.

[MarGul]

Thanks a lot @nickovs . I had to update the pyzbar/__init__.py to include this code

from .pyzbar import (
    decode
)

All is working now. I have the libzbar.so as a Layer instead. Thanks a lot for your help. Hope this issue get's resolved soon.

[DanyStinson]

@MarGul do you have a repo on how to make the the library work with layers? I only manage to get it working if I uploaded it with the lambda function code, but can't manage to get it working as a Layer 😢

[andrologixx]

I've tried following everything from above and I'm still not getting the zbar library found.

Was a layer's path ever worked out? If not, a bundled .zip file?

I've tried both. I've tried patching the zbar_library.py based on #41 and I've compiled and put the libzbar.so in a bundle, in a layer, etc......

I'm working perfectly in windows or ubuntu.

But I cant get past here on aws lambda.

import json
from pdf2image import convert_from_path
from pyzbar import decode

[DanyStinson]

@andrologixx I managed to take a bit of everyone's help on this issue, and created the following repository to create your own Lambda Function working with pyzbar.
I still have not managed to include the shared libraries as a layer, but included them in the bundle .zip and it seems to work that way, if someone manages to make it work as part of a layer happy to modify it!

[MarGul]

I'm using AWS SAM to create my function and resources. This is in my template.yaml

ZbarBinary:
    Type: AWS::Serverless::LayerVersion
    Properties:
      LayerName: zbar
      Description: The zbar package to be able to use pyzbar
      ContentUri: src/layers/zbar-binary
      CompatibleRuntimes:
        - python3.9
      RetentionPolicy: Retain
    Metadata:
      BuildMethod: makefile

The important part here is the BuildMethod: makerfile. This file basically just copy all the .so files into the layer. You can read more about this here. This is what the Makefile looks like:

build-ZbarBinary:
	mkdir -p "$(ARTIFACTS_DIR)/lib/zbar_binary"
	cp -a * "$(ARTIFACTS_DIR)/lib/zbar_binary"

My ZBAR_PATH environment variable is now /opt/lib/zbar_binary/libzbar.so.

Hope this helps.

[souravjamwal77]

@MarGul What I did was copied the inner pyzbar directory (i.e. the one that directly contains the Python files) into the base directory that I was packaging up for Lambda. I copied the libzbar.so.0.3.0 file into the same base directory and renamed it libzbar.so, so the files I package up look like this:

lambda_bundle
  - myscript.py
  - libzbar.so
  - pyzbar
    - __init__.py
    - ...

If you do this then you should be able to just use from pyzbar import decode from your Lambda script.

Hi @nickovs, How do I get a libzbar.so file for AMI-2 (CentOS 7 based image)? It looks like you're using ubuntu based .so file.

[nickovs]

@souravjamwal77 The gist script I posted builds the library for running on AWS Lambda instances, which run a Red Hat derived Linux, but if you need a copy for Centos7 you can just replace the Docker image name amazonlinux:2 with centos:7 and it seems to build just fine.

[mojoee]

I used the following solution for AWS, it's using a layer approach, which is a little different from @MarGul :

1. create the zip file for a lambda layer with the zbar libraries. You can follow @DanyStinson repo. This will give you the zip file for zbar in your s3 bucket
2. use the zip file to create a lambda layer on AWS and save the ARN adress
3. Adjust your SAM local template. You need to add the arn adress below the Layers header. It should look something like the following:

Template file:

Resources:
  OCR:
    Type: AWS::Serverless::Function 
    Properties:
      PackageType: Image
      Layers:
        - [Your ARN adress]

4. You should be able to build your serverless architecture now with sam build and then invoke it with sam invoke

[Water-Enjoyer]

For anyone attempting to get a standalone AWS layer, I was able to do so by using the following Dockerfile which follows the same approach from @DanyStinson above. Uploading pyzbar_layer.zip directly to an AWS layer worked correctly for me.

# use latest https://gallery.ecr.aws/sam/build-python3.12
FROM public.ecr.aws/sam/build-python3.12:1.121.0-20240730174605

WORKDIR /var/task

RUN dnf update -y && \
    dnf install -y autoconf gettext-devel automake pkgconfig libtool git make gcc-c++ zip

RUN pip install pyzbar -t python/lib/python3.12/site-packages/

RUN git clone https://github.com/mchehab/zbar.git && \
    cd zbar && \
    autoreconf -vfi || true && \
    ./configure && \
    make

RUN cp zbar/zbar/.libs/libzbar.so.0.3.0 python/lib/python3.12/site-packages/pyzbar/libzbar.so

RUN sed -i "s/find_library('zbar')/('\/opt\/python\/lib\/python3.12\/site-packages\/pyzbar\/libzbar.so')/g" python/lib/python3.12/site-packages/pyzbar/zbar_library.py

RUN zip -r pyzbar_layer.zip python