when slim_handler is used MySQLdb cannot be found by Django

[timj98]

this issue is similar to #755
when using the zappa_settings.json configuration to turn on "use_precompiled_packages":true and then turning on "slim_handler": true I get error below:

1491418915817] Error loading MySQLdb module: libmysqlclient.so.18: cannot open shared object file: No such file or directory: ImproperlyConfigured Traceback (most recent call last): File "/var/task/handler.py", line 508, in lambda_handler return LambdaHandler.lambda_handler(event, context) File "/var/task/handler.py", line 239, in lambda_handler handler = cls() File "/var/task/handler.py", line 142, in __init__ wsgi_app_function = get_django_wsgi(self.settings.DJANGO_SETTINGS) File "/var/task/django_zappa_app.py", line 20, in get_django_wsgi return get_wsgi_application() File "/private/var/folders/7c/lt6hvzr97z919lwy5yxyfs7w0000gn/T/pip-build-YiIZfn/django/django/core/wsgi.py", line 14, in get_wsgi_application File "/tmp/dailyfantasyser/django/__init__.py", line 18, in setup apps.populate(settings.INSTALLED_APPS) File "/tmp/dailyfantasyser/django/apps/registry.py", line 108, in populate app_config.import_models(all_models) File "/tmp/dailyfantasyser/django/apps/config.py", line 198, in import_models self.models_module = import_module(models_module_name) File "/usr/lib64/python2.7/importlib/__init__.py", line 37, in import_module __import__(name) File "/private/var/folders/7c/lt6hvzr97z919lwy5yxyfs7w0000gn/T/pip-build-YiIZfn/django/django/contrib/auth/models.py", line 41, in <module> File "/private/var/folders/7c/lt6hvzr97z919lwy5yxyfs7w0000gn/T/pip-build-YiIZfn/django/django/db/models/base.py", line 139, in __new__ File "/private/var/folders/7c/lt6hvzr97z919lwy5yxyfs7w0000gn/T/pip-build-YiIZfn/django/django/db/models/base.py", line 324, in add_to_class File "/private/var/folders/7c/lt6hvzr97z919lwy5yxyfs7w0000gn/T/pip-build-YiIZfn/django/django/db/models/options.py", line 250, in contribute_to_class File "/private/var/folders/7c/lt6hvzr97z919lwy5yxyfs7w0000gn/T/pip-build-YiIZfn/django/django/db/__init__.py", line 36, in __getattr__ File "/private/var/folders/7c/lt6hvzr97z919lwy5yxyfs7w0000gn/T/pip-build-YiIZfn/django/django/db/utils.py", line 240, in __getitem__ File "/private/var/folders/7c/lt6hvzr97z919lwy5yxyfs7w0000gn/T/pip-build-YiIZfn/django/django/db/utils.py", line 111, in load_backend File "/usr/lib64/python2.7/importlib/__init__.py", line 37, in import_module __import__(name) File "/private/var/folders/7c/lt6hvzr97z919lwy5yxyfs7w0000gn/T/pip-build-YiIZfn/django/django/db/backends/mysql/base.py", line 27, in <module> ImproperlyConfigured: Error loading MySQLdb module: libmysqlclient.so.18: cannot open shared object file: No such file or directory

I am running in virtualenv on macos10.12.3 with python2.7, Django 1.8.5.

Expected Behavior

the error should not occur and the library should be found in the proper location for Django to run properly.

Actual Behavior

the following error: ImproperlyConfigured: Error loading MySQLdb module: libmysqlclient.so.18: cannot open shared object file: No such file or directory

Possible Fix

Steps to Reproduce

1. update project to use_precompiled_packages:true and slim_handler:true
2. deploy large project using zappa update staging
3. run zappa tail to view the log

Your Environment

• Zappa version used: 0.40.0p
• Operating System and Python version: macos10.12.3
• Link to your project (optional):
• Your zappa_settings.py:

{ "staging": { "aws_region": "us-east-1", "django_settings": "daily.settings_lambda", "s3_bucket": "daily", "memory_size": 1024, "timeout_seconds": 120, "use_precompiled_packages": true, "slim_handler": true, "delete_local_zip": false, "log_level": "DEBUG", } }

[bxm156]

Does this change need any documentation updates? Will Django users' know they need to set some value here in order for their project to work under certain conditions?

[Miserlou]

It should JustWork^tm, at least for MySQL

[timj98]

The change does not affect anything if you do not use the slim_handler. If you do and you use Django and MySQLdb, it just works with the change…user doesn’t have to do anything. TimJ

…

On Apr 19, 2017, at 2:41 PM, Bryan Marty ***@***.***> wrote: Does this change need any documentation updates? Will Django users' know they need to set some value here in order for their project to work under certain conditions? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#776 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABvUM0nZ7cLCRQK026kn9o09bqld9vkBks5rxmN5gaJpZM4M0swS>.

[Miserlou]

I think this is fixed in 0.42.0

[gagandeep]

https://github.com/Miserlou/Zappa/blob/master/zappa/handler.py#L110
Why libmysqlclient.so.18 is in default.

I have a project that does not depend upon MySQL client but Postgres. And, it was failing at the runtime with a very less obvious error of Failed to find library...right filename?.

[timj98]

@gagandeep have you tried to use the "include" feature in your zappa_settings file to include the proper Postgres library? It seems like it might be preferable to have zappa check your Django settings and automatically include the proper library based on the database. Maybe this is a future enhancement.

[gagandeep]

@timj98 Yes, I had to change include. My only point is Library of any kind shouldn't assume things whether its MySQL or Postgres doesn't matter.
If someone needs to include mysql with Zappa it should be mentioned in include setting. Zappa shouldn't add libmysqlclient.so.18 as others might be using something else and that will break things.

Django & MySQL with slim handler should be part of the documentation as an example, not the library default.

[timj98]

All good points. I think that the reason it was included by default was so that it just worked when someone was using Django & MySQL. It sounds like it should be reworked to check the Django settings and include the default library from there so one doesn't have to manually include anything.

[gagandeep]

@timj98 let me tell you my fix. I had to mention include=[] in my zappa settings. Which will be non-obvious to many, if they don't follow the code. I let you or @Miserlou to decide on this one.

[ijey-lit]

@gagandeep Did you have to specify a specific postgresql library? Did you just have an empty include=[] in your zappa_settings.json?

[timj98]

You must include one for postgresql as this is not automatic. @gagandeep In the docs it states that you can include the library of your choice in the include statement. By default it automatically includes the library for MySQL.

…

On Apr 5, 2018, at 2:13 PM, ijey-lit ***@***.***> wrote: @gagandeep Did you have to specify a specific postgresql library? Did you just have an empty include=[] in your zappa_settings.json? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[gagandeep]

Yes, I have empty include in my zappa settings. Postgres is working fine with that.

[xncbf]

how is it going this issue? I have a same issue on zappa==0.51.0, django==3.0.5, python==3.7

[RavimoShark]

Same here.

[beittenc]

I am now having this problem as well. I have an existing Django 3/Zappa 0.51/Postgres 10 project that works fine. I created a new project, copying the old project's zappa_settings.json and tweaking values as appropriate, but it can't deploy at all. I'm now concerned that this problem will crop up on my old project at some point, which is an issue since it's a production critical environment. I have never had an include or exclude in my zappa_settings.json and adding one to the new project doesn't seem to help.

[LyleScott]

This is actually answered better over in #1834

I'd guess the "include": [] hack works due to

Zappa/zappa/handler.py

Line 105 in ba20c85

included_libraries = getattr(self.settings, 'INCLUDE', ['libmysqlclient.so.18'])

It will knock out the auto-included (with slim handler) mysql lib.

[gagandeep]

Yes, you got it absolutely right. That's why I wrote earlier

Why libmysqlclient.so.18 is in default.

[nabazm]

i still have issues tried all these solutions and no luck

[pedroraminhas]

I also still have issues. When I add include:[] it does not work and the code still tries to import libmysqlclient.so.18

[amsal-muhammad]

adding include : [] does not help. How can I add postgres to include?

[TheTechRealms]

"include": [] in the Zappa_settings.json file didn't work for me either.

[jessetarot]

I'd guess the "include": [] hack works due to

Zappa/zappa/handler.py

Line 105 in ba20c85

included_libraries = getattr(self.settings, 'INCLUDE', ['libmysqlclient.so.18'])

Looks like the "include": [] hack doesn't work for some (including me) due to this:

Zappa/zappa/cli.py

Lines 2378 to 2380 in 818f7de

	include = self.stage_config.get('include', [])
	if len(include) >= 1:
	settings_s += "INCLUDE=" + str(include) + '\n'

since the INCLUDE= var is not added regardless of whether you omit "include" or you write "include": [] in zappa_settings.json.

I propose using a sentinal value "include": null to signify that you don't want to include the default, error-causing value ['libmysqlclient.so.18']

Something like this:

                include = self.stage_config.get('include', [])
                if include is None:
                    settings_s += 'INCLUDE=[]\n'
                elif len(include) >= 1:
                    settings_s += "INCLUDE=" + str(include) + '\n'

Thoughts?

[TheTechRealms]

I remember trying “include”: null. It didn’t work for me either

[philmade]

This doesn't work for me either. It's trying to include a library I don't need which is hardcoded into line 105 of handler.py.

Zappa/zappa/handler.py

Line 105 in ba20c85

included_libraries = getattr(self.settings, 'INCLUDE', ['libmysqlclient.so.18'])

I'm using postgres and as far as I'm aware my project doesn't use this libmysqlclient, yet its causing my scripts to fail when it cannot find it,. Is there a way to remove it since it's not needed?