Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

error: sodium.h: No such file or directory #79

Closed
fletom opened this issue Jul 11, 2014 · 12 comments
Closed

error: sodium.h: No such file or directory #79

fletom opened this issue Jul 11, 2014 · 12 comments

Comments

@fletom
Copy link

fletom commented Jul 11, 2014

PyNaCl installs fine using pip, but when imported it bugs out saying it can't find a file called sodium.h.

From my understanding, PyNaCl includes libsodium and doesn't need it to be pre-installed on the system. I also tried pre-installing it just to make sure, and got the same error. I also verified that sodium.h was found in my LD_LIBRARY_PATH, but no matter what I do I seem to get the same error.

Has anybody ever seen this before?

If you would like to reproduce it the exact environment (Heroku) you can do so by running:

git init
echo "PyNaCl" > requirements.txt
echo "cffi" >> requirements.txt
heroku create --buildpack git://github.com/kennethjiang/heroku-buildpack-python-libffi.git
git add .
git commit -m "test"
git push heroku master
heroku run 'python -c "import nacl.hash"'

The full content of the error I get is as follows:

~ $ python -c "import nacl.hash"
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c:185:20: error: sodium.h: No such file or directory
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c: In function ‘_cffi_f_crypto_box’:
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c:269: warning: implicit declaration of function ‘crypto_box’
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c: In function ‘_cffi_f_crypto_box_afternm’:
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c:345: warning: implicit declaration of function ‘crypto_box_afternm’
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c: In function ‘_cffi_f_crypto_box_beforenm’:
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c:402: warning: implicit declaration of function ‘crypto_box_beforenm’
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c: In function ‘_cffi_f_crypto_box_beforenmbytes’:
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c:416: warning: implicit declaration of function ‘crypto_box_beforenmbytes’
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c: In function ‘_cffi_f_crypto_box_boxzerobytes’:
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c:430: warning: implicit declaration of function ‘crypto_box_boxzerobytes’
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c: In function ‘_cffi_f_crypto_box_keypair’:
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c:474: warning: implicit declaration of function ‘crypto_box_keypair’
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c: In function ‘_cffi_f_crypto_box_noncebytes’:
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c:488: warning: implicit declaration of function ‘crypto_box_noncebytes’
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c: In function ‘_cffi_f_crypto_box_open’:
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c:577: warning: implicit declaration of function ‘crypto_box_open’
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c: In function ‘_cffi_f_crypto_box_open_afternm’:
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c:653: warning: implicit declaration of function ‘crypto_box_open_afternm’
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c: In function ‘_cffi_f_crypto_box_publickeybytes’:
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c:667: warning: implicit declaration of function ‘crypto_box_publickeybytes’
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c: In function ‘_cffi_f_crypto_box_secretkeybytes’:
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c:681: warning: implicit declaration of function ‘crypto_box_secretkeybytes’
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c: In function ‘_cffi_f_crypto_box_zerobytes’:
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c:695: warning: implicit declaration of function ‘crypto_box_zerobytes’
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c: In function ‘_cffi_f_crypto_hash’:
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c:745: warning: implicit declaration of function ‘crypto_hash’
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c: In function ‘_cffi_f_crypto_hash_sha256’:
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c:795: warning: implicit declaration of function ‘crypto_hash_sha256’
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c: In function ‘_cffi_f_crypto_hash_sha256_bytes’:
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c:809: warning: implicit declaration of function ‘crypto_hash_sha256_bytes’
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c: In function ‘_cffi_f_crypto_hash_sha512’:
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c:859: warning: implicit declaration of function ‘crypto_hash_sha512’
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c: In function ‘_cffi_f_crypto_hash_sha512_bytes’:
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c:873: warning: implicit declaration of function ‘crypto_hash_sha512_bytes’
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c: In function ‘_cffi_f_crypto_scalarmult_base’:
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c:917: warning: implicit declaration of function ‘crypto_scalarmult_base’
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c: In function ‘_cffi_f_crypto_scalarmult_bytes’:
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c:931: warning: implicit declaration of function ‘crypto_scalarmult_bytes’
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c: In function ‘_cffi_f_crypto_scalarmult_scalarbytes’:
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c:945: warning: implicit declaration of function ‘crypto_scalarmult_scalarbytes’
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c: In function ‘_cffi_f_crypto_secretbox’:
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c:1021: warning: implicit declaration of function ‘crypto_secretbox’
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c: In function ‘_cffi_f_crypto_secretbox_boxzerobytes’:
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c:1035: warning: implicit declaration of function ‘crypto_secretbox_boxzerobytes’
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c: In function ‘_cffi_f_crypto_secretbox_keybytes’:
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c:1049: warning: implicit declaration of function ‘crypto_secretbox_keybytes’
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c: In function ‘_cffi_f_crypto_secretbox_noncebytes’:
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c:1063: warning: implicit declaration of function ‘crypto_secretbox_noncebytes’
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c: In function ‘_cffi_f_crypto_secretbox_open’:
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c:1139: warning: implicit declaration of function ‘crypto_secretbox_open’
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c: In function ‘_cffi_f_crypto_secretbox_zerobytes’:
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c:1153: warning: implicit declaration of function ‘crypto_secretbox_zerobytes’
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c: In function ‘_cffi_f_crypto_sign’:
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c:1229: warning: implicit declaration of function ‘crypto_sign’
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c: In function ‘_cffi_f_crypto_sign_bytes’:
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c:1243: warning: implicit declaration of function ‘crypto_sign_bytes’
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c: In function ‘_cffi_f_crypto_sign_keypair’:
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c:1287: warning: implicit declaration of function ‘crypto_sign_keypair’
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c: In function ‘_cffi_f_crypto_sign_open’:
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c:1363: warning: implicit declaration of function ‘crypto_sign_open’
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c: In function ‘_cffi_f_crypto_sign_publickeybytes’:
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c:1377: warning: implicit declaration of function ‘crypto_sign_publickeybytes’
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c: In function ‘_cffi_f_crypto_sign_secretkeybytes’:
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c:1391: warning: implicit declaration of function ‘crypto_sign_secretkeybytes’
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c: In function ‘_cffi_f_crypto_sign_seed_keypair’:
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c:1448: warning: implicit declaration of function ‘crypto_sign_seed_keypair’
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c: In function ‘_cffi_f_randombytes’:
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c:1484: warning: implicit declaration of function ‘randombytes’
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/app/.heroku/python/lib/python2.7/site-packages/nacl/hash.py", line 17, in <module>
    import nacl.c
  File "/app/.heroku/python/lib/python2.7/site-packages/nacl/c/__init__.py", line 16, in <module>
    from nacl.c.crypto_box import (
  File "/app/.heroku/python/lib/python2.7/site-packages/nacl/c/crypto_box.py", line 23, in <module>
    crypto_box_SECRETKEYBYTES = lib.crypto_box_secretkeybytes()
  File "/app/.heroku/python/lib/python2.7/site-packages/nacl/_lib/__init__.py", line 73, in __getattr__
    self._lib = self.ffi.verifier.load_library()
  File "/app/.heroku/python/lib/python2.7/site-packages/cffi/verifier.py", line 74, in load_library
    self._compile_module()
  File "/app/.heroku/python/lib/python2.7/site-packages/cffi/verifier.py", line 139, in _compile_module
    outputfilename = ffiplatform.compile(tmpdir, self.get_extension())
  File "/app/.heroku/python/lib/python2.7/site-packages/cffi/ffiplatform.py", line 25, in compile
    outputfilename = _build(tmpdir, ext)
  File "/app/.heroku/python/lib/python2.7/site-packages/cffi/ffiplatform.py", line 51, in _build
    raise VerificationError('%s: %s' % (e.__class__.__name__, e))
cffi.ffiplatform.VerificationError: CompileError: command 'gcc' failed with exit status 1
~ $

Thanks!

@davidblewett
Copy link

I'm seeing this same behavior when installing pynacl via buildout. What's interesting is that I cannot reproduce it by creating a new virtualenv (either pypy 2.3.0 or cpython 2.7). My only conjecture is that in the OP's environment (and under buildout), the cffi bindings are not getting generated at install time. When attempting to build them at run time, it fails because libsodium is no longer available. I would appreciate being able to point pynacl at a different libsodium anyway than the bundled one. This is possible with other cffi-using projects (pyzmq springs to mind). From an audit perspective, it would be much easier to ensure that the library is legitimate independently.

I am testing on OSX 10.9.3 using MacPorts.

@Ayrx
Copy link
Contributor

Ayrx commented Jul 13, 2014

@davidblewett I'm not sure about why this is failing (and I don't have time right now to hunt down the problem), but you can tell pynacl to use the system's libsodium instead of the bundled one by setting a environmental variable. https://github.com/pyca/pynacl/blob/master/setup.py#L80

@fletom
Copy link
Author

fletom commented Jul 13, 2014

I don't know if this is relevant, but if I run export SODIUM_INSTALL=system (when libsodium is installed system-wide) before running I still get this same error. Or does this variable have to be set during installation?

I really appreciate both of you looking at this. If there was some way I could get this working I'd be so grateful.

@davidblewett
Copy link

@fletom I was able to get this working with a non-bundled libsodium by passing C_INCLUDE_PATH and LIBRARY_PATH environment variables pointing to the correct locations the first time I imported an nacl module that uses a C interface (I chose nacl.signing). For an OS-supplied libsodium on Linux, those would probably be /usr/include and /usr/lib respectively. The first time nacl needs to access a C library, cffi will build the shared object, so you won't need to pass the environment variables explicitly after that.

What I'm not sure on is why I have to pass them at all the first time. It seems that the cffi shared object should be built at install time, using the environment passed then. When I install the package using buildout, I have the above 2 variables set so it should be able to do so.

@fletom
Copy link
Author

fletom commented Jul 13, 2014

@davidblewett That worked!

I had to set all four of the following:

export SODIUM_INSTALL=system
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:[libsodium's libs dir, where libsodium.a/.la/.so are found]
export LIBRARY_PATH=$LIBRARY_PATH:[libsodium's libs dir, same as above]
export C_INCLUDE_PATH=$C_INCLUDE_PATH:[libsodium's include dir, where sodium.h is found]

Each of these environment variables seems to be strictly necessary, as leaving out even one of them before importing something like nacl.hash causes some kind of compilation-related error to appear.

Anyways, if this gets fixed someday that's great, but for now I'm happy with having this workaround. Hopefully other people with the same problem will see this.

Thank you @davidblewett!

@mk-fg
Copy link

mk-fg commented Jul 17, 2014

Compilation on import itself seem to be bogus - don't think cffi should do that, and it usually can't do that with module installed system-wide (e.g. in /usr), unless you're importing it as root.

I've had the same issue installing PyNaCl from PKGBUILD (having simple python2 setup.py install --root="$pkgdir/" --optimize=1 inside) on current ARMv7 Arch, but only when I forgot to include python2-cffi in "depends=(...)" there, so it wasn't available during build.
Building with python2-cffi installed produced proper pre-built package that imports without any compilation (as far as I can tell).

Not sure if build working without cffi like that is supposed to be a bug or a feature.

@fletom
Copy link
Author

fletom commented Jul 18, 2014

@mk-fg Re compilation, I agree. If it's possible not to compile anything at all on import that would be ideal. I notice that PyNaCl takes a few seconds to import (longer than any other packages) because of this.

I'm also seeing this error now on my local machine, when I have 'pynacl' in my setup.py's install_requires and I run python setup.py develop. Strangely, if I run pip install pynacl instead, I don't get the error. Is there something I need to add to my setup.py to get it working perhaps?

@dlitz
Copy link

dlitz commented Sep 1, 2014

I'm actually seeing this same error when trying to install from the git repo using pip install --editable. The difference seems to be the presence of the --editable flag. I'm not sure if this is a separate issue, or if heroku create adds that flag somewhere.

If you have Docker installed, you can reproduce the error using:

docker build - <<'EOF'
FROM ubuntu:12.04
RUN apt-get -qy update
RUN apt-get -qy install build-essential git libffi-dev python-dev python-pip

# Uncomment one of the following 4 cases:

# 1. This fails:
RUN git clone https://github.com/pyca/pynacl && cd pynacl && pip install -e .
# 2. This also fails:
#RUN pip install -e 'git+https://github.com/pyca/pynacl@master#egg=pynacl'
# 3. This succeeds:
#RUN git clone https://github.com/pyca/pynacl && cd pynacl && pip install .
# 4. This also succeeds:
#RUN pip install 'git+https://github.com/pyca/pynacl@master#egg=pynacl'
EOF

This is with pynacl v0.2.3-97-g337a955.

dlitz added a commit to nylas/sync-engine that referenced this issue Sep 3, 2014
This removes -e (`pip install --editable`) from the pynacl line in
requirements.txt, and uses git+https:// because pip will break if the
URI isn't in the format `vcs+protocol://...`

This may be related to the following pynacl issue:

    pyca/pynacl#79
dlitz added a commit to nylas/sync-engine that referenced this issue Sep 3, 2014
This removes -e (`pip install --editable`) from the pynacl line in
requirements.txt, and uses git+https:// because pip will break if the
URI isn't in the format `vcs+protocol://...`

This may be related to the following pynacl issue:

    pyca/pynacl#79
dlitz added a commit to nylas/sync-engine that referenced this issue Sep 3, 2014
This removes -e (`pip install --editable`) from the pynacl line in
requirements.txt, and uses git+https:// because pip will break if the
URI isn't in the format `vcs+protocol://...`

This may be related to the following pynacl issue:

    pyca/pynacl#79
dlitz added a commit to nylas/sync-engine that referenced this issue Oct 22, 2014
Summary:
Workaround for the "error: sodium.h: No such file or directory" bug in
pynacl and/or cffi: pyca/pynacl#79

Fixes #95

Test Plan: `vagrant up --provision` from scratch

Reviewers: charles

Reviewed By: charles

Subscribers: mg

Differential Revision: https://review.inboxapp.com/D635
dlitz added a commit to nylas/sync-engine that referenced this issue Nov 4, 2014
Summary:
Workaround for the "error: sodium.h: No such file or directory" bug in
pynacl and/or cffi: pyca/pynacl#79

Fixes #95

Test Plan: `vagrant up --provision` from scratch

Reviewers: charles

Reviewed By: charles

Subscribers: mg

Differential Revision: https://review.inboxapp.com/D635
@mk-fg
Copy link

mk-fg commented Dec 10, 2014

As I'm bumping into this one again, and there doesn't seem to be a link to a related cffi packaging issue, here it is: https://bitbucket.org/cffi/cffi/issue/109/enable-sane-packaging-for-cffi

@warner
Copy link
Member

warner commented Mar 7, 2015

With the build changes that went into the recent 0.3.0 release, is this still a problem? I know that 0.3.0 fixed bug I was experiencing, in which a different package that had "install_requires=pynacl" could not be pip-installed. Could you check to see if anything's improved in the new version?

@mk-fg
Copy link

mk-fg commented Oct 25, 2015

Sorry for very delayed response, but latest releases indeed don't seem to have this issue for me - packaging pre-built module works fine and doesn't trigger rebuild on first use.

@warner
Copy link
Member

warner commented Oct 25, 2015

great, thanks, I'll close this out.

@warner warner closed this as completed Oct 25, 2015
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Oct 6, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Development

No branches or pull requests

6 participants