Broken on Mac OSX

[hz]

hz:> pip3 install bloscpack
hz:> blpk c utf.csv

Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.5/bin/blpk", line 7, in
from bloscpack.cli import main

File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/bloscpack/init.py", line 9, in
from .args import (BloscArgs,
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/bloscpack/args.py", line 6, in
import blosc
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/blosc/init.py", line 12, in
from blosc.toplevel import (
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/blosc/toplevel.py", line 16, in
from blosc import blosc_extension as _ext
ImportError: dlopen(/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/blosc/blosc_extension.cpython-35m-darwin.so, 2): Symbol not found: _aligned_alloc
Referenced from: /Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/blosc/blosc_extension.cpython-35m-darwin.so
Expected in: flat namespace
in /Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/blosc/blosc_extension.cpython-35m-darwin.so

[hz]

Python: 3.5.2. blosc (1.2.7).

[esc]

The bug is in python-blosc. Bloscpack currently only works with python-blosc 1.2.7. I.e. there is no simple fix, but you can look at: #45

[oogali]

The same issue happens with python-blosc 1.2.7.

Traceback (most recent call last):
  File "venv/bin/statrunner", line 9, in <module>
    load_entry_point('statrunner==0.2', 'console_scripts', 'statrunner')()
  File "venv/lib/python2.7/site-packages/pkg_resources.py", line 353, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "venv/lib/python2.7/site-packages/pkg_resources.py", line 2321, in load_entry_point
    return ep.load()
  File "venv/lib/python2.7/site-packages/pkg_resources.py", line 2048, in load
    entry = __import__(self.module_name, globals(),globals(), ['__name__'])
  File "statrunner/main.py", line 29, in <module>
    from statrunner.castra_api import *
  File "statrunner/castra_api.py", line 3, in <module>
    from castra import Castra
  File "venv/lib/python2.7/site-packages/castra/__init__.py", line 1, in <module>
    from .core import Castra
  File "venv/lib/python2.7/site-packages/castra/core.py", line 18, in <module>
    import blosc
  File "venv/lib/python2.7/site-packages/blosc/__init__.py", line 12, in <module>
    from blosc.toplevel import (
  File "venv/lib/python2.7/site-packages/blosc/toplevel.py", line 16, in <module>
    from blosc import blosc_extension as _ext
ImportError: dlopen(venv/lib/python2.7/site-packages/blosc/blosc_extension.so, 2): Symbol not found: _aligned_alloc
  Referenced from: venv/lib/python2.7/site-packages/blosc/blosc_extension.so
  Expected in: flat namespace
 in venv/lib/python2.7/site-packages/blosc/blosc_extension.so

The problem appears to be that python-blosc 1.2.7 includes c-blosc 1.6.1 in its sources, which in turn uses STDC_MACRO to determine support for the C11 aligned_alloc function instead of the ISOC11_SOURCE macro.

This causes clang (the default compiler shipped by Apple, as of Xcode 5/OS X 10.9 Mavericks), to compile against this function. However, C11 support is not enabled (to actually enable under clang, you must add -std=c11 to CFLAGS).

So the code appears to compile and link okay, but fails at runtime.

$ ( git clone [email protected]:Blosc/c-blosc 2>/dev/null && cd c-blosc && git checkout v1.7.0 2>/dev/null )
$ ( git clone [email protected]:Blosc/python-blosc 2>/dev/null && cd python-blosc && git checkout v1.2.7 2>/dev/null )
$ diff -U1 python-blosc/c-blosc/blosc/blosc.c c-blosc/blosc/blosc.c
--- python-blosc/c-blosc/blosc/blosc.c  2016-07-26 17:04:58.000000000 -0400
+++ c-blosc/blosc/blosc.c   2016-07-26 17:04:43.000000000 -0400

...

@@ -199,9 +200,9 @@
   void *block = NULL;
   int res = 0;

 /* Do an alignment to 32 bytes because AVX2 is supported */
-#if __STDC_VERSION__ >= 201112L
+#if _ISOC11_SOURCE
   /* C11 aligned allocation. 'size' must be a multiple of the alignment. */
   block = aligned_alloc(32, size);
 #elif defined(_WIN32)
   /* A (void *) cast needed for avoiding a warning with MINGW :-/ */

...

The path of least resistance would be passing in the necessary compiler flag viaCFLAGS, except setuptools doesn't differentiate between C (such as blosc) and C++ code (such as snappy), so when setuptools attempts to compile the snappy C++ sources with a C compiler flag, clang sternly says No.:

cc -DNDEBUG -g -fwrapv -Os -Wall -Wstrict-prototypes -std=c11 -arch i386 -arch x86_64 -pipe -DHAVE_LZ4=1 -DHAVE_SNAPPY=1 -DHAVE_ZLIB=1 -Ic-blosc/blosc -Ic-blosc/internal-complibs/lz4-1.6.0 -Ic-blosc/internal-complibs/snappy-1.1.1 -Ic-blosc/internal-complibs/zlib-1.2.8 -I/System/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c c-blosc/internal-complibs/lz4-1.6.0/lz4hc.c -o build/temp.macosx-10.11-intel-2.7/c-blosc/internal-complibs/lz4-1.6.0/lz4hc.o -std=c11 -DSHUFFLE_SSE2_ENABLED

cc -DNDEBUG -g -fwrapv -Os -Wall -Wstrict-prototypes -std=c11 -arch i386 -arch x86_64 -pipe -DHAVE_LZ4=1 -DHAVE_SNAPPY=1 -DHAVE_ZLIB=1 -Ic-blosc/blosc -Ic-blosc/internal-complibs/lz4-1.6.0 -Ic-blosc/internal-complibs/snappy-1.1.1 -Ic-blosc/internal-complibs/zlib-1.2.8 -I/System/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c c-blosc/internal-complibs/snappy-1.1.1/snappy-c.cc -o build/temp.macosx-10.11-intel-2.7/c-blosc/internal-complibs/snappy-1.1.1/snappy-c.o -std=c11 -DSHUFFLE_SSE2_ENABLED

error: invalid argument '-std=c11' not allowed with 'C++/ObjC++'

error: command 'cc' failed with exit status 1

----------------------------------------
Cleaning up...
Command venv/bin/python -c "import setuptools, tokenize;__file__='venv/build/blosc/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /var/folders/n2/gjc0slfs38q9g08mzxp0ckc40000gn/T/pip-wgt_wr-record/install-record.txt --single-version-externally-managed --compile --install-headers venv/include/site/python2.7 failed with error code 1 in venv/build/blosc
Storing debug log for failure in ~/.pip/pip.log

For OS X users, your current options for using bloscpack appear to be:
a) don't use OS X
b) use GCC

I believe you would call this a rock and a hard place.

[esc]

Thanks for the detailed low-level detective work, much appreciated!

Do you upgrading the python-blosc dependency for Bloscpack? As of the latest comments in #45 it appears that fixing the test-suit and upgrading the python-blosc dependency would fix this issue and make Bloscpack available for OSX again?

[FrancescAlted]

Thanks @oogali . I think the issue might be that Blosc is using different checks in:

https://github.com/Blosc/c-blosc/blob/master/blosc/blosc.c#L65

and:

https://github.com/Blosc/c-blosc/blob/master/blosc/blosc.c#L210

and that might be introducing problems.

At any rate, as requiring C11 is quite unlikely for the life of the Blosc1 project, I removed this check in:

Blosc/c-blosc@4d832aa

Could you please confirm that this works for you?

[oogali]

@FrancescAlted @esc This does appear to fix the issue with c-blosc, but should I be concerned that this brings in blosc 1.4.0 and not the 1.2.7 version that you previously pinned in python-blosc?

I forked and made changes in my repos:

oogali/python-blosc@c-blosc-4d832aa

oogali/bloscpack@unpin-blosc

[oogali]

Hmm, to truly test this, I should have probably made the change in tagged version v1.2.7 of python-blosc, right?

[FrancescAlted]

Thanks for testing this. I think @esc pinned v1.2.7 in bloscpack just because of some segfaults in previous versions of C-Blosc, but I don't think this is relevant anymore in recent releases. IMO, going to 1.4.x is pretty safe.

[esc]

Closing as of #51