Build Fails on MacOS #47

mhworth · 2019-04-21T21:39:27Z

Building python-rocksdb fails on recent version of MacOS. The final error you get is when linking with libstdc++:

clang: warning: libstdc++ is deprecated; move to libc++ with a minimum deployment target of OS X 10.9 [-Wdeprecated]
ld: library not found for -lstdc++
clang: error: linker command failed with exit code 1 (use -v to see invocation)
error: command 'g++' failed with exit status 1

Summary of how to work around right now:

brew install rocksdb
export MACOSX_DEPLOYMENT_TARGET=10.9
pip install python-rocksdb

The pandas folks ran into this too; here is a discussion on how to resolve the issue by passing additional args to Clang: pandas-dev/pandas#23424

The text was updated successfully, but these errors were encountered:

zzzhc · 2019-06-22T07:52:35Z

export CFLAGS=-stdlib=libc++
pip install python-rocksdb

this works for me on macOS 10.14.3

cvarma21 · 2019-10-04T18:30:29Z

Thanks. This worked perfectly

deed02392 · 2020-02-03T18:07:53Z

I'm getting rocksdb/_rocksdb.cpp:622:10: fatal error: 'rocksdb/slice.h' file not found

Above did not help.

vincentwyshan · 2020-04-13T03:48:29Z

I'm getting rocksdb/_rocksdb.cpp:622:10: fatal error: 'rocksdb/slice.h' file not found

Above did not help.

I get same problem, guess we need install rocksdb in mac and make complier link to proper headers

mrugacz95 · 2020-05-18T14:49:48Z

Installing rocksdb with brew helped:
brew install rocksdb

alexreg · 2022-02-22T02:08:53Z

None of the above solutions work on macOS 12.2, I'm afraid...

faberchri · 2022-02-25T11:31:42Z

The following procedure worked for me on:

MacBook Pro M1
macOS 12.2.1 Monterey
rocksdb 6.29.3

Install rocksdb with brew:

brew install rocksdb

Use brew to look up the path on your system of the libs required for the install of python-rocksdb:

brew info rocksdb lz4 snappy

I had all libs already installed, maybe due to the install of rocksdb. If libs are missing, I guess you need to install them.

Export (for gcc?) the paths of the libs with CPPFLAGS and LDFLAGS and install python-rocksdb with pip.
For me that was (adapt paths according to your setup):

export CPPFLAGS=-I/opt/homebrew/Cellar/rocksdb/6.29.3/include
export LDFLAGS="-L/opt/homebrew/Cellar/rocksdb/6.29.3/lib -L/opt/homebrew/Cellar/snappy/1.1.9/lib -L/opt/homebrew/Cellar/lz4/1.9.3/lib"
pip install python-rocksdb

The exports fixed also the poetry install of a poetry project with faust-streaming-rocksdb (faust-streaming = {extras = ["rocksdb"], version = "^0.8.2"}) as dependency.

alexreg · 2022-02-25T15:53:14Z

@faberchri Thank you for sharing, although that didn't work for me either. The only thing that worked in the end was to use the more recent fork of this library, patch that for the latest rocksdb (just removing a few definitions), and then build under brew sh. I'll try to get a PR submitted to that project.

Littlecowherd · 2022-03-04T11:05:26Z

The following procedure worked for me on:

MacBook Pro M1

macOS 12.2.1 Monterey

rocksdb 6.29.3

Install rocksdb with brew:
brew install rocksdb
Use brew to look up the path on your system of the libs required for the install of python-rocksdb:
brew info rocksdb lz4 snappy
I had all libs already installed, maybe due to the install of rocksdb. If libs are missing, I guess you need to install them.

Export (for gcc?) the paths of the libs with CPPFLAGS and LDFLAGS and install python-rocksdb with pip. For me that was (adapt paths according to your setup):
export CPPFLAGS=-I/opt/homebrew/Cellar/rocksdb/6.29.3/include
export LDFLAGS="-L/opt/homebrew/Cellar/rocksdb/6.29.3/lib -L/opt/homebrew/Cellar/snappy/1.1.9/lib -L/opt/homebrew/Cellar/lz4/1.9.3/lib"
pip install python-rocksdb
The exports fixed also the poetry install of a poetry project with faust-streaming-rocksdb (faust-streaming = {extras = ["rocksdb"], version = "^0.8.2"}) as dependency.

Thank you anyway, the installation succeeded but runs failed ……

rollue · 2022-03-24T06:46:58Z

@faberchri Thank you for sharing, although that didn't work for me either. The only thing that worked in the end was to use the more recent fork of this library, patch that for the latest rocksdb (just removing a few definitions), and then build under brew sh. I'll try to get a PR submitted to that project.

Could you share how this could be done in more detail?

I tried to install from your PR branch by adding the below line to poetry,

rocksdb = { git = "https://github.com/alexreg/python-rocksdb.git", branch = "update-api" }

but still getting this error

...
rocksdb/_rocksdb.cpp:796:10: fatal error: 'rocksdb/utilities/backupable_db.h' file not found
#include "rocksdb/utilities/backupable_db.h"
...

P.S: what do you mean by build under brew sh?

alexreg · 2022-03-24T15:57:51Z

@rollue I mean to start the Homebrew shell using brew sh and run the build from within that. But anyway, the way I recommend using my branch (and when it's merged, NightTsarina's fork), is to configure the rocksdb dependency exactly how you did above, but also, before running pip install (or the poetry equivalent), to export some environment variables:

export INCLUDE_PATH="$(brew --prefix)/include" LIBRARY_PATH="$(brew --prefix)/lib"

This presumes you installed rocksdb with Homebrew. Note that $(brew --prefix) is Bash syntax, but if you're using another shell, replace it with the equivalent (or just hard code it to /opt/homebrew or /usr/local, as appropriate).

rollue · 2022-03-25T00:53:28Z

@alexreg thanks for the explanation, but pardon my ignorance.
When I try to poetry install your branch after installing rocksdb with Homebrew (with brew install rocksdb).
I keep getting the following error. (I'm on M1 Mac pro)

.... 
    rocksdb/_rocksdb.cpp:819:10: fatal error: 'rocksdb/utilities/backupable_db.h' file not found
    #include "rocksdb/utilities/backupable_db.h"
             ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    3 warnings and 1 error generated.
    error: command '/usr/bin/clang' failed with exit code 1
    [end of output]
...

I've seen in your conversation with NightTsarina, especially in NightTsarina#13 (comment) that this may be deprecated.
But I can't find a workaround for this. Any ideas how to go about this?

alexreg · 2022-03-25T01:25:51Z

@rollue I don't use poetry so I can't say much about it, but if you set the environment variables like above and then pip install my branch, it seems to work fine on macOS. And yes, backupable_db is deprecated, though it still exists with rocksdb 6.

jrj-d · 2022-05-11T11:51:11Z

Hello @alexreg, I'm experiencing the same issue as @rollue.
We're blocked because there's no easy way to install rocksdb v6 on mac (with brew). I'll try to find a workaround

alexreg · 2022-05-11T14:49:39Z

Hello @alexreg, I'm experiencing the same issue as @rollue.

We're blocked because there's no easy way to install rocksdb v6 on mac (with brew). I'll try to find a workaround

brew install alexreg/dev/rocksdb@6

I tried to get the formula accepted in homebrew-core but sadly no luck. At least this should make it easy though.

KochankovID · 2022-05-24T03:42:09Z

@Littlecowherd I faced with the same problem and managed to solve it by moving from anaconda to simple venv or poetry. I'm not sure why it works so, but I suggest anyone to try it. It worked for me.

alexreg · 2022-05-25T01:30:22Z

@KochankovID Better to use NightTsarina's fork these days. Actively maintained.

faberchri · 2022-08-25T07:55:56Z

Regarding my previous comment in this thread:

The following procedure worked for me on:

MacBook Pro M1
macOS 12.2.1 Monterey
rocksdb 6.29.3
Install rocksdb with brew:

brew install rocksdb
Use brew to look up the path on your system of the libs required for the install of python-rocksdb:

brew info rocksdb lz4 snappy
I had all libs already installed, maybe due to the install of rocksdb. If libs are missing, I guess you need to install them.

Export (for gcc?) the paths of the libs with CPPFLAGS and LDFLAGS and install python-rocksdb with pip.
For me that was (adapt paths according to your setup):

export CPPFLAGS=-I/opt/homebrew/Cellar/rocksdb/6.29.3/include
export LDFLAGS="-L/opt/homebrew/Cellar/rocksdb/6.29.3/lib -L/opt/homebrew/Cellar/snappy/1.1.9/lib -L/opt/homebrew/Cellar/lz4/1.9.3/lib"
pip install python-rocksdb
The exports fixed also the poetry install of a poetry project with faust-streaming-rocksdb (faust-streaming = {extras = ["rocksdb"], version = "^0.8.2"}) as dependency.

I Followed this same procedure just now again in order to successfully install python-rocksdb on my new computer (Apple M1 Max, 2021).

Crucial missing information from my original post: the procedure works only for rocksdb v6. If you run now brew install rocksdb, you install version 7+ and for version 7+ the procedure does not work.

For me rocksdb v6 is fine, so I used this procedure to install rocksdb v6 with brew. I also had to change the path to the lz4 lib, now /opt/homebrew/Cellar/lz4/1.9.4/lib.

ccz5 · 2022-10-16T18:06:21Z

Here is how I made it to work.

Setup:

brew info rocksdb lz4 snappy find out if there is any missing package and install it.
brew install rocksdb (this is to install latest rocksdb, v7)
brew install alexreg/dev/rocksdb@6
Find out where you installed the rocksdb in step 2 and 3, for me it's /usr/local/Cellar/rocksdb@6 and /usr/local/Cellar/rocksdb
a. export CPPFLAGS="-I/usr/local/Cellar/rocksdb@6/6.29.5/include -I/usr/local/Cellar/rocksdb/7.7.2/include" (I had to include both 6 and 7 header files because the install seems to need both).
b. export LDFLAGS="-L/usr/local/Cellar/rocksdb@6/6.29.5/lib -L/usr/local/Cellar/rocksdb/7.7.2/lib -L/usr/local/Cellar/snappy/1.1.9/lib -L/usr/local/Cellar/lz4/1.9.4/lib"

Install

downloaded tar.gz file and unzip it https://pypi.org/project/python-rocksdb/#files
go to the directory and vim setup.py, change extra_compile_args "-std=c++" to "-std=c++17" because c++17 functions are used.
python3 setup.py install

Good luck!

alexreg · 2022-10-16T23:39:07Z

Glad it helped. That looks good for this repo. For anyone using (NightTsarina's more up-to-date fork (but still based on rocksdb v6), you don't need to make the change to extra_compile_args, and you can just modify PKG_CONFIG_PATH as indicated by my Homebrew formula.

export PKG_CONFIG_PATH="/opt/homebrew/opt/rocksdb@6/lib/pkgconfig"

jagalactic · 2022-11-06T16:37:43Z

Regarding my previous comment in this thread:

The following procedure worked for me on:
MacBook Pro M1
macOS 12.2.1 Monterey
rocksdb 6.29.3
Install rocksdb with brew:
brew install rocksdb
Use brew to look up the path on your system of the libs required for the install of python-rocksdb:
brew info rocksdb lz4 snappy
I had all libs already installed, maybe due to the install of rocksdb. If libs are missing, I guess you need to install them.
Export (for gcc?) the paths of the libs with CPPFLAGS and LDFLAGS and install python-rocksdb with pip.
For me that was (adapt paths according to your setup):
export CPPFLAGS=-I/opt/homebrew/Cellar/rocksdb/6.29.3/include
export LDFLAGS="-L/opt/homebrew/Cellar/rocksdb/6.29.3/lib -L/opt/homebrew/Cellar/snappy/1.1.9/lib -L/opt/homebrew/Cellar/lz4/1.9.3/lib"
pip install python-rocksdb
The exports fixed also the poetry install of a poetry project with faust-streaming-rocksdb (faust-streaming = {extras = ["rocksdb"], version = "^0.8.2"}) as dependency.

I Followed this same procedure just now again in order to successfully install python-rocksdb on my new computer (Apple M1 Max, 2021).

Crucial missing information from my original post: the procedure works only for rocksdb v6. If you run now brew install rocksdb, you install version 7+ and for version 7+ the procedure does not work.

For me rocksdb v6 is fine, so I used this procedure to install rocksdb v6 with brew. I also had to change the path to the lz4 lib, now /opt/homebrew/Cellar/lz4/1.9.4/lib.

Thank you! The procedure you linked worked for me! One caveat: I ended up installing a slightly older version of rocksdb (6.27.3_1 IIRC) because 2.29.3 had some sort of CRC error or something. So first try fail, second try success.

itadventurer · 2022-11-28T16:01:00Z

I had the same problem and for me it seems like the problem is, that python-rocksdb only supports rocksdb 6 and not the latest one (present in Homebrew).

The following approach worked for me (use at your own risk):

Install old Rocksdb

brew tap-new $USER/local-rocksdb
brew extract --version=6.29.3 rocksdb $USER/local-rocksdb
brew install [email protected]

Export Buildflags

export CPPFLAGS=-I/usr/local/Cellar/[email protected]/6.29.3/include
export LDFLAGS="-L/usr/local/Cellar/[email protected]/6.29.3/lib -Lusr/local/Cellar/snappy/1.1.9/lib -L/usr/local/Cellar/lz4/1.9.4/lib"

Install packages

pip install python-rocksdb

alexreg · 2022-11-28T17:40:07Z

@itadventurer This is all covered fully in the above comments.

ly2020888 · 2022-12-20T10:31:11Z

very useful , I appreciate it

Tsunaou · 2022-12-26T07:17:23Z

@itadventurer It's very useful to me.

marghidanu · 2023-05-26T17:09:50Z

This does not work in macOS at all. First, the slice error and then the backupable error.

utahkay · 2023-06-29T16:18:20Z

@itadventurer Your comment was the only thing that worked for me, thank you!

Eliacy · 2023-07-28T08:20:50Z

I tried the itadventurer way but have no success. So I put here some other walk-around choices.

Choice 1: Run x86 Conda environment in M1 Mac

Have a "Rosetta Terminal": In Finder, Right Click the "Terminal.app", Enable "Open using Rosetta" option in "Get Info". Then all command in Terminal will run under Rosetta mode. (If you run "arch", it will return "i386".)
Install the Intel version Anaconda.
Create Python environment, install python-rocksdb under Rosetta Termianl. (I tried Python version 3.9)

conda create -n "myenv" python=3.9
conda activate myenv
conda install -c conda-forge python-rocksdb

Choice 2: Use Arm64 Linux in Docker

I create a Docker image with tag "python:3.9-bullseye", and use the bullseye "python3-rocksdb" deb package. Such as:

docker run -it --name mypy --volume $(pwd):/usr/src/app --net=host py-dev:latest bash
apt install python3-rocksdb

Now /usr/lib/python3/dist-packages/rocksdb is the python-rocksdb lib.

brunneis · 2023-08-15T13:47:05Z

I managed to make it work on a MacBook Air M1 with RocksDB 6.29.3

git clone https://github.com/Homebrew/homebrew-core
cd homebrew-core
git checkout b3597b729a2
brew install Formula/rocksdb.rb
brew pin rocksdb

export CPPFLAGS="-I/opt/homebrew/Cellar/rocksdb/6.29.3/include"
export LDFLAGS="-L/opt/homebrew/Cellar/rocksdb/6.29.3/lib -L/opt/homebrew/Cellar/snappy/1.1.10/lib -L/opt/homebrew/Cellar/lz4/1.9.4/lib"

pip install python-rocksdb

For any other package and version you can check this gist.

e-compagno · 2024-10-10T11:16:26Z

After having rocksdb installed with brew

brew install rocksdb

I tried to install the library with

 export CPPFLAGS="-I/opt/homebrew/Cellar/rocksdb/9.6.1/include"
export LDFLAGS="-L/opt/homebrew/Cellar/rocksdb/9.6.1/lib -L/opt/homebrew/Cellar/snappy/1.2.1/lib -L/opt/homebrew/Cellar/lz4/1.10.0/lib"
pip install python-rocksdb

in MacOs Sequoia (15.0.1) and Python 3.11.9, but there are compiler errors, e.g.

 rocksdb/_rocksdb.pyx:104:11: 'Slice' is not a constant, variable or function identifier
 rocksdb/_rocksdb.pyx:645:40: cimported module has no attribute 'NewBlockBasedTableFactory'
 rocksdb/_rocksdb.pyx:671:55: cimported module has no attribute 'kPlain'
 rocksdb/_rocksdb.pyx:673:55: cimported module has no attribute 'kPrefix'
 rocksdb/_rocksdb.pyx:679:41: cimported module has no attribute 'NewPlainTableFactory'
...

wbarnha mentioned this issue Aug 1, 2022

Update version for 0.9.0 release faust-streaming/python-rocksdb#2

Merged

wbarnha mentioned this issue Jan 3, 2023

Installation instructions for Windows and macOS environments faust-streaming/python-rocksdb#11

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Build Fails on MacOS #47

Build Fails on MacOS #47

mhworth commented Apr 21, 2019

zzzhc commented Jun 22, 2019

cvarma21 commented Oct 4, 2019

deed02392 commented Feb 3, 2020

vincentwyshan commented Apr 13, 2020

mrugacz95 commented May 18, 2020

alexreg commented Feb 22, 2022

faberchri commented Feb 25, 2022 •

edited

Loading

alexreg commented Feb 25, 2022

Littlecowherd commented Mar 4, 2022 •

edited

Loading

rollue commented Mar 24, 2022 •

edited

Loading

alexreg commented Mar 24, 2022

rollue commented Mar 25, 2022

alexreg commented Mar 25, 2022

jrj-d commented May 11, 2022

alexreg commented May 11, 2022 •

edited

Loading

KochankovID commented May 24, 2022

alexreg commented May 25, 2022

faberchri commented Aug 25, 2022

ccz5 commented Oct 16, 2022

alexreg commented Oct 16, 2022

jagalactic commented Nov 6, 2022

itadventurer commented Nov 28, 2022

alexreg commented Nov 28, 2022

ly2020888 commented Dec 20, 2022

Tsunaou commented Dec 26, 2022

marghidanu commented May 26, 2023

utahkay commented Jun 29, 2023

Eliacy commented Jul 28, 2023 •

edited

Loading

brunneis commented Aug 15, 2023 •

edited

Loading

e-compagno commented Oct 10, 2024

Build Fails on MacOS #47

Build Fails on MacOS #47

Comments

mhworth commented Apr 21, 2019

zzzhc commented Jun 22, 2019

cvarma21 commented Oct 4, 2019

deed02392 commented Feb 3, 2020

vincentwyshan commented Apr 13, 2020

mrugacz95 commented May 18, 2020

alexreg commented Feb 22, 2022

faberchri commented Feb 25, 2022 • edited Loading

alexreg commented Feb 25, 2022

Littlecowherd commented Mar 4, 2022 • edited Loading

rollue commented Mar 24, 2022 • edited Loading

alexreg commented Mar 24, 2022

rollue commented Mar 25, 2022

alexreg commented Mar 25, 2022

jrj-d commented May 11, 2022

alexreg commented May 11, 2022 • edited Loading

KochankovID commented May 24, 2022

alexreg commented May 25, 2022

faberchri commented Aug 25, 2022

ccz5 commented Oct 16, 2022

alexreg commented Oct 16, 2022

jagalactic commented Nov 6, 2022

itadventurer commented Nov 28, 2022

alexreg commented Nov 28, 2022

ly2020888 commented Dec 20, 2022

Tsunaou commented Dec 26, 2022

marghidanu commented May 26, 2023

utahkay commented Jun 29, 2023

Eliacy commented Jul 28, 2023 • edited Loading

Choice 1: Run x86 Conda environment in M1 Mac

Choice 2: Use Arm64 Linux in Docker

brunneis commented Aug 15, 2023 • edited Loading

e-compagno commented Oct 10, 2024

faberchri commented Feb 25, 2022 •

edited

Loading

Littlecowherd commented Mar 4, 2022 •

edited

Loading

rollue commented Mar 24, 2022 •

edited

Loading

alexreg commented May 11, 2022 •

edited

Loading

Eliacy commented Jul 28, 2023 •

edited

Loading

brunneis commented Aug 15, 2023 •

edited

Loading