bundle librdkafka binaries for aarch64 #1326
Comments
|
From the errors, looks it's related to gcc, have you already installed it? |
|
I can confirm that My assamption is that some binaries are bundled for Possibly related issues for Go version of the package: |
|
Why are there no prebuilt wheels for arm64? |
|
It's not a perfect world. But if you wanna test Docker images in Apple Silicon (M1), you have to compile in ARM64 binaries using multi-stages: ARG SPARK_IMAGE=spark-base:latest
FROM ${SPARK_IMAGE}
USER root
RUN apt update && apt install -y git
# Need gcc, make and these stuffs to compile
RUN cd /tmp && git clone https://github.com/edenhill/librdkafka.git && \
cd librdkafka && git checkout tags/v1.9.0 && \
./configure && make && make install && \
cd ../ && rm -rf librdkafkaThen you copy to your last build image before COPY --from=0 /usr/local/lib /usr/local/lib
COPY --from=0 /usr/local/include /usr/local/include
COPY --from=0 /usr/local/share/doc/librdkafka /usr/local/share/doc/librdkafka
RUN pip install confluent-kafka |
|
hey @sibeream , this is very valid request! I'm thinking of creating a wrapper for this package that will have both binaries
|
|
I have built confluent kafka wheels for different platforms (Mac ARM, linux ARM) and made instructions on how to do that in this forked repo |
I tested your wheel for aarch64 and getting the following error message on start:
|
|
@b3n3w thx for noticing |
|
This run successfully on my Apple M1. Could someone confirm if this is useful? (based on @ignitz 's comment) |
|
This allows me to install on my M1 Mac: |
|
tracking aarch64 binary wheel support in #1439 |
Thanks for these. I am using Apple M1 and It makes the installing working but still i got I added these lines in my dockerfile and it works |
|
Looks like version And version |
Description
docker run -ti python:3.8-slim-buster /bin/bashpip install confluent-kafka==1.8.2The above commands produce different results on x86_64 and aarch64 platforms.
Operating systems in use:
For x86_64 confluent-kafka is just installed inside the container, for aarch64 the output is the following:
I assume the reason for the discrepancy in the behaviour is that some binaries are bundled for x86_64, but are missing for aarch64.
Is it possible to prebuild and bundle those binaries from your side?
The discrepancy in deliverables make it far more complicated to support different architectures for users of the package.
How to reproduce
Run the following commands on aarch64 platform (Apple M1 for instance)
docker run -ti python:3.8-slim-buster /bin/bashpip install confluent-kafka==1.8.2Checklist
Please provide the following information:
confluent_kafka.version()andconfluent_kafka.libversion()):{...}'debug': '..'as necessary)The text was updated successfully, but these errors were encountered: