Runtime missing for Apple Silicon (M1)

[igorgiovannini]

Description

The NuGet package for .NET projects doesn't include the ARM runtime for macOS (osx-arm64) and therefore it doesn't work.

https://nuget.info/packages/librdkafka.redist/1.7.0

How to reproduce

Runtime is missing

Checklist

IMPORTANT: We will close issues where the checklist has not been completed.

Please provide the following information:

• librdkafka version (release number or git tag): 1.7.0
• Apache Kafka version: <REPLACE with e.g., 0.10.2.3>
• librdkafka client configuration: <REPLACE with e.g., message.timeout.ms=123, auto.reset.offset=earliest, ..>
• Operating system: macOS 11.4, Apple M1
• Provide logs (with debug=.. as necessary) from librdkafka
• Provide broker log excerpts
• Critical issue

[mikebell90]

Remains the case for librdkafka 1.8.2

[JSanchezIO]

Will this be included in the 1.9.0 release?

[edenhill]

We're currently transitioning CI systems, so M1 builds will unfortunately miss the upcoming 1.9 release, but will be included in the next release after that.

[niemyjski]

@edenhill any updates? How can m1 users get unblocked?

[edenhill]

No update. We're probably moving to Semaphore CI, but it too lacks M1 workers, so we'll see.

[niemyjski]

@edenhill Regardless of CI, looks like tons of people are having this issue including core team members. Any reason this can't be fixed regardless of CI status? I think most of the community would help test this in the mean time. Even call it alpha support on a stable release if you have to.

[edenhill]

We use CIs to verify builds&tests, as well as create build artifacts, as they provide an isolated, dependable, audited, and integrated (with Github, etc) environment.
So until there's a viable CI that provides M1 workers, there isn't much we can do. But we're actively working on getting such support on Semaphore to unblock you all.

[niemyjski]

I get that, one could also commit prebuilt from a signed commit that could be trusted and audited for the short term, not ideal... also why not use this actions/runner-images#2187 with self hosted runner?

[rhenwood-arm]

Hi, I've just noticed this thread... I'm curious how significant the M1 hardware is - and if a general AArch64 Linux platform would help fill a gap?

[timmc-edx]

M1 is significant because Apple is pushing it quite hard, and apparently it's now hard for developers to get new Intel-based Macs even though so many things are not yet compatible with M1.

linux-aarch64 would be perfect for us because the M1 users would actually be running this under Docker, on a Linux VM.

[adamdecaf]

M1 is significant because Apple is pushing it quite hard, and apparently it's now hard for developers to get new Intel-based Macs even though so many things are not yet compatible with M1.

Can confirm this impacts a lot of development teams.

[jonchiu]

M1 support is available as of https://github.com/edenhill/librdkafka/releases/tag/v1.9.1

[mhowlett]

M1 support is available OOTB in 1.9.1 for OSX. We don't yet bundle a linux-aarch64 librdkafka build. It seems there is sufficient demand though, so we probably will in the future. Confluent.Kafka should work with a custom linux-aarch64 librdkafka build.

[adamdecaf]

Gotcha. We're trying to release Docker images for Intel and M1 macbooks to run. Running the code (tests) locally has been working for a while, but the ARM containers fail to find librdkafka.so.1

[ChillarAnand]

M1 support is available OOTB in 1.9.1 for OSX. We don't yet bundle a linux-aarch64 librdkafka build. It seems there is sufficient demand though, so we probably will in the future.

Any ETA for this? @mhowlett

[ChillarAnand]

@mhowlett What needs to be done to provide support for linux-aarch64 builds?

[ChillarAnand]

@jonchiu Any plans/roadmap on adding linux-aarch64 builds? Let me know if I can be of any help.

[ChillarAnand]

@edenhill I am not familiar with the codebase. Not sure if this is a minor or major issue.

Just checking if there are any plans to take it up soon.

[ChillarAnand]

With ChatGPT help and after a bit of trial and error, I managed to build linux-aarch64 wheels using the below Dockerfile

FROM ubuntu:22.04


ARG DEBIAN_FRONTEND=noninteractive

RUN apt update && apt install -y wget git curl g++ make postgresql-client \
                                 nano less shared-mime-info openjdk-17-jre-headless \
                                 libpq-dev vim tzdata python3 python3-dev


RUN apt install -y python3-pip
RUN python3 -m pip install setuptools

WORKDIR /
RUN git clone https://github.com/confluentinc/confluent-kafka-python
WORKDIR confluent-kafka-python

COPY . /app
WORKDIR /app
RUN ./configure --arch=aarch64 --prefix=/usr
RUN make
RUN make install

WORKDIR /confluent-kafka-python
RUN python3 setup.py install