[fix] Fix issues with Pulsar Alpine docker image stability: remove glibc-compat

[lhotari]

Fixes #23717 #23306

Motivation

The Pulsar Alpine docker image has stability issues due to including glibc-package solution which includes glibc library into the Alpine docker image.

The stability issue causes JVM crashes. This happens mainly in Netty native library loading and usage. It's also a problem in other native libraries such as Conscrypt (#23364 (comment)) and Snappy (#22804). There are also Netty stability issues causing JVM crashes which could be caused by problems in native code (#23612 (comment)).

Alpine maintainers don't recommend adding glibc to Alpine since mixing real glibc in Alpine will result in an unstable environment. In Alpine maintainer Ariadne Conill's words: "Combining glibc and musl runtimes is basically all but guaranteed to create an unstable environment, unless the system is appropriately configured (glibc side uses glibc binaries only, and vice versa)."

• blog post by Ariadne Conill, there is no such thing as a "glibc based alpine image"
• comments explaining reasons:
  • Add Alpine glibc images docker-library/official-images#10779 (comment)
  • Add Alpine glibc images docker-library/official-images#10779 (comment)
  • https://gitlab.alpinelinux.org/alpine/aports/-/merge_requests/24647#note_176723

The reason why the glibc solution was initially added was to support Pulsar IO Kinesis connector. The Amazon Kinesis Producer Library isn't fully Java. The Java API calls a native executable which is built for glibc. Amazon doesn't provide a binary for Alpine, however the native executable source code is provided. In order to use Amazon Kinesis Producer Library on Alpine, the stable solution is to compile this binary specifically for Alpine.

Additional context

Mailing list thread

Modifications

This PR includes changes to:

• remove the glibc-package solution

• provide a Amazon Kinesis Producer Library (KPL) executable compiled for Alpine

  • There's a Dockerfile that downloads and compiles KPL 0.15.12 version for Alpine.
    • The resulting docker image has been published to apachepulsar/pulsar-io-kinesis-sink-kinesis_producer:0.15.12.
      • This will need to be updated only when Kinesis Producer Library is upgraded within Pulsar.
  • Changes to build on Alpine have been submitted as a PR to Amazon Kinesis Producer Library so that Amazon would hopefully start supporting Alpine directly in the future: Support building on Alpine Linux musl awslabs/amazon-kinesis-producer#620

• The kinesis_producer executable is copied from apachepulsar/pulsar-io-kinesis-sink-kinesis_producer:0.15.12 image to the apachepulsar/pulsar-all image and an environment variable PULSAR_IO_KINESIS_KPL_PATH is set to the executable path.

• Amazon Kinesis Producer Library has been upgraded from 0.14.13 version to 0.15.12.

  • The 0.14.13 contains several critical issues such as a potential data loss issue
  • In 0.15.12, the implementation uses AWS STS (Security Token Service) under the covers.
    • It is necessary to update the unit and integration tests Localstack configuration to include STS support and overriding the endpoints in the Pulsar IO Kinesis Sink connector

• The Pulsar IO Kinesis Sink connector has been modified to support the AWS Kinesis Producer Library parameter nativeExecutable. When the PULSAR_IO_KINESIS_KPL_PATH env var is set, it will be set to the nativeExecutable parameter as the default value. This is how the Pulsar IO Kinesis Sink connector will use the kinesis_producer binary compiled for Alpine when using the pulsar-all image.

• In the Pulsar docker image, LD_PRELOAD=/lib/libgcompat.so.0 is set. This is required to support loading Netty native libraries in Alpine unless the JVM already loads the gcompat library which provides a glibc compatibility layer for Alpine. The Alpine gcompat library is the recommended option for Netty native libraries on Alpine.

  • More details in comments:
    • grpc-java upgrade from 1.53.0 to 1.54.0 crashes JVM on Alpine grpc/grpc-java#10096 (comment)
    • __getauxval symbols cannot be found after 2.0.66 on musl Linux netty/netty-tcnative#907 (comment)
    • [Bug] Pulsar 4.0.1 image crashed when loading the native SSL library #23717 (comment)

Documentation

• doc
• doc-required
• doc-not-needed
• doc-complete

[heesung-sn]

In order to use Amazon Kinesis Producer Library on Alpine, the stable solution is to compile this binary specifically for Alpine.

I think it would be hard to maintain this build script in this pulsar repo.

nit: are we planning to export those IO connectors from the repo and maintain them separately?

[lhotari]

In order to use Amazon Kinesis Producer Library on Alpine, the stable solution is to compile this binary specifically for Alpine.

I think it would be hard to maintain this build script in this pulsar repo.

@heesung-sn sure, it's possible to say that. The Amazon Kinesis Producer library doesn't provide Alpine support and there aren't other feasible options. I have contributed changes to the AWS repository so hopefully support would be added later and we could remove our custom solution. Please continue the review.

nit: are we planning to export those IO connectors from the repo and maintain them separately?

Yes, that was decided already in 2020. 😀

[heesung-sn]

LGTM.

[heesung-sn]

nit: please provide more reference links(like dependency location list ) as comments in the build scripts for future updates.