Include kinesis_producer binary for Alpine in pulsar-functions-java-runner image

[lhotari]

Background

Running Kinesis Sink with the current Alpine image will fail with this type of errors:

WARN  com.amazonaws.services.kinesis.producer.LogInputStreamReader - Error relocating /tmp/amazon-kinesis-producer-native-binaries/kinesis_producer_B2A87159E60ABF99DE719370CD9483CFC815BAC9: _Unwind_Resume_or_Rethrow: symbol not found
WARN  com.amazonaws.services.kinesis.producer.LogInputStreamReader - Error relocating /tmp/amazon-kinesis-producer-native-binaries/kinesis_producer_B2A87159E60ABF99DE719370CD9483CFC815BAC9: sys_siglist: symbol not found

The Kinesis Java client uses a binary called Kinesis Producer Library which is compiled for glibc and requires also multiple libraries. It doesn't support Alpine musl and related libraries.

Solution

The issue has been fixed in Apache Pulsar with apache/pulsar#23762.

In Apache Pulsar, the Kinesis Producer Library is compiled for Alpine musl with this Dockerfile:
https://github.com/apache/pulsar/tree/master/docker/kinesis-producer-alpine

This binary is installed in the pulsar-all image in the Dockerfile in this way:

# Copy the kinesis_producer native executable compiled for Alpine musl to the pulsar-all image
# This is required to support the Pulsar IO Kinesis sink connector
COPY --from=pulsar-io-kinesis-sink-kinesis_producer /opt/amazon-kinesis-producer/bin/kinesis_producer /opt/amazon-kinesis-producer/bin/.os_info /opt/amazon-kinesis-producer/bin/.build_time /opt/amazon-kinesis-producer/bin/.revision /opt/amazon-kinesis-producer/bin/.system_info /opt/amazon-kinesis-producer/bin/.version /opt/amazon-kinesis-producer/bin/
# Set the environment variable to point to the kinesis_producer native executable
ENV PULSAR_IO_KINESIS_KPL_PATH=/opt/amazon-kinesis-producer/bin/kinesis_producer
# Install the required dependencies for the kinesis_producer native executable
USER 0
RUN apk update && apk add --no-cache \
    brotli-libs \
    c-ares \
    libcrypto3 \
    libcurl \
    libgcc \
    libidn2 \
    libpsl \
    libssl3 \
    libunistring \
    nghttp2-libs \
    zlib \
    zstd-libs \
    libuuid
USER 10000

The pulsar Docker image contains these lines to install gcompat which is also necessary.

The Kinesis Sink has been modified to use the PULSAR_IO_KINESIS_KPL_PATH for choosing the native binary:
https://github.com/apache/pulsar/blob/a4a34091cbf58bfa080b42c101bd5094d82c41f3/pulsar-io/kinesis/src/main/java/org/apache/pulsar/io/kinesis/KinesisSinkConfig.java#L107-L115
That's how it uses the native binary compiled for Alpine and which is included in the image at /opt/amazon-kinesis-producer/bin/kinesis_producer.

Similar changes should be made in images/pulsar-functions-java-runner/Dockerfile so that Pulsar IO Kinesis Sink connector would be supported.

[lhotari]

Backport for PULSAR_IO_KINESIS_KPL_PATH for Pulsar 3.0.x: apache/pulsar#24349

[lhotari]

AWS Kinesis Producer Library (KPL) versions <1.0 will be deprecated at the end of January 2026. To address this, the KPL version is being updated to 1.0.4 in apache/pulsar#24661.
The pulsar-functions-java-runner image should include a matching version 1.0.4 of the kinesis_producer binary for Alpine.

[lhotari]

It seems that this was addressed in #809, but this issue wasn't closed.