[Bug]: fluvio-socket v0.14.6 doesn't compile in Docker unless git is installed.

[marvin-hansen]

What happened

I am trying to compile fluvio in Docker while building a release container.

However, the compile in Docker fails UNLESS git is installed.

FIX / Workaround: Install git in the build container.

See relevant Dockerfile at the bottom.

Expected behavior

I would expect fluvio to compile in Docker without a hard
dependency on git. I don't think I ever had to install git in a build container to compile any other published crate.

The error originates from the Build script in the fluvio-socket crate that uses a git checkout.

What puzzles me is that the build script in question is pinned to HEAD of the remote repo instead of a specific commit hash,
which makes the entire build non-hermetic and non-deterministic. This is problematic because you never know
with which commit fluvio was actually build so this really is a problem that only waits to bite you one day.

The way I see the situation, this might be relatively easy to fix with a proper versioning of dependencies
instead of using git and pinning to head.

However, if a proper solution is infeasible for any reason, please update the fluvio-socket build script
in such a way that it tests whether git is installed and if not, issues a meaningful error message
that makes it clear that git must be installed for the build to succeed. This would have saved me a bit of time
tracing down the root cause of the build fail.

Describe the setup

• Local install.
• Rust Fluvio: 0.21
• What version of Fluvio are you using? fluvio version.
  Fluvio CLI : 0.11.2
  Fluvio CLI Arch : aarch64-apple-darwin
  Fluvio CLI SHA256 : 2db084880a36e9aeed5b28e67f78fa2df3531558e6d8bd9e36a355ca970654ac
  Fluvio channel frontend SHA256 : 2db084880a36e9aeed5b28e67f78fa2df3531558e6d8bd9e36a355ca970654ac
  Fluvio Platform : 0.11.2 (local)
  Git Commit : cddc053
  OS Details : Darwin 14.2 (kernel 23.2.0)

How to reproduce it (as minimally and precisely as possible)
Steps to reproduce the behavior:

1. Make a minimal fluvion app or copy over a code example.
2. Copy the Docker file from below
3. Build the app in Docker with the Dockerfile

The build script that causes the error:

Lines 15 - 21
https://github.com/infinyon/fluvio/blob/master/crates/fluvio-socket/build.rs

Log output

106.2 error: failed to run custom build command for `fluvio-socket v0.14.6`
106.2 note: To improve backtraces for build dependencies, set the CARGO_PROFILE_RELEASE_BUILD_OVERRIDE_DEBUG=true environment variable to enable debug information generation.
106.2
106.2 Caused by:
106.2   process didn't exit successfully: `/app/target/release/build/fluvio-socket-bcc9b2c7ebbaf90d/build-script-build` (exit status: 101)
106.2   --- stdout
106.2   cargo:rerun-if-changed=build.rs
106.2
106.2   --- stderr
106.2   thread 'main' panicked at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/fluvio-socket-0.14.6/build.rs:18:10:
106.2   should run 'git rev-parse HEAD' to get git hash: Os { code: 2, kind: NotFound, message: "No such file or directory" }
106.2   stack backtrace:
106.2      0:     0xaaaabd83d560 - <unknown>
106.2      1:     0xaaaabd85c7d4 - <unknown>
106.2      2:     0xaaaabd83ad5c - <unknown>
106.2      3:     0xaaaabd83d394 - <unknown>
106.2      4:     0xaaaabd83e8a0 - <unknown>
106.2      5:     0xaaaabd83e5b4 - <unknown>
106.2      6:     0xaaaabd83ee5c - <unknown>
106.2      7:     0xaaaabd83ed24 - <unknown>
106.2      8:     0xaaaabd83da44 - <unknown>
106.2      9:     0xaaaabd83ea7c - <unknown>
106.2     10:     0xaaaabd819d34 - <unknown>
106.2     11:     0xaaaabd81a01c - <unknown>
106.2     12:     0xaaaabd81adec - <unknown>
106.2     13:     0xaaaabd81a354 - <unknown>
106.2     14:     0xaaaabd81a368 - <unknown>
106.2     15:     0xaaaabd837a44 - <unknown>
106.2     16:     0xaaaabd81af60 - <unknown>
106.2     17:     0xffffbc6a1da4 - __libc_start_main
106.2     18:     0xaaaabd81a27c - <unknown>
106.2 warning: build failed, waiting for other jobs to finish...

Environment (please complete the following information):

• OS: MacOS 14.2 / M3 Macbook
• Docker Desktop 4.26.0 (130397)
• Rust: 1.74
  fluvio cluster status
  📝 Running cluster status checks with profile local
  ✅ SC is ok
  ✅ (1/1) SPUs are online
  ✅ 1 topic using 0 B

Additional context

Dockerfile:

############################
# Build image
# This Dockerfile runs relative to the repo root.
# Therefore, copy it there before running.
############################
FROM rust:1.74-slim-buster as builder

# Create the user and group files to run the binary
# in the final scratch container as an unprivileged user.
RUN mkdir /user && \
    echo 'nobody:x:65534:65534:nobody:/:' > /user/passwd && \
    echo 'nobody:x:65534:' > /user/group

# Install protobuf compiler so that
RUN apt-get update && apt-get install -y protobuf-compiler libprotobuf-dev

## FIX: Ucomment to make the build work by installing git 
# RUN apt-get update && apt-get install -y git protobuf-compiler libprotobuf-dev

# Install build target musel for compiling a static binary
RUN rustup target add aarch64-unknown-linux-musl

# Set workspace directory
WORKDIR /app

# Copy over the entire source code. Data are excluded in .Dockerignore
COPY . ./

# Fix linker issue with musel that affects both, AMD64 and ARM64 (aarch64).
# https://github.com/rust-lang/stacker/issues/80
ARG CARGO_TARGET_AARCH64_UNKNOWN_LINUX_MUSL_LINKER=aarch64-linux-gnu-gcc
ARG CC=aarch64-linux-gnu-gcc

# Download all dependencies
RUN cargo fetch

# Because of Issue #10174 in SurrealDB, Docker requires at least 4GB of memory to build the image.
# When memory is low, the docker build aborts after 15+ min with (signal: 9, SIGKILL: kill)
# Just set memory limit in Docker to at least 4GB, better 12GB, and the issue is gone.
# https://github.com/google/oss-fuzz/issues/10174

# Run the release build.
RUN RUST_BACKTRACE=full cargo build -p qdgw --release --target aarch64-unknown-linux-musl

# Move binary up to root level directory for easy access
RUN mv /app/target/aarch64-unknown-linux-musl/release/qdgw /qdgw

############################
# Scratch image
############################
# Create minimal docker image
FROM scratch as runner

# Import user and group files from the build stage.
COPY --from=builder /user/group /user/passwd /etc/

# Import the CAcertificates from the build stage to enable HTTPS.
COPY --from=builder /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/

# Copy binary from build output directory
COPY --from=builder /qdgw /

# Port number must match deployment.yaml and specs/*/service_info
# service port: 4040
# metrics port: 8080
EXPOSE 4040 8080

# Run binary as unprivileged user
USER nobody:nobody

# Hard coded start command b/c no shell in scratch image
CMD ["/qdgw"]

[ajhunyady]

@marvin-hansen thanks for reporting, we'll take a look.

Just an FYI, we also have pre-compiled Docker images. https://infinyon.com/docs/tutorials/docker-installation/

[marvin-hansen]

That's cool, thank you. To be clear, my release container compiles and works. It's just the git dependency that raises an eyebrow. Frankly I've never seen this before. Out of > 650 total dependencies that get compiled for my project, only this one has this weirdo git build script and I think there might be a different way without requiring git.

Otherwise, great job. This project rocks.

[morenol]

If am not mistaken we use git only to get the git rev here and also in the fluvio crate:

fluvio/crates/fluvio-socket/build.rs

Line 15 in 682930c

let git_version_output = Command::new("git")

I think that we could replace that with gitoxide

[marvin-hansen]

Thanks for the fast resolution.

…

On Wed, Dec 20, 2023 at 10:15 PM morenol ***@***.***> wrote: Closed #3787 <#3787> as completed via #3789 <#3789>. — Reply to this email directly, view it on GitHub <#3787 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFYR7XGV6ZKL36TMEBGV5M3YKLXJNAVCNFSM6AAAAABA24KFXSVHI2DSMVQWIX3LMV45UABCJFZXG5LFIV3GK3TUJZXXI2LGNFRWC5DJN5XDWMJRGMYDGMJYG42DOMI> . You are receiving this because you were mentioned.Message ID: ***@***.***>