Docker image >= 1.2.0 broken on M1 Mac?

[m-o-e]

Bug Report

Trying to build in docker with latest-alpine, nightly-alpine
or v1.2.*-alpine gives me the following error:

$ echo 'puts "Hello World!"' > hello-world.cr
$ docker run --platform linux/amd64 --rm -it -v $(pwd):/workspace -w /workspace crystallang/crystal:latest-alpine crystal build hello-world.cr
Missing executable path to expand $ORIGIN path (Exception)
  from ???
  from ???
  from ???
  from ???
  from ???
  from ???
  from ???
  from ???
  from ???
  from ???
Error: you've found a bug in the Crystal compiler. Please open an issue, including source code that will allow us to reproduce the bug: https://github.com/crystal-lang/crystal/issues

Building with v1.1.1-alpine works fine.

[straight-shoota]

I can't reproduce this using the same command but with explicit version for the docker image (crystallang/crystal:1.2.2-alpine) for consistency. Same for current nightly image (Crystal 1.3.0-dev [5455e807b] (2022-01-04)).

This error means the compiler can't figure out the path of its executable, which on Linux is just resolving the symlink /proc/self/exe. That's unlikely to fail. If it does, I'd expect file system issues.

[m-o-e]

Hm, fwiw, I'm using Docker on a M1 Mac (but as seen above, docker runs
in amd64 emulation, so I think in theory it shouldn't make a difference).

Can consistently reproduce it here. v1.1.1-alpine works, anything higher does not.
(using the exact same command, changing only the image version number)

Maybe it's indeed just a weird Docker-on-M1-Mac issue (wouldn't be the first). 🤔
I've added that info to the title for now - would be great if someone else on a Mac
could try to reproduce.

[beta-ziliani]

Can't reproduce in an Intel Mac (macOS 11.6.2)

[m-o-e]

Someone on a M1 Mac I should add. (but thanks nonetheless! 🙂)

Some new findings (also updated the description above accordingly):

• It's actually not related to --static at all.
  Trying to build without that flag gives me the same error.

• It seems to be a general M1 Mac issue
  (has the underlying alpine image been updated between 1.1.1 -> 1.2.0? 🤔 )

• crystal --help works fine for me in all image versions.
  crystal build and crystal run give the above error.

• The /proc/self/exe symlink looks the same to me in all image versions:

$ docker run --rm -it -v $(pwd):/workspace -w /workspace crystallang/crystal:1.2.2-alpine ls -l /proc/self/exe
lrwxrwxrwx    1 root     root             0 Jan  5 17:13 /proc/self/exe -> /bin/busybox
$ docker run --rm -it -v $(pwd):/workspace -w /workspace crystallang/crystal:1.1.1-alpine ls -l /proc/self/exe
lrwxrwxrwx    1 root     root             0 Jan  5 17:13 /proc/self/exe -> /bin/busybox

[straight-shoota]

To investigate further: Using the working 1.1.1 image, you could build a dev version of the compiler with some strategic puts calls to log what's going on. Just make sure to link it statically, then mount it in the 1.2.2 image.
A simple program with just puts Process.executable_path could already be helpful as well.

The base image has actually been downgraded from alpine:3.13 to alpine:3.12 between 1.1.1 and 1.2.2 (crystal-lang/distribution-scripts#127).

[m-o-e]

@straight-shoota

Using the working 1.1.1 image, you could build a dev version of the compiler

After some more dabbling I'm afraid that may not provide further insight.
For curiosity I grabbed a copy of /usr/bin/crystal from the 1.1.1 and 1.2.2 images,
copied them to the other one and tried to run them there.

27c9d7e0edd78c61da08713e9676e89b  /usr/bin/crystal (from 1.1.1-alpine)
45c45385525d8517adbd3f9d7995dd9a  /usr/bin/crystal (from 1.2.2-alpine)

The 1.1.1 binary works fine in both containers.
The 1.2.2 binary fails in both containers.

The bug seems to be in the crystal-binary, so a dev-version
built in the 1.1.1 image would likely behave just fine
when run under the 1.2.2 image.

tldr, it looks like:

• When crystal is built inside the 1.1.1 image on a x86-host
  it runs fine on a m1-host.

• When crystal is built inside the 1.2.2 image on a x86-host
  it fails on a m1-host.

• Might be interesting to try what happens when crystal
  is built inside the 1.2.2 image on a m1-host.
  (Will it then work on m1 but fail on x86? 🤔)

Unfortunately don't have time to dig deeper into this atm
(for my own usage I'll just stick with 1.1.1 for now).

Would also be inclined to suggest that nobody else
spend more time on this before someone other than me
can confirm that it fails on M1 for them, too. 🙈

(don't want to send you down rabbit holes and in the end it turns
out it was just some qemu/docker weirdness on my particular machine here,
i guess docker on M1 is still to be considered experimental after all...)

[noahlh]

Just want to chime in here - I'm on an M1 Mac trying to build a dev version of the compiler and I'm running into this same error. I'm coming at this somewhat laterally, so apologies if I'm jumping in at the wrong point. But I'm running Docker Desktop 4.3.2 (72729) on an M1 MacBook Pro (2021). Grabbed the 1.3.0-dev-build docker image, the latest source, and tried running simply make and get the following output:

root@03cc6f9723a9:~/crystal# make 
Using /usr/bin/llvm-config-10 [version= 10.0.0]
g++ -c  -o src/llvm/ext/llvm_ext.o src/llvm/ext/llvm_ext.cc -I/usr/lib/llvm-10/include -std=c++14   -fno-exceptions -D_GNU_SOURCE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS
CRYSTAL_CONFIG_BUILD_COMMIT="a3ee70ca0" CRYSTAL_CONFIG_PATH='$ORIGIN/../share/crystal/src' SOURCE_DATE_EPOCH="1641471908" CRYSTAL_CONFIG_LIBRARY_PATH='$ORIGIN/../lib/crystal' ./bin/crystal build -D strict_multi_assign -Dwithout_interpreter  -o .build/crystal src/compiler/crystal.cr -D without_openssl -D without_zlib
Missing executable path to expand $ORIGIN path (Exception)
  from ???
  from ???
  from ???
  from ???
  from ???
  from ???
Error: you've found a bug in the Crystal compiler. Please open an issue, including source code that will allow us to reproduce the bug: https://github.com/crystal-lang/crystal/issues
/usr/bin/ld: cannot find -lgc (this usually means you need to install the development package for libgc)
collect2: error: ld returned 1 exit status
Error: execution of command failed with code: 1: `cc "${@}" -o /root/.cache/crystal/root-crystal-src-ecr-process.cr/macro_run  -rdynamic -lpcre -lm -lgc -lpthread -levent -lrt -ldl`
make: *** [Makefile:193: .build/crystal] Error 1

[alexkutsan]

Any progress on it?
I can't stay on 1.1.1 anymore so I found a workaround for me in https://github.com/84codes/crystal-container-images

I don't know why but it works fine :

➜  echo 'puts "Hello World!"' > hello-world.cr
➜  docker run --platform linux/amd64 --rm -it -v $(pwd):/workspace -w /workspace 84codes/crystal:1.3.1-alpine-latest hello-world.cr
Hello World!

Also there are aarch64 images available and they work faster on M1

amd64:

➜  time docker run --platform linux/amd64 --rm -it -v $(pwd):/workspace -w /workspace 84codes/crystal:1.3.1-alpine-latest hello-world.cr
Hello World!
docker run --platform linux/amd64 --rm -it -v $(pwd):/workspace -w /workspace  0.10s user 0.05s system 1% cpu 11.461 total

arm64:

➜  time docker run --platform linux/arm64 --rm -it -v $(pwd):/workspace -w /workspace 84codes/crystal:1.3.1-alpine-latest hello-world.cr
Hello World!
docker run --platform linux/arm64 --rm -it -v $(pwd):/workspace -w /workspace  0.10s user 0.04s system 3% cpu 3.491 total

[Sija]

Also there are aarch64 images available and they work faster on M1

@alexkutsan That's because anything other than arm64 will use QEMU under-the-hood.

[raphj]

I'm seeing the same problem in an amd64 Debian Bullseye systemd container running on an ARM64 board running Armbian.
The container is run by qemu-user.

I see the problem with crystal 1.3 and 1.2 but crystal 1.1 works.

It's probably unrelated to Docker since I don't use it. It might be related to qemu though.

[raphj]

Same problem with crystal 1.4.

However, copying the following files from the x86_64 Debian bullseye docker image from https://github.com/84codes/crystal-container-images (thanks @alexkutsan]) in my x86_64 container worked:

• /usr/bin/crystal
• /usr/bin/shards
• /usr/share/crystal

I suppose directly using the arm64 version in a native container or in the main system would work too.
edit: yes, it works!

Could it be that the ORIGIN / CRYSTAL_CONFIG_PATH values are somehow incorrect? It seems like the error is coming from

crystal/src/compiler/crystal/crystal_path.cr

Line 51 in ef05e26

raise "Missing executable path to expand $ORIGIN path"

The difference I can see between running crystal natively and in qemu is that the executable_path value could be different (pointing to qemu when we are in a x86 container).

[martinmares]

I don't know why, but building my docker image (multistage build, on Apple M1 Max chip) with latest docker hub crystal (tagged "crystallang/crystal:latest") ending with this error:

...
#0 1.446 Error target some-binnary-app1 failed to compile:
#0 1.446 Missing executable path to expand $ORIGIN path (Exception)
#0 1.446   from ???
#0 1.446   from ???
#0 1.446   from ???
#0 1.446   from ???
#0 1.446   from ???
#0 1.446   from ???
#0 1.446   from ???
#0 1.446   from ???
#0 1.446   from ???
#0 1.446   from ???
#0 1.446 Error: you've found a bug in the Crystal compiler. Please open an issue, including source code that will allow us to reproduce the bug: https://github.com/crystal-lang/crystal/issues
...

And building with docker hub image from "84codes" (tagged 84codes/crystal:latest-ubuntu-22.04) ending with result OK.

My Dockerfile look like this:

FROM 84codes/crystal:latest-ubuntu-22.04 AS utils

RUN ... git clone some repos from github ... && \
    ... and building some binary apps (staticaly linked) ... with command like this:
    ... shards build --production --static ...

FROM alpine

RUN apk update --no-cache && \
    apk upgrade --no-cache && \
    apk add --quiet --no-cache git curl ruby ruby-dev build-base tzdata ca-certificates docker && \
    rm -rf /var/cache/apk/* && \
    echo 'gem: --no-document' > $HOME/.gemrc

COPY --from=utils /app/bin/some-binnary-app1 /bin/some-binnary-app1
COPY --from=utils /app/bin/some-binnary-app2 /bin/some-binnary-app2
COPY --from=utils /app/bin/some-binnary-app3 /bin/some-binnary-app3

RUN gem update --system && \
    gem install bundler && \
    bundle

Difference is only with this line FROM crystallang/crystal:latest AS utils or FROM 84codes/crystal:latest-ubuntu-22.04 AS utils.

Build command is:

$ docker buildx build --platform linux/amd64 -t sample/base-ruby-image .

[defido]

Got same issue M1 Air, also having issues with fly.io. Might be the docker config.

[mloughran]

Just read through this issue and can confirm that this issue affects both the alpine & non-alpine builds (M1 mac, Docker version 20.10.16).

~ % docker run crystallang/crystal:1.1.1-alpine crystal eval 'puts "hello"'
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
hello

~ % docker run crystallang/crystal:1.1.1 crystal eval 'puts "hello"'
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
hello

~ % docker run crystallang/crystal:1.2.0-alpine crystal eval 'puts "hello"'
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
Missing executable path to expand $ORIGIN path (Exception)
  from ???
  from ???
  from ???
  from ???
  from ???
  from ???
  from ???
  from ???
  from ???
  from ???
  from ???
  from ???
Error: you've found a bug in the Crystal compiler. Please open an issue, including source code that will allow us to reproduce the bug: https://github.com/crystal-lang/crystal/issues

~ % docker run crystallang/crystal:1.2.0 crystal eval 'puts "hello"'
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
Missing executable path to expand $ORIGIN path (Exception)
  from ???
  from ???
  from ???
  from ???
  from ???
  from ???
  from ???
  from ???
  from ???
  from ???
  from ???
  from ???
Error: you've found a bug in the Crystal compiler. Please open an issue, including source code that will allow us to reproduce the bug: https://github.com/crystal-lang/crystal/issues

Same issue on crystal 1.4.1.

Where can the Dockerfiles used to generate docker hub images be found (in case these contain a clue)?

[beta-ziliani]

Thanks for looking into it @mloughran , here are the docker files.

[jgaskins]

Been running into this, too. Looks like it happens here, which looks to be caused by Process.executable_path returning falsy.

It works fine on my M1 without Docker, so maybe since this involves a system call, this bug could be specific to the combination of arm64+linux, and not necessarily Docker+M1?

[daliborfilus]

I'd like to help with this issue. Crystal arm64 m1 builds from macports work beautifully, the issues I have are during docker image build, ending with the same message as this issue.
I tried to build my images on top of crystallang/crystal using both amd64 and arm64 platform as a base (using docker buildx build --platform...; apt-get install curl pulls amd64 or arm64 packages, so I know I'm on the right platform), but the crystallang/crystal images are amd64 only, so it seems that doesn't matter.

To build my own arm64 linux crystal image, should I try to use as the base these Dockerfiles and Makefile https://github.com/crystal-lang/distribution-scripts/tree/master/docker
or these? https://github.com/crystal-lang/distribution-scripts/blob/master/linux/

I don't know what's their respective purposes. It looks to me like the first one is meant to built the crystallang/crystal docker images, and the second is using docker only to isolate the build process from the rest of the system?

Or is the crystallang/crystal image built some different way?

[straight-shoota]

I don't know what's their respective purposes. It looks to me like the first one is meant to built the crystallang/crystal docker images, and the second is using docker only to isolate the build process from the rest of the system?

Exactly.

[daliborfilus]

Ok. So I wanted to try building crystal using the distribution-scripts/linux (even though it doesn't allow me to patch the source code without modifications), just to see if I can even build the thing.

All these builds use default docker platform, i.e. arm64/v8 on the M1 mac.

Building with PREVIOUS_CRYSTAL_VERSION > 1.1.1 (I tried 1.2.0, 1.3.1) ends up with crystal compilation error:

make CRYSTAL_VERSION=1.4.1 PREVIOUS_CRYSTAL_VERSION=1.2.0
# or: make CRYSTAL_VERSION=1.4.1 PREVIOUS_CRYSTAL_VERSION=1.3.1
# or: make CRYSTAL_VERSION=1.4.1 PREVIOUS_CRYSTAL_VERSION=1.4.0

# [ cut ]

> [stage-1  8/11] RUN git clone https://github.com/crystal-lang/crystal  && cd crystal  && git checkout 1.4.1   && make crystal stats=true static=true ${release:+release=true}                  CRYSTAL_CONFIG_TARGET=x86_64-unknown-linux-gnu  && ([ "$(ldd .build/crystal | wc -l)" -eq "1" ] || { echo './build/crystal is not statically linked'; ldd .build/crystal; exit 1; }):                                                                                             

# [ cut ]

#13 14.41 HEAD is now at b7377c041 Add Changelog for 1.4.1 (#12017)
#13 14.86 Using /usr/bin/llvm-config [version= 10.0.0]g++ -c  -o src/llvm/ext/llvm_ext.o src/llvm/ext/llvm_ext.cc -I/usr/lib/llvm10/include -std=c++14   -fno-exceptions -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS
#13 36.52 CRYSTAL_CONFIG_BUILD_COMMIT="b7377c041" CRYSTAL_CONFIG_PATH='$ORIGIN/../share/crystal/src' SOURCE_DATE_EPOCH="1650653870" CRYSTAL_CONFIG_LIBRARY_PATH='$ORIGIN/../lib/crystal' ./bin/crystal build -D strict_multi_assign --stats --static -Dwithout_interpreter  -o .build/crystal src/compiler/crystal.cr -D without_openssl -D without_zlib
Parse:                             00:00:00.004335208 (   0.76MB)
#13 38.21 Failed to raise an exception: END_OF_STACK
#13 38.21 [0x4000c1ad56] ???
#13 38.21 [0x40002916e8] ???
#13 38.21 [0x40002a50f3] ???
#13 38.21 [0x4000c1a6c4] ???
#13 38.21 [0x4001c98cf8] ???
#13 38.21 
#13 38.21 Tried to raise:: Error writing file: Broken pipe (IO::Error)
#13 38.21   from ???
#13 38.21   from ???
#13 38.21   from ???
#13 38.21   from ???
#13 38.22 make: *** [Makefile:198: .build/crystal] Error 5

-EDIT: Is this the same issue as #8738?- EDIT2: this crashed on non-musl build, so nevermind. Shards are using musl, though... and that's amd64, this occured during arm64/v8 build.

PREVIOUS_CRYSTAL_VERSION=1.1.1 (and 1.0.0) fails with:

 => ERROR [stage-1  8/11] RUN git clone https://github.com/crystal-lang/crystal  && cd crystal  && git checkout 1.4.1   && make crystal stats=true   137.3s
------                                                                                                                                                      
 > [stage-1  8/11] RUN git clone https://github.com/crystal-lang/crystal  && cd crystal  && git checkout 1.4.1   && make crystal stats=true static=true ${release:+release=true}                  CRYSTAL_CONFIG_TARGET=x86_64-unknown-linux-gnu  && ([ "$(ldd .build/crystal | wc -l)" -eq "1" ] || { echo './build/crystal is not statically linked'; ldd .build/crystal; exit 1; }):                                                                                             
#13 0.142 Cloning into 'crystal'...                                                                                                                         
#13 4.643 Note: switching to '1.4.1'.                                                                                                                       
#13 4.643 
#13 4.643 You are in 'detached HEAD' state. You can look around, make experimental
#13 4.643 changes and commit them, and you can discard any commits you make in this
#13 4.643 state without impacting any branches by switching back to a branch.
#13 4.643 
#13 4.643 If you want to create a new branch to retain commits you create, you may
#13 4.643 do so (now or later) by using -c with the switch command. Example:
#13 4.643 
#13 4.643   git switch -c <new-branch-name>
#13 4.643 
#13 4.643 Or undo this operation with:
#13 4.643 
#13 4.643   git switch -
#13 4.643 
#13 4.643 Turn off this advice by setting config variable advice.detachedHead to false
#13 4.643 
#13 4.643 HEAD is now at b7377c041 Add Changelog for 1.4.1 (#12017)
#13 4.690 Using /usr/bin/llvm-config [version= 10.0.0]g++ -c  -o src/llvm/ext/llvm_ext.o src/llvm/ext/llvm_ext.cc -I/usr/lib/llvm10/include -std=c++14   -fno-exceptions -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS
#13 7.064 CRYSTAL_CONFIG_BUILD_COMMIT="b7377c041" CRYSTAL_CONFIG_PATH='$ORIGIN/../share/crystal/src' SOURCE_DATE_EPOCH="1650653870" CRYSTAL_CONFIG_LIBRARY_PATH='$ORIGIN/../lib/crystal' ./bin/crystal build -D strict_multi_assign --stats --static -Dwithout_interpreter  -o .build/crystal src/compiler/crystal.cr -D without_openssl -D without_zlib
Parse:                             00:00:00.004414041 (   0.75MB)
Semantic (top level):              00:00:02.865040793 ( 137.68MB)
Semantic (new):                    00:00:00.015248291 ( 137.68MB)
Semantic (type declarations):      00:00:00.231574916 ( 137.68MB)
Semantic (abstract def check):     00:00:00.259165958 ( 153.68MB)
Semantic (ivars initializers):     00:00:43.129813562 (1037.62MB)
Semantic (cvars initializers):     00:00:00.062373042 (1037.62MB)
#13 137.2 /usr/lib/gcc/aarch64-alpine-linux-musl/9.3.0/../../../../aarch64-alpine-linux-musl/bin/ld: _main.o: Relocations in generic ELF (EM: 62)
#13 137.2 /usr/lib/gcc/aarch64-alpine-linux-musl/9.3.0/../../../../aarch64-alpine-linux-musl/bin/ld: _main.o: Relocations in generic ELF (EM: 62)
#13 137.2 /usr/lib/gcc/aarch64-alpine-linux-musl/9.3.0/../../../../aarch64-alpine-linux-musl/bin/ld: _main.o: Relocations in generic ELF (EM: 62)
#13 137.2 /usr/lib/gcc/aarch64-alpine-linux-musl/9.3.0/../../../../aarch64-alpine-linux-musl/bin/ld: _main.o: Relocations in generic ELF (EM: 62)
#13 137.2 /usr/lib/gcc/aarch64-alpine-linux-musl/9.3.0/../../../../aarch64-alpine-linux-musl/bin/ld: _main.o: Relocations in generic ELF (EM: 62)
#13 137.2 /usr/lib/gcc/aarch64-alpine-linux-musl/9.3.0/../../../../aarch64-alpine-linux-musl/bin/ld: _main.o: error adding symbols: file in wrong format
#13 137.2 collect2: error: ld returned 1 exit status
#13 137.2 Error: execution of command failed with code: 1: `cc "${@}" -o /root/.cache/crystal/crystal-src-ecr-process.cr/macro_run  -rdynamic -static -L/bdwgc/.libs/ -lpcre -lgc -levent`
make: *** [Makefile:198: .build/crystal] Error 1
------
executor failed running [/bin/sh -c git clone https://github.com/crystal-lang/crystal  && cd crystal  && git checkout ${crystal_sha1}   && make crystal stats=true static=true ${release:+release=true}                  CRYSTAL_CONFIG_TARGET=${gnu_target}  && ([ "$(ldd .build/crystal | wc -l)" -eq "1" ] || { echo './build/crystal is not statically linked'; ldd .build/crystal; exit 1; })]: exit code: 2
make: *** [build/crystal-1.4.1-1-linux-x86_64.tar] Error 1

I see the build is failing because it tries to build arm64 (aarch64) vs x86_64 targets and gets confused.
Is there a way to build arm64 linux binary? There is a Github Action/CI pipeline which builds aarch64, can I use that somehow and create a docker image with crystal/shards arm64 versions?

---

So I tried different strategy - use docker multiarch build to build amd64 binaries. After patching Makefile with this:

diff --git a/linux/Makefile b/linux/Makefile
index d055d5b..bb03911 100644
--- a/linux/Makefile
+++ b/linux/Makefile
@@ -77,7 +77,7 @@ build: $(OUTPUT_BASENAME64).tar ## Build the raw uncompressed tarball
 
 $(OUTPUT_BASENAME64).tar: Dockerfile
        mkdir -p $(OUTPUT_DIR)
-       docker build $(BUILD_ARGS64) -t crystal-build-temp .
+       docker buildx build --platform linux/amd64 $(BUILD_ARGS64) -t crystal-build-temp .
        container_id="$$(docker create crystal-build-temp)" \
          && docker cp "$$container_id":/output/crystal-$(CRYSTAL_VERSION)-$(PACKAGE_ITERATION).tar $@ \
          && docker rm -v "$$container_id"
@@ -93,7 +93,7 @@ $(OUTPUT_BASENAME64)-bundled: $(OUTPUT_BASENAME64).tar $(OUTPUT_DIR)/bundled-lib
 
 $(OUTPUT_DIR)/bundled-libs.tar: bundled.dockerfile
        mkdir -p $(OUTPUT_DIR)
-       docker build $(BUILD_ARGS64_BUNDLED) -t crystal-build-temp -f bundled.dockerfile .
+       docker buildx build --platform linux/amd64 $(BUILD_ARGS64_BUNDLED) -t crystal-build-temp -f bundled.dockerfile .
        container_id="$$(docker create crystal-build-temp)" \
          && docker cp "$$container_id":/output/bundled-libs.tar $@ \
          && docker rm -v "$$container_id"

both versions 1.0.0 and 1.1.1 successfuly compile crystal, but fail on shards.
Should that fail? Should I use different PREVIOUS_CRYSTAL_VERSION?
(Using 1.2.0, 1.3.1, 1.4.0 results in the same error as in arm64 build, so at least that's consistent.)

Failure during the shards build:

# using the amd64 buildx:
make CRYSTAL_VERSION=1.4.1 PREVIOUS_CRYSTAL_VERSION=1.0.0
# or with same result: make CRYSTAL_VERSION=1.4.1 PREVIOUS_CRYSTAL_VERSION=1.1.1

 => CACHED [stage-1  8/11] RUN git clone https://github.com/crystal-lang/crystal  && cd crystal  && git checkout 1.4.1   && make crystal stats=true s  0.0s
 => CACHED [debian 2/4] RUN echo "deb http://deb.debian.org/debian buster-backports main" >> /etc/apt/sources.list  && apt-get update  && apt-get ins  0.0s
 => CACHED [debian 3/4] RUN apt-get install -y --allow-downgrades libcurl3-gnutls=7.64.0-4+deb10u2                                                     0.0s
 => CACHED [debian 4/4] RUN git clone https://github.com/ivmai/bdwgc  && cd bdwgc  && git checkout v8.2.0  && ./autogen.sh  && ./configure --disable-  0.0s
 => ERROR [stage-1  9/11] RUN git clone https://github.com/crystal-lang/shards  && cd shards  && git checkout v0.17.0  && touch shard.lock  && make   27.5s
---
 > [stage-1  9/11] RUN git clone https://github.com/crystal-lang/shards  && cd shards  && git checkout v0.17.0  && touch shard.lock  && make SHARDS=false CRYSTAL=/crystal/bin/crystal          FLAGS="--stats --target x86_64-linux-musl --static ${release:+--release}"   && ([ "$(ldd bin/shards | wc -l)" -eq "1" ] || { echo 'shards is not statically linked'; ldd bin/shards; exit 1; }):                                                                             
#0 0.090 Cloning into 'shards'...                                                                                                                           
#0 1.210 Note: switching to 'v0.17.0'.                                                                                                                      
#0 1.210                                                                                                                                                    
#0 1.210 You are in 'detached HEAD' state. You can look around, make experimental                                                                           
#0 1.210 changes and commit them, and you can discard any commits you make in this                                                                          
#0 1.210 state without impacting any branches by switching back to a branch.
#0 1.210 
#0 1.210 If you want to create a new branch to retain commits you create, you may
#0 1.210 do so (now or later) by using -c with the switch command. Example:
#0 1.210 
#0 1.210   git switch -c <new-branch-name>
#0 1.210 
#0 1.210 Or undo this operation with:
#0 1.210 
#0 1.210   git switch -
#0 1.210 
#0 1.210 Turn off this advice by setting config variable advice.detachedHead to false
#0 1.210 
#0 1.212 HEAD is now at 85b30b5 Add changelog for 0.17.0 (#549)
#0 1.610 Using compiled compiler at /crystal/.build/crystal
#0 19.19 Failed to raise an exception: END_OF_STACK
#0 19.19 [0x40002fb816] ???
#0 19.20 [0x40002a94fb] ???
#0 19.20 [0x40004404ec] ???
#0 19.20 [0x4000443d6e] ???
#0 19.20 [0x400044cb3d] ???
#0 19.20 [0x400030719a] ???
#0 19.20 [0x4000444477] ???
#0 19.20 [0x40002c9705] ???
#0 19.20 [0x4002515244] ???
#0 19.20 
#0 19.20 Tried to raise:: Error writing file: Broken pipe (IO::Error)
#0 19.20 Failed to raise an exception: END_OF_STACK
#0 19.20 [0x40002fb816] ???
#0 19.20 [0x40002a94fb] ???
#0 19.20 [0x40002f1bb4] ???
#0 19.20 [0x40002b9166] ???
#0 19.20 [0x400250bde1] ???
#0 19.20 
#0 19.20 Tried to raise:: Invalid magic number (Crystal::ELF::Error)
#0 19.20   from ???
#0 19.20   from ???
#0 19.20   from ???
#0 19.20   from ???
#0 19.20   from ???
#0 19.20   from ???
#0 19.20   from ???
#0 19.20   from ???
#0 19.20   from ???
#0 19.20   from ???
#0 26.35 mkdir -p lib/molinillo
#0 26.36 false install || (curl -L "https://github.com/crystal-lang/crystal-molinillo/archive/v.tar.gz" | tar -xzf - -C lib/molinillo --strip-components=1)
#0 26.42   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
#0 26.42                                  Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100    14  100    14    0     0     19      0 --:--:-- --:--:-- --:--:--    19
#0 27.12 tar: invalid magic
#0 27.12 tar: short read
#0 27.15 make: *** [Makefile:63: lib] Error 1

I don't understand why crystal compiled and shards doesn't.

[straight-shoota]

Is there a way to build arm64 linux binary?

Yes, it basically works the same way as on any other platform. There is currently no recipe for this in distribution-scripts, though.

There is a Github Action/CI pipeline which builds aarch64, can I use that somehow and create a docker image with crystal/shards arm64 versions?

We're planning to use CI for building aarch64 distribution packages, but it's not yet implemented.

You can use the docker images provided by 84codes: https://hub.docker.com/r/84codes/crystal
They also provide some aarch64 packages at https://packagecloud.io/84codes/crystal

---

Any version since 1.0.0 should work for PREVIOUS_CRYSTAL_VERSION. Ideally the most recent one, of course.

The error messages are weird. Might be running out of memory?

[daliborfilus]

The docker desktop VM has 8 GB limit currently and no other containers are running. I can try to increase the limit.

The errors are probably caused by the use of QEMU, since these are the amd64 build variants.
But... that really can be related to this issue, since the issue author wanted to run crystal docker image on M1, and the docker image is amd64. I got these errors above when I, too, try to run amd64 crystal > 1.2.0 inside docker build, so the symptoms are the same, since <= 1.1.1 amd64 crystal can build crystal source (via qemu), but >= 1.2.0 cannot?

So maybe it can be a clue, even if the error message is completely different?

[ysbaddaden]

It seems to be a known bug in qemu. See docker/for-mac#6049 for some pointers. There doesn't seem to be any fix planned for the time being 😭

Given this is crashing the compiler, maybe we'd like to have a hardcoded fallback in Process.executable_path() when /proc/self/exe can't be accessed (e.g. default to search PATH, as for BSDs)?

[m-o-e]

(this comment can be ignored, was user error)

It seems the 84codes images also no longer work. :(

$ docker run --rm -it --platform linux/amd64 84codes/crystal:latest-alpine /bin/sh
In bin/sh:1:1

 1 | �ELF>��@��
               @8
                 @@@@h�����������	�	�
     ^
Error: unknown token: '\u007F'

(doesn't matter which of their images I try, all the binaries in them give me this error)

[carlhoerberg]

@m-o-e please try $ docker run --rm -it --platform linux/amd64 --entrypoint /bin/sh 84codes/crystal:latest-alpine otherwise you give /bin/sh as an argument to the crystal binary which is the default entrypoint.

[m-o-e]

@carlhoerberg Uargh, yes, this works. Thanks. 🙈
I've redacted my comment above to avoid confusing others.

[daliborfilus]

Just noticed this issue is still open. I think we can close it, since Docker Desktop officially supports Rosetta 2, which isn't experimental anymore. I think it's even enabled by default? https://www.docker.com/blog/docker-desktop-4-25/ says it is:

System requirements: Rosetta for Linux is available on macOS version 13.0 and above, specifically for Apple silicon devices. Notably, it’s enabled by default on macOS 14.1 and newer, making it even more accessible.

I had enabled "the Rosetta checkbox" in Docker Desktop for over a year and a half now and I haven't encountered any issues.

EDIT: docker/roadmap#384 shows people are still having some issues when using Rosetta in Docker Desktop, but that shouldn't be Crystal's concern, in my opinion.