Cant build with cross

[bombsimon]

What happened:

When running make build-image (or more exactly build-linux-binary) I get

make: *** [Makefile:94: build-linux-binary] Error 127

This also happens when running the failing step without make:

docker run --rm \
  -w /workspace \
  -v /path/to/quilkin:/workspace \
  -v ~/.cargo/registry:/usr/local/cargo/registry \
  -v /var/run/docker.sock:/var/run/docker.sock \
   --entrypoint=cross quilkin-build build --target x86_64-unknown-linux-gnu ; echo $?
127

What you expected to happen:

Quilkin to build successfully with cross.

How to reproduce it (as minimally and precisely as possible):

docker run --rm \
  -w /workspace \
  -v /path/to/quilkin:/workspace \
  -v ~/.cargo/registry:/usr/local/cargo/registry \
  -v /var/run/docker.sock:/var/run/docker.sock \
   --entrypoint=cross quilkin-build build --target x86_64-unknown-linux-gnu

Just changing the entrypoint to cargo makes it work.

Anything else we need to know?:

I don't get any output from the Makefile or when running Docker manually but if I attach to the image and try to run cross manually I get the following:

docker run --rm -it \
  -v /path/to/quilkin:/workspace \
  -v ~/.cargo/registry:/usr/local/cargo/registry \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -w /workspace \
  --entrypoint=/bin/bash quilkin-build 

root@6f37b81e4dcc:/workspace#  echo $PATH
/usr/local/cargo/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

root@6f37b81e4dcc:/workspace# echo $CARGO_HOME
/usr/local/cargo

root@6f37b81e4dcc:/workspace# cross build --target x86_64-unknown-linux-gnu
info: syncing channel updates for '1.59.0-x86_64-unknown-linux-gnu'

  1.59.0-x86_64-unknown-linux-gnu unchanged - rustc 1.59.0 (9d1b2106e 2022-02-23)

info: checking for self-updates
info: downloading component 'rust-src'
info: installing component 'rust-src'
sh: 1: cargo: not found

Environment:

• Quilkin version: 0.4.0
• Execution environment (binary, container, etc): -
• Operating system: WSL Ubuntu 20.04 under Windows 11
• Custom filters? (Yes/No - if so, what do they do?): No
• Log(s): -
• Others: -

[markmandel]

Hrmnn - fascinating. This is working on Linux (obviously, since CI is passing), and since everything is Docker based, in theory it should just work.

I wonder if it's a Docker-in-docker issue 🤔

Within WSL, if you stat /var/run/docker.sock what do you get?

I might have to fire up a windows machine to test.

[markmandel]

As an aside, an easy way to get a shell within the container - make shell (although that doesn't mount the docker.sock, maybe it should).

Cargo is in my path. But strangely, not for you 🤔 From your shell, if you run cargo --version I assume it works? or is it just the cross can't find it for some reason?

On research, code 127 is:

The 127 error code indicates “command not found”. This occurs when any given command within your Bash script or on Bash command line is not found in any of the paths defined by PATH system environment variable.

(source)

Which does lead me to the belief that for some reason, something can't find cargo, which is very odd.

I also just found this: cross-rs/cross#260

Which is, concerning. I'll keep digging in, see what I can find.

[markmandel]

I was hoping that this was a "things break if you start from scratch" problem, but I cleared out all my docker images and tried it again on my Debian machine, and it all still worked for me.

Sounds like I might need to provision a Windows machine and get a Docker Desktop licence to work through this (unless you have some bandwidth to investigate?)

[bombsimon]

I was hoping that this was a "things break if you start from scratch" problem, but I cleared out all my docker images and tried it again on my Debian machine, and it all still worked for me.

Sounds like I might need to provision a Windows machine and get a Docker Desktop licence to work through this (unless you have some bandwidth to investigate?)

Interesting, I would've guessed this was platform independent since it's in Docker. 🤔

I can do some investigation tomorrow or early next week so feel free to put this on hold on your side for a while!

[markmandel]

Interesting, I would've guessed this was platform independent since it's in Docker. thinking

Yeah same. I'm also confused. Something is weird somewhere.

[bombsimon]

Seems 100% related to cross, added a comment and MRE in the issue you linked. I can't make it work in Docker in Docker in Windows or WSL. Well, with one weird exception that I can run any alpine based image (vanilla or f.ex. docker:dind) and have cross build successfully, even if Docker is not even available but I have no clue how that works so I don't trust it.

[markmandel]

Yeah, definitely does seem related to cross - one thought I had was - I can spin up a blank Ubuntu instance, and see if that is the issue, rather than WSL. Far easier and faster to check to see if it's the Ubuntu layer in your stack rather than the WSL layer.

I am wondering if the workaround

RUN printf '#!/bin/sh\nexport CARGO_HOME=/opt/cargo\nexec /bin/sh "$@"\n' >/usr/local/bin/sh
RUN chmod +x /usr/local/bin/sh

From cross-rs/cross#260 (comment) will solve the issue for us. If you have a chance to drop it into the Dockerfile (it will rebuild automatically on next run) and see if it works for you, that would be great - otherwise I'll see if I can replicate on Ubuntu as a next step, before going full Windows (and having to get a Docker Desktop licence).

[bombsimon]

After a reboot of my computer and restart of Rancher Desktop, I actually can build with cross in DinD from Windows so it seems only related to WSL.

I can spin up a blank Ubuntu instance, and see if that is the issue, rather than WSL

It sounded unlikely given it works on Debian and Manjaro but just to confirm this I created a VM instance with Ubuntu 22.04 and that worked as expected. Tried both vanilla Ubuntu on the VM and also DinD in the VM with the steps from my comment.

I also tried to test this on macOS but I can't successfully even install cross in Docker on my M1 MacBook.

# cargo install cross
    Updating crates.io index
Killed Fetch [======>                  ]  28.46%, 4.52MiB/s

I am wondering if the workaround

I tried explicitly setting CARGO_HOME but when reproducing I'm not using the Quilkin image but rather this: cross-rs/cross#260 (comment). However I tried doing this anyway to have it tested (both in my example and in the Dockerfile) without any success.

[markmandel]

Uh. I wish cross compilation with Rust was easier (or maybe I'm missing something).

https://www.reddit.com/r/rust/comments/r60fzb/m1_users_how_are_you_cross_compiling/ seems like a whole thread on cross and M1.

You know, I never actually tried using cargo build --target - since I assumed we weren't going to be pure rust always (are we pure rust now?), but maybe that's something to go back to?

[markmandel]

From the meeting today -- it sounds like it's worth experimenting to see if we can build with cargo build --target rather than cross, since that would simplify things considerably.

[bombsimon]

This is the result of me investigating WSL: cross-rs/cross#728

TL;DR: Doesn't seem to be related to cross but rather Docker in Docker in WSL.

Is Windows supposed to be supported? Thinking if you have access to a WSL host you probably have Windows access as well so we might not have to support WSL right now. The problem is I can't get this to build on Windows either although that's not due to cross.

[markmandel]

So I always believed it was supposed to work:
For example: https://tomgregory.com/running-docker-in-docker-on-windows/

But maybe WSL is causing issue? 🤔

On WSL, I assume you can stat /var/run/docker.sock ?

[markmandel]

So just running some tests, and, unfortunately, it looks like we'll still need cross for cross compilation - or we'll need to include the appropriate linkers in the Docker image (maybe that is better?):

cargo build --target=x86_64-pc-windows-gnu
...
   Compiling hyper v0.14.18
   Compiling hyper-timeout v0.4.1
   Compiling tonic v0.6.2
error: linker `x86_64-w64-mingw32-gcc` not found
  |
  = note: No such file or directory (os error 2)

Either that, or maybe we setup some time on a video call, and try and debug this together?

[markmandel]

Was doing some hunting around, and found this recent article:

https://kerkour.com/rust-cross-compilation

Which shows some clever ways to avoid the need for cross. Definitely worth taking for a spin and seeing if we can make it work.

[markmandel]

Actually, this wasn't very hard at all - been pottering on this this evening. WIP here:
https://github.com/markmandel/quilkin/tree/feature/remove-cross

I expect we can leave the cross compile to mac in place, since none of that uses Docker in Docker.

I just need to make sure the generation of the Docker image still works (some of the paths of binary creation may have changed), then I can submit PRs.