Invalid cross-device link (os error 18) when building inside pods

[huguesBouvier]

Problem

Hello!

I developing my application on kubernetes. For development, I use minikube, I mount my host repo inside a pod and I build/run from inside it.
However, I have the following issue when building:
Compiling libc v0.2.118
error: failed to write /home/azureuser/..../target/debug/deps/libunicode_xid-8bb8457dab823b6f.rmeta: Invalid cross-device link (os error 18)

error: could not compile unicode-xid due to previous error.

Thanks!

Steps

Install and start minikube.
mount any rust repo:
minikube mount /repo:/repo

Create a deployment to mount the repo inside a pod.
Do cargo build inside the pod

Possible Solution(s)

No response

Notes

No response

Rustup version

rustup 1.24.3 (ce5817a94 2021-05-31)

Installed toolchains

rustc 1.57.0 (f1edd0429 2021-11-29)

[rbtcollins]

I suspect that this isn't a rustup problem, sauce you are getting this error from the compiler.

What's the full path in the error?

[g00nix]

I can confirm the issue. I am running rustup inside a podman container. Code to the full setup can be found here.

~ # ❯❯❯ rustup update
info: syncing channel updates for 'stable-x86_64-unknown-linux-gnu'
info: latest update on 2022-02-24, rust version 1.59.0 (9d1b2106e 2022-02-23)
info: downloading component 'rust-std' for 'wasm32-unknown-unknown'
info: downloading component 'cargo'
info: downloading component 'clippy'
info: downloading component 'rust-docs'
info: downloading component 'rust-std'
info: downloading component 'rustc'
 53.8 MiB /  53.8 MiB (100 %)  30.8 MiB/s in  1s ETA:  0s
info: downloading component 'rustfmt'
info: removing previous version of component 'rust-std' for 'wasm32-unknown-unknown'
info: removing previous version of component 'cargo'
info: rolling back changes
error: could not rename component file from '/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/share/zsh/site-functions' to '/root/.rustup/tmp/4jpq5_2ziff3ygx8_dir/bk'
info: syncing channel updates for 'nightly-x86_64-unknown-linux-gnu'
info: latest update on 2022-03-19, rust version 1.61.0-nightly (1bfe40d11 2022-03-18)
info: downloading component 'rust-analyzer-preview'
info: downloading component 'rust-src'
info: downloading component 'cargo'
info: downloading component 'clippy'
info: downloading component 'rust-docs'
info: downloading component 'rust-std'
info: downloading component 'rustc'
 55.2 MiB /  55.2 MiB (100 %)  25.9 MiB/s in  2s ETA:  0s
info: downloading component 'rustfmt'
info: removing previous version of component 'rust-analyzer-preview'
info: rolling back changes
error: could not rename component file from '/root/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/share/doc/rust-analyzer' to '/root/.rustup/tmp/b62wowa4hn89yplv_dir/bk'
info: checking for self-updates

   stable-x86_64-unknown-linux-gnu update failed - rustc 1.58.1 (db9d1b20b 2022-01-20)
  nightly-x86_64-unknown-linux-gnu update failed - rustc 1.61.0-nightly (3b348d932 2022-02-19)

info: cleaning up downloads & tmp directories
~ # ❯❯❯ rustup update nightly
info: syncing channel updates for 'nightly-x86_64-unknown-linux-gnu'
info: latest update on 2022-03-19, rust version 1.61.0-nightly (1bfe40d11 2022-03-18)
info: downloading component 'rust-analyzer-preview'
info: downloading component 'rust-src'
info: downloading component 'cargo'
info: downloading component 'clippy'
info: downloading component 'rust-docs'
info: downloading component 'rust-std'
info: downloading component 'rustc'
 55.2 MiB /  55.2 MiB (100 %)  23.4 MiB/s in  2s ETA:  0s
info: downloading component 'rustfmt'
info: removing previous version of component 'rust-analyzer-preview'
info: rolling back changes
error: could not rename component file from '/root/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/share/doc/rust-analyzer' to '/root/.rustup/tmp/qb9tafd8b28fe6to_dir/bk': Invalid cross-device link (os error 18)
~ # ❯❯❯                   

The solution in my case was to delete the container and the image and to run the install automation again, forcing all OS and rust-related packages to install from scratch.

[huguesBouvier]

Thanks I'll try that!

[huguesBouvier]

My solution, which is not really one was to build on host with musl and mount only the executable....

[kinnison]

I don't recommend running rustup update or similar inside containers. Sadly container runtimes don't offer full filesystem semantics which results in things going bad. Instead ensure that you have rustup, and the correct toolchain, installed in a single layer.

[sajattack]

I'm struggling with this too. Not running rustup update, but even just trying to build a crate on a minikube mount. Should I re-log it on https://github.com/rust-lang/rust ?

[rbtcollins]

the behaviour of rustup and cargo/rustc in this area is independent. As long as you're not running a custom toolchain, if rustup succeeded installing the toolchain, then the problem is very unlikely to be rustup, and you should debug cargo/rustc

[clarfonthey]

This isn't just a container issue; the issue is that rustup is assuming that the file can simply be renamed in cases where it may actually be moving between filesystems. For example, I use different ZFS datasets for ~/.rustup/downloads, ~/.rustup/tmp, and ~/.rustup/toolchains (so I can backup toolchains, but ignore downloads/tmp) and I get this error:

error: could not rename component file from '/home/ltdk/.rustup/tmp/tz7f320ij__0ru3o_file' to '/home/ltdk/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/components'
error: could not rename components file from '/home/ltdk/.rustup/tmp/d_ded9_oq2y4lbm7_file' to '/home/ltdk/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/components': Invalid cross-device link (os error 18)

I'll try and take a look in the code to see if this can be fixed.

[Diggsey]

@huguesBouvier @sajattack Your issues (which are the one described in the issue body) are not related to rustup. The error is coming from rustc although the cause is likely due to your container's configuration.

@g00nix @clarfonthey Your issues are different, and are related to rustup. As you correctly point out they are due to the rustup directories existing on different filesystems. However, this is not a supported configuration - it's expected that everything within the .rustup directory resides on a single filesystem, because rustup expects to be able to rename files between the downloads/tmp/toolchains folders. Adding support for this may be possible, but may have performance and correctness implementations that should be considered before that.

(For that matter, backing up toolchains seems like a waste of space since they are immutable... Unless these are custom toolchains, in which case it might be better to have only the custom toolchains in the ZFS dataset?)

[clarfonthey]

Toolchains aren't immutable; updating a version of stable, for example, will overwrite the data in the toolchains folder. While they aren't super important from the perspective of being able to re-download them, so is most of my root filesystem, which I also back up and snapshot for the purpose of being able to debug problems when they happen. Sure, in the particular case of Rust, I don't have to do this, but I didn't question it until now because I had just assumed that things would be copy-removed if needed, like the behaviour of mv and others.

[gdennie]

What worked for me... running with nightly...

rustup toolchain uninstall nightly
rustup toolchain install nightly

[dylanahsmith]

The PR description does seem like it is coming from rustc rather than rustup, even though rustup does experience run into the same system error which #2949 (comment) commented does show. Should a separate issue be created for the rustup specific issue?

I don't recommend running rustup update or similar inside containers. Sadly container runtimes don't offer full filesystem semantics which results in things going bad. Instead ensure that you have rustup, and the correct toolchain, installed in a single layer.

As long as you're not running a custom toolchain, if rustup succeeded installing the toolchain, then the problem is very unlikely to be rustup

Unfortunately, using a rust-toolchain.toml file makes this much more likely to occur when installing rustup and the stable toolchain in a generic base layer then using cargo to build code in an application layer on top of that. In that case, the rust-toolchain.toml can introduce an implicit rustup update to install a new component/target by invoking cargo in any ways, resulting in this error when it picks up a new version of rust. This is the failure scenario we ran into.

The RUSTUP_PERMIT_COPY_RENAME environment variable was introduced specifically to fix this issue, although it needs #3292 to fix it and it is documented as being unstable. That would make for a convenient fix for the issue, since the environment variable can be inherited through the base layer to apply to any images based on it.

[mxchist]

I've got the same error when I was removing a rustup target. It's easy to reproduce:

1. Get compose.yml

services:
  web_application:
    build: 
      context: .
      dockerfile_inline: |
        FROM rust:alpine
        
        RUN rustup target add wasm32-unknown-unknown

    command:
      - sleep
      - "86400"

2. Run docker compose up -d web_application
3. Connect to a running docker container:
   docker compose exec -it web_application sh
4. Remove a target:

/ # rustup target remove wasm32-unknown-unknown
info: removing component 'rust-std' for 'wasm32-unknown-unknown'
info: rolling back changes
error: could not rename component file from '/usr/local/rustup/toolchains/1.82.0-x86_64-unknown-linux-musl/lib/rustlib/wasm32-unknown-unknown/lib' to '/usr/local/rustup/tmp/lloywzebcq3vsy6h_dir/bk': Cross-device link (os error 18)