Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

(Platform: WSL 2) "error: No such file or directory (os error 2)" during cargo publish #8439

Closed
Cokemonkey11 opened this issue Jul 2, 2020 · 11 comments · Fixed by #8950
Closed
Labels
A-filesystem Area: issues with filesystems C-bug Category: bug O-windows OS: Windows

Comments

@Cokemonkey11
Copy link

Problem

cargo publish yields error: No such file or directory (os error 2) without any further diagnostics.

Steps

Unclear. On my machine:

chillup $ cargo clean
chillup $ cargo build && cargo build --release
(snip)
chillup $ cargo publish -vvv --no-verify
    Updating crates.io index
   Packaging chillup v0.3.0 (/mnt/c/Users/Cokem/workspace/chillup)
   Archiving .cargo_vcs_info.json
   Archiving .gitignore
   Archiving Cargo.lock
   Archiving Cargo.toml
   Archiving Cargo.toml.orig
   Archiving README.md
   Archiving src/main.rs
   Uploading chillup v0.3.0 (/mnt/c/Users/Cokem/workspace/chillup)
error: No such file or directory (os error 2)

Possible Solution(s)
n/a - need better diags

Notes

Output of cargo version:

cargo 1.44.1 (88ba85757 2020-06-11)

rustc 1.44.1 (c7087fe00 2020-06-17)

wsl -l -v
  NAME                   STATE           VERSION
* Ubuntu                 Running         2
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 18.04.2 LTS
Release:        18.04
Codename:       bionic

Note: This package already exists - I expect publish to update it.

Note: I have successfully published on WSL with this machine already, but this is my first time trying since upgrading to WSL 2.

Note: I generated a new crates.io auth key as a guess for the issue, which did not solve the problem

@Cokemonkey11 Cokemonkey11 added the C-bug Category: bug label Jul 2, 2020
@alexcrichton
Copy link
Member

Thanks for the report! I think we may have merged some improvements to the error message in more recent versions of Cargo, can you try publication with nightly Cargo and see if the error message changes?

@Cokemonkey11
Copy link
Author

Hey @alexcrichton , thanks for the response.

I've done rustup default nightly and rustup update, but it's unclear to me if I'm actually on a recent nightly release:

chillup $ cargo -V
cargo 1.46.0-nightly (c26576f9a 2020-06-23)
chillup $ cargo publish -vvv --no-verify
    Updating crates.io index
   Packaging chillup v0.3.0 (/mnt/c/Users/Cokem/workspace/chillup)
   Archiving .cargo_vcs_info.json
   Archiving .gitignore
   Archiving Cargo.lock
   Archiving Cargo.toml
   Archiving Cargo.toml.orig
   Archiving README.md
   Archiving src/main.rs
   Uploading chillup v0.3.0 (/mnt/c/Users/Cokem/workspace/chillup)
error: No such file or directory (os error 2)

Is this in-line with your expectations?

@ehuss
Copy link
Contributor

ehuss commented Jul 4, 2020

@alexcrichton do you have access to a Windows machine with WSL2? For some reason, mine won't let me upgrade. I tried publishing in WSL1, and it seemed fine. However, AIUI, WSL2 has completely redone how the filesystem mounting works, and I would be suspicious if something related to that is an issue.

@alexcrichton
Copy link
Member

I do have a Windows machine but apparently it won't let me update to the latest Windows to get WSL2. I'm not entirely sure why (it just points me here and says "no action needed", but I can try this in the future if it lets me update.

@Cokemonkey11
Copy link
Author

Hey @alexcrichton, I had the same experience. The easiest way to force the upgrade is to join the windows insider program on one of the release rings.

To be honest, I don't recommend doing it. Microsoft forces you to enable telemetry to join insider ring. The only reason I've done this is to get access to docker wsl backend.

I would be happy to gather more diagnostics, or if someone does want to force this upgrade, happy to provide help with that. I'm @cokemonkey111:matrix.org.

Cheers,

@mati865
Copy link
Contributor

mati865 commented Jul 17, 2020

You can force update with Update Assistant: https://www.microsoft.com/en-gb/software-download/windows10
I have access to WSL2 so I can help if you need.

I believe this fails in WSL2 because you are in fact using "remote" directory. Unlike WSL, WSL2 is a virtual machine so it cannot directly access host directories. To workaround that Windows partitions are available as /mnt/<letter> through 9P protocol. This has serious downsides like very small throughput, big latency and issues with file locks.
My general recommendation with WSL2 is to do all the work on directories inside the image and use /mnt/<letter> only for copying between the VM and host.

@Cokemonkey11
Copy link
Author

Cokemonkey11 commented Jul 20, 2020

@mati865 thanks for the insights. I've managed to publish from a directory in ~ on first attempt.

Leaving this issue open as cargo should provide better diags when this happens (and fix the bug itself). For next steps it's probably needed to isolate what area of code breaks down. I can easily reproduce this, and I'm happy to help.

Cheers,

@ehuss ehuss added A-filesystem Area: issues with filesystems O-windows OS: Windows labels Jul 29, 2020
@calavera
Copy link
Contributor

calavera commented Dec 5, 2020

I bumped into this same issue yesterday and I did some digging. I traced the problem to the point where the publish code checks the tarball metadata, but I don't really know how to fix it:

diff --git i/crates/crates-io/lib.rs w/crates/crates-io/lib.rs
index 3b8b30b30..2561b27c9 100644
--- i/crates/crates-io/lib.rs
+++ w/crates/crates-io/lib.rs
@@ -169,7 +169,14 @@ impl Registry {
         //      <json request> (metadata for the package)
         //      <le u32 of tarball>
         //      <source tarball>
-        let stat = tarball.metadata()?;
+        let stat = match tarball.metadata() {
+            Ok(stat) => stat,
+            Err(err) => bail!(
+                "failed to read tarball metadata for file {:?}: {}",
+                tarball,
+                err
+            ),
+        };
         let header = {
             let mut w = Vec::new();
             w.extend(&(json.len() as u32).to_le_bytes());
(END)

Applying that diff, I get the following output:

   Uploading netlify_toml v1.0.0-alpha.1 (/mnt/c/Users/David/source/repos/netlify-toml-rs)
error: failed to read tarball metadata for file File { fd: 3, path: "/mnt/c/Users/David/source/repos/netlify-toml-rs/target/package/netlify_toml-1.0.0-alpha.1.crate", read: true, write: true }: No such file or directory (os error 2)

The supposed missing file can be read correctly otherwise:

➜  cargo git:(master) ✗ file /mnt/c/Users/David/source/repos/netlify-toml-rs/target/package/netlify_toml-1.0.0-alpha.1.crate
/mnt/c/Users/David/source/repos/netlify-toml-rs/target/package/netlify_toml-1.0.0-alpha.1.crate: gzip compressed data, was "netlify_toml-1.0.0-alpha.1.crate", max compression, original size modulo 2^32 36352

@ehuss
Copy link
Contributor

ehuss commented Dec 5, 2020

Thanks @calavera for digging in. I can finally update my Windows machine to WSL2. I have posted a proposed fix at #8950, which has some more details on why this is happening.

@chrismooredev
Copy link

Hey all, it seems this bug showed up again. Happy to start another issue for this if we want since I believe it's a different part of the code, just same symptoms.

user@wsl-system:/mnt/e/projects/mylibrary$ cargo version
cargo 1.67.0-nightly (ba607b23d 2022-11-22)
user@wsl-system:/mnt/e/projects/mylibrary$ cargo publish --dry-run --verbose
<...output snipped...>
Finished dev [optimized + debuginfo] target(s) in 7.61s
error: could not learn metadata for: `/mnt/e/projects/mylibrary/target/package/mylibrary-0.1.1.crate`

Caused by:
  No such file or directory (os error 2)
user@wsl-system:/mnt/e/projects/mylibrary$

Relevant strace:

lseek(3, 0, SEEK_SET)                   = 0
rename("/mnt/e/projects/mylibrary/target/package/.mylibrary-0.1.1.crate", "/mnt/e/projects/mylibrary/target/package/mylibrary-0.1.1.crate") = 0
statx(3, "", AT_STATX_SYNC_AS_STAT|AT_EMPTY_PATH, STATX_ALL, 0x7ffe02e1fc90) = -1 ENOENT (No such file or directory)
flock(3, LOCK_UN)                       = 0
close(3)                                = 0

It looks like it was reintroduced late last month, in commit 81f0f63. I figure this could be solved most easily by:

  1. obtaining file metadata from the file by the new name (std::fs::metadata(&dst_path))
  2. obtaining file metadata from the file before renaming it (dst.file().metadata() after the dst.seek() call above)

Happy to submit a PR with one of those solutions, it may just take me a bit to get a cargo dev environment setup.

I agree with @mati865's general conclusion that WSL handles 9p files oddly - it seems to lose the ability to stat a file through a file descriptor after renaming it (see strace above).

I've wrote a small test for it below, along with the output for my system.
Summary:

Result Test Case
Pass Windows (NTFS)
Pass WSL2 Linux (ext4)
Pass WSL2 Linux (ext4 - mounted over SSHFS from a native Linux system)
Fail WSL2 Linux (9p - Windows drive)
Fail WSL2 Linux (9p - mounted over SSHFS from localhost)

renamed_file_metadata.rs:

use std::fs;
#[cfg(unix)]
use std::os::unix::fs::MetadataExt;

fn main() {
    // create file
    // rename it by path to something else
    // get file metadata from original file descriptor

    const FILE_SRC: &str = "test_file_src.txt";
    const FILE_DST: &str = "test_file_dst.txt";

    eprintln!("cwd: {}", std::env::current_dir().unwrap().display());
    let f = fs::File::create(FILE_SRC).expect("error creating original file");
    
    let fm = f.metadata().expect("unable to get new file's metadata");
    #[cfg(unix)]
    eprintln!("current file's dev/inode before rename: {}/{}", fm.dev(), fm.ino());
    fs::rename(FILE_SRC, FILE_DST).expect("error renaming file");

    #[cfg(unix)] {
        eprintln!("current file's dev/inode after rename: {}/{}", fm.dev(), fm.ino());
        
        match fs::File::open(FILE_DST) {
            Ok(fr) => {
                // seems to be allowed on ext4, not p9
                let fr = fr.metadata().expect("unable to fetch metadata for newly renamed file");
                eprintln!("renamed file's dev/inode: {}/{}", fr.dev(), fr.ino());
            },
            Err(e) => {
                eprintln!("renamed file's dev/inode: (failed: {:?})", e);
            }
        }
    }
    eprintln!("Result: able to get renamed file's metadata from original FD/Handle: {:?}", f.metadata().is_ok());
}

My system versions + test output

PS E:\> cmd /c ver
Microsoft Windows [Version 10.0.19045.2130]
PS E:\> rustc renamed_file_metadata.rs
PS E:\> .\renamed_file_metadata.exe
cwd: E:\
able to get renamed file's metadata from original FD/Handle: true
PS E:\>
user@wsl-system:~$ uname -a
Linux wsl-system 5.10.16.3-microsoft-standard-WSL2 #1 SMP Fri Apr 2 22:23:49 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
user@wsl-system:~$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu Jammy Jellyfish (development branch)
Release:        22.04
Codename:       jammy
user@wsl-system:~$ df -T
Filesystem     Type   1K-blocks       Used Available Use% Mounted on
/dev/sdb       ext4   263174212   14469452 235266604   6% /
tmpfs          tmpfs    9815288          0   9815288   0% /mnt/wsl
tools          9p     402653180  329823164  72830016  82% /init
none           tmpfs    9815288          4   9815284   1% /run
none           tmpfs    9815288          0   9815288   0% /run/lock
none           tmpfs    9815288          0   9815288   0% /run/shm
none           tmpfs    9815288          0   9815288   0% /run/user
tmpfs          tmpfs    9815288          0   9815288   0% /sys/fs/cgroup
drivers        9p     402653180  329823164  72830016  82% /usr/lib/wsl/drivers
lib            9p     402653180  329823164  72830016  82% /usr/lib/wsl/lib
C:\            9p     402653180  329823164  72830016  82% /mnt/c
D:\            9p    1885995004 1597227076 288767928  85% /mnt/d
E:\            9p     536869884  503601612  33268272  94% /mnt/e
user@wsl-system:~$ rustc renamed_file_metadata.rs
user@wsl-system:~$ ~/renamed_file_metadata
cwd: /home/user
current file's dev/inode before rename: 2064/68907
current file's dev/inode after rename: 2064/68907
renamed file's dev/inode: 2064/68907
able to get renamed file's metadata from original FD/Handle: true
user@wsl-system:~$ cd /mnt/e
user@wsl-system:/mnt/e$ ~/renamed_file_metadata
cwd: /mnt/e
current file's dev/inode before rename: 48/3377699721142538
current file's dev/inode after rename: 48/3377699721142538
renamed file's dev/inode: (failed: Os { code: 2, kind: NotFound, message: "No such file or directory" })
able to get renamed file's metadata from original FD/Handle: false
user@wsl-system:/mnt/e$

@jtmoon79
Copy link

jtmoon79 commented Mar 18, 2023

tl;dr appears related to WSL NTFS mounts, workaround is git clone elsewhere then cargo publish

Bug Observation

Using Ubuntu 22 on WSL2, I attempted to cargo publish (within Ubuntu 22) using a working directory that has an underlying NTFS file system managed by Windows 11.

$ cargo clean
$ cargo publish --verbose --dry-run
...
error: could not learn metadata for: `/mnt/c/Users/user1/Projects/super-speedy-syslog-searcher/target/package/super_speedy_syslog_searcher-0.3.54.crate`

The .crate file is accessible and touchable within Ubuntu 22.

$ file ./target/package/super_speedy_syslog_searcher-0.3.54.crate
./target/package/super_speedy_syslog_searcher-0.3.54.crate: gzip compressed data, was "super_speedy_syslog_searcher-0.3.54.crate", max compression, original size modulo 2^32 1919488

$ ls -las ./target/package/super_speedy_syslog_searcher-0.3.54.crate
-rwxrwxrwx 1 ulug ulug 303591 Mar 17 22:11 ./target/package/super_speedy_syslog_searcher-0.3.54.crate

$ touch ./target/package/super_speedy_syslog_searcher-0.3.54.crate

I tried cargo publish from different path under the same NTFS mount (git clone the repository, run cargo publish --dry-run). The same error occurred.

This appears related to NTFS mounts used by WSL2 Linux.

A week ago using the same project path cargo publish succeeded. I don't recall updating rust since then. There had been one significant Windows Update 2023-03 Cumulative Update for Windows 11 Version 22H2 for x64-based Systems (KB5023706).

Workaround

git clone elsewhere. I used a path under /tmp which is not backed by Windows NTFS. From there, cargo publish succeeded.

Context

$ cargo --version
cargo 1.67.1 (8ecd4f20a 2023-01-10)

$ uname -a
Linux host1 5.10.16.3-microsoft-standard-WSL2 #1 SMP Fri Apr 2 22:23:49 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-filesystem Area: issues with filesystems C-bug Category: bug O-windows OS: Windows
Projects
None yet
Development

Successfully merging a pull request may close this issue.

7 participants