Wrong file ownership on root filesystem

[jibel]

Windows Version

Microsoft Windows [Version 10.0.22621.1555]

WSL Version

1.2.2.0

Are you using WSL 1 or WSL 2?

• WSL 2
• WSL 1

Kernel Version

5.15.90.1

Distro Version

Ubuntu and all releses of it

Other Software

No response

Repro Steps

This has been observed on all the releases of Ubuntu.
To reproduce the issue, install a WSL instance from the store.

While investigating issues with failing services of systemd, we found that some files and directories are not owned by the right users.

For instance:
/usr/bin/crontab root/77 instead of root/crontab (0/104)
/var/spool/rsyslog 18/16 instead of syslog/adm (104/4)
/var/log/journal/ root/78 instead of root/systemd-journal (0/101)
usr/lib/dbus-1.0/dbus-daemon-launch-helper root/18 instead of root/messagebus (0/105)
...

users and groups on an installed WSL instance sometimes differ from users and groups of the root filesystem (the initial tarball); some do not match the correct id from the root filesystem, others are nonexistent.

A guess is that when the rootfs tarball is extracted during the installation of the instance, the tar command uses the passwd and group databases of the root WSL VM instead of the passwd and group files of the root filesystem and wrongly maps user names and group names.

The consequence is that the system will not operate properly. System units are not starting, or commands won't execute properly, for example.

Expected Behavior

User and groups are unchanged compared to the original root filesystem

Actual Behavior

Some files have incorrect ownership.

Here is the list of files that we found have the wrong ownership

--- rootfs.list	2023-04-14 16:16:50.770067493 +0200
+++ wsl.list	2023-04-14 16:16:10.365804172 +0200
-0/104 usr/bin/crontab
+0/77 usr/bin/crontab

-0/105 usr/lib/dbus-1.0/dbus-daemon-launch-helper
+0/18 usr/lib/dbus-1.0/dbus-daemon-launch-helper

-0/43 usr/lib/x86_64-linux-gnu/utempter/utempter
+0/13 usr/lib/x86_64-linux-gnu/utempter/utempter

-0/43 var/log/btmp
+0/13 var/log/btmp

-0/101 var/log/journal
+0/78 var/log/journal

-0/43 var/log/lastlog
+0/13 var/log/lastlog

-0/4 var/log/unattended-upgrades
+0/16 var/log/unattended-upgrades

-0/43 var/log/wtmp
+0/13 var/log/wtmp

-0/8 var/mail
+0/34 var/mail

-0/104 var/spool/cron/crontabs
+0/77 var/spool/cron/crontabs

-104/4 var/spool/rsyslog
+18/16 var/spool/rsyslog

Diagnostic Logs

No response

[benhillis]

@jibel - I'm a bit confused here. The from a UID / GID perspective, tar is just storing numbers and setting those values when extracting. Have you validated locally that when you extract the tar file the UID / GID matches what you expect?

[jibel]

I am confused too. I would understand the name doesn't match, but not a change of the UID/GID.

I did the following test:

1. Download the rootfs of 22.04 and Ubuntu: https://cloud-images.ubuntu.com/wsl/jammy/current/ubuntu-jammy-wsl-amd64-wsl.rootfs.tar.gz
2. check the content of the tar file for one of the erroneous file for example usr/bin/crontab

j-lallement@boedic:~/tmp.notshared/wsl$ tar tvzf ubuntu-jammy-wsl-amd64-wsl.rootfs.tar.gz  usr/bin/crontab
-rwxr-sr-x root/systemd-resolve 39568 2022-03-23 14:49 usr/bin/crontab
j-lallement@boedic:~/tmp.notshared/wsl$ tar tvzf ubuntu-jammy-wsl-amd64-wsl.rootfs.tar.gz --numeric-owner usr/bin/crontab
-rwxr-sr-x 0/104         39568 2022-03-23 14:49 usr/bin/crontab

(on my system crontab is 105 and 104 is systemd-time, and systemd-resolve is 103)

3. Extract the file with the same-owner flag
   sudo tar xvzf ubuntu-jammy-wsl-amd64-wsl.rootfs.tar.gz --same-owner usr/bin/crontab
4. Check the owner

j-lallement@boedic:~/tmp.notshared/wsl$ ls -l usr/bin/crontab 
-rwxr-sr-x 1 root systemd-resolve 39568 mars  23  2022 usr/bin/crontab
j-lallement@boedic:~/tmp.notshared/wsl$ ls -ln usr/bin/crontab 
-rwxr-sr-x 1 0 103 39568 mars  23  2022 usr/bin/crontab

It is now owned by systemd-resolve (same name than the tar output) but the GID is 103.

The GID of the same file in WSL is 77.

If I mount /dev/sda, which contains Mariner and search /etc/group for GID 77, it's systemd-resolve.

[benhillis]

I'm still confused - we run bsdtar as root so --same-owner would be the default.

[jibel]

Same with bsdtar, the GID is modified on extract:

j-lallement@boedic:~/tmp.notshared/wsl$ sudo bsdtar --numeric-owner -tvf ubuntu-jammy-wsl-amd64-wsl.rootfs.tar.gz usr/bin/crontab
-rwxr-sr-x  0 0      104     39568 mars 23  2022 usr/bin/crontab
j-lallement@boedic:~/tmp.notshared/wsl$ sudo bsdtar -xf ubuntu-jammy-wsl-amd64-wsl.rootfs.tar.gz usr/bin/crontab
j-lallement@boedic:~/tmp.notshared/wsl$ ls -ln usr/bin/crontab 
-rwxr-sr-x 1 0 103 39568 mars  23  2022 usr/bin/crontab

It's 104 in the archive and 103 once extracted. I'll dig further tomorrow.

[benhillis]

I'm seeing something similar locally:

root@BENHILL-DEV:~# stat usr/bin/crontab
  File: usr/bin/crontab
  Size: 39568           Blocks: 80         IO Block: 4096   regular file
Device: 820h/2080d      Inode: 30694       Links: 1
Access: (2755/-rwxr-sr-x)  Uid: (    0/    root)   Gid: (  103/systemd-resolve)
Access: 2023-04-17 10:08:00.473964561 -0700
Modify: 2022-03-23 06:49:13.000000000 -0700
Change: 2023-04-17 10:08:00.473964561 -0700
 Birth: 2023-04-17 10:08:00.473964561 -0700
root@BENHILL-DEV:~# stat /usr/bin/crontab
  File: /usr/bin/crontab
  Size: 39568           Blocks: 80         IO Block: 4096   regular file
Device: 820h/2080d      Inode: 1322        Links: 1
Access: (2755/-rwxr-sr-x)  Uid: (    0/    root)   Gid: (   77/ UNKNOWN)
Access: 2023-04-12 14:35:19.000000000 -0700
Modify: 2022-03-23 06:49:13.000000000 -0700
Change: 2023-04-17 10:06:04.543573403 -0700

[benhillis]

You're right, it looks like ownership information is being inherited from the /etc/passwd file inside the system distro. (you can confirm this by launching wsl.exe --system -u root and looking at /etc/passwd). Looks like we should probably be using the --numeric-owner option.

[benhillis]

@jibel - I think I have a fix for this. How did you identify the files that have the wrong owner so I can run that locally to see if the issue is resolved?

[benhillis]

@jibel - could you try 1.2.4 and see if that resolves what you're seeing? You'll have to re-register the distro after installing the updated package.

https://github.com/microsoft/WSL/releases/tag/1.2.4

[jibel]

@jibel - I think I have a fix for this. How did you identify the files that have the wrong owner so I can run that locally to see if the issue is resolved?

I created a list of files with owners in WSL, excluding owner root/root (run as root):

find / -xdev -printf "%U/%G %p\n"|grep -v 0/0|sort -k2 > wsl.list

Same with the tarball (add leading and strip trailing slashes):

tar tvzf ubuntu-jammy-wsl-amd64-wsl.rootfs.tar.gz --numeric-owner |awk '!/0\/0/ {print $2, "/" $6}'|sed -e 's/\/$//'|sort -k2 > tar.ubuntu.files

Then diff the two files.

diff -u tar.ubuntu.files wsl.ubuntu.files

There are some extra lines but the diff is small enough to identify the wrong ones.

[jibel]

@jibel - could you try 1.2.4 and see if that resolves what you're seeing? You'll have to re-register the distro after installing the updated package.

https://github.com/microsoft/WSL/releases/tag/1.2.4

@benhillis I confirm that 1.2.4 fixes the issue. Thanks for the quick fix!