Make Silverblue more robust against SELinux labels change breakage

[jkonecny12]

Hi everyone.

Is your feature request related to a problem? Please describe.
It's pretty easy to break the SELinux on SilverBlue system by running restorecon -RV /. I think this is totally valid command on normal workstation when you know that you have broken labels somewhere but don't know where exactly. However, running this on SilverBlue will relabel most of the files (maybe all of them) on /sysroot because there is no policy to specify the correct labels. After this command, you are not able to log into the system because systemd user session is denied and probably most of all other parts of the system.

Workaround (not really)
The solution is to disable SELinux and reinstall the system completely. Based on this and my other tests with @cmurf we wasn't able to find a way out of this. The system seems to be doomed from the SELinux PoV.

Describe the solution you'd like
There should be a command on rpm-ostree or ostree which would set the correct labels for the /sysroot/ based on the remote repository.

Describe alternatives you've considered
Store the SELinux labels as metadata object in the OSTree repository but that seems much more complicated than the above solution.

Additional context
I broke my system because I tried to solve the current issue with SELinux: https://bugzilla.redhat.com/show_bug.cgi?id=2042157 seems that this is still an issue and there are multiple people complaining about this but it's not reproducible on the VM.
I really don't think that command like restorecon -RF / should break the system on SilverBlue so hard.

[tpopela]

@cgwalters @jlebon @dustymabe any idea? Also this doesn't look like it's Silverblue specific as it could affect other (rpm-)ostree based distributions.

[travier]

I think that running restorecon -RV / is never a good idea and should probably be an error in restorecon.

• On Workstation, you should run either fixfiles onboot and reboot or retorecon -RFv /path/to/a/specific/folder.
• On Silverblue, you should never touch /sysroot as it is managed by rpm-ostree. You should only run restorecon on files under /var or /etc: retorecon -RFv /var/...

On Fedora CoreOS, /sysroot is mounted RO and we should import that in Silverblue as that should have prevented this issue.

[travier]

ostreedev/ostree#1265

[jkonecny12]

I think that running restorecon -RV / is never a good idea and should probably be an error in restorecon.

* On Workstation, you should run either `fixfiles onboot` and reboot or `retorecon -RFv /path/to/a/specific/folder`.

On Workstation I would do touch /.autorelabel.

* On Silverblue, you should never touch `/sysroot` as it is managed by rpm-ostree. You should only run restorecon on files under `/var` or `/etc`: `retorecon -RFv /var/...`

Is that documented somewhere? I agree with that now but wasn't aware of what it could done before.

On Fedora CoreOS, /sysroot is mounted RO and we should import that in Silverblue as that should have prevented this issue.

Is there any reason why it's not RO now? I thought it is like that the whole time. It's pretty fragile to change anything there and really easy way to break SilverBlue by many other ways here.

[travier]

I think that running restorecon -RV / is never a good idea and should probably be an error in restorecon.

• On Workstation, you should run either fixfiles onboot and reboot or retorecon -RFv /path/to/a/specific/folder.

On Workstation I would do touch /.autorelabel.

This is what fixfiles onboot does.

• On Silverblue, you should never touch /sysroot as it is managed by rpm-ostree. You should only run restorecon on files under /var or /etc: retorecon -RFv /var/...

Is that documented somewhere? I agree with that now but wasn't aware of what it could done before.

There is a note in the Fedora CoreOS docs but I agree that it could be more visible: https://docs.fedoraproject.org/en-US/fedora-coreos/storage/#_immutable_read_only_usr

On Fedora CoreOS, /sysroot is mounted RO and we should import that in Silverblue as that should have prevented this issue.

Is there any reason why it's not RO now? I thought it is like that the whole time. It's pretty fragile to change anything there and really easy way to break SilverBlue by many other ways here.

I don't think there is a specific reason. It just hasn't been done yet.

[jkonecny12]

So I was able to resolve the issue without reinstall by:
https://docs.fedoraproject.org/en-US/fedora-silverblue/troubleshooting/#_running_restorecon

[travier]

⚠️⚠️ Only try the following if you are confortable debugging an un-bootable system and have made backups ⚠️⚠️

How to try this, pending work on making this the default:

sudo rpm-ostree kargs --append-if-missing=rw
sudo ostree config --repo=/sysroot/ostree/repo set "sysroot.readonly" "true"
sudo systemctl reboot

Note that you can not "rollback" to the previous deployment to undo this change. You will have to boot into a Live ISO and edit the config file in the ostree repo to remove this config option.

Feedback welcomed!

[travier]

@tpopela, if you're OK with that, I'll make a change request for Fedora 37 as this involves Anaconda changes (setting default kernel arguments and setting option in the ostree repo) and workstation-ostree-config changes (for in-place updates for existing users, which is trickier).

[tpopela]

If it might prevent breakages (done by user) and aligns us more with CoreOS, then I'm definitely in favor.

[travier]

Great, I've started: https://fedoraproject.org/wiki/Changes/Silverblue_Kinoite_readonly_sysroot. Feel free to edit.

[jkonecny12]

@tpopela, if you're OK with that, I'll make a change request for Fedora 37 as this involves Anaconda changes (setting default kernel arguments and setting option in the ostree repo) and workstation-ostree-config changes (for in-place updates for existing users, which is trickier).

Great thanks. I can do the Anaconda implementation or someone else from the team if I don't have a time for that :).
Also if you want someone from the Anaconda team in the change - feel free to add me there.

[travier]

Great thanks. I can do the Anaconda implementation or someone else from the team if I don't have a time for that :).

Great. I'm not exactly sure what we should change in Anaconda: we will have to ensure that a kernel command line option is set (rw) and set a property on the ostree repo. Maybe we can do this directly in the Kickstart templates but I'm not familiar enough with how that's done.

Also if you want someone from the Anaconda team in the change - feel free to add me there.

Thanks! Will do.

[HarryMichal]

warningwarning Only try the following if you are confortable debugging an un-bootable system and have made backups warningwarning

How to try this, pending work on making this the default:

sudo rpm-ostree kargs --append-if-missing=rw
sudo ostree config --repo=/sysroot/ostree/repo set "sysroot.readonly" "true"
sudo systemctl reboot

Note that you can not "rollback" to the previous deployment to undo this change. You will have to boot into a Live ISO and edit the config file in the ostree repo to remove this config option.

Feedback welcomed!

Just tried the mentioned steps and my system successfully booted without any problems. I already applied the fix from https://docs.fedoraproject.org/en-US/fedora-silverblue/troubleshooting/#_selinux_problems to make my system usable after being hit by the SELinux issue (#234).

[travier]

This is being implemented in:

• https://pagure.io/workstation-ostree-config/pull-request/277
• rpm-ostree: Setup readonly sysroot for ostree & rw karg rhinstaller/anaconda#4240

[richiedaze]

Workaround

Adding these labels manually to selinux policy makes these relabeling problems obsolete.

ostree.cil

(roleattributeset cil_gen_require system_r)
(filecon "/ostree" symlink (system_u object_r usr_t ((s0) (s0))))
(filecon "/sysroot/ostree(/.*)?" any (system_u object_r usr_t ((s0) (s0))))
(filecon "/sysroot/ostree/boot(.*)/(.*)/(.*)/0/(.*)?" any ())
(filecon "/sysroot/ostree/boot(.*)/(.*)/(.*)/0" symlink (system_u object_r usr_t ((s0) (s0))))
(filecon "/sysroot/ostree/boot(.*)" symlink (system_u object_r usr_t ((s0) (s0))))
(filecon "/sysroot/ostree/deploy/(.*)/deploy/(.*)/(.*)?" any ())
(filecon "/sysroot/ostree/deploy/(.*)/deploy/(.*)\.origin" file (system_u object_r system_conf_t ((s0) (s0))))
(filecon "/sysroot/ostree/deploy/(.*)/deploy/(.*)" dir (system_u object_r root_t ((s0) (s0))))
(filecon "/sysroot/ostree/deploy/(.*)/var/(.*)?" any ())
(filecon "/sysroot/ostree/deploy/(.*)/var" dir (system_u object_r var_t ((s0) (s0))))
(filecon "/sysroot/ostree/deploy(/.*)?" dir (system_u object_r usr_t ((s0) (s0))))
(filecon "/sysroot/ostree/repo/extensions/rpmostree/private/(.*)?" any ())
(filecon "/sysroot/ostree/repo/objects/../(.*)\.dirmeta" file (system_u object_r system_conf_t ((s0) (s0))))
(filecon "/sysroot/ostree/repo/objects/../(.*)\.dirtree" file (system_u object_r system_conf_t ((s0) (s0))))
(filecon "/sysroot/ostree/repo/objects/../(.*)\.file" any ())
(filecon "/sysroot/ostree/repo/objects/../(.*)\.filez" any ())
(filecon "/sysroot/ostree/repo/state/(.*)\.commitpartial" file (system_u object_r system_conf_t ((s0) (s0))))
(filecon "/sysroot/ostree/repo(/.*)?" any (system_u object_r system_conf_t ((s0) (s0))))
(filecon "/sysroot/run" dir (system_u object_r var_run_t ((s0) (s0))))
(filecon "/sysroot/tmp" dir (system_u object_r tmp_t ((s0) (s0))))
(filecon "/sysroot(/.*)?" dir (system_u object_r root_t ((s0) (s0))))

Testing

Test relabeling with the -n option (to show would label tags):
restorecon -vrn /sysroot/*

Note: Every file without these labels would be relabeled to default_t as per files.fc default labeling.

Copy the above to a file named ostree.cil

Install policy rules:
sudo semodule -v --install=ostree.cil

Test relabeling again with the -n option (to show would label tags):
restorecon -vrn /sysroot/*

Question

• Should these labels be pushed upstream to ostreedev/ostree in a selinux folder like flatpak does.
  or
• Should these labels be pushed upstream to fedora-selinux adding it to the files.fc
  ???

[travier]

I'm not sure this will work. The content in /sysroot/ostree/deploy/... is specific to each deployment. Although unlikely, the policy and labels might differ from a deployment to the other thus a given version of the policy for the currently booted deployment might not correspond to what's in a previous or later /etc.

Thus the only safe thing to do here is to not touch those files and leave them as-is, which is the goal of the read only sysroot work.

[cgwalters]

Yes, only ostree should be changing things in /ostree.

(In the future actually I'd like to go one step farther and fully hide the data in /sysroot from the default mount namespace)

[jkonecny12]

I'm not happy about hiding things. Let's make them more robust to change but Linux distribution should be transparent so you can easily learn how stuff is working.

[richiedaze]

/usr mounted "readonly"

• has policy covering filesystem labels
• restorecon is blocked from changing labels

/usr remounted to "read/write" (by admin)

• has policy covering filesystem labels
• restorecon would relabel them accordingly to the policy

/sysroot mounted "readonly"

• does not have policy covering filesystem labels
• restorecon is blocked from changing labels

/sysroot remounted to "read/write" (by admin)

• does not have policy covering filesystem labels
• restorecon would relabel them accordingly to the policy

In the first example relabeling works as it was configured to do. In the second example, the /sysroot filesystem is not properly labeled. Remounting and relabeling would relabel most of /sysroot to the default_t labels again bypassing this remedy. This proposal does help a lot, but in my opinion is not the correct way to fix this issue.

Assuming anaconda sets the labels correctly from install and ostree pulls everything with a proper label, not having a policy will not maintain it's label consistency, rendering labeling redundant.

In my example policy, I labeled the root filesystems built by OSTree in it's build order before it mounts them to the system's root directory.

• Folders are labeled.
• Links are labeled.
• Source archives are labeled.
• OSTree's var and etc folders are labeled with the <<none>> tag
  • but when mounted can be relabeled properly
• All deployments are labeled with the <<none>> tag
  • ostree created deployments would retain it's correct labels
• All repo Objects and Hardlinks are labeled with the <<none>> tag
  • ostree created repos would retain it's correct labels

*** The <<none>> labels protect relabeling from normal ostree operations.
*** The directory/link labels I declared, protect relabeling anacondas set labels

IMHO OSTree should have standard filesystem labels for all it's filesystem before they are mounted in the ostree deployment, the same way flatpak and containers have done.

In the end, users are expecting the relabeling program(s) to correct any incorrect labels caused by the system or by the user.

[travier]

This is not a question whether or not files in /sysroot should have a label. They have one. The point is that you can not know which one they should have when running a given deployment. rpm-ostree/ostree manages those already. I don't think it's worth adding more complexity for something no one will hit once we've made that RO.

[travier]

This change is now available in Rawhide and F37: https://bugzilla.redhat.com/show_bug.cgi?id=2060976

[jkonecny12]

Thanks a lot for implementing this. I think it is another great step to make SilverBlue unbreakable system :).

Closing this issue now. I think the implemented change is great solution to go.

[travier]

For folks that want to get this early, it's part of the Fedora 37 Beta that has been recently released.