why remount-secure even needed? fix insecure mount options at the root, not here

[adrelanos]

Which software / source code sets the initial insecure mount options anyhow?

Could we fix it there instead of adding a band-aid on top?

[monsieuremre]

Well, actually there is a better way. A solution that is no band-aid. And it is modifiying the fstab. There is also another solution, that is direct and involves no remounting. We can write custom systemd mount targets and ship them with the package. The only problem with the latter is, fstab entries take precedence over those units. So we can't harden what is covered in fstab with this way.

[monsieuremre]

Like if the user has a seperate /home volume, we can't harden it with our custom systemd unit. I think the reason behind this is, systemd dynamically overrides our unit with the unit it auto generates from fstab on startup.

[adrelanos]

That doesn't really answer why it's insecure by Debian (or most Linux distributions) default.

[monsieuremre]

Because by default debian and most distributions don't do too much disk partitioning. If you want a seperate tmp or var partition, most of the time you manually configure them when installing. And bind mounting to itself is clever workaround to get all the good mounting option benefits without having the partition itself, but it is really not that surprising that no distro would mount a directory to itself by default.

[monsieuremre]

A very highly unrelated sidenote: the only distribution that I see who ships secure defaults by default is openSUSE. Debian by default ships with very slack file permissions for anything. OpenSUSE however ships with already mostly hardened defaults. Tumbleweed gets a score of 87 on lynis after a default install. Debian and most other distros get like 60. And this isn't surprising, because Debian does not try to be better desktop or server os. Debian tries to be the universal os, compatible with every device ever. So actually makes sense for them to ship slack defaults. A price to pay for universal compatibility.

[adrelanos]

Where is the source code where it says "use these default mount options for this and that partition"?

[adrelanos]

Like if the user has a seperate /home volume, we can't harden it with our custom systemd unit. I think the reason behind this is, systemd dynamically overrides our unit with the unit it auto generates from fstab on startup.

Created #161 for it.

[adrelanos]

quote https://lists.freedesktop.org/archives/systemd-devel/2019-December/043845.html

On a standard Debian system, the three tmpfs mounts (/run, /tmp, /dev/shm) already have the nosuid and nodev options – this is hardcoded in mount-setup.c. So you should first figure out why they are not present in your case to begin with.

This might be the referenced source file:
https://github.com/hramrach/systemd-1/blob/master/src/core/mount-setup.c

a) The remount-secure.service unit file should run alone. Non-parallel.
Until remount-secure.service is done, no other systemd unit files should
be run. Is that possible with systemd?

Have it run before systemd (i.e. do it from the initramfs). If I remember
correctly, if the initramfs pre-mounts /run &c. with the correct options,
then systemd will just use that as-is and won't downgrade to the hardcoded
options.

This implies that initramfs (initramfs-tools, dracut) is responsible for mount options too.

[monsieuremre]

Ok I tell you the source. Systemd calls the binart mount with the option --all on boot. And mount, unless the mount options are overriden with the option --option or -o, looks for the options in the fstab file, which by default has the path /etc/fstab. What we can do is:

• We modify the mount package. Uninstall the one packaged by debian, package our own, modify the source, make the fstab path default to /something/kicksecure/fstab, and the world is good.
• Modify the calling of mount --all, and make it be called with mount --all --fstab /something/kicksecure/fstab.
• Don't play any games and just override the fstab, for simplicity.

These would not be necessary for stuff that have no entries in fstab. Like if the use has no seperate partition for var, we can create a var.mount systemd unit and define our default options there and life would be easy. But unfortunately, if there is a var entry in fstab, our unit will be overridden.

[adrelanos]

But dracut earlier also already uses mount? Or do you mean the systemd which already runs during dracut?

I couldn't find yet the mount options that dracut uses.

[monsieuremre]

When you mount, if you do -o or --options, then you define options, overriding what was the default. If you don't override, then mount looks into fstab for the options. When no options there, then it looks to the respective systemd unit. If it doesn't exist too, then it gives out an error. This is my understanding.

[adrelanos]

At least 3 different things we could accomplish here.

• A) Harden mount options of newly built Kicksecure (or Whonix) VM images. And/or,
• B) Harden mount options of newly built Kicksecure (or Whonix) ISO images. And/or,
• C) Harden mount options of arbitrary systems (such as a Debian based server which gets distribution morphed into Kicksecure).

In case of A),

• for non-Qubes Kicksecure based, the question is why isn't /etc/fstab hardened by default. It's because it's built by grml-debootstrap's (in Debian) chroot-script.
  • We could contribute to grml-debootstrap to harden /etc/fstab as much as possible as upstream accepts as well as add an option --hardened-fstab or --custom-fstab or something like this with an alternative version that hardens it even more (such as in case upstream does not want to mount /home with noexec by default).
• in case of Qubes, their /etc/fstab is here and I've recently sent a PR to improve formatting of /etc/fstab QubesOS/qubes-core-agent-linux#472 which will maybe be the basis for further improvements.
  • Qubes may or may not accept /home mounted with noexec by default.
  • Related tickets:
    • Mount /rw and /home with nosuid + nodev QubesOS/qubes-issues#5263
    • mount /tmp /var/tmp /dev/shm with nodev nosuid QubesOS/qubes-issues#5329

In case of B),

• When installing, it will be probably calamares based. So calamares installer would create /etc/fstab, we'd need to look into how calamares handles that, contribute upstream or write a custom calamares module. But that is blocked until an ISO emerges. (Btw progress was made there recently on ISO building, see deriative-maker help-steps/image-to-iso.)

In case of C),

directly writing to /etc/fstab from security-misc is risky. Hence such a systemd unit + script makes sense.

[adrelanos]

dracut also supports /etc/fstab.sys. Not well documented. But can be seen in dracut source code.

[monsieuremre]

Yeah I don't think that will work. Pushing to debian upstream. Like it might, but the debian team is very unresponsive unless it is a bug fix. It would happen probably, but like, in about 3 debian releases later. We can just make our own good defaults ISO. Which isn't that hard to be honest. I feel like we are on our own for the most part tho.

I don't want to take every opportunity to shill suse, so I'm sorry. But opensuse puts in effort to harden their default fstab entries.

[adrelanos]

Pushing to debian upstream. Like it might, but the debian team is very unresponsive unless it is a bug fix. It would happen probably, but like, in about 3 debian releases later.

Debian isn't a conglomerate. It depends on the individual packages, maintainers. I was talking about grml-debootstrap. Pushing fixes to grml-debootstrap seems entirely doable. I managed to land a few patches in grml-debootstrap. The only condition is that it needs to be clean, high quality code. You can see my commits here:
https://github.com/grml/grml-debootstrap/commits/master

It doesn't take long either until it reaches derivative-maker. I copied grml-debootstrap to derivative-maker source code folder. A soft fork if you want so. (Not with the intention splitting from upstream due to disagreement or running a separate project. Just to make the updated code available earlier in derivative-maker waiting for the next major Debian release. No big story.)

grml-debootstrap recently learned arm64 support which was contributed by a Whonix contributor. Upstream grml-debootstrap took even did the cleanup / refactoring commits. Building VM images with EFI and Secure Boot was also recently contributed and merged by another contributor.

Until now, grml-debootstrap upstream is entirely reasonable. I might even be capable of running a "full fork" of grml-debootstrap should upstream disappear which I don't hope because they've been doing great work for years.

Improving grml-debootstrap fstab will result in improved Kicksecure VM images. (Would also be possible to leave grml-debootstrap as is and add a better fstab at the derivative-maker level only but that would be more hacky and less beneficial for the overall ecosystem, less use and testing, less bugs only happening inside Kicksecure and derivatives.)

Non-responsiveness of other Debian maintainers is inapplicable here.

[monsieuremre]

Debian isn't a conglomerate. It depends on the individual packages, maintainers.

You are right. I was talking about changes you would suggest debian to make in their iso installer. In that case they would not be involved anyway.

And also the unresponsiveness of debian maintainers are not meant in a bad way. They are very responsive on security fixes and bug fixes, but thats pretty much it. If they wanted, they could be responsive to feature requests, if they weren't shipping one billion packages in their repositories, which even for them is really impossible to maintain. Instead they choose the way of trying to be the universal os. Which I think is something that holds kicksecure back. There are two things I see that are holding kicksecure back. First is QubesOS compatibility. Which is understandable and there isn't much to do against this. The second is debian. Debian's package freezing is really not we want. Like even if you fix your thing upstream, good luck waiting another 2 years till the next debian, in which that version still might not be in the repos anyway. And they even microcode updates too, between point releases. This is also, a no no for a security system. Actually imma open an issue in a sec with a suggestion.

[adrelanos]

Debian isn't a conglomerate. It depends on the individual packages, maintainers.

You are right. I was talking about changes you would suggest debian to make in their iso installer. In that case they would not be involved anyway.

Btw the installer would be calamares based which is also a project independent of Debian which apparently is popular, used by many Linux distributions nowadays, configurable, supports plugins, is responsive. It's also packaged for Debian.

Any fstab issues could be reported there. However, I am not sure how great the chances are of fstab hardening feature requests to be implemented without also sending a pull request doing the actual implementation. Testing calamares fstab and feature request if applicable would be a helpful contribution nonetheless to see what upstream calamares thinks about this.

(A calamare based installer is entirely realistic. There once was a Whonix-Host installer. Implying that the same would be possible for Kicksecure. The calamares config files can nowadays be found in Kicksecure package live-config-dist. It was deprecated for other reasons because it was based on isolinux bootloader, only supported legacy BIOS booting, EFI booting was unsupported with no known solution on how to fix that. The new upcoming implementation won't have these issues.)

[adrelanos]

A completely different approach would be having no fstab at all. I've had a chat with chatgpt on this topic:
https://chat.openai.com/share/0c573e00-06bf-41a1-8f86-010624172bdc

In summary, use bootloader, dracut and systemd mounts to mount the root disk and everything else.

Not sure that is possible with grub. systemd-boot might be capable of that. But systemd-boot isn't signed by Debian yet. [1] For rather mysterious reasons if you read that ticket. [2]

Dunno if the Boot Loader Specification (BLS) is required which grub might not support. I couldn't find a ticket for grub. Seems like some distributions as Fedora are patching grub.

In summary, seems difficult to research and create a stable implementation to me.

[1] https://groups.google.com/g/linux.debian.bugs.dist/c/n0-eRL6eVtQ
https://salsa.debian.org/systemd-team/systemd/-/merge_requests/132
[2] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=996202

[monsieuremre]

In summary, seems difficult to research and create a stable implementation to me.

Well then I am really pleased to share that I have found a way. Real, no bandaid, no workaround. Just mounting stuff the way we want it. I tested it too. It's just writing mount units, but they have to be under etc.

From the manpage:

If a mount point is configured in both /etc/fstab and a unit file that is stored below
/usr, the former will take precedence. If the unit file is stored below /etc, it will take
precedence. This means: native unit files take precedence over traditional configuration
files, but this is superseded by the rule that configuration in /etc will always take
precedence over configuration in /usr.

It means exactly what it means. Tho fstab takes precedence over mount units, mount units under etc take precedence even over fstab. So we just create mount unis under etc. We can even do this under a something.mount.d directory to allow the user to override us. The only thing that is problematic is, the mount options. We can not preemptively know before shipping a mount unit for home if needs bind or not. If we tackle that, I think we got a solid solution.

[adrelanos]

How about /usr/lib/systemd/system folder?

[monsieuremre]

Unfortunately doesn't work. Gets overriden by fstab. You can see it in the manpage section I quoted.

[adrelanos]

The only thing that is problematic is, the mount options. We can not preemptively know before shipping a mount unit for home if needs bind or not.

systemd supports many conditionals:
https://www.freedesktop.org/software/systemd/man/latest/systemd.unit.html#Conditions%20and%20Asserts

Such as ConditionPathExists, ConditionPathIsSymbolicLink. ConditionPathIsDirectory, ConditionPathIsMountPoint, ConditionDirectoryNotEmpty and many others.

Does that work? If not, please post a systemd feature request. 1) The feature might exist already anyhow 2) they might implement the feature.

[adrelanos]

I'd still prefer /usr/lib/systemd/system folder.

Then how about /etc/fstab? Well, that's "just a detail". "Just part of the migration plan to this feature."

• A) For new Kicksecure (VM) builds: They would come without an /etc/fstab, or with an empty /etc/fstab or ideally if possible with an /etc/fstab that only contains comments. (phase 1)
• B) For existing Kicksecure VM builds that are upgraded: If /etc/fstab is unmodified which it probably is for most users, it might be possible to safely delete it to upgrade users to the new method. This could be done at a later time. (phase 2)
• C) For distribution morphed installations of Kicksecure such as servers: The migration is more difficult here as somebody or something (an installer) else might have created /etc/fstab. In that case, maybe security-misc should just show a warning during installation and recommend deprecation of /etc/fstab.

These migration details can be tackled later. First priority are the systemd units as a proof of concept.

[adrelanos]

Yeah, a systemd based implementation would be best. We could deploy the more restrictive mount settings by default. (/home mounted as noexec.) If users want to undo that, they could just use the standard systemd drop-in file mechanism to overrule this. This would be documented in the wiki.

Another detail: For Whonix, tb-starter would need a sudoers exception to run a script which mounts Tor Browser's folder (unfortunately resides in /home) with exec. But that's doable. Got similar code for that already.

[monsieuremre]

Actually we kinda need the fstab to read the mount points. At least initially. Else how would we know where to mount /boot for example? If we write mount units under /usr/lib with --bind options, this might be a good enough solution. If there is no partition, it gets binded to itself with the secure defaults. If there is a partition, it gets overriden by fstab anyway, so the unit will do nothing. This won't harden a the /home directory if it is on a seperate partition for example.

[adrelanos]

Yeah, a systemd based implementation would be best. We could deploy the more restrictive mount settings by default. (/home mounted as noexec.) If users want to undo that, they could just use the standard systemd drop-in file mechanism to overrule this. This would be documented in the wiki.

To avoid user complaints and get everyone on board:
https://www.kicksecure.com/wiki/Noexec

[adrelanos]

Actually we kinda need the fstab to read the mount points. At least initially. Else how would we know where to mount /boot for example?

That is what I meant by

In summary, seems difficult to research and create a stable implementation to me.

in #157 (comment) which also mentions how that might be possible to go completely fstab-less. (grub kernel parameter, dracut to mount the root disk, systemd to handle all other mounts)

It might be the case to go fstab-less would only be possible for (new) Kicksecure (VM) builds where we know the (simple) partitioning. Next version will be built using grml-debootstrap using --vmefi, GPT partition table, separate EFI partition (unfortunately, would prefer without but impossible), no separate /home or /boot partition needed). In that case, even disk UUID is a non-issue thanks to grml-debootstrap FIXED_DISK_IDENTIFIERS=yes build env var.

I had in mind here a completely static systemd mount files configuration. Nothing auto generated by any scripts. Just simple systemd config snippets. And no fstab at all.

This won't harden a the /home directory if it is on a seperate partition for example.

There could be two different systemd mount drop-ins? One with ConditionPathIsDirectory the other with ConditionPathIsMountPoint?

[monsieuremre]

Addressed in the new pull request. For the most part it is solved.

[monsieuremre]

A little off topic:

By the way the guide you provided, I have also gone through it several times before, but it should be noted that it is super duper old. Mounting /tmp with noexec will not cause any problems almost ever. What they are mentioning is a legacy issue, that some old old software used to install itself by running scripts in /tmp. This is super prone to exploits and no legit program should ever run anything on /tmp, ever, there is no legit reason for this. The software that breaks because of this is the one thing that would need fixing. And also, no one does this anymore anyway. As such, this has not been a problem for like ever. The issue they mention on that link is from 2001. So I think it isn't much of a stretch to assume this is no more relevant.

The only case where running some kind of anything on /tmp that I can think of is, maybe, software development and debugging. Like some sort of compilation stuff while debugging, I don't know, maybe then, it might be used legitimately. But even then I think it is not the only way. But this won't be the only thing that will break given our hardened kernel parameters. So no, we should go nexec for tmp all the way, which I think we are, so nice.

[monsieuremre]

By the way, while we are at it, I wanna refer to #178, where you said:

Compatibility with the kernel version in Debian stable is interesting but not a blocker for getting merged as long as nothing breaks.

In that case I would suggest we set the following parameters in grub, as they will work for the newer systemd versions.
systemd.mount-extra=udev:/dev:devtmpfs:defaults,nosuid,noexec
systemd.mount-extra=tmpfs:/dev/shm:tmpfs:defaults,nosuid,noexec,nodev
systemd.mount-extra=tmpfs:/tmp:tmpfs:defaults,nosuid,noexec,nodev

This has the same behavior as adding new lines to fstab. And since these are universal and system independent, they can be set this way. They won't change based on the users partitioning scheme.

[adrelanos]

I am also surprised we're the first ones inventing a hardened fstab file and publishing it.

There are some problems.

First of all, /run is mounted with noexec by default on debian 12.

What is "Debian"? I mean, where is it down, how, source file? This could be specific to initramfs-tools vs dracut vs other init systems vs booting a system for real vs systemd-nspawn etc.

If it is different on a debian qube, that is a qubes os thing and not a debian thing.

Also Qubes is also a legitimate user of Debian and Qubes most likely didn't purposefully downgrade /run. Rather something else (something similar to initramfs-tools vs dracut).

Secondly, anything that is not on disk should never be bound. Not to itself, not to anywhere.

Why?

For example hardening /dev would be like this:

udev                                                         /dev             devtmpfs  defaults,nosuid,noexec            0      0

Added and then from the running system sudo mount -a. Error message:

mount: /dev: udev already mounted or mount point busy.
dmesg(1) may have more information after failed mount system call.

Works for me only when using bind.

Could say your syntax works when booting but not from the running system. But using bind seems easier to to work with since then no reboot is required every time.

Maybe the current syntax would fail to boot (or actually do nothing).

In any case, a completely hardened fstab file that is fully tested would be most helpful.

I would recommend using the defaults option at the beginning.

Ok.

Also nofail should not be used when not debugging. We probably want the mounting to happen for sure. If not happening, we want it to fail and not boot.

If mounting /var as noexec,nosuid fails, I'd prefer a booting system. Way easier to debug and fix than a non-booting system.

Unit testing can be done in other places such as in systemcheck.

In any case, nofail can stay for the first iteration and maybe removed later when this is stable.

Also, is this planned to be a complete fstab file?

Yes, a complete fstab that can be used in Kicksecure VMs. Would a great start.

Because there are no lines for the regular real partitions, like the root partition /.

There are no partitions in the VM except for:

/dev/disk/by-uuid/26ada0c0-1165-4098-884d-aafd2220c2c6       /                auto      nofail,defaults,errors=remount-ro       0      1

And also /boot/efi but the latter is simply not done yet. I don't know if it has a stable uuid yet or maybe blocked by migration to (maybe) mkosi which creates stable uuids presumably.

A little off topic:

By the way the guide you provided, I have also gone through it several times before, but it should be noted that it is super duper old. Mounting /tmp with noexec will not cause any problems almost ever. What they are mentioning is a legacy issue, that some old old software used to install itself by running scripts in /tmp. This is super prone to exploits and no legit program should ever run anything on /tmp, ever, there is no legit reason for this. The software that breaks because of this is the one thing that would need fixing. And also, no one does this anymore anyway. As such, this has not been a problem for like ever. The issue they mention on that link is from 2001. So I think it isn't much of a stretch to assume this is no more relevant.

The only case where running some kind of anything on /tmp that I can think of is, maybe, software development and debugging. Like some sort of compilation stuff while debugging, I don't know, maybe then, it might be used legitimately. But even then I think it is not the only way. But this won't be the only thing that will break given our hardened kernel parameters. So no, we should go nexec for tmp all the way, which I think we are, so nice.

Great to know!

By the way, while we are at it, I wanna refer to #178, where you said:

Compatibility with the kernel version in Debian stable is interesting but not a blocker for getting merged as long as nothing breaks.

In that case I would suggest we set the following parameters in grub, as they will work for the newer systemd versions. systemd.mount-extra=udev:/dev:devtmpfs:defaults,nosuid,noexec systemd.mount-extra=tmpfs:/dev/shm:tmpfs:defaults,nosuid,noexec,nodev systemd.mount-extra=tmpfs:/tmp:tmpfs:defaults,nosuid,noexec,nodev

This has the same behavior as adding new lines to fstab. And since these are universal and system independent, they can be set this way. They won't change based on the users partitioning scheme.

I want to test a hardened /etc/fstab first over several releases. Then everything is in 1 expected place and easy for users to customize.

When nothing breaks, maybe systemd kernel parameters can be used.

Maybe /etc/fstab could contain a comment to mention how /dev/, /dev/shm, /tmp is hardened (by using kernel parameters). That would be good too. Feel free to send a PR. That could remain open until the next Debian release.

Compatibility with the kernel version in Debian stable is interesting but not a blocker for getting merged as long as nothing breaks.

True, I said that. However, in this case I would make an exception. This is the second step. Would be rushing ahead and confusing myself to merge that right now. Also because of the potential regressions when porting to the next major version of Debian. Could take a while to remember and figure out what is causing it.

Because /etc/fstab hardening is realistic in this Debian release cycle so that should be done as first step.

[monsieuremre]

I am also surprised we're the first ones inventing a hardened fstab file and publishing it.

I don't think we are. Debian already does some hardening for api file systems. For example run is fully hardened and /dev/shm is partially hardened on debian, which would not be the case by default if debian devs didn't set it that way.

What is "Debian"? I mean, where is it down, how, source file?

On a debian system run cat /usr/share/initramfs-tools/init. This is where the mount options for api file systems are set. You can manually edit this script file to set whatever you want. In this file for example, there is a line mount -t sysfs -o nodev,noexec,nosuid sysfs /sys. This line is the choice of debian devs. By the way, the difference in /run hardening may be different on a system with dracut, but on debian dracut is not the default.

Also, I have discovered that, api file systems are still remounted if we modify the fstab. Because when these file systems are mounted the first time, there is no universal way of accessing the fstab file, so the init script just runs and does its thing. If there is a line for an api file system in fstab, they are remounted by systemd-remount-fs.service. So the ultra clear solution would be modifying the init script, which I think is possible for packages to do as well.

You asked

Why?
not to use bind mounts for non disks things. Because, first, these file systems are on ram for a reason. They are volatile and fast. Binding them to themselves or to somewhere else introduces unneccessary extra mounts and binds. This is extra complexity. Not to mention that some sub-directories under these paths are already bound to other places for legacy reasons, so when we bind the parent or the child, this may introduce a lot of multiple bind mounts and one path being accessible through several ways. Which, at best is just superfluous functionality, at worst is a vector that slow us down for no reason.

Could say your syntax works when booting but not from the running system. But using bind seems easier to to work with since then no reboot is required every time.

Because it is meant to work on boot only. Bind mounting provides no solution because we never use fstab to mount things midway. But it creates new problems on its own. This would also be true if we had a line for proc there. We can also use bind, but then if a process was accessing that path before we bind, it won't be affected by our hardening at all, including init I think, which obviously can't be good if anything.

So for the ultimate solution, I suggest we find a way to solve the api file systems in initramfs. For other, real partitions, we already have a working solution that uses drop in config files, from my old pull request. So combining these would be, I think, optimal.

[adrelanos]

What is "Debian"? I mean, where is it down, how, source file?

On a debian system run cat /usr/share/initramfs-tools/init.

This is only 1 way to use Debian. That is initramfs-tools, which Kicksecure doesn't use. At time of writing, Kicksecure uses dracut by default. Reference:
https://forums.whonix.org/t/replacing-initramfs-tools-with-dracut/4487

It is also possible to have an initrdless system (a kernel with all required modules compiled-in), perhaps other init systems (in the future maybe mkosi-initrd, which is a systemd based initrd without special init queues by initramfs-tools or dracut, which might simplify initrd a lot), and certain types of virtualization such as systemd-nspawn does not use and initrd. Might be similar for docker, OpenVZ.

They all however should be honoring /etc/fstab.

Also, I have discovered that, api file systems are still remounted if we modify the fstab. Because when these file systems are mounted the first time, there is no universal way of accessing the fstab file, so the init script just runs and does its thing. If there is a line for an api file system in fstab, they are remounted by systemd-remount-fs.service. So the ultra clear solution would be modifying the init script, which I think is possible for packages to do as well.

There's a dracut module right in security-misc:
usr/lib/dracut/modules.d-disabled

It was working but it was said to be slow. Maybe the slowness issues could be worked on by debugging the script. Tracing, debug output execution times throughout its run. Perhaps parallelization, i.e. running the mount commands into the background.

If you're interested in initramfs-tools or any other init system, you could contribute fstab hardening upstream. That would also get way more scrutiny and probably stability than anything invented and only in Kicksecure.

So for the ultimate solution, I suggest we find a way to solve the api file systems in initramfs.

Also seems we're going in circles here (as the dracut module was previously enabled) and without writing a technical design with all the different approaches, pros and cons of each, solution impartial research this will probably keep going in circles.

[adrelanos]

Created an overview now:
https://www.kicksecure.com/wiki/Dev/remount-secure

Contributions welcome.

There are at least 6 different approaches to mount options hardening, 3 different types of mount points (API, partition, folder) and various different environment.

[monsieuremre]

This is only 1 way to use Debian. That is initramfs-tools, which Kicksecure doesn't use. At time of writing, Kicksecure uses dracut by default. Reference:

Ok very interesting. I have said this is the debian default because debian by default does not use dracut. Because systemd is also just one way to use debian, since they offer other init systems in their repos. But since it is the default, I think it is reasonable to say debian by default uses systemd, and by default uses initramfs-tools. I see the reasoning the porject had to switch to dracut is to support a live system. Is this necessary? I'm pretty sure Ubuntu uses initramfs and they offer a live system. So I don't know why you switched there. Does this make development somehow easier?

If you're interested in initramfs-tools or any other init system, you could contribute fstab hardening upstream. That would also get way more scrutiny and probably stability than anything invented and only in Kicksecure.

I agree. But for this, the place to contribute is the package maintainer of the distribution. Because even this semi-hardening is the choice of debian packagers, and not the upstream devs, far as I know. So instead of bothering with that, we might as well fix it for us ourselves.

Created an overview now:
https://www.kicksecure.com/wiki/Dev/remount-secure

Good. This is a good way to keep track of this long discussion. It is more complicated than the our good old usual hardening and there are many ways.

[adrelanos]

btrfs

https://www.kicksecure.com/wiki/Dev/filesystem

[adrelanos]

I am getting confused by the many different approaches, tickets, pull requests on this topic.

Wasn't there some git branch you wanted to restore, older approach which you wanted to re-vitalize? Instead #195 was done?

[adrelanos]

found the comment here:
#165 (comment)

Ok good. Soon when I find the time I will restore the branch, check everything again and reopen the pull.

[adrelanos]

This is only 1 way to use Debian. That is initramfs-tools, which Kicksecure doesn't use. At time of writing, Kicksecure uses dracut by default. Reference:

Ok very interesting. I have said this is the debian default because debian by default does not use dracut. Because systemd is also just one way to use debian, since they offer other init systems in their repos. But since it is the default, I think it is reasonable to say debian by default uses systemd, and by default uses initramfs-tools. I see the reasoning the porject had to switch to dracut is to support a live system. Is this necessary? I'm pretty sure Ubuntu uses initramfs and they offer a live system. So I don't know why you switched there. Does this make development somehow easier?

Live mode (grub-live) indeed also works with initramfs-tools.

The full story about porting to dracut is here:
https://forums.whonix.org/t/replacing-initramfs-tools-with-dracut/4487

Main reasons here:
https://forums.whonix.org/t/replacing-initramfs-tools-with-dracut/4487/15

Updated summary why here:
https://forums.whonix.org/t/replacing-initramfs-tools-with-dracut/4487/32

The very main reason is:
https://www.kicksecure.com/wiki/Ram-wipe

[adrelanos]

After long consideration of the many different options, I've decided that /etc/fstab file based mount options hardening is the most suitable way to implement secure mount options for Kicksecure. This is the way forward.

Contributions to /usr/share/doc/security-misc/fstab-vm are welcome.

[wryMitts]

Will there be a way to control some mount options or control the timing on mounting? If it will use systemd to trigger the fstab read, being able to add an override.d like After=zfs-mount.service to the unit file would be useful.

With current configuration released to end users, I believe I am having a similar issue to this one:
(I have a ZFS dataset for /var/ and /var/log, etc)

Caused quite an interesting scenario where all my data became hidden with a recent patch.

https://old.reddit.com/r/zfs/comments/hpnmiw/systemd_mount_order_between_zfs_and_etcfstab_bind/

After long consideration ofthe many different options, I've decided that /etc/fstab file based mount options hardening is the most suitable way to implement secure mount options for Kicksecure. This is the way forward.

Contributions to /usr/share/doc/security-misc/fstab-vm are welcome.

[adrelanos]

Mounting /tmp with noexec would break initramfs-tools as reported here:

• pam-tmpdir-helper breaks certain initramfs-update actions on systems with noexec on the /tmp mount #198
• https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1062756

[adrelanos]

Will there be a way to control some mount options or control the timing on mounting? If it will use systemd to trigger the fstab read, being able to add an override.d like After=zfs-mount.service to the unit file would be useful.

No. There's no need for it. Simply a modified /etc/fstab configuration file by default. So any settings can be easily customized by the user as usual. No extra complexity, extra features, extra changes.

[monsieuremre]

No. There's no need for it. Simply a modified /etc/fstab configuration file by default. So any settings can be easily customized by the user as usual. No extra complexity, extra features, extra changes.

fstab is a user artefact. So the solution there would be to somehow configure the calamares installer or whatever we plan to use to set these secure defaults in the installer. But even then, there won't be binding anything. You can not add bind lines to fstab in the installer. So package would need to do that. And we are back to here.

Will there be a way to control some mount options or control the timing on mounting? If it will use systemd to trigger the fstab read, being able to add an override.d like After=zfs-mount.service to the unit file would be useful.

If you messed a little with these stuff you probably know that this gets really complicated real quickly. Even with /var and /var/log the order is not possible to configure in the fstab file. In a unit file, it is possible, but then it is almost always guaranteed to slow up the boot massively (when we are binding, especially nested binding).

[adrelanos]

No. There's no need for it. Simply a modified /etc/fstab configuration file by default. So any settings can be easily customized by the user as usual. No extra complexity, extra features, extra changes.

fstab is a user artefact.

• Well supported,
• not going away,
• any bugs are easily reproducible on Debian and be forwarded to upstream.

So the solution there would be to somehow configure the calamares installer or whatever we plan to use to set these secure defaults in the installer.

Calamares supports fstab configuration already.

Quote https://github.com/calamares/calamares/blob/calamares/src/modules/fstab/fstab.conf

options: "defaults,noatime,mode=1777"

But even then, there won't be binding anything. You can not add bind lines to fstab in the installer.

Why not?

[monsieuremre]

Why not?

Because the installer is a dynamic program. In like the expected scenario the user plugs in a usb and live boots to kicksecure, there is the live environment, he looks around for a minute, ok nice, now he decides to install. Calamares installer, ok nice. When you are at the place where you do the partitions and stuff, the installer does not do everything for you. The installer just makes suggestions basically, but the user still chooses the partitions and whatnot. In an ideal scenarion, we modify the default suggestions made by calamares to use secure defaults. Ok nice, but then there can't be any bind mounts. Which binds are we going to include? Does calamares even have an interface to suggest bind mounts in the installer?

There we need some program to make decisions with elaborate if else statements. Ok if boot is not a separate partition then do this, if it is, do this, if it is like a different file system, then obviously do these stuff because it is different on this. If the user changes the installation or formats one of the partitions or like a billion other things, we still have to at some point decide dynamically. The "let's put the config and forget about it and it works universally" logic only works for api file systems with kernel params and real mounts with real partitions with drop in configs. For binds, there is no such solution that I could find, at least not without creating several layers of abstraction.

The other issue is the fact that bind mounts are already problematic on their own. I remeber I encountered this problem. There is no way to make nested binds work universally. Firstly, having the wrong order with nested binds will result in two valid points of accessing a directory, one of which will easily circumvent the hardened mount options. To remove this possible work around, they have to be mounted in cordination with the order from inner to outer. And that, requires some big after before requires lines in the systemd config file because for some reason I remember it always tried to mount /var before /var/tmp. And then, once you find a solution with before after stuff, it just breaks after you install auditd. I remember I encountered this issue with nested bind mounts and I said ok, it is not worth it doing this with mount units. Is fstab going to be any less problematic? Doubt it. Maybe we can just avoid the nested binds all together? Yeah then, maybe fstab will work, but then, at that point the mount units will also work too.

Sorry for the long answer. Think of it as if I'm brainstorming loudly.

[adrelanos]

When you are at the place where you do the partitions and stuff, the installer does not do everything for you.

Partioning is automatic.

Which binds are we going to include?

Same as #157 (comment)

Does calamares even have an interface to suggest bind mounts in the installer?

No, and it's probably not needed either.
Seems like https://github.com/calamares/calamares/blob/calamares/src/modules/fstab/fstab.conf can do that.

And if needed, then calamares seems easily extensible with python modules.
This is also a functionality which might be welcome if contributed upstream.
But even just the fstab.conf (which is a calamares fstab.conf, not a literal /etc/fstab) might be sufficient.
Calamares already has some code for custom mount options.

There we need some program to make decisions with elaborate if else statements. Ok if boot is not a separate partition then do this, if it is, do this, if it is like a different file system, then obviously do these stuff because it is different on this. If the user changes the installation or formats one of the partitions or like a billion other things, we still have to at some point decide dynamically.

I am not sure Calamares supports custom partitioning or if that should be supported. But if Calamares can do that, then fstab.conf (speak calamares fstab module) can probably handle that as well.

The "let's put the config and forget about it and it works universally" logic only works for api file systems with kernel params and real mounts with real partitions with drop in configs. For binds, there is no such solution that I could find, at least not without creating several layers of abstraction.

The other issue is the fact that bind mounts are already problematic on their own. I remeber I encountered this problem. There is no way to make nested binds work universally.

If that's true, then this ticket is a cannot implement anyhow.

Firstly, having the wrong order with nested binds will result in two valid points of accessing a directory, one of which will easily circumvent the hardened mount options. To remove this possible work around, they have to be mounted in cordination with the order from inner to outer. And that, requires some big after before requires lines in the systemd config file because for some reason I remember it always tried to mount /var before /var/tmp. And then, once you find a solution with before after stuff, it just breaks after you install auditd. I remember I encountered this issue with nested bind mounts and I said ok, it is not worth it doing this with mount units. Is fstab going to be any less problematic? Doubt it. Maybe we can just avoid the nested binds all together? Yeah then, maybe fstab will work, but then, at that point the mount units will also work too.

Then upstream needs to be consulted regarding these issues.

[monsieuremre]

Once again. Calamares is a solution, no doubt, at least when installing. If we want a universal solution that ships with a package tho, the remount service is really a good solution. I have done some reading of systemd source code. For api file systems, changing fstab entries just makes systemd remount them with the new options using a service. The initial mounting still takes places with the options of the init script, written by debian mostly. So like, having our own service is not so different than modifying fstab for these file systems at least.

The trickster here is, the remount service systemd has is written in c. Shell scripting a bunch of remounts is slow. Also shell scripts are non universal. Bash or zsh, synthax changes. We should avoid having shell executables whenever possible, especially for services. This is not only for speed, but also for security.

The suid-disabler-permission-hardener deserves a separate repo. So does the remount-secure service.

Having the mount service and permission thing trigger on boot would be enough. Also we would trigger the permission thing with a post apt hook (better than using stat-override for various reasons).

I can personally write these things. I can write makefiles, and dedicated apparmor profiles and like at the end, you would just run make all and you would have everything in a conventional structure under a directory. Or literally like github workflows to build them binaries and put them in a good structure. Packaging that directory then would be the classical good old stuff that you already know. This is just my suggestion. Moving away from shell scripts is a better and cleaner way, and provides modularity. Having several services under a single package is not developer friendly. and if you think this would be complicated than necessary, well, it wouldn't be because the source code is gonna really short and minimalistic.

Either way, moving away from shell scripts or not, I think these services deserve separate repos.

[adrelanos]

The initial mounting still takes places with the options of the init script, written by debian mostly.

That one I would like to see the source code for.

The trickster here is, the remount service systemd has is written in c. Shell scripting a bunch of remounts is slow.

Why slow? The slowness isn't inherited from being shell scripting. If you run the remount-secure from within the running system it should complete in seconds. Perhaps 1-3 seconds. Please measure that. But during boot, it might be a lot slower slower. 10, 20 seconds? But that's not because of the shell. There must be other reasons for that. So first, please compare the speeds: - A) shell, from running system - B) shell, using systemd or dracut - C) versus compiled language

Bash or zsh, synthax changes.

In 12 years, bash never required a syntax change that I can remember.

We should avoid having shell executables whenever possible, especially for services. This is not only for speed, but also for security.

I am yet to see a threat model where remount-secure being shell versus compiled language results in a significant difference.

The suid-disabler-permission-hardener deserves a separate repo. So does the remount-secure service.

Either way, moving away from shell scripts or not, I think these services deserve separate repos. Yes, these functionalities are really heavy-weight and ideally had separate repositories. milestone: "one day"

Having the mount service and permission thing trigger on boot would be enough. Also we would trigger the permission thing with a post apt hook (better than using stat-override for various reasons).

For permission hardener that is already implemented. It has an dpkg trigger that runs wherever any package on the system is installed or updated. dpkg-statoverride is the tool to configure DPKG to prevent restoration of layer permissions when packages that contain files with SUID that are removed are upgraded. If you think there's something missing, please open a separate issue for that.

I can personally write these things. I can write makefiles, and dedicated apparmor profiles and like at the end, you would just run make all and you would have everything in a conventional structure under a directory. Or literally like github workflows to build them binaries and put them in a good structure. Packaging that directory then would be the classical good old stuff that you already know. This is just my suggestion.

Please maintain as independent upstream: source code, binaries, popularity, review, etc. That would help so I don't have to look into it. If it's independently used and reviewed, then integration into Kicksecure will be much simpler.

[adrelanos]

@monsieuremre in #203 (comment):

Once again, I want to give my opinion here. I am not a big fan of bind mounting things. My suggested implementation was, how would it be done if we absolutely wanted bind units. It goes without saying that remounting anything is completely unnecessary and does nothing but add extra overhead and complexity. These can be hardened with drop in files, as I suggested previously. Api file systems can be hardened from the kernel command line, again, a drop-in file. Bind mounting stuff is where we can't use drop in files. Actually, if we wanted, we could. But I already am not a fan of bind mounts. We can create bind units under /usr/lib/systemd, but really, should we be the one to have the var.mount unit? Like someone else might want to own that file, I don't know. My suggestion is, let's do drop in's boot parameters. I don't support bind mounts. But if it is desired, it can be done with either

• a service such as this (messy)
• boot parameter, again, fstab lines (preferred)
• systemd units under /usr/lib (less messy, but still is messy)

For API file systems and real partitions, a service or a hook should not be used. This is not only unnecessary, but introduces overhead.

@adrelanos above:

After long consideration of the many different options, I've decided that /etc/fstab file based mount options hardening is the most suitable way to implement secure mount options for Kicksecure. This is the way forward.

Contributions to /usr/share/doc/security-misc/fstab-vm are welcome.

Solutions are all messy but the least messy I've settled with is

• 1. this remount-secure systemd unit file, or
  • test remount-secure script and systemd unit #203
• 2. perhaps a Kicksecure default image / installation fstab file.

Otherwise there are too many mount points.

• A) API file systems
• B) real partitions
• C) directories

And each requirement a different mechanism to change these would make it even more messy.

let's do drop in's boot parameters

It cannot really cover all mount points A), B), C)?

[monsieuremre]

It cannot really cover all mount points A), B), C)?

It surely can cover everything apart from API file systems, and they can be covered with just drop in kernel parameters, but its just not a good idea.

[adrelanos]

It cannot really cover all mount points A), B), C)?

It surely can cover everything apart from API file systems, and they can be covered with just drop in kernel parameters, but its just not a good idea.

Anything missing here?

https://www.kicksecure.com/wiki/Dev/remount-secure#systemd_kernel_parameter_based_mount_options_hardening