Nix Support (And maybe Fedora Silverblue) (due 2024-02)

[directionless]

• Hardcoded paths to binaries everywhere #833
• osqueryd is not statically linked #770
• Running from a read-only device messes up the autoupdate #638
• Table Request: NixOS Installed Packages #868
• Nix will need some kind of finalization step on downloaded osqueryd binaries #1200
• Autoupdate path needs to be configurable (See TUF work)
• Launcher needs new and improved enrollment secret handling #1473

[znewman01]

Hi all! Thanks for your patience with our weird OS 😄 I've just filed an issue for adding the launcher to the Nix package repository. We'd love to work with you on getting support for NixOS in a good place.

I can send a PR (no concrete timeline; let me know if it becomes urgent) for #833. I'll also chime in on a couple of the other issues. Let me know if I can help otherwise (especially testing things out).

[directionless]

Bouncing to this ticket... If Nix is like the other distros, getting osquery in the formal packages will be quite hard. Osquery is designed to build with its own third party libraries and versions. And most distros really want software to build with the shipped libraries. I've seen a couple of distros ship and then drop osquery because of this.

Kolide ships our own packages for things, because we don't need to follow all the os packaging rules. So we can just ship an osquery binary.

I can do a little to try to help get these into Nix itself, but I think that's a hard road. I'd love to get a Kolide distributed nix package for launcher. But I know very little about how that would work. (Like, what's a package here? Is there something like a build thing, like a deb, or is it closer to only distributing source manifests?)

[terlar]

Regarding having specifically pinned third party libraries and versions for a specific package is not something that would be a problem for a nix package as far as I understand. If needed the package set can contain several different versions of the same package, especially if required to build another package. What kind of libraries are we talking about here?

A package are instructions on how to build a piece of software/library in the end it will result in some outputs that will be stored in the Nix store. It will include all required inputs, build instructions and "install" instructions.

For example:

• https://github.com/NixOS/nixpkgs/blob/master/pkgs/applications/version-management/git-and-tools/ghq/default.nix
• Or a more complicated beast (with 2 versions also): https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/libraries/openssl/default.nix

[directionless]

You're asking about the osquery build? It links to things via git submodules. Nose around https://github.com/osquery/osquery/tree/master/libraries/cmake

[terlar]

Yes. I see, a quite extensive list. It is definitely possible, but it could perhaps be a maintenance burden. Not sure if it would be decided to have another set of libs/packages or if the route would be to adapt osquery to work with the stuff that is already packaged. Since a lot of packaging usually requires specific modification for Nix. I'm not super familiar with cmake. But perhaps all these libraries are built as part of osquery? Then perhaps they wouldn't need to be built separately...

[directionless]

The nature of what osquery does makes it quite sensitive to versions and build parameters. Over the last many years we've pulled the dependancies into osquery. It gives the osquery maintainers the control needed to keep it all working. The cmake build downloads all the source, and builds a static binary.

A couple years ago, someone did a bunch of work to get the osquery builds to support building for Arch. But looking at https://github.com/archlinux/svntogit-community/blob/packages/osquery/trunk/PKGBUILD it looks like it still uses the same libraries.

[znewman01]

Osquery is designed to build with its own third party libraries and versions.
The nature of what osquery does makes it quite sensitive to versions and build parameters. Over the last many years we've pulled the dependancies into osquery. It gives the osquery maintainers the control needed to keep it all working. The cmake build downloads all the source, and builds a static binary.

That's really good context, thanks. As @terlar points out, this may or may not be an issue. We'll see how much wailing/gnashing-of-teeth it takes to get it working. There's definitely a path forward, the questions are "how much work will it take to maintain?" and "will it be merged upstream?"

I'd love to get a Kolide distributed nix package for launcher. But I know very little about how that would work. (Like, what's a package here? Is there something like a build thing, like a deb, or is it closer to only distributing source manifests?)

A "Nix package" is really a recipe for building a package (usually a wrapper around standard build tools), though as an escape hatch the recipe can be "download this .deb and unpack it." I think the Nix community (including myself) is likely willing to maintain such a package, especially if you all remain this responsive 🙂

Here's an example from a good while back (this is not "production ready," but it contains the main ideas): https://github.com/znewman01/kolide-launcher/blob/c48b577d814c17a3aaeb1b89288c9c1509b87cf9/flake.nix#L17-L44

It's also pretty easy to maintain a fork of the Nix packages repository and have folks install from there while we're hashing these issues out.

---

It's sounding to me like, in the long run, we probably want to support both autoupdate options: users can manually pass an osquery in, or use an autoupdated version. For autoupdates, we need to:

1. Allow configuring the location for the osquery updates (Running from a read-only device messes up the autoupdate #638)
2. Figure out a hack to get the non-statically-linked osquery to work (osqueryd is not statically linked #770). IIRC, the launcher downloads a .deb file and extracts it locally, then runs it. We'd need to intercept that process. Ideally we could, during extraction, patch the binaries to point at the correct library versions (it works a lot better than you'd think). We could also do the same thing when it tries to run these binaries.

Once that's done, we should be pretty close to getting it working. Then it's just a matter of quality-of-life improvements (#868, #833).

I can try to prototype something on the Nixpkgs end that will evolve into the package that gets merged. Then once the above comes together, it should just be a matter of testing/configuration and ongoing maintenance.

[directionless]

Generally speaking, there is no need to unpack debs or rpms -- Kolide's autoupdate works off tar.gz binaries.

Osquery also distributes linux binaries in simple tar balls.

[directionless]

Fedora Silverblue also uses an immutable filesystems. Much like Nix, launcher won't work as is. I don't know what it would need to work

[markododa]

I have sucessfully installed the agent on fedora silverblue by using rpmrebuild.
I ran rpmrebuild -enp *_kolide-launcher.rpm
Changed /usr/local/ to /usr/share inside the opened spec file.
then from the the BuildRoot directory (3rd line in the spec file) ran the following commands

mkdir usr/share
mv usr/local/kolide-k2 usr/share
rmdir usr/local/
sed 's#/usr/local#/usr/share#' etc/kolide-k2/launcher.flags usr/lib/systemd/system/launcher.kolide-k2.service -i
rm usr/lib/.build-id/OSQUERYD_UUID
ln -s ../../../../usr/share/kolide-k2/bin/osqueryd usr/lib/.build-id/OSQUERYD_UUID
rm usr/lib/.build-id/LAUNCHER_UUID
ln -s ../../../../usr/share/kolide-k2/bin/launcher usr/lib/.build-id/LAUNCHER_UUID

[directionless]

Chatting with a nix user (internal slack thread , they said they were able to get launcher and osquery running inside an FHS, but that the entire FHS thing is a bit of a hack.

They thought NixOS/nixpkgs#195965 was the better approach

[James-Pickett]

Some users have managed to get launcher running on NixOS and we got reports of seeing multiple desktop items. I was attempting to recreate this issue using a NixOS vm, but was unable to get desktop to work at all. I tried using gnome and plasma versions of NixOS that are availalbe in default installer.

On gnome the error was:

systray error: failed to register our icon with the notifier watcher (maybe no tray is running?): The name org.kde.StatusNotifierWatcher was not provided by any .service files

On Plasma the error was:

systray error: failed to connect to DBus

On both machines I can see and interact with the tray on the desktop, but launcher still fails to run desktop. I also tried installing various other tray applications, but no success.

[James-Pickett]

Made a connection while looking at another issue.

It looks like NixOS ships with a compatible appindicator, but it's not enabled by default.

After enabling it with gnome-extensions enable [email protected] I was able to get the icon to appear.

[aasseman]

I have sucessfully installed the agent on fedora silverblue by using rpmrebuild. I ran rpmrebuild -enp *_kolide-launcher.rpm Changed /usr/local/ to /usr/share inside the opened spec file. then from the the BuildRoot directory (3rd line in the spec file) ran the following commands

mkdir usr/share
mv usr/local/kolide-k2 usr/share
rmdir usr/local/
sed 's#/usr/local#/usr/share#' etc/kolide-k2/launcher.flags usr/lib/systemd/system/launcher.kolide-k2.service -i
rm usr/lib/.build-id/OSQUERYD_UUID
ln -s ../../../../usr/share/kolide-k2/bin/osqueryd usr/lib/.build-id/OSQUERYD_UUID
rm usr/lib/.build-id/LAUNCHER_UUID
ln -s ../../../../usr/share/kolide-k2/bin/launcher usr/lib/.build-id/LAUNCHER_UUID

Thanks @markododa! I confirm that works on Fedora Silverblue 39.
For those confused with rpmrebuild, you have to edit the temporary build root directory while the spec file is open.
Also, I had to manually enable the service afterwards:

sudo systemctl enable --now launcher.kolide-k2.service

Since the tweaks are pretty simple, any chance we get official rpm-ostree support soon?

[directionless]

Hi @aasseman I'm not familiar with rpm-ostree can you say more about it? A quick search makes it unclear to me if it's a package format, or a package specification file.

[aasseman]

rpm-ostree is the software that manages base system images and packages for the Red Hat / Fedora immutable distros: RHCOS and Fedora CoreOS, Silverblue, Kinoite, etc.

[RebeccaMahany]

NixOS support is newly available via https://github.com/kolide/nix-agent, with setup instructions here.

[directionless]

I think we can close this? We're shipping nix, and while there's always more improvements, we probably don't need this issue.