[RFC22, RFC78] NixOS a la carte (proof of concept)

[roberth]

Motivation for this change

• RFC22: a minimal NixOS module list to improve performance. A 3× eval performance improvement is realistic based on earlier experimentation on the postgresql test.
• RFC78: independent modules that can be composed (almost) like functions, for use outside of conventional NixOS.
• Allow alternative implementations to coexist

These are big goals that this PR does not achieve by itself, but works towards. The actual goal for this PR is to provide an example of the kinds of refactoring that can be done to make NixOS more modular.

What it does already achieve, is an example of how to build a dockerTools image using the NixOS /etc code:

    let
      nixosCore = (lib.nixos.core ({ config, modules, ... }: {
        imports = [ pkgs.nixosModule modules.etc ];
        environment.etc."hosts" = {
          text = ''
            127.0.0.1 localhost
            ::1 localhost
          '';
          mode = "0444";
        };
      }));
    in pkgs.dockerTools.streamLayeredImage {
      name = "poc";
      tag = "latest";
      fakeRootCommands = ''
        mkdir -p /etc
        ${nixosCore.config.system.build.etcActivationCommands}
      '';
      config.Cmd = pkgs.writeScript "poc-cmd" ''
        #!${pkgs.busybox}/bin/sh
        ${pkgs.busybox}/bin/cat /etc/hosts
      '';
    }

So there you have it, reusing a bit of NixOS with great evaluation performance. When more modules are refactored this way, it becomes actually useful, solving a real problem in practical dockerTools use such as #94636) and replacing incomplete solutions like #105685.

I've found that the best workflow for this refactoring is test driven development.

Things done

• Built on platform(s)
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
• Tested, as applicable:
  • NixOS test(s) (look inside nixos/tests)
  • and/or package tests
  • or, for functions and "core" functionality, tests in lib/tests or pkgs/test
  • made sure NixOS tests are linked to the relevant packages
• Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
• Tested basic functionality of all binary files (usually in ./result/bin/)
• 22.05 Release Notes (or backporting 21.11 Release notes)
  • (Package updates) Added a release notes entry if the change is major or breaking
  • (Module updates) Added a release notes entry if the change is significant
  • (Module addition) Added a release notes entry if adding a new NixOS module
  • (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes
• Fits CONTRIBUTING.md.

[roberth]

@ofborg test docker-tools etc nixpkgs openssh postgresql.postgresql_12 postgresql-container

[pennae]

interesting! we've been trying to get to a faster nixos another way, by marking most modules as "optional" and having the module system not evaluate them after the initial import of their nix file unless a configurable path in config has values (in our plan this path would usually be a module enable option). the docs work we've been doing is part of this, since docs currently evaluate all modules anyway and users probably will not be very happy if most of the configuration.nix manpage disappears because they've selected only a subset of modules for their system :<

[roberth]

@pennae That's also a good approach; a bit more pragmatic, but seems to need a more complicated module system and does not necessarily stimulate improvements towards RFC78 for example. Could you show what changes you had to make?

[pennae]

@roberth we don't have anything for this yet since we're concentrating on doing the docs bit first (which in itself yields about a 40% improvement in system build time on this machine). this being a highly experimental endeavour we just haven't yet had the energy to spend on doing things that might end up rejected.

the idea was to tag each module that isn't essential with an extra toplevel key that describes properties of the modules, leading to eg

{ lib, config, ... }:

{
  module.enabledBy = [ "services" "theThing" "enable" ];

  options.services.theThing.enable = mkEnableOption "the thing";

  # ...
}

the module system imports all modules as before, but does not evaluate such optional modules except for their enabledBy. it then checks the resulting options (not config!) for enabler paths and adds those to the evaluation. repeat until fixed point is reached. for modules that enable other modules we'd go ahead and allow dependency declarations, eg module.requires to avoid unnecessary iterations. our testing has shown that module parse time is about 5% of the total instantiation time for our configuration, and if nixos modules declare their dependencies correctly we'd only have to do a single iteration to enable all necessary modules. as a test we had minimized module-list.nix for our config and (with the current docs build enabled) also arrived at a ~40% build time reduction.

[roberth]

~40% build time reduction.

You should be able to get about 3× faster, which is your reduction applied twice. That discrepancy can be explained by the choice of test and I was quite radical with my pruning, even going into some modules to set values to sensible defaults in order to remove more modules. It doubt whether those wins are feasible without refactoring those modules.

[pennae]

oh yes, a lot of the remaining time was most likely due to a large environment.systemPackages from our config. we had also pruned the config itself for a different test and got a 4x improvement with the pruned config. there's clearly a lot of improvement still possible!

[aanderse]

@roberth awesome! I'm looking forward to seeing more examples so I can better wrap my mind around this.
@pennae also awesome! I hope to hear more about that coming up.

I thought I would include this link here as well because it is relevant.

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/a-faster-dockertools-buildimage-prototype/16922/4

[roberth]

Idea: migrate the implementations of nixos/modules/profiles to new nixos/modules/archetypes, so we have a better name and a new namespace to rethink the profiles.

[roberth]

I'm closing this because the work now happens in other PRs, which tend to be linked here.

[roberth]

Tracking which changes have been cleaned up and merged

Note to self: might want to bundle up some of the similar and simple changes, like module extractions and adding imports.

Things not to do:

• Create an attribute set of NixOS modules. This would be premature as the set of relevant modules is yet to be determined.
• Create more modules than necessary.
• Blindly rebase. Make sure the refactors are clean.

The commits of this PR and their relation to derived PRs or vice versa:

• Add minimal NixOS entrypoint #153211
  • 18672bf lib.nixos: init
  • 62e7f0e nixos/nixpkgs.nix: Make independent
• nixos: Refactor to allow etc unit test #155892 (pkgsModule)
  • dd6d8e3 pkgs.nixosModule: init
  • 56c283e nixos/etc.nix: Make independent
• 7f129e3 nixos/build.nix: Make independent
• 9b2af86 dockerTools: Add example of using NixOS' etc
• nixos/top-level: maintenance #199595
  • 4ec415c nixos/top-level.nix: Make extensible
  • 2bf147c nixos/specialisation.nix: Extract module
  • db05855 nixos/top-level.nix: Move configurationName to grub.nix
• nixos/system: Support pre-activated images #237040
  • baa96b8 nixos/system-with-activation.nix: Extract module
• 8da928c nixos/bootable.nix: Extract module
• c78d41a nixos: Move top-level stage2 def to stage-2.nix
• c8da2d4 nixos: Move system.name logic to network-interfaces.nix
• c6ea188 nixos/system-path-core.nix: Extract module
• dc2b396 nixos/users-groups.nix: Make independent
• 42b59e3 nixos/update-users-groups.pl: Fall back to root group for shadow
• b9cb182 nixos: Move specialfs activation script to filesystems.nix
• 8313fd3 nixos/top-level.nix: Make independent of systemd.package
• 8772485 nixos/user-profiles.nix: Extract module
• f2cd714 nixos/systemd-user-activation.nix: Extract module
• a7a1438 nixos/top-level.nix: Make independent of boot modules
• 7c0c2e2 nixos: Add system.activation.externalActivationScript
• 413be2c nixos/top-level.nix: Declare dependencies
• b692a92 nixos/users-groups.nix: Declare dependencies
• cc74885 nixos/activation-script: Declare dependencies
• 226d20e lib.nixos: Add users module
• 77e8c73 dockerTools.streamLayeredImage: Make robust against cd in fakeRootCommands
• bc6808e nixos/nixos-core.nix: Add etcActivation module
• eb3fc9a dockerTools: Update etc example
• 6adab4c nixos/top-level.nix: Add system.checks NixOS: add system.checks #231316
• 25093d7 nixos/binsh.nix: Extract module
• 2f3e9f4 nixos/environment-variables.nix: Extract module
• d9e94ba (hercules-ci/nixos-a-la-carte-poc-2) nixos/postgresql.nix: Split module, create container, test it

---

related changes

• lib.modules: Let module declare options directly in bare submodule #156533
• nixos: Add system.build.{toplevel,installBootLoader}, improve error message #156503

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/genodepkgs-extending-nixpkgs-nixos-to-genode/8779/7

[roberth]

Latest PR is for having a separate activation script, that can run during the build of an image

• nixos/system: Support pre-activated images #237040

EDIT: review appreciated.