Configurable derivations #6583

edolstra · 2022-05-27T15:10:53Z

WIP

To show all the options of a configuration:

# nix describe github:edolstra/configurables-test#nixosConfigurations2.test

To build a configuration:

# nix build github:edolstra/configurables-test#nixosConfigurations2.test

To override the host name from the command line:

# nix build github:edolstra/configurables-test#nixosConfigurations2.test --argstr networking.hostName xyzzy

To override the system type from the command line:

# nix build github:edolstra/configurables-test#nixosConfigurations2.test ---argstr nixpkgs.system i686-linux

roberth · 2022-05-28T13:54:57Z

Intriguing :)

Mutually exclusive or mix-in?

I think the usability would be better of configurability was a mix-in interface rather than an entirely different type.
That way, expression can use the default configuration as usual. This is especially true if the wiring of the configuration happens in the flake fixpoint, so that self.* is always the configured thing.

Overriding other parts of the flake

It'd also be more powerful to have this at the level of the flake fixpoint. This would allow the user to override a package in packages while deploying a nixosConfigurations.

A bonus feature would be to make these configurations first class, so they can be interpreted by the frameworks that build on Flakes, allowing the configuration of framework-specific concepts. This could be done by adding, say, self.configurationOverrides.<attrName>.<configurableItem> = <configuredValue>. Frameworks can declare that all possible <attrName>s are validated and handled by the framework by setting outputs.nixConfig.checkConfigurationOverridesAttrNames = false (or similar; maybe not nixConfig?), as this is already within their capability to do, without bothering the user.

`nixosConfigurations` meaning

I've always understood the nixosConfigurations as the final "real-world" configurations that are ready to be applied. Your example violates this understanding, making nixosConfigurations more like nixosModules.
We'd have:

nixosModules: services, image logic, profiles (I'd call them archetypes), complete but not instantiated system configs
nixosConfigurations: instantiated real systems, instantiated but not completely configured "real" systems

Configuring NixOS after the fact

While I'm not super happy about the fact that the nixosConfigurations are now almost conceptually redundant (if it wasn't for the choice of NixOS entrypoint (which source, which lib.nixosSystem-equivalent), I should point out a recent addition that will make your configurability a bit cleaner.

In the upcoming release (and recent backport to 21.11) it is possible to do

nixosConfigurations.foo = lib.nixosSystem { ..... };
nixosConfigurations.foo-with-hello =
  self.nixosConfigurations.foo.extendModules { modules = [
    { environment.systemPackages = [ pkgs.hello ]; }
  ]; };

This cleanly copies the effect of all lib.nixosSystem parameters to the new instance. That also fixes the little problem with specialArgs for you.

edolstra · 2022-05-30T10:53:07Z

A bit off-topic, but I feel that .extendModules perpetuates the .override anti-pattern, where you have to call something (in this case a NixOS configuration) just to be able to override it.

It'd also be more powerful to have this at the level of the flake fixpoint. This would allow the user to override a package in packages while deploying a nixosConfigurations.

I think this would require a change to the flakes spec, since it would mean that the flake outputs attribute has to be a configurable (i.e. implement an interface that allows its configuration options to be discoverable and settable). But it should be possible to do in a backwards-compatible way.

roberth · 2022-05-30T11:25:36Z

A bit off-topic, but I feel that .extendModules perpetuates the .override anti-pattern, where you have to call something (in this case a NixOS configuration) just to be able to override it.

In this case it is slightly better. It exists on a rather cheap value that does not require any options to be computed; not even imports to be resolved. Now this is going to look like a party trick, but it actually demonstrates that this extension mechanism is lazier than the usual .override ones. The crux: without a value for later, the module fixpoint can not be computed, yet we don't get an error, so it has not been computed before producing the final configuration.

nix-repl> ((lib.evalModules { prefix = []; modules = [ ({ later, ... }: { imports = [ later ]; }) ]; }).extendModules { specialArgs.later = { }; }).config
{ }

The reason why this mechanism is lazy is that unlike most overrides, the returned set off attribute names does not depend on the actual values. It is always { _module, config, options, extendModules, type } so the only cost of overriding here is producing the other 4 thunks; next to nothing.

At first, I only intended extendModules for internal use in specialisations and such (through module arguments), but the surprisingly negligible cost made me reconsider that. In fact, this is the cheaper one, as when you use it through module arguments, you're by definition duplicating the configuration, as is required for such features as specialisations.

But it should be possible to do in a backwards-compatible way.

While backcompat is generally nice, this should be no issue as flakes are experimental. I'm all for it though.

OT: A solution for #3843 can probably also be backwards compatible, but after accumulating a bunch of backcompat, it seems like a good idea to clear most of that out, before finalizing the RFC that specifies the first non-experimental flakes release.

roberth · 2022-05-30T11:57:04Z

We've talked about generalizing this from output attributes to all outputs, but it could be generalized further to encompass the inputs as well. Doing so will solve NixOS/nixpkgs#166220, or I guess this PR already does that, but only for the specific case of the nixpkgs flake; not yet other flakes that depend on nixpkgs.

At that point, we might consider setting system with the same means, thereby solving #3843 in a way, although such a solution would suggest being able to work with only a single system though, at least when ignoring the ability to invoke the good old nixpkgs function outside the flake paradigm. This could be a bit of a problem for dependency flakes that expose data (such as docs or a website) which isn't buildable everywhere, but can be used everywhere by virtue of being data (which is in a binary cache, presumably). There are other uses cases for cross-system dependencies as well, such as for natively built VM images.

edolstra · 2022-05-30T12:00:48Z

While backcompat is generally nice, this should be no issue as flakes are experimental. I'm all for it though.

Flake configurability is more of a flakes 2.0 feature. It shouldn't delay Nix 3.0.

nixos-discourse · 2022-06-03T09:48:52Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/tweag-nix-dev-update-31/19481/1

nixos-discourse · 2022-06-11T17:54:45Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/nixos-configuration-nix-with-argument/19653/4

domenkozar · 2022-06-18T01:15:28Z

It would be cool if it accepted a JSON, as a file or a string.

matthewbauer · 2022-10-25T01:33:38Z

src/nix/describe.cc

+
+        auto type = cursor->getAttr(state->sType)->getString();
+
+        if (type != "configurable")


Maybe it's overloading the concept, but it seems like you might expect nix describe to also work on flakes.

shlevy

Overall looks pretty good! The one big picture question I have, referenced in a few of my comments: Why limit configurability to derivations? Why not configurable apps, configurable NixOS configurations, etc.?

I may pick this PR up and open a PR against your repo bringing it up to date with master and completing the work (documentation, testing, etc.)

shlevy · 2023-08-14T10:52:01Z

src/libcmd/installables.cc

+                for (const auto & [n, s] : enumerate(ss)) {
+                    if (n + 1 == ss.size()) {
+                        if (pos->children.find(s) != pos->children.end())
+                            throw Error("definition of '%s' clashes with a previous definition", state->symbols[attr.name]);


Many tools with non-repeatable flags have the convention that the later can be used to override the former. Is there a reason we went with this approach instead?

shlevy · 2023-08-14T11:00:12Z

src/libcmd/installables.cc

+            }
+        }
+
+        std::function<Value * (const ValueTree & tree)> buildAttrs;


Might be worth a comment that this declared here for recursive calling, it surprised me.

shlevy · 2023-08-14T11:02:56Z

src/libcmd/installables.cc

+
+        ValueTree tree;
+
+        if (cmd) {


If we want other things besides drvs/installables to be configurable, we should probably extract everything from here to line 713 into a separate function that can be used for other interfaces.

shlevy · 2023-08-14T11:05:32Z

src/libcmd/installables.cc

+        auto vArgs = buildAttrs(tree);
+
+        auto vRes = state->allocValue();
+        state->callFunction(build, *vArgs, *vRes, noPos);


Shouldn't the pos be wherever we found attr?

shlevy · 2023-08-14T11:11:18Z

src/libcmd/installables.cc

    }

-    if (outputsToInstall.empty())
-        outputsToInstall.insert("out");
+    else if (type == "configurable") {


If we want other things besides drvs/installables to be configurable, should we allow type to be some kind of structured value? E.g. [ "configurable" "derivation" ] or { configurable = true; configured = "derivation"; } or something?

The alternative is to dynamically resolve this, i.e. check type after resolving the configurable.

nixos-discourse · 2023-09-07T02:44:01Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/experimental-does-not-mean-unstable-detsyss-perspective-on-nix-flakes/32703/2

ggPeti · 2024-01-17T20:21:39Z

made a workaround

edolstra added 3 commits May 27, 2022 13:50

Add operator for concatenating strings and string_views

634ce41

Add 'nix describe' command

db35580

Allow configurables to be overriden from the command line

e851b05

edolstra added the flakes label May 30, 2022

edolstra mentioned this pull request Jun 1, 2022

Migrate modules docs to markdown NixOS/nixpkgs#175586

Closed

edolstra mentioned this pull request Sep 6, 2022

Provide builtins.currentSystem as a input to flakes or make system a parameter #3843

Open

matthewbauer reviewed Oct 25, 2022

View reviewed changes

stale bot added the stale label May 21, 2023

roberth mentioned this pull request Jul 1, 2023

Feature request: Flake templates with variables #4017

Open

infinisil mentioned this pull request Jul 14, 2023

Present nix configurations nixpkgs-architecture/pkgs-modules#26

Open

edolstra mentioned this pull request Aug 14, 2023

Allow auto-calling installables. #8815

Closed

8 tasks

shlevy reviewed Aug 14, 2023

View reviewed changes

stale bot removed stale labels Aug 14, 2023

roberth mentioned this pull request Aug 30, 2023

Fine-grained impurity #8865

Open

RyanGibb mentioned this pull request Sep 11, 2023

Flake arguments #2861

Closed

Dessix mentioned this pull request Dec 3, 2023

Any interest in bringing this project under the NixOS organization? nmattia/niv#374

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Configurable derivations #6583

Configurable derivations #6583

edolstra commented May 27, 2022

roberth commented May 28, 2022 •

edited

Loading

edolstra commented May 30, 2022

roberth commented May 30, 2022

roberth commented May 30, 2022

edolstra commented May 30, 2022

nixos-discourse commented Jun 3, 2022

nixos-discourse commented Jun 11, 2022

domenkozar commented Jun 18, 2022

matthewbauer Oct 25, 2022

shlevy left a comment

shlevy Aug 14, 2023

shlevy Aug 14, 2023

shlevy Aug 14, 2023

shlevy Aug 14, 2023

shlevy Aug 14, 2023

nixos-discourse commented Sep 7, 2023

ggPeti commented Jan 17, 2024


		auto type = cursor->getAttr(state->sType)->getString();

		if (type != "configurable")

Configurable derivations #6583

Are you sure you want to change the base?

Configurable derivations #6583

Conversation

edolstra commented May 27, 2022

roberth commented May 28, 2022 • edited Loading

Mutually exclusive or mix-in?

Overriding other parts of the flake

nixosConfigurations meaning

Configuring NixOS after the fact

edolstra commented May 30, 2022

roberth commented May 30, 2022

roberth commented May 30, 2022

edolstra commented May 30, 2022

nixos-discourse commented Jun 3, 2022

nixos-discourse commented Jun 11, 2022

domenkozar commented Jun 18, 2022

matthewbauer Oct 25, 2022

Choose a reason for hiding this comment

shlevy left a comment

Choose a reason for hiding this comment

shlevy Aug 14, 2023

Choose a reason for hiding this comment

shlevy Aug 14, 2023

Choose a reason for hiding this comment

shlevy Aug 14, 2023

Choose a reason for hiding this comment

shlevy Aug 14, 2023

Choose a reason for hiding this comment

shlevy Aug 14, 2023

Choose a reason for hiding this comment

nixos-discourse commented Sep 7, 2023

ggPeti commented Jan 17, 2024

roberth commented May 28, 2022 •

edited

Loading

`nixosConfigurations` meaning