-
-
Notifications
You must be signed in to change notification settings - Fork 14.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
nixos/nixpkgs: add override/merging capabilities from the module system to the nixpkgs.config
option
#80582
Conversation
…em to the `nixpkgs.config` option The `nixpkgs.config`[1] option is just a plain attribute-set that is directly passed to `<nixpkgs/pkgs/top-level/default.nix>`. This has the following drawbacks: * Missing override/merging capabilities: you can't use module-system features like `mkOverride`/`mkOrder`/`mkIf`/etc. * Conflicting definitions don't cause an evaluation error. The merging is basically a `fold recursiveUpdate {} defs` which means that the last declaration of a value inside `nixpkgs.config` "wins". For example, ``` nix { pkgs, lib, ... }: { nixpkgs = lib.mkMerge [ { config.allowUnfree = false; } { config.allowUnfree = true; } ]; environment.systemPackages = [ pkgs.spotify /* an unfree package */ ]; } ``` would evaluate. Another attempt to use the module-system for nixpkgs-config stalled after a long discussion[2]. This change aims to be minimally-invasive by applying the filters from `<nixpkgs/lib/modules.nix>` onto the attribute-set without constraining the attributes with a predefined option structure (or in other words: you don't need to change anything in the module-system to add a new nixpkgs-config value). This is done by replacing the old `configType` with a new option-type that emulates the merging behavior of the module-system: * All values are merged together into a single attr-set to ensure that all values that are defined in any included module are taken into account. * Each sub-attribute that is an attr-set and has a `_type`-field (which defines which `mk*`-filter was used for the declaration) will be passed through priority/discharge/ordering filters to resolve all `mk*` statements in `nixpkgs.config`. * The result will be merged together to ensure that no conflicting definitions exist while sequential values will be merged together. Due to the validation mechanisms from the module-system, the following case will break with an evaluation error now: * When declaring values of different types to a single nixpkgs-config option, the eval will fail now (rather than silently replacing the values in `lib.recursiveUpdate`): For example, ``` nix { lib, ... }: { nixpkgs = lib.mkMerge [ { config.chromium.enableWidevine = true; } { config.chromium = 23; } ]; } ``` will break with the following error: ``` Cannot merge definitions of `nixpkgs.config.chromium' given in `<unknown-file>' and `<unknown-file>'. ``` (Please note that there might be some more edge-cases I'm just missing while writing the commit-message). Additionally the following issues aren't fixed yet: * There's no support for anything like `types.separatedString`. As string-concatenation during merge-phase may be harmful[3] (and there's no way to differentiate a separatedString from a "normal" str without using `<nixpkgs/lib/types.nix>`), evaluations like this will always fail for now: ``` nix { lib, ... }: { nixpkgs = lib.mkMerge [ { config.random = '' foo ''; } { config.random = '' bar ''; } ]; } ``` ``` The unique option `nixpkgs.config.random' is defined multiple times, in: - <unknown-file> - <unknown-file>. ``` Please note that the only reason why this yields `<unknown-file>` is because the values were defined in a `mkMerge`. Since `loc` is passed properly to the error-handler, the actual file where a value is declared would be shown if the options were defined in different files. * Expressions inside a `mkMerge` aren't evaluated yet. For instance, in the following expression not all `mk*` would be resolved properly: ``` nix { lib, ... }: { nixpkgs.config = { chromium = lib.mkMerge [ { randomValue = mkIf true "something"; } ]; }; } ``` ``` { chromium = { randomValue = { _type = "if"; condition = true; content = "something"; }; }; } ``` Please note that this only applies to ordering & conditionals, overrides inside a `mkMerge` are resolved without any problems. [1] https://nixos.org/nixpkgs/manual/#idm140737322665904 [2] NixOS#56227 [3] NixOS@03391cd
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I know that this change is rather hacky, but IMHO it's the simplest solution to enable module-system features in nixpkgs.config
without breaking too much existing code. If you know better approaches or found some edge-cases that break with this patch, please let me know, I'm happy to take a look at this (and probably fix those), but please take a look at the commit-message first, I may have thought about your issue already.
@infinisil you're one of the folks where I know for sure that they know fairly much about the module-system, so I hope it's okay that I requested a review from you here as well :)
# | ||
# FIXME This is missing support for separatedString! | ||
merge = let h = (head result).value or null; in | ||
if isList h || isAttrs h then mergeDefaultOption else mergeOneOption; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
As mentioned in the comment, separatedString
doesn't work here. As recursiveUpdate
did silent overrides, this never worked before, but are there any suggestions how to fix this as well? :)
appendVal = all: value_: next: | ||
let | ||
# FIXME support `mkMerge [{ snens = mkIf cond val; }]` | ||
discharged = dischargeProperties value_; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
As mentioned in the comment, mk{If,Merge,Before,...}
aren't evaluated inside a mkMerge
(mkMerge [ (mkIf cond { /* ... */ } ]
) works fine though). IMHO this would make the code even more complex, but are there any suggestions how fix this? :)
let | ||
# Ensure that each attribute from nixpkgs.config appears at the | ||
# first iteration to make sure that nothing is lost. | ||
allValues = fold recursiveUpdate {} defs; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm aware that this is a gross hack to make sure that any attr-names that are defined in any moduel taken into account is known to the merger. Are there any suggestions for a better workaround? :)
|
||
# Gather all declarations of a value in the `nixpkgs.config` attr-set. | ||
allDeclarations = foldl | ||
(all: next: appendVal all (optCall (attrByPath path null next) finalPkgs) next) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The optCall
in the recursive mapAttrs
replaces the merging behavior for {perlP,p}ackageOverrides
. Please note that I'm now using finalPkgs
rather than pkgs
as this seems more appropriate to me (since finalPkgs
also takes overlays into account). If this is a wrong decision, please let me know, we probably might want to use pkgs
rather than finalPkgs
here.
What we need here is a new module system feature. Being able to define unspecified options is more generally useful. For example
By supporting it in the module system, we don't have to maintain one or more hacky reimplementations of this feature. |
So you're basically suggesting that this should be made generally useful, right? I don't see a problem with it, I kept it in the |
I just gave implementing partially typed options a go. The basic use-case works, but the semantics of this implementation are inconsistent. The work is in this branch: https://github.com/Infinisil/nixpkgs/tree/partially-typed, and an example of using it: { pkgs, config, lib, ... }: with lib;
{
options.foo = lib.mkOption {
type = with types; types.partialSubmodule [ bool int ] ({ config, ... }: {
options.bar = lib.mkOption {
type = bool;
default = false;
};
options.b = lib.mkOption {
type = types.int;
default = 0;
};
# Doesn't work
config.test = mkIf config.bar "test";
# Doesn't work
config.a = config.baz;
});
};
config.foo = {
bar = true;
baz = 10;
};
} I implemented it by writing a type that combines two other types (one of which is an So I think a better approach to implement this would be to dynamically generate internal options for all untyped values that are specified, such that doing I might give this a go too soon. If this would work well, it could simplify NixOS/rfcs#42 considerably. |
You could rename
The module system can then gather up the undeclared definitions, check and merge them. The default value for I liked the elegance of something like |
I did another run for this, and I seem to have a nicely working prototype now! The work is in https://github.com/Infinisil/nixpkgs/tree/partially-typed-v2, and here's an example of how it looks: let
lib = import ./lib;
inherit (lib) types;
in
lib.evalModules {
modules = [
({ lib, config, ... }: {
options.services.foo = {
enable = lib.mkEnableOption "service";
settings = lib.mkOption {
type = with types; submoduleWith {
unstructuredOption = lib.mkOption {
type = oneOf [ str int ];
};
modules = [{
options.port = lib.mkOption {
type = types.int;
};
}];
};
default = {};
};
};
options.result = lib.mkOption {
type = types.str;
default = "";
};
config = lib.mkIf config.services.foo.enable {
services.foo.settings.bar = lib.mkDefault "bar";
result = lib.concatStringsSep "\n" (lib.mapAttrsToList (name: value: "${name} = ${toString value}") config.services.foo.settings);
};
})
{
services.foo = {
enable = true;
settings.port = 80;
# This throws an error that port isn't of type int
# settings.port = "80";
settings.some-value = "value";
settings.bar = lib.mkForce 10;
};
}
];
} This now works directly through the module system: Instead of it only trying to evaluate declared options, it now also looks at all definitions without a declaration (if |
If you bundle them up and treat them as a single, potentially nested attribute set, you can leave the choice of nesting semantics to the caller by means of their choice of type for the whole attrset. This also gives the caller more power over merging. |
Since the fallback-type patches seem like a better and more mature idea to me, I'm going to close this for now. |
Closes NixOS#194056 Alternative implementation to NixOS#80582 `nixpkgs.config` is now a freeform-type which imports the `<nixpkgs/pkgs/top-level/config.nix>`-module. This brings the following benefits: * Proper merging (to a certain degree, more details later) of several `nixpkgs.config` declarations. * `allowUnfree` and friends appear in the NixOS manual. To be precise, this means that something like this is now possible: { nixpkgs = mkMerge [ { allowUnfree = false; } { allowUnfree = mkForce true; } ]; } Previously this would lead to a bogus attr-set, namely { _type = "override"; content = true; priority = 50; } since no merging for this option-type was implemented. This however has certain limitations. For instance nixpkgs.config.allowUnfree = mkForce false; works fine whereas nixpkgs.config.virtualbox.enableExtensionPack = mkForce true; doesn't. This is because we use `types.raw` inside the config-module for `nixpkgs` which doesn't merge undeclared attribute-sets (such as `virtualbox`). This is because we don't know if any attribute-set is actually mergeable, for further details see the previous discussion[1]. A basic type has been implemented for the (actually deprecated) mechanisms `packageOverrides`/`perlPackageOverrides`. This one basically applies the `pkgs` argument onto each definition of this function, so e.g. the following expression { nixpkgs.config = mkMerge [ { packageOverrides = pkgs: { randomattr = 23; }; } { packageOverrides = pkgs: { randomattr = pkgs.randomattr + 19; }; } ]; } adds an attribute `randomattr` of value `42` to `_module.args.pkgs`. [1] NixOS#194207 (comment)
Motivation for this change
The
nixpkgs.config
[1] option is just a plain attribute-set that is directlypassed to
<nixpkgs/pkgs/top-level/default.nix>
. This has the followingdrawbacks:
Missing override/merging capabilities: you can't use module-system features
like
mkOverride
/mkOrder
/mkIf
/etc.Conflicting definitions don't cause an evaluation error. The merging
is basically a
fold recursiveUpdate {} defs
which means that thelast declaration of a value inside
nixpkgs.config
"wins".For example,
would evaluate.
Another attempt to use the module-system for nixpkgs-config stalled
after a long discussion[2].
This change aims to be minimally-invasive by applying the filters from
<nixpkgs/lib/modules.nix>
onto the attribute-set without constrainingthe attributes with a predefined option structure (or in other words:
you don't need to change anything in the module-system to add a new
nixpkgs-config value).
This is done by replacing the old
configType
with a new option-typethat emulates the merging behavior of the module-system:
All values are merged together into a single attr-set to ensure that
all values that are defined in any included module are taken into
account.
Each sub-attribute that is an attr-set and has a
_type
-field (whichdefines which
mk*
-filter was used for the declaration) will bepassed through priority/discharge/ordering filters to resolve all
mk*
statements innixpkgs.config
.The result will be merged together to ensure that no conflicting
definitions exist while sequential values will be merged together.
Due to the validation mechanisms from the module-system, the following
case will break with an evaluation error now:
When declaring values of different types to a single nixpkgs-config
option, the eval will fail now (rather than silently replacing the
values in
lib.recursiveUpdate
):For example,
will break with the following error:
(Please note that there might be some more edge-cases I'm just missing
while writing the commit-message).
Additionally the following issues aren't fixed yet:
There's no support for anything like
types.separatedString
. Asstring-concatenation during merge-phase may be harmful[3] (and there's no
way to differentiate a separatedString from a "normal" str without
using
<nixpkgs/lib/types.nix>
), evaluations like this will alwaysfail for now:
Please note that the only reason why this yields
<unknown-file>
isbecause the values were defined in a
mkMerge
. Sinceloc
is passedproperly to the error-handler, the actual file where a value is
declared would be shown if the options were defined in different
files.
Expressions inside a
mkMerge
aren't evaluated yet. For instance, inthe following expression not all
mk*
would be resolved properly:Please note that this only applies to ordering & conditionals, overrides
inside a
mkMerge
are resolved without any problems.[1] https://nixos.org/nixpkgs/manual/#idm140737322665904
[2] #56227
[3] 03391cd
Things done
sandbox
innix.conf
on non-NixOS linux)nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
./result/bin/
)nix path-info -S
before and after)