Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rework deployment mechanism for installations #78

Merged
merged 20 commits into from
Jul 8, 2024

Conversation

Kreyren
Copy link
Member

@Kreyren Kreyren commented Jun 28, 2024

Refer to nix-community/disko#657 (comment) for implementation notes

@Kreyren Kreyren added the help wanted Extra attention is needed label Jun 28, 2024
@Kreyren Kreyren self-assigned this Jun 28, 2024
@Kreyren Kreyren requested a review from Tanvir1337x June 28, 2024 03:40
@Kreyren
Copy link
Member Author

Kreyren commented Jun 28, 2024

@Kreyren Kreyren mentioned this pull request Jun 30, 2024
@Kreyren
Copy link
Member Author

Kreyren commented Jul 1, 2024

nix build 'github:kreyren/nixos-config/6c6587c402f1e2e7d381ab7d1da16da3c0e4ffe2#devShells.x86_64-linux.default' --show-trace
warning: Git tree '/nix/persist/NiXium' is dirty
error:
       … while calling the 'derivationStrict' builtin

         at /builtin/derivation.nix:9:12: (source not available)

       … while evaluating derivation 'NiXium-devshell'
         whose name attribute is located at /nix/store/giq1qbcwfx5gq0g9jf6id0b3m55a860j-source/pkgs/stdenv/generic/make-derivation.nix:331:7

       … while evaluating attribute 'nativeBuildInputs' of derivation 'NiXium-devshell'

         at /nix/store/giq1qbcwfx5gq0g9jf6id0b3m55a860j-source/pkgs/stdenv/generic/make-derivation.nix:375:7:

          374|       depsBuildBuild              = elemAt (elemAt dependencies 0) 0;
          375|       nativeBuildInputs           = elemAt (elemAt dependencies 0) 1;
             |       ^
          376|       depsBuildTarget             = elemAt (elemAt dependencies 0) 2;

       … from call site

         at /nix/store/giq1qbcwfx5gq0g9jf6id0b3m55a860j-source/pkgs/stdenv/generic/make-derivation.nix:302:58:

          301|       (map (drv: getDev drv.__spliced.buildBuild or drv) (checkDependencyList "depsBuildBuild" depsBuildBuild))
          302|       (map (drv: getDev drv.__spliced.buildHost or drv) (checkDependencyList "nativeBuildInputs" nativeBuildInputs'))
             |                                                          ^
          303|       (map (drv: getDev drv.__spliced.buildTarget or drv) (checkDependencyList "depsBuildTarget" depsBuildTarget))

       … while calling 'checkDependencyList''

         at /nix/store/giq1qbcwfx5gq0g9jf6id0b3m55a860j-source/pkgs/stdenv/generic/make-derivation.nix:274:43:

          273|   checkDependencyList = checkDependencyList' [];
          274|   checkDependencyList' = positions: name: deps:
             |                                           ^
          275|     imap1

       … from call site

         at /nix/store/giq1qbcwfx5gq0g9jf6id0b3m55a860j-source/pkgs/stdenv/generic/make-derivation.nix:275:5:

          274|   checkDependencyList' = positions: name: deps:
          275|     imap1
             |     ^
          276|       (index: dep:

       … while calling 'imap1'

         at /nix/store/giq1qbcwfx5gq0g9jf6id0b3m55a860j-source/lib/lists.nix:334:14:

          333|   */
          334|   imap1 = f: list: genList (n: f (n + 1) (elemAt list n)) (length list);
             |              ^
          335|

       … from call site

         at /nix/store/giq1qbcwfx5gq0g9jf6id0b3m55a860j-source/pkgs/build-support/mkshell/default.nix:41:36:

           40|   buildInputs = mergeInputs "buildInputs";
           41|   nativeBuildInputs = packages ++ (mergeInputs "nativeBuildInputs");
             |                                    ^
           42|   propagatedBuildInputs = mergeInputs "propagatedBuildInputs";

       … while calling 'mergeInputs'

         at /nix/store/giq1qbcwfx5gq0g9jf6id0b3m55a860j-source/pkgs/build-support/mkshell/default.nix:17:17:

           16| let
           17|   mergeInputs = name:
             |                 ^
           18|     (attrs.${name} or [ ]) ++

       … while calling anonymous lambda

         at /nix/store/giq1qbcwfx5gq0g9jf6id0b3m55a860j-source/lib/lists.nix:1883:30:

         1882|   */
         1883|   subtractLists = e: filter (x: !(elem x e));
             |                              ^
         1884|

       … while evaluating derivation '-'
         whose name attribute is located at /nix/store/giq1qbcwfx5gq0g9jf6id0b3m55a860j-source/pkgs/stdenv/generic/make-derivation.nix:331:7

       … while evaluating attribute 'text' of derivation '-'

         at /nix/store/giq1qbcwfx5gq0g9jf6id0b3m55a860j-source/pkgs/build-support/trivial-builders/default.nix:103:16:

          102|       ({
          103|         inherit text executable checkPhase allowSubstitutes preferLocalBuild;
             |                ^
          104|         passAsFile = [ "text" ]

       … from call site

         at /nix/store/giq1qbcwfx5gq0g9jf6id0b3m55a860j-source/pkgs/build-support/trivial-builders/default.nix:278:9:

          277|             runtimeEnv))
          278|       + lib.optionalString (runtimeInputs != [ ]) ''
             |         ^
          279|

       … while calling 'optionalString'

         at /nix/store/giq1qbcwfx5gq0g9jf6id0b3m55a860j-source/lib/strings.nix:268:5:

          267|     # String to return if condition is true
          268|     string: if cond then string else "";
             |     ^
          269|

       … from call site

         at /nix/store/giq1qbcwfx5gq0g9jf6id0b3m55a860j-source/pkgs/build-support/trivial-builders/default.nix:280:24:

          279|
          280|         export PATH="${lib.makeBinPath runtimeInputs}:$PATH"
             |                        ^
          281|       '' + ''

       … while calling 'makeSearchPathOutput'

         at /nix/store/giq1qbcwfx5gq0g9jf6id0b3m55a860j-source/lib/strings.nix:195:5:

          194|     # List of packages
          195|     pkgs: makeSearchPath subDir (map (lib.getOutput output) pkgs);
             |     ^
          196|

       … from call site

         at /nix/store/giq1qbcwfx5gq0g9jf6id0b3m55a860j-source/lib/strings.nix:195:11:

          194|     # List of packages
          195|     pkgs: makeSearchPath subDir (map (lib.getOutput output) pkgs);
             |           ^
          196|

       … while calling 'makeSearchPath'

         at /nix/store/giq1qbcwfx5gq0g9jf6id0b3m55a860j-source/lib/strings.nix:175:5:

          174|     # List of base paths
          175|     paths:
             |     ^
          176|     concatStringsSep ":" (map (path: path + "/" + subDir) (filter (x: x != null) paths));

       … while calling anonymous lambda

         at /nix/store/giq1qbcwfx5gq0g9jf6id0b3m55a860j-source/lib/strings.nix:176:32:

          175|     paths:
          176|     concatStringsSep ":" (map (path: path + "/" + subDir) (filter (x: x != null) paths));
             |                                ^
          177|

       … while evaluating derivation 'install'
         whose name attribute is located at /nix/store/giq1qbcwfx5gq0g9jf6id0b3m55a860j-source/pkgs/stdenv/generic/make-derivation.nix:331:7

       … while evaluating attribute 'text' of derivation 'install'

         at /nix/store/giq1qbcwfx5gq0g9jf6id0b3m55a860j-source/pkgs/build-support/trivial-builders/default.nix:103:16:

          102|       ({
          103|         inherit text executable checkPhase allowSubstitutes preferLocalBuild;
             |                ^
          104|         passAsFile = [ "text" ]

       … while calling anonymous lambda

         at /nix/store/wxjsfgkkd93fwqn1g49srpj9gms656wn-source/attrsets.nix:1205:18:

         1204|         mapAttrs
         1205|           (name: value:
             |                  ^
         1206|             if isAttrs value && cond value

       … from call site

         at /nix/store/wxjsfgkkd93fwqn1g49srpj9gms656wn-source/attrsets.nix:1208:18:

         1207|             then recurse (path ++ [ name ]) value
         1208|             else f (path ++ [ name ]) value);
             |                  ^
         1209|     in

       … while calling anonymous lambda

         at /nix/store/wxjsfgkkd93fwqn1g49srpj9gms656wn-source/modules.nix:242:72:

          241|           # For definitions that have an associated option
          242|           declaredConfig = mapAttrsRecursiveCond (v: ! isOption v) (_: v: v.value) options;
             |                                                                        ^
          243|

       … while evaluating the option `perSystem.x86_64-linux.mission-control.scripts.install.exec':

       … while calling anonymous lambda

         at /nix/store/wxjsfgkkd93fwqn1g49srpj9gms656wn-source/modules.nix:824:28:

          823|         # Process mkMerge and mkIf properties.
          824|         defs' = concatMap (m:
             |                            ^
          825|           map (value: { inherit (m) file; inherit value; }) (builtins.addErrorContext "while evaluating definitions from `${m.file}':" (dischargeProperties m.value))

       … while evaluating definitions from `/nix/store/ik95dzhanyzmg2s18d22rhkyha3fq6bs-source/tasks/administration/install, via option perSystem':

       … from call site

         at /nix/store/wxjsfgkkd93fwqn1g49srpj9gms656wn-source/modules.nix:825:137:

          824|         defs' = concatMap (m:
          825|           map (value: { inherit (m) file; inherit value; }) (builtins.addErrorContext "while evaluating definitions from `${m.file}':" (dischargeProperties m.value))
             |                                                                                                                                         ^
          826|         ) defs;

       … while calling 'dischargeProperties'

         at /nix/store/wxjsfgkkd93fwqn1g49srpj9gms656wn-source/modules.nix:896:25:

          895|   */
          896|   dischargeProperties = def:
             |                         ^
          897|     if def._type or "" == "merge" then

       error: attribute 'install-task' missing

       at /nix/store/ik95dzhanyzmg2s18d22rhkyha3fq6bs-source/tasks/administration/install/default.nix:14:22:

           13|                          # FIXME-QA(Krey): This makes the declaration more functional, but looks like an ugly hack
           14|                          exec = toString ((import ./script.nix { inherit pkgs inputs; }).install-task + /bin/install-task);
             |                      ^
           15|                  };

@Kreyren
Copy link
Member Author

Kreyren commented Jul 6, 2024

CC Flake-parts developers: @roberth @shlevy

Could you help with this issue? I don't know what should be the way that this is handled via flake-parts and the NixOS community lacks experience with flake-parts to help.

Thanks for anything relevant! <3

@roberth
Copy link

roberth commented Jul 6, 2024

👋

Looks like hercules-ci/flake-parts#101. If you get inputs from the flake-parts top level instead of perSystem, that should work.

Comment on lines 1 to 8
{ pkgs, ... }:

# Task to INSTALL the specified derivation on current system including the firmware in a fully declarative way

# Refer to https://github.com/nix-community/disko/issues/657#issuecomment-2146978563 for implementation notes

{
perSystem = { system, inputs, ... }: {
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
{ pkgs, ... }:
# Task to INSTALL the specified derivation on current system including the firmware in a fully declarative way
# Refer to https://github.com/nix-community/disko/issues/657#issuecomment-2146978563 for implementation notes
{
perSystem = { system, inputs, ... }: {
{ inputs, ... }:
# Task to INSTALL the specified derivation on current system including the firmware in a fully declarative way
# Refer to https://github.com/nix-community/disko/issues/657#issuecomment-2146978563 for implementation notes
{
perSystem = { system, pkgs, inputs', ... }: {

disk="$2"

[ -n "$AGE" ] || AGE="${pkgs.age}/bin/age"
[ -n "$DISKO_INSTALL" ] || DISKO_INSTALL="${inputs.disko.packages.${system}.disko-install}/bin/disko-install"
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you change the parameter name to inputs' you get to use some un-system-ing logic:

Suggested change
[ -n "$DISKO_INSTALL" ] || DISKO_INSTALL="${inputs.disko.packages.${system}.disko-install}/bin/disko-install"
[ -n "$DISKO_INSTALL" ] || DISKO_INSTALL="${inputs'.disko.packages.disko-install}/bin/disko-install"

@Kreyren
Copy link
Member Author

Kreyren commented Jul 6, 2024

$ nix build 'github:kreyren/nixos-config/09fefa3a7b90b70b22d61ba9e7db7b83840245c0#devShells.x86_64-linux.default' --show-trace
error:
       … while calling the 'derivationStrict' builtin

         at /builtin/derivation.nix:9:12: (source not available)

       … while evaluating derivation 'NiXium-devshell'
         whose name attribute is located at /nix/store/giq1qbcwfx5gq0g9jf6id0b3m55a860j-source/pkgs/stdenv/generic/make-derivation.nix:331:7

       … while evaluating attribute 'nativeBuildInputs' of derivation 'NiXium-devshell'

         at /nix/store/giq1qbcwfx5gq0g9jf6id0b3m55a860j-source/pkgs/stdenv/generic/make-derivation.nix:375:7:

          374|       depsBuildBuild              = elemAt (elemAt dependencies 0) 0;
          375|       nativeBuildInputs           = elemAt (elemAt dependencies 0) 1;
             |       ^
          376|       depsBuildTarget             = elemAt (elemAt dependencies 0) 2;

       … from call site

         at /nix/store/giq1qbcwfx5gq0g9jf6id0b3m55a860j-source/pkgs/stdenv/generic/make-derivation.nix:302:58:

          301|       (map (drv: getDev drv.__spliced.buildBuild or drv) (checkDependencyList "depsBuildBuild" depsBuildBuild))
          302|       (map (drv: getDev drv.__spliced.buildHost or drv) (checkDependencyList "nativeBuildInputs" nativeBuildInputs'))
             |                                                          ^
          303|       (map (drv: getDev drv.__spliced.buildTarget or drv) (checkDependencyList "depsBuildTarget" depsBuildTarget))

       … while calling 'checkDependencyList''

         at /nix/store/giq1qbcwfx5gq0g9jf6id0b3m55a860j-source/pkgs/stdenv/generic/make-derivation.nix:274:43:

          273|   checkDependencyList = checkDependencyList' [];
          274|   checkDependencyList' = positions: name: deps:
             |                                           ^
          275|     imap1

       … from call site

         at /nix/store/giq1qbcwfx5gq0g9jf6id0b3m55a860j-source/pkgs/stdenv/generic/make-derivation.nix:275:5:

          274|   checkDependencyList' = positions: name: deps:
          275|     imap1
             |     ^
          276|       (index: dep:

       … while calling 'imap1'

         at /nix/store/giq1qbcwfx5gq0g9jf6id0b3m55a860j-source/lib/lists.nix:334:14:

          333|   */
          334|   imap1 = f: list: genList (n: f (n + 1) (elemAt list n)) (length list);
             |              ^
          335|

       … from call site

         at /nix/store/giq1qbcwfx5gq0g9jf6id0b3m55a860j-source/pkgs/build-support/mkshell/default.nix:41:36:

           40|   buildInputs = mergeInputs "buildInputs";
           41|   nativeBuildInputs = packages ++ (mergeInputs "nativeBuildInputs");
             |                                    ^
           42|   propagatedBuildInputs = mergeInputs "propagatedBuildInputs";

       … while calling 'mergeInputs'

         at /nix/store/giq1qbcwfx5gq0g9jf6id0b3m55a860j-source/pkgs/build-support/mkshell/default.nix:17:17:

           16| let
           17|   mergeInputs = name:
             |                 ^
           18|     (attrs.${name} or [ ]) ++

       … while calling anonymous lambda

         at /nix/store/giq1qbcwfx5gq0g9jf6id0b3m55a860j-source/lib/lists.nix:1883:30:

         1882|   */
         1883|   subtractLists = e: filter (x: !(elem x e));
             |                              ^
         1884|

       … while evaluating derivation '-'
         whose name attribute is located at /nix/store/giq1qbcwfx5gq0g9jf6id0b3m55a860j-source/pkgs/stdenv/generic/make-derivation.nix:331:7

       … while evaluating attribute 'text' of derivation '-'

         at /nix/store/giq1qbcwfx5gq0g9jf6id0b3m55a860j-source/pkgs/build-support/trivial-builders/default.nix:103:16:

          102|       ({
          103|         inherit text executable checkPhase allowSubstitutes preferLocalBuild;
             |                ^
          104|         passAsFile = [ "text" ]

       … from call site

         at /nix/store/giq1qbcwfx5gq0g9jf6id0b3m55a860j-source/pkgs/build-support/trivial-builders/default.nix:278:9:

          277|             runtimeEnv))
          278|       + lib.optionalString (runtimeInputs != [ ]) ''
             |         ^
          279|

       … while calling 'optionalString'

         at /nix/store/giq1qbcwfx5gq0g9jf6id0b3m55a860j-source/lib/strings.nix:268:5:

          267|     # String to return if condition is true
          268|     string: if cond then string else "";
             |     ^
          269|

       … from call site

         at /nix/store/giq1qbcwfx5gq0g9jf6id0b3m55a860j-source/pkgs/build-support/trivial-builders/default.nix:280:24:

          279|
          280|         export PATH="${lib.makeBinPath runtimeInputs}:$PATH"
             |                        ^
          281|       '' + ''

       … while calling 'makeSearchPathOutput'

         at /nix/store/giq1qbcwfx5gq0g9jf6id0b3m55a860j-source/lib/strings.nix:195:5:

          194|     # List of packages
          195|     pkgs: makeSearchPath subDir (map (lib.getOutput output) pkgs);
             |     ^
          196|

       … from call site

         at /nix/store/giq1qbcwfx5gq0g9jf6id0b3m55a860j-source/lib/strings.nix:195:11:

          194|     # List of packages
          195|     pkgs: makeSearchPath subDir (map (lib.getOutput output) pkgs);
             |           ^
          196|

       … while calling 'makeSearchPath'

         at /nix/store/giq1qbcwfx5gq0g9jf6id0b3m55a860j-source/lib/strings.nix:175:5:

          174|     # List of base paths
          175|     paths:
             |     ^
          176|     concatStringsSep ":" (map (path: path + "/" + subDir) (filter (x: x != null) paths));

       … while calling anonymous lambda

         at /nix/store/giq1qbcwfx5gq0g9jf6id0b3m55a860j-source/lib/strings.nix:176:32:

          175|     paths:
          176|     concatStringsSep ":" (map (path: path + "/" + subDir) (filter (x: x != null) paths));
             |                                ^
          177|

       … while evaluating derivation 'install'
         whose name attribute is located at /nix/store/giq1qbcwfx5gq0g9jf6id0b3m55a860j-source/pkgs/stdenv/generic/make-derivation.nix:331:7

       … while evaluating attribute 'text' of derivation 'install'

         at /nix/store/giq1qbcwfx5gq0g9jf6id0b3m55a860j-source/pkgs/build-support/trivial-builders/default.nix:103:16:

          102|       ({
          103|         inherit text executable checkPhase allowSubstitutes preferLocalBuild;
             |                ^
          104|         passAsFile = [ "text" ]

       … while calling anonymous lambda

         at /nix/store/wxjsfgkkd93fwqn1g49srpj9gms656wn-source/attrsets.nix:1205:18:

         1204|         mapAttrs
         1205|           (name: value:
             |                  ^
         1206|             if isAttrs value && cond value

       … from call site

         at /nix/store/wxjsfgkkd93fwqn1g49srpj9gms656wn-source/attrsets.nix:1208:18:

         1207|             then recurse (path ++ [ name ]) value
         1208|             else f (path ++ [ name ]) value);
             |                  ^
         1209|     in

       … while calling anonymous lambda

         at /nix/store/wxjsfgkkd93fwqn1g49srpj9gms656wn-source/modules.nix:242:72:

          241|           # For definitions that have an associated option
          242|           declaredConfig = mapAttrsRecursiveCond (v: ! isOption v) (_: v: v.value) options;
             |                                                                        ^
          243|

       … while evaluating the option `perSystem.x86_64-linux.mission-control.scripts.install.exec':

       … while calling anonymous lambda

         at /nix/store/wxjsfgkkd93fwqn1g49srpj9gms656wn-source/modules.nix:824:28:

          823|         # Process mkMerge and mkIf properties.
          824|         defs' = concatMap (m:
             |                            ^
          825|           map (value: { inherit (m) file; inherit value; }) (builtins.addErrorContext "while evaluating definitions from `${m.file}':" (dischargeProperties m.value))

       … while evaluating definitions from `/nix/store/5knbd9wlg1w6sgnbcf89lnyf2pf0fidd-source/tasks/administration/install, via option perSystem':

       … from call site

         at /nix/store/wxjsfgkkd93fwqn1g49srpj9gms656wn-source/modules.nix:825:137:

          824|         defs' = concatMap (m:
          825|           map (value: { inherit (m) file; inherit value; }) (builtins.addErrorContext "while evaluating definitions from `${m.file}':" (dischargeProperties m.value))
             |                                                                                                                                         ^
          826|         ) defs;

       … while calling 'dischargeProperties'

         at /nix/store/wxjsfgkkd93fwqn1g49srpj9gms656wn-source/modules.nix:896:25:

          895|   */
          896|   dischargeProperties = def:
             |                         ^
          897|     if def._type or "" == "merge" then

       error: attribute 'install-task' missing

       at /nix/store/5knbd9wlg1w6sgnbcf89lnyf2pf0fidd-source/tasks/administration/install/default.nix:14:22:

           13|                          # FIXME-QA(Krey): This makes the declaration more functional, but looks like an ugly hack
           14|                          exec = toString ((import ./script.nix { inherit pkgs inputs; }).install-task + /bin/install-task);
             |                      ^
           15|                  };

@roberth thanks for looking into this problem! Very appreciated! <3

The issue seems to be that the install-task module fails evaluation when using perSystem from flake-parts and i can't figure out why, to highlight the issue i changed the script body to simple hello-world and implemented your proposal to use flake-parts's variables without relying on the top-level module.

@Kreyren Kreyren removed the help wanted Extra attention is needed label Jul 7, 2024
@Kreyren
Copy link
Member Author

Kreyren commented Jul 7, 2024

Sorry for the spam, i was trying new vscodium extension and it screwed up the whole thing..

@TanvirOnGH This solution seems as the most functional, thoughts?

We can call it with nix run without the need of a cloned repo, it's distribution-independent and flexible for special requirements of the systems e.g. arm will need different deployment and integrated with mission control so that we can do , install nixos SYSTEM

@Tanvir1337x
Copy link

It's not elegant, but if it's functional, I guess it works for now.

@Kreyren
Copy link
Member Author

Kreyren commented Jul 8, 2024

It's not elegant, but if it's functional, I guess it works for now. -- @TanvirOnGH (#78 (comment))

It seems elegant and robust to me, can you be more specific? Should be request a peer-review?

@Kreyren Kreyren marked this pull request as ready for review July 8, 2024 02:03
@Kreyren Kreyren merged commit cf9ff1a into Arcanyx-org:central Jul 8, 2024
@Kreyren Kreyren deleted the install-task branch July 8, 2024 20:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants