Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

lib/sources: remove 2 usages of toString on a path which will be read using fileContents #199812

Merged
merged 3 commits into from
Nov 17, 2022

Conversation

Artturin
Copy link
Member

@Artturin Artturin commented Nov 6, 2022

Check commit messages

Description of changes
Things done
  • Built on platform(s)
    • x86_64-linux
    • aarch64-linux
    • x86_64-darwin
    • aarch64-darwin
  • For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
  • Tested, as applicable:
  • Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
  • Tested basic functionality of all binary files (usually in ./result/bin/)
  • 22.11 Release Notes (or backporting 22.05 Release notes)
    • (Package updates) Added a release notes entry if the change is major or breaking
    • (Module updates) Added a release notes entry if the change is significant
    • (Module addition) Added a release notes entry if adding a new NixOS module
    • (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes
  • Fits CONTRIBUTING.md.

…al0000000000000000000000005-source''

happens on lazy-trees branch of Nix (NixOS/nix#6530)
@ofborg ofborg bot added 10.rebuild-darwin: 0 This PR does not cause any packages to rebuild on Darwin 10.rebuild-linux: 1-10 labels Nov 6, 2022
@Artturin
Copy link
Member Author

Artturin commented Nov 6, 2022

Getting the following in nix flake check / nix build -f ./pkgs/top-level/release.nix tarball

nixpkgs-tarball> checking eval-release.nix
nixpkgs-tarball> trace: lib.crossLists is deprecated, use lib.cartesianProductOfSets instead
nixpkgs-tarball> trace: warning: literalExample is deprecated, use literalExpression instead, or use literalDocBook for a non-Nix description.
nixpkgs-tarball> trace: warning: lib.mkFixStrictness has no effect and will be removed. It returns its argument unmodified, so you can just remove any calls.
nixpkgs-tarball> trace: `lib.nixpkgsVersion` is deprecated, use `lib.version` instead!
nixpkgs-tarball> trace: warning: lib.readPathsFromFile is deprecated, use a list instead
nixpkgs-tarball> trace: Warning: `showVal` is deprecated and will be removed in the next release, please use `traceSeqN`
nixpkgs-tarball> trace: lib.zip is deprecated, use lib.zipAttrsWith instead
nixpkgs-tarball> error: getting status of '/build/source/.git': No such file or directory
nixpkgs-tarball>        … while evaluating 'readCommitFromFile'
nixpkgs-tarball>        at /build/source/lib/sources.nix:177:36:
nixpkgs-tarball>           176|   commitIdFromGitRepo =
nixpkgs-tarball>           177|     let readCommitFromFile = file: path:
nixpkgs-tarball>              |                                    ^
nixpkgs-tarball>           178|         let fileName       = path + "/" + file;
nixpkgs-tarball>        … from call site
nixpkgs-tarball>        at /build/source/lib/sources.nix:169:35:
nixpkgs-tarball>           168|
nixpkgs-tarball>           169|   pathIsGitRepo = path: (tryEval (commitIdFromGitRepo path)).success;
nixpkgs-tarball>              |                                   ^
nixpkgs-tarball>           170|
nixpkgs-tarball>        … while evaluating 'pathIsGitRepo'
nixpkgs-tarball>        at /build/source/lib/sources.nix:169:19:
nixpkgs-tarball>           168|
nixpkgs-tarball>           169|   pathIsGitRepo = path: (tryEval (commitIdFromGitRepo path)).success;
nixpkgs-tarball>              |                   ^
nixpkgs-tarball>           170|
nixpkgs-tarball>        … from call site
nixpkgs-tarball>        at /build/source/lib/trivial.nix:218:11:
nixpkgs-tarball>           217|       gitRepo      = ./.. + "/.git";
nixpkgs-tarball>           218|     in if lib.pathIsGitRepo gitRepo
nixpkgs-tarball>              |           ^
nixpkgs-tarball>           219|        then lib.commitIdFromGitRepo gitRepo
nixpkgs-tarball>        … while evaluating 'revisionWithDefault'
nixpkgs-tarball>        at /build/source/lib/trivial.nix:214:5:
nixpkgs-tarball>           213|     # Default value to return if revision can not be determined
nixpkgs-tarball>           214|     default:
nixpkgs-tarball>              |     ^
nixpkgs-tarball>           215|     let
nixpkgs-tarball>        … from call site
nixpkgs-tarball>        at /build/source/doc/doc-support/lib-function-locations.nix:3:14:
nixpkgs-tarball>             2| let
nixpkgs-tarball>             3|   revision = pkgs.lib.trivial.revisionWithDefault (nixpkgs.revision or "master");
nixpkgs-tarball>              |              ^
nixpkgs-tarball>             4|
nixpkgs-tarball>        … while evaluating anonymous lambda
nixpkgs-tarball>        at /build/source/doc/doc-support/lib-function-locations.nix:62:8:
nixpkgs-tarball>            61|   xmlstrings = (nixpkgsLib.strings.concatMapStrings
nixpkgs-tarball>            62|       ({ name, value }:
nixpkgs-tarball>              |        ^
nixpkgs-tarball>            63|       ''
nixpkgs-tarball>        … from call site
nixpkgs-tarball>        … while evaluating 'concatMapStrings'
nixpkgs-tarball>        at /build/source/lib/strings.nix:54:25:
nixpkgs-tarball>            53|   */
nixpkgs-tarball>            54|   concatMapStrings = f: list: concatStrings (map f list);
nixpkgs-tarball>              |                         ^
nixpkgs-tarball>            55|
nixpkgs-tarball>        … from call site
nixpkgs-tarball>        at /build/source/doc/doc-support/lib-function-locations.nix:61:17:
nixpkgs-tarball>            60|   urlPrefix = "https://github.com/NixOS/nixpkgs/blob/${revision}";
nixpkgs-tarball>            61|   xmlstrings = (nixpkgsLib.strings.concatMapStrings
nixpkgs-tarball>              |                 ^
nixpkgs-tarball>            62|       ({ name, value }:
nixpkgs-tarball>        … while evaluating the attribute 'text' of the derivation 'locations.xml'
nixpkgs-tarball>        at /build/source/pkgs/stdenv/generic/make-derivation.nix:270:7:
nixpkgs-tarball>           269|     // (lib.optionalAttrs (attrs ? name || (attrs ? pname && attrs ? version)) {
nixpkgs-tarball>           270|       name =
nixpkgs-tarball>              |       ^
nixpkgs-tarball>           271|         let
nixpkgs-tarball>        … while evaluating the attribute 'buildCommand' of the derivation 'doc-support'
nixpkgs-tarball>        at /build/source/pkgs/stdenv/generic/make-derivation.nix:270:7:
nixpkgs-tarball>           269|     // (lib.optionalAttrs (attrs ? name || (attrs ? pname && attrs ? version)) {
nixpkgs-tarball>           270|       name =
nixpkgs-tarball>              |       ^
nixpkgs-tarball>           271|         let
nixpkgs-tarball>        … while evaluating the attribute 'postPatch' of the derivation 'nixpkgs-manual'
nixpkgs-tarball>        at /build/source/pkgs/stdenv/generic/make-derivation.nix:270:7:
nixpkgs-tarball>           269|     // (lib.optionalAttrs (attrs ? name || (attrs ? pname && attrs ? version)) {
nixpkgs-tarball>           270|       name =
nixpkgs-tarball>              |       ^
nixpkgs-tarball>           271|         let
nixpkgs-tarball>        … while evaluating the attribute 'drvPath'
nixpkgs-tarball>        at /build/source/lib/customisation.nix:210:7:
nixpkgs-tarball>           209|     in commonAttrs // {
nixpkgs-tarball>           210|       drvPath = assert condition; drv.drvPath;
nixpkgs-tarball>              |       ^
nixpkgs-tarball>           211|       outPath = assert condition; drv.outPath;
nixpkgs-tarball>        … while evaluating 'recurse'
nixpkgs-tarball>        at /build/source/maintainers/scripts/eval-release.nix:13:19:
nixpkgs-tarball>            12|   # nix-instantiate recurses into nested attribute sets.
nixpkgs-tarball>            13|   recurse = path: attrs:
nixpkgs-tarball>              |                   ^
nixpkgs-tarball>            14|     if (builtins.tryEval attrs).success then
nixpkgs-tarball>        … from call site
nixpkgs-tarball>        at /build/source/maintainers/scripts/eval-release.nix:21:69:
nixpkgs-tarball>            20|       else { recurseForDerivations = true; } //
nixpkgs-tarball>            21|            mapAttrs (n: v: let path' = path ++ [n]; in trace path' (recurse path' v)) attrs
nixpkgs-tarball>              |                                                                     ^
nixpkgs-tarball>            22|     else { };
nixpkgs-tarball>        … while evaluating anonymous lambda
nixpkgs-tarball>        at /build/source/maintainers/scripts/eval-release.nix:7:76:
nixpkgs-tarball>             6| let
nixpkgs-tarball>             7|   trace = if builtins.getEnv "VERBOSE" == "1" then builtins.trace else (x: y: y);
nixpkgs-tarball>              |                                                                            ^
nixpkgs-tarball>             8|
nixpkgs-tarball>        … from call site
nixpkgs-tarball>        at /build/source/maintainers/scripts/eval-release.nix:21:56:
nixpkgs-tarball>            20|       else { recurseForDerivations = true; } //
nixpkgs-tarball>            21|            mapAttrs (n: v: let path' = path ++ [n]; in trace path' (recurse path' v)) attrs
nixpkgs-tarball>              |                                                        ^
nixpkgs-tarball>            22|     else { };
nixpkgs-tarball>        … while evaluating anonymous lambda
nixpkgs-tarball>        at /build/source/maintainers/scripts/eval-release.nix:21:25:
nixpkgs-tarball>            20|       else { recurseForDerivations = true; } //
nixpkgs-tarball>            21|            mapAttrs (n: v: let path' = path ++ [n]; in trace path' (recurse path' v)) attrs
nixpkgs-tarball>              |                         ^
nixpkgs-tarball>            22|     else { };
nixpkgs-tarball>        … from call site
nixpkgs-tarball>        … while evaluating the attribute 'manual'
nixpkgs-tarball> build times:
nixpkgs-tarball> user time for the shell             0m0.000s
nixpkgs-tarball> system time for the shell           0m0.000s
nixpkgs-tarball> user time for all child processes   0m0.000s
nixpkgs-tarball> system time for all child processes 0m0.000s

remember to use NixOS/nix#6530

lib/sources.nix Outdated Show resolved Hide resolved
… using fileContents

It gives a warning on the lazy-trees branch of Nix
(NixOS/nix#6530)

"warning: applying 'toString' to path '...' and then accessing it is deprecated, at '...'"

'else toString (/. + "${base}/${path}");' at line 183 may still cause a warning but i don't know how
to reach that codepath and test so im leaving it untouched

changing it to 'else /. + "${base}/${path}";'
caused this error
```
error: a string that refers to a store path cannot be appended to a path

       at /home/systems/nixpkgs/lib/sources.nix:183:20:

          182|               then path
          183|               else /. + "${base}/${path}";
             |                    ^
          184|         in if pathIsRegularFile path
```
@Artturin Artturin force-pushed the removeusagesoftostringonpath1 branch from 4c7ddd2 to 8c1b019 Compare November 6, 2022 18:00
@Artturin
Copy link
Member Author

Artturin commented Nov 6, 2022

Getting the following in nix flake check / nix build -f ./pkgs/top-level/release.nix tarball
...
remember to use NixOS/nix#6530

those commands work on master

@Artturin Artturin marked this pull request as draft November 6, 2022 19:11
This requires us to avoid the `tryEval` + `throw` combination,
because throw is strict in its error message, and we don't want
to drop our single clue when `commitIdFromGitRepo` is used
incorrectly.
@roberth
Copy link
Member

roberth commented Nov 7, 2022

The remaining problem was the pathIsGitRepo was using tryEval, but throw is strict in its argument, so it failed to + path, which should have been + toString path anyway. I've taken the liberty to push a fix.

@tobiasBora
Copy link
Contributor

Is this PR supposed to solve the fact that nixos-rebuild takes a lot of RAM (~1.3G) when the input is a flake (not tried without flake) as it tries to evaluate the whole nixpkgs repository? Or is it unrelated?

@Artturin Artturin marked this pull request as ready for review November 13, 2022 21:30
@Artturin
Copy link
Member Author

Artturin commented Nov 13, 2022

Is this PR supposed to solve the fact that nixos-rebuild takes a lot of RAM (~1.3G) when the input is a flake (not tried without flake) as it tries to evaluate the whole nixpkgs repository? Or is it unrelated?

Thats unrelated

@tobiasBora
Copy link
Contributor

Ok thanks @Artturin . For reference I created a separate issue there NixOS/nix#7308

@Artturin
Copy link
Member Author

Thanks @roberth

@Artturin Artturin merged commit e3bd5d1 into NixOS:master Nov 17, 2022
@Artturin Artturin deleted the removeusagesoftostringonpath1 branch November 17, 2022 13:18
@zowoq
Copy link
Contributor

zowoq commented Nov 18, 2022

https://github.com/NixOS/nixpkgs/actions/runs/3494613825/jobs/5850544462

This PR seems to have broken nixpkgs manual build.

https://hydra.nixos.org/build/199090592/nixlog/1

It's blocking nixpkgs-unstable.

@vcunat
Copy link
Member

vcunat commented Nov 18, 2022

Right, bisected to ec8f8f6 in particular. I don't understand why this happens, at least for now. But if noone does very soon, we should revert.

@vcunat
Copy link
Member

vcunat commented Nov 18, 2022

Well, removing the doc-comment for commitIdFromGitRepoOrError makes the error go away, but I have no idea what's wrong about it and even simplifying to one simple sentence didn't help 🤯 (the XML errors here don't help me)

@vcunat
Copy link
Member

vcunat commented Nov 18, 2022

Apparently this is related to the combination of this new function not getting exported from the file, while still getting documented. It's actually not good to get nixpkgs docs for a function that can't be used (from outside this file), so we can either

A. export it

--- a/lib/sources.nix
+++ b/lib/sources.nix
@@ -281,2 +281,3 @@
     commitIdFromGitRepo
+    commitIdFromGitRepoOrError
 

B. or not include it in the docs. It's nice to keep description though, so e.g. this experiment worked for me:

  # Get the commit id of a git repo.
  #
  # Returns `{ value = commitHash }` or `{ error = "... message ..." }`.
  #
  # Example: commitIdFromGitRepo <nixpkgs/.git>

@Artturin
Copy link
Member Author

#201779

@amaxine
Copy link
Contributor

amaxine commented Nov 22, 2022

I cannot rebuild any of my systems since this PR was merged, had to revert to commit~1 to have working systems again. #201779 accurately represents my issue, but I'm unaware of my system being a particularly esoteric configuration.

Shouldn't this be reverted? It's been 4 days and master is, at least to me, not possible to build:

error: a string that refers to a store path cannot be appended to a path

       at /home/systems/nixpkgs/lib/sources.nix:193:30:

          192|               then path
          193|               else toString (/. + "${base}/${path}");
             |                              ^
          194|         in if pathIsRegularFile path
(use '--show-trace' to show detailed location information)

(I'm not pasting a full trace because the issue afaik covers it)

@Artturin
Copy link
Member Author

#202370

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
10.rebuild-darwin: 0 This PR does not cause any packages to rebuild on Darwin 10.rebuild-linux: 1-10
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants