nixpkgs-manual: init

[philiptaron]

Motivation

Make it easier and more natural to build the contents of the doc/ directory.

Description of changes

• Reshape the derivations from doc/ to callPackage style.
• When needed, extract a derivation into its own file.
• Organize them under the nixpkgs-manual name.
• Make doc/default.nix and doc/shell.nix forward to them.

The diff is pretty chunky, but it's not too bad when you have the old and new side by side... sadly GitHub makes this pretty impossible.

Incidental changes:

• nixfmt on new files
• stdenvNoCC instead of stdenv as appropriate
• Derivations accessible through passthru

Why not pkgs/by-name?

Because it has a rule that packages in pkgs/by-name cannot refer to files outside their own directory, which this obviously needs to do. I've opened NixOS/nixpkgs-vet#64 to ideate about this use case -- it's not broad, but it would be nice -- but in the meantime, this derivation can't use pkgs/by-name.

The CI failure says that merging won't break the main branch.

Things done

• Built on platform(s)
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)
  • sandbox = relaxed
  • sandbox = true
• Tested, as applicable:
  • nix-build doc/
  • nix-build -A nixpkgs-manual
  • pushd doc/ && nix-shell
  • On master, nix-build doc/, on branch nix-build doc/, and then diffoscope <original> <new> (no changes except nix-support/hydra-build-products)
• Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
• Tested basic functionality of all binary files (usually in ./result/bin/)
• Fits CONTRIBUTING.md.

[hsjobeki]

Maybe we should not add the docs with pkgs-by-name, but via an overlay or add an exclude from checks since it will always import files from /doc ?

[philiptaron]

Maybe we should not add the docs with pkgs-by-name, but via an overlay or add an exclude from checks since it will always import files from /doc ?

I've put this PR in draft mode while figuring out the way forward with nixpkgs-check-by-name rules. No such exclusion mechanism exists today, and there's a real use case for it here, I think.

I'm going to have better availability to contribute to open source during the summer. That's when I plan on re-engaging with this.

[philiptaron]

@hsjobeki said:

Maybe we should not add the docs with pkgs-by-name, but via an overlay or add an exclude from checks since it will always import files from doc/?

This is the direction I went. The current nixpkgs-check-by-name failure says this:

- Attribute `pkgs.nixpkgs-manual` is a new top-level package using `pkgs.callPackage ./pkgs/data/documentation/nixpkgs-manual { /* ... */ }`.
  Please define it in pkgs/by-name/ni/nixpkgs-manual/package.nix instead.
  See `pkgs/by-name/README.md` for more details.
  Since the second `callPackage` argument is `{ }`, no manual `callPackage` in pkgs/top-level/all-packages.nix is needed anymore.

This PR introduces additional instances of discouraged patterns as listed above.
Merging is discouraged but would not break the base branch.

This is now ready for review, I think.

[philiptaron]

There is one thing I could do that would satisfy nixpkgs-check-by-name and the existing code: move everything, web assets and all, into pkgs/by-name/ni/nixpkgs-manual and make a symlink to that directory replacing the doc/ directory.

[fricklerhandwerk]

Re-doing the construction to be callPackage style is defintely a good thing, but I fear that hiding the sources in pkgs will make it pretty hard to find. The manual is pretty important and keeping it in /doc would be vital to make it as likely as possible for people to find it on their own.

[eclairevoyant]

The manual is pretty important and keeping it in /doc would be vital to make it as likely as possible for people to find it on their own.

Can we add a symlink? (doc/nixpkgs-manual -> pkgs/.../nixpkgs-manual) Even the GH web ui can follow symlinks.

[philiptaron]

The manual is pretty important and keeping it in /doc would be vital to make it as likely as possible for people to find it on their own.

Can we add a symlink? (doc/nixpkgs-manual -> pkgs/.../nixpkgs-manual) Even the GH web ui can follow symlinks.

This it what I suggested in #311459 (comment) -- and yeah, it's quite possible, I believe.

[infinisil]

You have my blessing to ignore the pkgs/by-name check failure! This kind of failure won't break master, and it's not worth trying to work around it when it's such a special case.

The real solution imo is to get away from the messy single pkgs namespace, which currently contains both packages, but also a whole lot else. But that's for another day :)

[infinisil]

I reviewed all commits, looking good except one thing!

[infinisil]

I noticed you changed this to an assert when it was this before:

nixpkgs/doc/default.nix

Lines 89 to 93 in 6a03708

	if hasPrefix (toString ../..) (toString decl)
	then
	let subpath = removePrefix "/" (removePrefix (toString ../.) (toString decl));
	in { url = "https://github.com/NixOS/nixpkgs/blob/master/${subpath}"; name = subpath; }
	else decl)

Notably I'd expect this to break manual rendering when building non-Nixpkgs modules using documentation.nixos.includeAllModules

[philiptaron]

Yes! I thought that for the nixpkgs manual, there wouldn't be ... non-nixpkgs modules. Am I wrong there?

Notably this would be terribly incorrect for the NixOS manual.

[infinisil]

Ah yes you're right! Mixed things up here :)

[infinisil]

Ah yes you're right! Mixed things up here :)

[philiptaron]

Thank you as always @infinisil!

[Artturin]

This shows up in the rebuild count by ofborg in unrelated packages https://gist.github.com/GrahamcOfBorg/e02bf2c3d2c6a52e1bfb155f6462a9a4

[philiptaron]

This shows up in the rebuild count by ofborg in unrelated packages https://gist.github.com/GrahamcOfBorg/e02bf2c3d2c6a52e1bfb155f6462a9a4

I'll be able to take a look tomorrow. Likely something is taking a dependency on the whole Nixpkgs tree somewhere.

[philiptaron]

OK, I root-caused this rebuild. Step to repro, in the nixpkgs root:

git checkout master
nix-build -A nixpkgs-manual
mv result result-on-master
git commit --allow-empty -m "testing"
nix-build -A nixpkgs-manual
nix-diff ./result-on-master ./result

Here's the diff:

- ./result-on-master:{out}
+ ./result:{out}
• The environments do not match:
    buildPhase=''
        /nix/store/k26j1ix7r2sj83f7rdrjjnsw3gb0zfyi-documentation-highlighter/loader.js

      cp -t out ./style.css ./anchor.min.js ./anchor-use.js

      nixos-render-docs manual html \
        --manpage-urls ./manpage-urls.json \
    -   --revision f4f322d1424aa547eba9cb092f905f5ceb9b639c \
    +   --revision 65abf7019bd37b6ad20ea008843ab4a9d316eaa5 \
        --stylesheet style.css \
        --stylesheet highlightjs/mono-blue.css \
        --script ./highlightjs/highlight.pack.js \
        --script ./highlightjs/loader.js \
''

The revision is being incorporated with these lines:

nixpkgs/doc/doc-support/package.nix

Lines 61 to 65 in 4089e29

	nixos-render-docs manual html \
	--manpage-urls ./manpage-urls.json \
	--revision ${lib.trivial.revisionWithDefault (nixpkgs.rev or "master")} \
	--stylesheet style.css \
	--stylesheet highlightjs/mono-blue.css \

lib.trivial.revisionWithDefault has this implementation:

nixpkgs/lib/trivial.nix

Lines 437 to 445 in 4089e29

	revisionWithDefault =
	default:
	let
	revisionFile = "${toString ./..}/.git-revision";
	gitRepo = "${toString ./..}/.git";
	in if lib.pathIsGitRepo gitRepo
	then lib.commitIdFromGitRepo gitRepo
	else if lib.pathExists revisionFile then lib.fileContents revisionFile
	else default;

Which makes the whole thing make sense. Every git revision makes this rebuild, because the documentation includes the git revision it was built with.

No other package includes the git revision it was built with in nixpkgs; the other usage is in nixos/modules/misc/version.nix, which uses it as a default for options.system.revision.

[philiptaron]

My proposed fix is here:

• nixpkgs-manual: use injected revision only #330915