Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Better documentation #52

Merged
merged 3 commits into from
Apr 22, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions .github/workflows/main.yml
Original file line number Diff line number Diff line change
Expand Up @@ -35,3 +35,13 @@ jobs:
run: scripts/release.sh
env:
GH_TOKEN: ${{ github.token }}

# Make sure that all links in Markdown documents are valid
xrefcheck:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4

# While this has a Nix build available, it needs to evaluate and build so much
# that I don't think it's worth adding it to the nix-build.
- uses: serokell/xrefcheck-action@v1
philiptaron marked this conversation as resolved.
Show resolved Hide resolved
69 changes: 69 additions & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,69 @@
# Contributor documentation

[@infinisil](https://github.com/infinisil) is the admin and main developer of this repository,
while everybody in [@NixOS/nixpkgs-check-by-name](https://github.com/orgs/NixOS/teams/nixpkgs-check-by-name) has write access.

## Development

Enter the development environment in this directory either automatically with
[`direnv`](https://github.com/direnv/direnv) or manually with
```bash
nix-shell
```

The most important tools and commands in this environment are:
- [rust-analyzer](https://rust-analyzer.github.io/) to have an IDE-like experience for your own editor.
- Running tests:
```bash
cargo test
```
- Linting and formatting:
```bash
cargo clippy --all-targets
treefmt
```
- Running the [main CI checks](./.github/workflows/main.yml) locally:
```bash
nix-build -A ci
```

### Integration tests

Integration tests are declared in [`./tests`](./tests) as subdirectories imitating Nixpkgs with these files:
- `default.nix`:
Always contains
```nix
import <test-nixpkgs> { root = ./.; }
```
which makes
```
nix-instantiate <subdir> --eval -A <attr> --arg overlays <overlays>
```
work very similarly to the real Nixpkgs, just enough for the program to be able to test it.
- `pkgs/by-name`:
The `pkgs/by-name` directory to check.

- `pkgs/top-level/all-packages.nix` (optional):
Contains an overlay of the form
```nix
self: super: {
# ...
}
```
allowing the simulation of package overrides to the real [`pkgs/top-level/all-packages.nix`](https://github.com/NixOS/nixpkgs/blob/master/pkgs/top-level/all-packages.nix).
The default is an empty overlay.

- `base` (optional):
Contains another subdirectory imitating Nixpkgs with potentially any of the above structures.
This is used to test [ratchet checks](./README.md#ratchet-checks).

- `expected` (optional):
A file containing the expected standard output.
The default is expecting an empty standard output.

## Automation

Pinned dependencies are [regularly updated automatically](./.github/workflows/update.yml).

Releases are [automatically created](./.github/workflows/release.yml) when the `version` field in [`Cargo.toml`](./Cargo.toml)
is updated from a push to the main branch.
112 changes: 53 additions & 59 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,26 +1,69 @@
# Nixpkgs pkgs/by-name checker

This directory implements a program to check the [validity](#validity-checks) of the [`pkgs/by-name` Nixpkgs directory](https://github.com/NixOS/nixpkgs/tree/master/pkgs/by-name).
This is part of the implementation of [RFC 140](https://github.com/NixOS/rfcs/pull/140).
This repository implements a program to check [Nixpkgs' `pkgs/by-name` directory](https://github.com/NixOS/nixpkgs/tree/master/pkgs/by-name) as part of [RFC 140](https://github.com/NixOS/rfcs/pull/140).

This code in this repository was originally [part of Nixpkgs](https://github.com/NixOS/nixpkgs/commits/55bf02190ee57fcf83490fd7b6bf7834e28c9c86/pkgs/test/nixpkgs-check-by-name), but has since been [moved into this separate repository](https://github.com/NixOS/nixpkgs/issues/286559).
See [`CONTRIBUTING.md`](./CONTRIBUTING.md) for contributor documentation.
Below is the user documentation.

To see how it is used in Nixpkgs, see the [`check-by-name.yml` workflow](https://github.com/NixOS/nixpkgs/blob/master/.github/workflows/check-by-name.yml).
Currently the only intended user for this program is [Nixpkgs](https://github.com/NixOS/nixpkgs).
So the interface may be changed in breaking ways as long as Nixpkgs is adjusted to deal with it.
See [the `pkgs/by-name` Readme](https://github.com/NixOS/nixpkgs/blob/master/pkgs/by-name/README.md#validation)
for how it's used in Nixpkgs.

[@infinisil](https://github.com/infinisil) is the admin and main developer of this repository, while everybody in [@NixOS/nixpkgs-check-by-name](https://github.com/orgs/NixOS/teams/nixpkgs-check-by-name) has write access.
## Nix derivations

## Interface
The source code contains a `default.nix` file, which defines a Nix function.

The interface of the tool is shown with `--help`:
The function takes an attribute set with at least these attributes as its argument:
- `system` (String, defaults to [`builtins.currentSystem`](https://nixos.org/manual/nix/stable/language/builtin-constants.html#builtins-currentSystem)):
The [`system`](https://nixos.org/manual/nix/stable/language/derivations#attr-system)
to build the resulting derivation with.

The function returns an attribute set with at least these attributes:
- `build` ([Package attribute set](https://nixos.org/manual/nix/stable/glossary#package-attribute-set)):
A derivation that can be built with the given `system`.

There is no guarantee that the derivation succeeds on systems that don't have [prebuilt store paths](#prebuilt-store-paths),
but it can be attempted with

```bash
nix-build https://github.com/NixOS/nixpkgs-check-by-name/tarball/master -A build
```
cargo run -- --help

## Prebuilt store paths

The [GitHub releases](https://github.com/NixOS/nixpkgs-check-by-name/releases)
contain a [gzip](https://www.gnu.org/software/gzip/)-compressed
[Nix Archive](https://nixos.org/manual/nix/stable/command-ref/nix-store/export.html)
of the [build closure](https://nixos.org/manual/nix/stable/glossary#gloss-closure)
of the [Nix derivation](#nix-derivations) with `x86_64-linux` as the `system`.

This release artifact is named `x86_64-linux.nar.gz`
and can be imported into a local Nix store using:

```bash
storePath=$(gzip -cd x86_64-linux.nar.gz | nix-store --import | tail -1)
# To prevent it from being garbage-collected
nix-store --realise "$storePath" --add-root result
```

The interface may be changed over time only if the CI workflow making use of it is adjusted to deal with the change appropriately.
Compared to building the [Nix derivations](#nix-derivations),
this has the benefit that no Nix evaluation needs to take place
and is therefore much faster and less storage intensive.

## Binary interface

The store path acquired from the above methods contains
a `system`-specific binary under `$storePath/bin/nixpkgs-check-by-name`.

The public interface of this binary is printed by calling
```bash
result/bin/nixpkgs-check-by-name --help
```

## Validity checks

These checks are performed by this tool:
The following checks are performed when calling the binary:

### File structure checks
- `pkgs/by-name` must only contain subdirectories of the form `${shard}/${name}`, called _package directories_.
Expand Down Expand Up @@ -52,52 +95,3 @@ The current ratchets are:
(see [nix evaluation checks](#nix-evaluation-checks)) must not be introduced.
- New top-level packages defined using `pkgs.callPackage` must be defined with a package directory.
- Once a top-level package uses `pkgs/by-name`, it also can't be moved back out of it.

## Development

Enter the development environment in this directory either automatically with `direnv` or with
```
nix-shell
```

Then use `cargo`:
```
cargo build
cargo test
cargo fmt
cargo clippy
```

## Tests

Tests are declared in [`./tests`](./tests) as subdirectories imitating Nixpkgs with these files:
- `default.nix`:
Always contains
```nix
import <test-nixpkgs> { root = ./.; }
```
which makes
```
nix-instantiate <subdir> --eval -A <attr> --arg overlays <overlays>
```
work very similarly to the real Nixpkgs, just enough for the program to be able to test it.
- `pkgs/by-name`:
The `pkgs/by-name` directory to check.

- `all-packages.nix` (optional):
Contains an overlay of the form
```nix
self: super: {
# ...
}
```
allowing the simulation of package overrides to the real [`pkgs/top-level/all-packages.nix`](https://github.com/NixOS/nixpkgs/blob/master/pkgs/top-level/all-packages.nix).
The default is an empty overlay.

- `base` (optional):
Contains another subdirectory imitating Nixpkgs with potentially any of the above structures.
This is used for [ratchet checks](#ratchet-checks).

- `expected` (optional):
A file containing the expected standard output.
The default is expecting an empty standard output.