Skip to content

Commit

Permalink
pkgs/README.md: add a chapter about version testing
Browse files Browse the repository at this point in the history
  • Loading branch information
AndersonTorres committed Oct 30, 2024
1 parent 0837dc8 commit 092b9aa
Showing 1 changed file with 28 additions and 0 deletions.
28 changes: 28 additions & 0 deletions pkgs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -741,6 +741,34 @@ stdenv.mkDerivation {
}
```

### Version Testing

Many programs provide a command-line flag that emits version information. It provides a straightforward test procedure, namely, run the program with that specific flag and verify the output.

Nixpkgs provides two tools for this task: [`testers.testVersion`](#tester-testVersion) and [`versionCheckHook`](#versioncheckhook).

#### Comparison between `versionCheckHook` and `testers.testVersion`

The most notable difference between `versionCheckHook` and `testers.testVersion` is that `versionCheckHook` runs at the build time of the package, while `passthru.tests.versions` executes a whole derivation that depends on the package.

The advantages of `testers.testVersion` over `versionCheckHook` are listed at [Package tests](#var-passthru-tests-packages) already. Below we will list some advantages of `versionCheckHook` over `testers.testVersion`:

- Since [`nixpkgs-review`](https://github.com/Mic92/nixpkgs-review) does not execute `passthru.tests` attributes, it will not execute `passthru.tests.version`.

On the other hand, `versionCheckHook` will be executed as a regular phase during the build time of the derivation, therefore it will be indirectly catched by `nixpkgs-review`, working around this limitation.

- When a pull request is opened against Nixpkgs repository, [ofborg](https://github.com/NixOS/ofborg)'s CI will automatically run `passthru.tests` attributes for the packages that are [directly changed by your PR (according to your commits' messages)](https://github.com/NixOS/ofborg?tab=readme-ov-file#automatic-building).

However, ofBorg does not run the `passthru.tests` attributes for _transitive dependencies_ of those packages. To execute them, commands like [`@ofborg build dependency1.tests dependency2.tests ...`](https://github.com/NixOS/ofborg?tab=readme-ov-file#build) are needed.

On the other hand, `versionCheckHook` will be executed as a regular phase during the build time of the derivation, therefore it will be indirectly catched by ofBorg, working around this limitation.

- Sometimes a package triggers no errors while being built - especially when the upstream provides no unit tests-, however it fails at runtime. When such a package is not used in a regular basis, it leaves room for a silent breakage that rots in the system / profile configuration, not being noticed for long periods of time until the next usage of this package.

Although `passthru.tests` fills the same purpose, it is more prone to be forgotten by human beings; on the other hand, `versionCheckHook` will be executed as a regular phase during the build time of the derivation, therefore it will not be accidentally ignored.

Despite having an almost identical functioning and a huge overlap, `versionCheckHook` and `testers.testVersion` have complementary roles, and there are no impediments for using both at the same time.

## Automatic package updates
[automatic-package-updates]: #automatic-package-updates

Expand Down

0 comments on commit 092b9aa

Please sign in to comment.