-
-
Notifications
You must be signed in to change notification settings - Fork 14.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
doc: add instructions for creating package tests #120534
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change | ||||
---|---|---|---|---|---|---|
|
@@ -512,3 +512,73 @@ If you do need to do create this sort of patch file, one way to do so is with gi | |||||
```ShellSession | ||||||
$ git diff > nixpkgs/pkgs/the/package/0001-changes.patch | ||||||
``` | ||||||
|
||||||
## Package tests {#sec-package-tests} | ||||||
|
||||||
Tests are important to ensure quality and make reviews and automatic updates easy. | ||||||
|
||||||
Nix package tests are a lightweight alternative to [NixOS module tests](https://nixos.org/manual/nixos/stable/#sec-nixos-tests). They can be used to create simple integration tests for packages while the module tests are used to test services or programs with a graphical user interface on a NixOS VM. Unittests that are included in the source code of a package should be executed in the `checkPhase`. | ||||||
|
||||||
### Writing package tests {#ssec-package-tests-writing} | ||||||
|
||||||
This is an example using the `phoronix-test-suite` package with the current best practices. | ||||||
|
||||||
Add the tests in `passthru.tests` to the package definition like this: | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. i would like to link There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Good idea, current level of interlinking in the manual leaves a lot to be desired. If you just use the anchor name, it will work now and we will migrate it:
Suggested change
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I rember the first time I saw Maybe one small phrase to introduce that concept here would be in order
(If that makes sense) There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @blaggacao yeah, that's true. there is reference documentation at https://nixos.org/manual/nixpkgs/stable/#sec-standard-meta-attributes |
||||||
|
||||||
```nix | ||||||
{ stdenv, lib, fetchurl, callPackage }: | ||||||
|
||||||
stdenv.mkDerivation { | ||||||
… | ||||||
|
||||||
passthru.tests = { | ||||||
simple-execution = callPackage ./tests.nix { }; | ||||||
}; | ||||||
|
||||||
meta = { … }; | ||||||
} | ||||||
``` | ||||||
|
||||||
Create `tests.nix` in the package directory: | ||||||
|
||||||
```nix | ||||||
{ runCommand, phoronix-test-suite }: | ||||||
|
||||||
let | ||||||
inherit (phoronix-test-suite) pname version; | ||||||
in | ||||||
|
||||||
runCommand "${pname}-tests" { meta.timeout = 3; } | ||||||
'' | ||||||
# automatic initial setup to prevent interactive questions | ||||||
${phoronix-test-suite}/bin/phoronix-test-suite enterprise-setup >/dev/null | ||||||
# get version of installed program and compare with package version | ||||||
if [[ `${phoronix-test-suite}/bin/phoronix-test-suite version` != *"${version}"* ]]; then | ||||||
echo "Error: program version does not match package version" | ||||||
exit 1 | ||||||
fi | ||||||
# run dummy command | ||||||
${phoronix-test-suite}/bin/phoronix-test-suite dummy_module.dummy-command >/dev/null | ||||||
# needed for Nix to register the command as successful | ||||||
touch $out | ||||||
'' | ||||||
``` | ||||||
|
||||||
### Running package tests {#ssec-package-tests-running} | ||||||
|
||||||
davidak marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||
You can run these tests with: | ||||||
|
||||||
```ShellSession | ||||||
$ cd path/to/nixpkgs | ||||||
$ nix-build -A phoronix-test-suite.tests | ||||||
``` | ||||||
|
||||||
### Examples of package tests {#ssec-package-tests-examples} | ||||||
|
||||||
Here are examples of package tests: | ||||||
|
||||||
- [Jasmin compile test](https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/compilers/jasmin/test-assemble-hello-world/default.nix) | ||||||
- [Lobster compile test](https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/compilers/lobster/test-can-run-hello-world.nix) | ||||||
- [Spacy annotation test](https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/python-modules/spacy/annotation-test/default.nix) | ||||||
- [Libtorch test](https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/libraries/science/math/libtorch/test/default.nix) | ||||||
- [Multiple tests for nanopb](https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/libraries/nanopb/default.nix) | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. It might make sense to link to a specific git revision here. Otherwise we'll risk ending up with a bunch of dead links. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. That makes sense for the long-term perspective where someone reads this manual version in 20 years. All examples are terrible and will be replaced tho. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We can handle graphical test just fine using Xvfb. The NixOS tests are primarily integration tests that depend on other services or to test NixOS modules themselves.
passthru.tests
should really be preferred whenever possible since the NixOS tests really are heavy.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
great. do you have an example?
i need it for #120453 (comment)
can it replace https://github.com/NixOS/nixpkgs/blob/master/nixos/tests/shattered-pixel-dungeon.nix ?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The invocation is
nixpkgs/pkgs/development/libraries/libdazzle/default.nix
Lines 26 to 28 in b7e08de
Sometimes the dbus session part can be omitted.
Yeah, that should not be NixOS test.
But xvfb-run just creates a virtual frame buffer so that you can start graphical programs – you will still need to write your own tests or use upstream ones. We should probably create some helpers for passthru tests so that we can use tesseract like in the NixOS test.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Shouldn't it be
installCheckPhase
is you want to e.g. test the final result? (IIRC that was wan of the ideas behind the NixCon 2017 talk as well).There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Clarification on that is added with #119731.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@jtojnar thanks!
that's probably what #36842 tried, but it was rejected because it duplicated nixos tests... maybe it can be generalized somehow so it works for both