doc: add instructions for creating package tests #120534

davidak · 2021-04-24T15:03:44Z

Motivation for this change

This is a big milestone in the effort to add tests to packages that was started by @Profpatsch in 2017 (see NixCon talk ). With documentation, people can actually create tests that can be used to ensure quality and help reviews and automatic updates!

Closes #73076
Closes #27604
Closes #46632

This is not perfect, but better than not having it documented.

I would like to link to better examples, but we don't have them yet. That changes hopefully with this documentation.

This introduces the term "Nix package tests" in addition to "NixOS module tests", or "package tests", "nixos tests", "module tests" for short.

Things done

jtojnar

Looks great, just a few comments.

doc/contributing/submitting-changes.chapter.md

doc/contributing/coding-conventions.chapter.md

davidak · 2021-04-24T16:45:53Z

doc/contributing/coding-conventions.chapter.md

+
+This is an example using the `phoronix-test-suite` package with the current best practices.
+
+Add the tests in `passthru.tests` to the package definition like this:


i would like to link checkPhase and passthru.tests inside the document, but i don't know how and if its worth adding now when the build pipeline changes soon and it might brake

Good idea, current level of interlinking in the manual leaves a lot to be desired.

If you just use the anchor name, it will work now and we will migrate it:

Suggested change

Add the tests in `passthru.tests` to the package definition like this:

Add the tests in [`passthru.tests`](#var-meta-tests) to the package definition like this:

I rember the first time I saw passthru, I went like: "?!?!!?!? (It's not even an english word)"

Maybe one small phrase to introduce that concept here would be in order

By convention, we use the passthru attribute which allows us to store arbitrary data, ...

(If that makes sense)

@blaggacao yeah, that's true. there is reference documentation at https://nixos.org/manual/nixpkgs/stable/#sec-standard-meta-attributes

jtojnar

Looks great thanks.

doc/contributing/coding-conventions.chapter.md

jtojnar · 2021-04-24T16:56:16Z

doc/contributing/coding-conventions.chapter.md

+
+Tests are important to ensure quality and make reviews and automatic updates easy.
+
+Nix package tests are a lightweight alternative to [NixOS module tests](https://nixos.org/manual/nixos/stable/#sec-nixos-tests). They can be used to create simple integration tests for packages while the module tests are used to test services or programs with a graphical user interface on a NixOS VM. Unittests that are included in the source code of a package should be executed in the `checkPhase`.


module tests are used to test services or programs with a graphical user interface on a NixOS VM

We can handle graphical test just fine using Xvfb. The NixOS tests are primarily integration tests that depend on other services or to test NixOS modules themselves. passthru.tests should really be preferred whenever possible since the NixOS tests really are heavy.

great. do you have an example?
i need it for #120453 (comment)
can it replace https://github.com/NixOS/nixpkgs/blob/master/nixos/tests/shattered-pixel-dungeon.nix ?

The invocation is

nixpkgs/pkgs/development/libraries/libdazzle/default.nix

Lines 26 to 28 in b7e08de

xvfb-run -s '-screen 0 800x600x24' dbus-run-session \

--config-file=${dbus.daemon}/share/dbus-1/session.conf \

meson test --print-errorlogs

Sometimes the dbus session part can be omitted.

can it replace https://github.com/NixOS/nixpkgs/blob/master/nixos/tests/shattered-pixel-dungeon.nix ?

Yeah, that should not be NixOS test.

But xvfb-run just creates a virtual frame buffer so that you can start graphical programs – you will still need to write your own tests or use upstream ones. We should probably create some helpers for passthru tests so that we can use tesseract like in the NixOS test.

Shouldn't it be installCheckPhase is you want to e.g. test the final result? (IIRC that was wan of the ideas behind the NixCon 2017 talk as well).

Clarification on that is added with #119731.

@jtojnar thanks!

We should probably create some helpers for passthru tests so that we can use tesseract like in the NixOS test.

that's probably what #36842 tried, but it was rejected because it duplicated nixos tests... maybe it can be generalized somehow so it works for both

teto · 2021-04-24T17:04:09Z

nice improvement. There are also tests in pkgs/tests, should these be moved on to their respective software directory ?

davidak · 2021-04-24T17:10:16Z

@teto thanks!

There are also tests in pkgs/tests

oh, i see that for the first time (it exists since 2004 😮 )

should these be moved on to their respective software directory ?

if possible, yes.

i would like to have a general test-version.nix, that would fit there well

or if a test includes multiple packages, like many nixosTests

nixos-discourse · 2021-04-24T19:09:27Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/improve-a-pkgs-tests-for-nixpkgs-update/9530/17

Ma27 · 2021-04-24T19:15:34Z

doc/contributing/coding-conventions.chapter.md

+
+Tests are important to ensure quality and make reviews and automatic updates easy.
+
+Nix package tests are a lightweight alternative to [NixOS module tests](https://nixos.org/manual/nixos/stable/#sec-nixos-tests). They can be used to create simple integration tests for packages while the module tests are used to test services or programs with a graphical user interface on a NixOS VM. Unittests that are included in the source code of a package should be executed in the `checkPhase`.


Shouldn't it be installCheckPhase is you want to e.g. test the final result? (IIRC that was wan of the ideas behind the NixCon 2017 talk as well).

doc/contributing/coding-conventions.chapter.md

Ma27 · 2021-04-24T19:17:06Z

doc/contributing/coding-conventions.chapter.md

+- [Lobster compile test](https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/compilers/lobster/test-can-run-hello-world.nix)
+- [Spacy annotation test](https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/python-modules/spacy/annotation-test/default.nix)
+- [Libtorch test](https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/libraries/science/math/libtorch/test/default.nix)
+- [Multiple tests for nanopb](https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/libraries/nanopb/default.nix)


It might make sense to link to a specific git revision here. Otherwise we'll risk ending up with a bunch of dead links.

That makes sense for the long-term perspective where someone reads this manual version in 20 years.

All examples are terrible and will be replaced tho.

roberth · 2021-04-24T19:28:37Z

This seems a bit more elaborate than #119731 but perhaps you could 'steal' a line or two.

Also check out #119942, which creates a better way to reference the package, supporting overrides.

abathur · 2021-04-24T19:51:17Z

@davidak A few days ago in the contributor-retention discourse thread I sketched out some questions I could imagine gaining perspective on via "what is meaningful to test" documentation. trying to keep my brain focused on something today so I haven't read this yet, but I wanted to go ahead and link it in case it has any utility for you here: https://discourse.nixos.org/t/contributor-retention/12412/75

Apologies if it's mostly overlap.

davidak · 2021-04-24T19:56:11Z

@abathur thanks. this just documents the best practices as of nov. 2019. when it's merged, we can extend the scope

we need documentation on what to test and how

blaggacao · 2021-04-24T20:43:38Z

I want to crosslink to my WIP effort to provide a nixpkgs-devshell. My vision is that we can provide templates per language subsytem and that those templates would include a test skeleton, as well.

Very early stage, nothing works, but from looking at the code, one might be able to get the general idea:

https://github.com/blaggacao/nixpkgs-devshell#readme

(... in search for capable contributors to submit those templated best practices)

lukegb · 2021-05-01T18:15:05Z

Discussion on this seems to have slowed down: @davidak @jtojnar are we ready to 🚢 this?

davidak · 2021-05-01T20:18:44Z

@lukegb thanks for noticing. i'm also afraid this will stall. from my perspective it is ready to merge. more reviews would be nice

lukegb · 2021-05-01T21:31:11Z

knedlsepp · 2021-07-09T12:00:17Z

Are such passthru.tests being built on hydra?

roberth · 2021-07-09T12:46:15Z

They're run by ofborg, but not by hydra, as far as I know.

This means problems in changed dependencies can go unnoticed until someone/something changes the package with a PR.

github-actions bot added the 8.has: documentation This PR adds or changes documentation label Apr 24, 2021

ofborg bot added 10.rebuild-darwin: 0 This PR does not cause any packages to rebuild on Darwin 10.rebuild-linux: 0 This PR does not cause any packages to rebuild on Linux labels Apr 24, 2021

jtojnar reviewed Apr 24, 2021

View reviewed changes

doc/contributing/submitting-changes.chapter.md Outdated Show resolved Hide resolved

davidak force-pushed the package-tests branch 2 times, most recently from 2194f30 to 1c35fb8 Compare April 24, 2021 16:18

jtojnar reviewed Apr 24, 2021

View reviewed changes

doc/contributing/coding-conventions.chapter.md Outdated Show resolved Hide resolved

jtojnar reviewed Apr 24, 2021

View reviewed changes

doc/contributing/coding-conventions.chapter.md Outdated Show resolved Hide resolved

jtojnar reviewed Apr 24, 2021

View reviewed changes

doc/contributing/coding-conventions.chapter.md Outdated Show resolved Hide resolved

davidak force-pushed the package-tests branch from 381c966 to 49270be Compare April 24, 2021 16:31

doc: add instructions for creating package tests

c357fbe

davidak force-pushed the package-tests branch from 49270be to c357fbe Compare April 24, 2021 16:39

davidak commented Apr 24, 2021

View reviewed changes

jtojnar approved these changes Apr 24, 2021

View reviewed changes

jtojnar reviewed Apr 24, 2021

View reviewed changes

doc/contributing/coding-conventions.chapter.md Show resolved Hide resolved

jtojnar reviewed Apr 24, 2021

View reviewed changes

This was referenced Apr 24, 2021

nixpkgs-docs: when to prefer passthru.tests over installCheckPhase #119731

Merged

seaweedfs: add version check to passthru.tests #119636

Merged

Ma27 reviewed Apr 24, 2021

View reviewed changes

lukegb merged commit 147c701 into NixOS:master May 1, 2021

davidak deleted the package-tests branch May 1, 2021 21:38

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

doc: add instructions for creating package tests #120534

doc: add instructions for creating package tests #120534

davidak commented Apr 24, 2021 •

edited

Loading

jtojnar left a comment

davidak Apr 24, 2021

jtojnar Apr 24, 2021

blaggacao Apr 24, 2021

davidak Apr 24, 2021

jtojnar left a comment

jtojnar Apr 24, 2021

davidak Apr 24, 2021

jtojnar Apr 24, 2021

Ma27 Apr 24, 2021

davidak Apr 24, 2021

davidak Apr 24, 2021

teto commented Apr 24, 2021

davidak commented Apr 24, 2021 •

edited

Loading

nixos-discourse commented Apr 24, 2021

Ma27 Apr 24, 2021

Ma27 Apr 24, 2021

davidak Apr 24, 2021

roberth commented Apr 24, 2021

abathur commented Apr 24, 2021

davidak commented Apr 24, 2021

blaggacao commented Apr 24, 2021 •

edited

Loading

lukegb commented May 1, 2021

davidak commented May 1, 2021

lukegb commented May 1, 2021

knedlsepp commented Jul 9, 2021

roberth commented Jul 9, 2021


		This is an example using the `phoronix-test-suite` package with the current best practices.

		Add the tests in `passthru.tests` to the package definition like this:

	Add the tests in `passthru.tests` to the package definition like this:
	Add the tests in [`passthru.tests`](#var-meta-tests) to the package definition like this:


		Tests are important to ensure quality and make reviews and automatic updates easy.

		Nix package tests are a lightweight alternative to [NixOS module tests](https://nixos.org/manual/nixos/stable/#sec-nixos-tests). They can be used to create simple integration tests for packages while the module tests are used to test services or programs with a graphical user interface on a NixOS VM. Unittests that are included in the source code of a package should be executed in the `checkPhase`.

	xvfb-run -s '-screen 0 800x600x24' dbus-run-session \
	--config-file=${dbus.daemon}/share/dbus-1/session.conf \
	meson test --print-errorlogs

doc: add instructions for creating package tests #120534

doc: add instructions for creating package tests #120534

Conversation

davidak commented Apr 24, 2021 • edited Loading

Motivation for this change

Things done

jtojnar left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jtojnar left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

teto commented Apr 24, 2021

davidak commented Apr 24, 2021 • edited Loading

nixos-discourse commented Apr 24, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

roberth commented Apr 24, 2021

abathur commented Apr 24, 2021

davidak commented Apr 24, 2021

blaggacao commented Apr 24, 2021 • edited Loading

lukegb commented May 1, 2021

davidak commented May 1, 2021

lukegb commented May 1, 2021

knedlsepp commented Jul 9, 2021

roberth commented Jul 9, 2021

davidak commented Apr 24, 2021 •

edited

Loading

davidak commented Apr 24, 2021 •

edited

Loading

blaggacao commented Apr 24, 2021 •

edited

Loading