Add integration with GLib Tests suite

[ignatenkobrain]

GLib Tests have some description for tests, i.e. we have 1 .c file with many functions and descriptions added to test suite by:
g_test_add_func ("/misc/assertions", test_assertions);

So it would be great to have ability to read it instead of generating meson descriptions and counting like

[1/1] Running test suite.
1/1 /Person                                 OK   ( 0.00 s)

[jpakkane]

Gtest has the same thing. The problem here is that this is not easy to determine. You can't know it beforehand because the tests are specified in C so you'd need a full C parser to make it reliable. On the other hand you want to print the name of the test before the test has managed to print anything.

A final problem is that if you want to do this after the fact you need to parse the output of the binary. Different test suites have different outputs and, presumably, they can change from version to version. Writing parsers for these takes a lot of effort for this. Maybe if they could be told to output some standardised format like json or maybe even xml. I'm not saying this can't be done but it seems like a lot of effort for little gain.

Note that all stdout and stderr go to the log files so in case of errors you can easily look up the details there.

[ignatenkobrain]

GLib tests has https://developer.gnome.org/glib/stable/gtester.html

gtester is a utility to run unit tests that have been written using the GLib test framework.

When called with the -o option, gtester writes an XML report of the test results, which can be converted into HTML using the gtester-report utility. 

[ignatenkobrain]

<?xml version="1.0"?>
<gtester>
  <testbinary path="tests/test-gn-person">
    <binary file="tests/test-gn-person"/>
    <random-seed>R02S1af8c84ddbf4356fd67227e04bc7b5a1</random-seed>
    <testcase path="/Person/Print Birthday">
      <duration>0.000169</duration>
      <status exit-status="0" n-forks="0" result="success"/>
    </testcase>
    <duration>0.003521</duration>
  </testbinary>
  <testbinary path="tests/test-gn-pythagorean-square">
    <binary file="tests/test-gn-pythagorean-square"/>
    <random-seed>R02S102d405eb05539675f20d11aed41fa1a</random-seed>
    <testcase path="/Pythagorean Square/Working Numbers">
      <duration>0.000363</duration>
      <status exit-status="0" n-forks="0" result="success"/>
    </testcase>
    <testcase path="/Pythagorean Square/Calculate Square">
      <duration>0.000266</duration>
      <status exit-status="0" n-forks="0" result="success"/>
    </testcase>
    <duration>0.004688</duration>
  </testbinary>
</gtester>

[pwithnall]

The recommended way to do this in GLib (which is also what installed-tests use) is TAP.

[ptomato]

So, would it be desirable to have a TAP test harness built in to Meson? Should not be too hard since TAP is designed to be easy to parse. What sort of parameter to test() would indicate that the test would be expected to produce TAP output?

As an aside, a proper TAP runner will exit with 0 even if some tests failed, since the harness is expected to recognize the not ok messages printed to stdout. Exiting with a nonzero code is reserved for when there was an error other than tests failing. This means that well-behaved TAP tests can't be run in Meson right now without a wrapper script because failing tests will register as passing.

[jpakkane]

Meson exports all information needed to run tests as JSON. Therefore writing test runner applications is simple, just slurp that data and run the individual tests in whatever manner you like.

I'm hesitant to add parsers for custom test frameworks because there are a ton of those and it is a never ending swamp because of things like this:

a proper TAP runner will exit with 0 even if some tests failed, since the harness is expected to recognize the not ok messages printed to stdout.

[jpakkane]

Alternatively you can specify a wrapper binary to use when running tests with add_test_setup()`.

[pwithnall]

TAP is pretty widely used (and by projects which aren’t GLib or using GTest); it would be a pain if every project had to implement its own TAP-based test runner.

[nirbheek]

In that case, I think it would make sense to add support for specific protocols/formats in test() that are widely-used like TAP. Something like:

test('name', exe,
  ...,
  protocol: 'tap')

+ support in the test runner for modular stdout/stderr/exit status parsing to evaluate test status. The existing test runner wrapper is modular enough to allow this.

[ptomato]

In case it's useful for implementing this feature in Meson, here's a TAP wrapper that I adapted from https://github.com/python-tap/tappy and made standalone: https://github.com/endlessm/webhelper/blob/3e8de45634c084cfaf4f038ffa30013731175800/test/tap.py

[TingPing]

It looks like if we want really nice Meson integration it has to be GLib specific, just implementing TAP only handles output but for mesontest we need to pass other GLib specific flags to list available tests and run only specific ones.

[TingPing]

How reasonable would it be to have something like gnome.test()? It could expose the test suites and set common vars like G_TEST_BUILDDIR/G_TEST_SRCDIR, etc.

Could also implement the installed-tests mentioned above.

[nirbheek]

Specifically, G_TEST_BUILDDIR/G_TEST_SRCDIR is set by glib.mk, so it works out-of-the-box with Autotools. This is a significant stumbling block for people trying to port to Meson.

We should begin by adding support for basic things like this at least, and think of an API that can expose this or a way we can auto-detect this.

[ebassi]

Additionally, glib-tap.mk and glib.mk automatically handle installed tests, which are composed of two elements:

• a binary, installed under $libexecdir/installed-tests/$project_name
• a key/value file, installed under $datadir/installed-tests/$project_name, which specifies the environment of the test, and the binary to exec
• a configuration option to enable/disable the installation of tests

This allows projects like Continuous to build a VM and run all the installed tests inside it, as part of our Continuous Integration/Delivery pipeline.

This would call for a special gnome.test() method, or a profile: gnome profile for the test() method.

[pwithnall]

This would call for a special gnome.test() method, or a profile: gnome profile for the test() method.

Nitpick: installed-tests are theoretically not GNOME-specific (even if in practice they are only used by GNOME projects, or projects heavily using GNOME technology at the moment), so it would be good to not tie them to the name gnome.

[TingPing]

I think in the short term it is probably fine to be under gnome because the gnome.test() method will have other glib specific features. If they become more generically useful it shouldn't be overly hard to refactor it into the main test() function.

Also how is test data handled? Just thrown into the same dir? Or is test-only data not used?

[TingPing]

Hmm, gnome-desktop-testing-runner does have some overlap with mesontest so maybe it could become an installed-test runner...

[pwithnall]

installed-tests documentation is here: https://wiki.gnome.org/Initiatives/GnomeGoals/InstalledTests

[ebassi]

@pwithnall I used "gnome", here, because we added various GNOME-only semantics, like the environment variables and the test metadata. Well, technically is GLib, not GNOME, but same difference.

@TingPing installed data is generally installed inside alongside the binary, as per glibtests.m4:

    AC_SUBST(installed_test_metadir, [${datadir}/installed-tests/]AC_PACKAGE_NAME)
    AC_SUBST(installed_testdir, [${libexecdir}/installed-tests/]AC_PACKAGE_NAME)

[TingPing]

So the usage of installed-tests and normal tests is basically entirely different, so perhaps it should be a separate function entirely.

gnome.installed_test('Tests foo', test_exe,
  subdir: 'foo-1.0',
  data: ['file.txt'],
  exclusive: false,
)

[TingPing]

So initial work is in this branch: https://github.com/mesonbuild/meson/tree/tingping/test-profile

• Automatically set G_TEST_SRCDIR and G_TEST_BUILDDIR
• Add mesontest support to split tests up into test paths (This seems like a decent amount of refactoring and changing core behavior of mesontest)
• Expose same features for benchmark()
• Expose installed-tests feature in some form (Need to refactor installing executables)

[smcv]

Expose installed-tests feature in some form

The only thing needed is to install the tests and their metadata, because the whole point of installed-tests is that you run them other than at build time.

glib-tap.mk separates the tests into three categories, which would probably make sense for Meson too:

• uninstalled-only tests: can only be run at build time. Meson should run them in meson test, but not install them. (Traditional non-GLib unit tests are like this too.)
• installed-only tests: can only be run on a properly installed system, either because they are integration tests or because they need capabilities that aren't normally available at build time (for example root privileges or a full GUI). Meson should install them and their metadata, but not run them at build time.
• tests that work both ways: Meson should run them, and also install them and their metadata.

[TheAifam5]

What about code coverage report? Looks like does not work too.