The directory contains test list files which can be sent to NayDuck to
request a run of the tests. Most notably, nightly.txt file contains
all the tests that NayDuck runs once a day on the head of the master
branch of the repository.
Nightly build results are available on NayDuck.
Each list file is read line-by-line. Empty lines and lines starting with a hash are ignored. The rest either specifies a test to run or a list file to include. The general syntax of a line defining a test is:
<category> [--skip-build] [--timeout=<timeout>] [--release] [--remote] <args>... [--features <features>]
<category> specifies the category of the test and can be pytest or
expensive. The meaning of <args> depends on the test category.
The pytest tests are run by executing a file within pytest/tests
directory. The file is executed via python interpreter (and
confusingly not via pytest command) so it must actually run the
tests rather than only defining them as test functions.
In the test specification path to the file needs to be given
(excluding the pytest/tests prefix) and anything that follows is
passed as arguments to the script. For example:
pytest sanity/lightclnt.py
pytest sanity/state_sync_routed.py manytx 115
Note: NayDuck also handles mocknet test category. It is now
deprecated and is treated like pytest with --skip-build flag
implicitly set.
The expensive tests run a test binary and execute specific test in
it. (Test binaries are those built via cargo test --no-run). While
this can be used to run any Rust test, the intention is to run
expensive tests only. Those are the tests which are ignored unless
expensive_tests crate feature is enabled. Such tests should be
marked with a cfg_attr macro, e.g.:
#[test]
#[cfg_attr(not(feature = "expensive_tests"), ignore)]
fn test_gc_boundaries_large() {
/* ... */
}
The arguments of an expensive test specify package in which the test is defined, test binary name and the full path to the test function. For example:
expensive nearcore test_tps_regression test::test_highload
(Currently the package name is ignored but it may change in the future
so make sure it’s set correctly). The path to the test function must
match exactly an the test binary is called with --exact argument.
As mentioned, there are additional arguments that can go between the
test category and the test specification arguments. Those are
--skip-build, --timeout, --release and --remote.
--skip-build causes build step to be skipped for the test. This
means that the test doesn’t have access to build artefacts (located in
target/debug or target/release) but also doesn’t need to wait for
the build to finish and thus can start faster.
--timeout=<timeout> specifies the time after which the test will be
stopped and considered failed. <timeout> is an integer with an
optional s, m or h suffix. If no suffix is given, s is
assumed. The default timeout is three minutes. For example, the
following increases timeout for a test to four minutes:
pytest --timeout=4m sanity/validator_switch.py
--release makes the build use a release profile rather than a dev
profile. In other words, all cargo invocations are passed
additional --release argument. This is supported but currently not
used by any of the nightly tests.
--remote configures pytest tests to use Near test nodes started on
spot GCP machines rather than executing a small cluster on host where
the test is running. No nightly test uses this feature and to be
honest I can’t vouch whether it even works.
Lastly, at the end of the test specification line additional features
can be given in the form of --features <features> arguments.
Similarly to --release, this results in given features being enabled
in builds. Note that the test_features Cargo feature is always
enabled so there's no need to specify it explicitly.
Note that with --skip-build switch the --release and --features
flags are essentially ignored since they only affect the build and are
not passed to the test.
To help organise tests, the file format also supports ./<path>
syntax for including contents of other files in the list. The
includes are handled recursively though at the moment there’s a limit
of three levels before the parser starts ignoring the includes.
For example, nightly.txt file may just look as follows:
./sandbox.txt
./pytest.txt
./expensive.txt
with individual tests listed in each of the referenced files. This makes the files more tidy.
Note that any includes accessible from nightly.txt file must live
within the nightly directory and use .txt file extension. Using
arbitrary paths and extensions will work locally but it will break
NayDuck’s nightly runs.
Every 24 hours NayDuck checks if master branch has changed and if it
has schedules a new run including all tests listed in the
nightly.txt file. It’s also possible to request a run manually in
which case arbitrary set of tests can be run on an arbitrary commit
(so long as it exists in the near/nearcore repository).
This can be done with nayduck.py script which takes the list file as
an argument. For example, to run spec tests one might invoke:
./scripts/nayduck.py -t nightly/pytest-spec.txt
With no other arguments the tests will be run against checked out
commit in the current branch. It is possible to specify which branch
and commit to run against with --branch and --sha arguments. For
full usage refer to ./scripts/nayduck.py --help output.
NayDuck cannot run tests against local working directory or even commits in a private fork of the nearcore repository. To schedule a NayDuck run, the commit must first be pushed to nearcore. The commit does not need to be on master branch; testing a feature branch is supported.
On success the script outputs link to the page which can be used to see status of the run. Depending on which tests were scheduled the run can take over an hour to finish.
New tests can be created either as a Rust test or a pytest.
If a Rust test is long-running (or otherwise requires a lot of
resources) and intended to be run as a nightly test on NayDuck it
should be marked with a #[cfg(feature = "expensive_tests")]
directive (either on the test function or module containing it). With
that, the tests will not be part of a cargo test run performed on
every commit, but NayDuck will be able to execute them. Apart from
that, expensive Rust tests work exactly the same as any other Rust
tests.
pytests are defined as scripts in the pytest/tests directory. As
previously mentioned, even though the directory is called pytest, when
run on NayDuck the scripts are run directly via python. This means
that they need to execute the tests when run as the main module rather
than just defining the tests function. To make that happen it’s best
to define test_<foo> functions with test bodies and then execute all
those functions in a code fragment guarded by if __name__ == '__main__' condition.
Unless tests are included (potentially transitively) in nightly.txt
file, NayDuck won’t run them. As part of pull request checks,
verification is performed to make sure that no test is forgotten and
all new tests are included in the nightly list. That’s done by
scripts/check_nightly.txt and scripts/check_pytest.txt scripts.
They list all the expensive and pytest tests defined in the repository
and then check whether they are all mentioned in the nightly list.
The scripts recognise commented out tests so if a test is broken it can be removed from the list by commenting it out. However, such a test must be proceeded by a TODO comment mentioning an issue which tracks the breakage. For example:
# TODO(#2411): Enable them when we fix the issue with proxy shutdown
#pytest --timeout=900 sanity/sync_ban.py true
#pytest --timeout=900 sanity/sync_ban.py false
The include directive can be commented out like that as well though crucially there must be no space between the hash sign and dot. For example:
# TODO(#2411): Working on a fix.
#./bridge.txt