Development Testing and Debugging

Testing

Listing Tests

You can easily list all available tests

$ tfb --list-tests
activeweb
activeweb-raw
aspnet
aspnet-jsonnet
aspnet-mongodb-raw
aspnet-mono
aspnet-mono-jsonnet
aspnet-mono-mongodb-raw
<snip>

Running Tests

There are a number of options that can be specified:

# Run a verification for test beego
$ tfb --test beego --mode verify

# Run a test in debug mode, which skips verification and leaves the server up for testing endpoints
$ tfb --test beego --mode debug

# Run the default benchmark for the beego test
$ tfb --test beego

# Specify which test types are run during benchmark
$ tfb --test beego --type json
$ tfb --test beego --type db
$ tfb --test beego --type fortune

# Additional options to allow for greater control while developing. `--extra-docker-runtime-args` will pass additional arguments to the test container
$ tfb --test spring --test-container-memory 1G --extra-docker-runtime-args cpu_period:10000 cpu_quota:50000

Note: The results directory must be removed after each test has been run in order to run the test again.

GitHub Actions

Use GitHub Actions on your project fork. This is as simple as pushing a commit. New frameworks and languages will be picked up automatically. When your development is done and your changes pass the GitHub Actions verification, submit a pull request with confidence that it can be merged quickly. Read more about TFB and GitHub Actions.

Important Note About GitHub Actions

If you make changes to configuration files, or files outside the frameworks directory, GitHub Actions will run tests on all existing frameworks. For this reason, it may appear that your tests have failed. Be sure to check your GitHub Actions workflow by clicking on the checkmark or red 'X' to dig into your specific test/language.

Debugging

Finding output logs

Logs file locations use the format results/latest/logs/wt/err.txt. The general structure is results/<run name>/<timestamp>/logs/<test name>/<file>. You can use the --name flag to change the <run name>. If you re-run the same test multiple times, you will get a different folder for each <timestamp>, although the latest folder will be kept up to date. The <test name> is simply the name of the test type you ran, and <file> is either out.txt or err.txt (these are the logout and logerr arguments passed into each setup.py file.

Note: If you're looking for logs from our official benchmark rounds, they can be found here