The SBML Test Suite is a conformance testing system for SBML, a file and data format used to store and exchange models in computational systems biology.
Main Authors: Sarah Keating, Michael Hucka, Lucian P. Smith, Frank T. Bergmann, Bruce Shapiro, Thomas W. Evans, Colin S. Gillespie, Darren J. Wilkinson, Brett Olivier, Andrew Finney.
Home page: https://sbml.org/software/sbml-test-suite/
Repository: https://github.com/sbmlteam/sbml-test-suite
Developers' discussion group: https://groups.google.com/forum/#!forum/sbml-discuss
Pivotal tracker: https://www.pivotaltracker.com/n/projects/68714
License: For full license information, please refer to the file LICENSE.html for details. Briefly, the test case distributions of the SBML Test Suite are distributed under the terms of the LGPL v2.1; the overall SBML Test Suite (including the software components) are distributed under the LGPL but include components from other sources licensed under other open-source terms. (However, none of the terms are more restrictive than the LGPL.)
Computational modeling has become a crucial aspect of biological research, and SBML (the Systems Biology Markup Language) has become the de facto standard open format for exchanging models between software tools in systems biology. At last count, over 280 software tools have provided support for SBML. However, full support for SBML (especially for numerical simulations of models described by SBML) is a complex undertaking.
The SBML Test Suite can be used by developers to test their understanding of SBML and their software's implementation of SBML support; likewise, the Test Suite can be used by modelers and other users to test the correctness and completeness of SBML support in a software system. The SBML Test Suite consists of (1) a collection of SBML models, each with associated simulated results where appropriate; (2) a testing framework for running software tools through the suite; and (3) basic documentation on the test cases and the use of the suite.
The SBML Test Suite test cases are divided into 3 sets and distributed separately. Each tests a different aspect of SBML understanding and compliance.
-
Semantic: The semantic test suite contains valid SBML models with known, deterministic simulation results. These models can be used to test the ability of a software system to understand the meaning of different SBML constructs and properly simulate the models in a deterministic fashion. An example of a deterministic simulator is a system using a numerical differential-algebraic solver that supports discontinuous events. Each test consists of a directory containing the model (or models, if that model can be translated to other SBML Levels + Version combinations without semantic loss), together with instructions on how to simulate that model, and the expected results. For more information about the semantic test cases, please look in the file named README.md in the cases/semantic/ subdirectory.
-
Stochastic: The stochastic test suite contains valid SBML models with known, stochastic simulation results. These tests exercise fewer features of SBML, and are instead intended to test the accuracy of systems that employ discrete stochastic simulation algorithms. Each test consists of a directory containing the model (or models, if that model can be translated to other SBML Levels + Version combination without semantic loss), together with instructions on how to simulate that model, and the expected results. Because the expected results are stochastic, each test case is designed to be performed multiple times, with summary statistics collected for each, which are then compared to the expected summary statistics using a formula derived from the number of times the test was repeated. For more information about the stochastic test cases, please look in the file named README.md in the cases/stochastic/ subdirectory.
-
Syntactic: The syntactic test suite consists of valid and invalid SBML models. Each test is designed to check a particular SBML validation rule. These rules are defined in the SBML specification documents. Each case model in this part of the Test Suite is expected to be recognized by a software system as being either valid or invalid – nothing more. The validity is indicated in the file name of a model. (Example:
01002-fail-01-01-sev2-l2v1.txtshould fail.) Details about the error and the error message produced by libSBML are included with each test case description, as are incidental warnings that libSBML may also produce for a given model/test case. For more information about the stochastic test cases, please look in the file named README.md in the cases/syntactic/ subdirectory.
The SBML Test Runner can be used to run an application against the collection of semantic test cases in the Test Suite, and report successes and failures. If you have downloaded a release of the standalone version of the SBML Test Runner, your distribution will consist of a few files and a program named SBML Test Runner. It comes complete with the test cases already included, and it can automatically check for updates to the test cases. The image below shows an example run from the SBML Test Runner:
More information can be found in a separate README file for the SBML Test Runner
The SBML Test Suite is under active development by a distributed team. If you have any questions, please feel free to post or email on the (https://groups.google.com/forum/#!forum/sbml-interoperability) forum, or contact the SBML Team directly.
Contributions to the SBML Test Suite are very much welcome in all areas. Please feel free to contact the developers. A quick way to find out what is currently on people's plates and our near-term plans is to look at the Pivotal Tracker for this project.
Notes about build procedures and other information are kept in the GitHub repository wiki for this project.
If you like this software, don't forget to give this repo a star on GitHub to show your support!
Funding for the continued development of the core of the SBML Test Suite currently comes from the National Institute of General Medical Sciences via grant GM070923 (Principal Investigator: Michael Hucka). Past development of different parts of the Test Suite have been funded by NIH grants GM077671 and GM070923 as well as BBSRC grants BEP 17042, BBS/B/16550, and BBC0082001 (Principal Investigator: Darren J. Wilkinson). Additional support for this and other projects of the SBML Team have been provided by the following organizations since the year 2000: the International Joint Research Program of NEDO (Japan); the JST ERATO-SORST Program (Japan); the Japanese Ministry of Agriculture; the Japanese Ministry of Education, Culture, Sports, Science and Technology; the BBSRC e-Science Initiative (UK); the DARPA IPTO Bio-Computation Program (USA); the Army Research Office's Institute for Collaborative Biotechnologies (USA); the Air Force Office of Scientific Research (USA); the California Institute of Technology (USA); the University of Hertfordshire (UK); the Molecular Sciences Institute (USA); the Systems Biology Institute (Japan); and the University of Keio (Japan).
The SBML Test Suite has been in development for many years, and many people have contributed in various ways large and small. The following are the primary authors of different parts:
-
Semantic test cases:
Sarah M. Keatinga,b, Lucian P. Smithb,c, Bruce Shapiro b, Michael Huckab, Frank T. Bergmannd, Brett Olivierg, Andrew Finneyb
-
Stochastic test cases:
Thomas W. Evanse, Colin S. Gillespief, Darren J. Wilkinsonf, Lucian P. Smithb,c
-
Syntactic test cases:
Sarah M. Keatinga,b, Lucian P. Smithb,c
-
SBML Test Runners:
Frank T. Bergmannd, Michael Huckab, Kimberley Begley
-
Online SBML Test Suite Database:
Frank T. Bergmannd
Additional contributions are gratefully acknowledged from Ralph Gaugesh, Chris Myersi, Akira Funahashij, Andreas Drägerk, Roland Kellerk, Fedor Kolpakovl, Stanley Guc, and others in the SBML community.
Institutions:
a EMBL-EBI, Hinxton, Cambridgeshire, UK
b California Institute of Technology, Pasadena, CA, US
c University of Washington, Seattle, WA, US
d University of Heidelberg, Heidelberg, DE
e University of Liverpool, Liverpool, UK
f Newcastle University, Newcastle, UK
g Vrije Universiteit Amsterdam
h Heidelberg Institute for Theoretical Studies, Heidelberg, DE
i University of Utah, Salt Lake City, UT, US
j Systems Biology Institute, Tokyo, JP
k University of Tuebingen, Tübingen, DE
l Institute of Systems Biology, Novosibirsk, RU
For full license information, please refer to the file LICENSE.txt for details. Briefly, the test case distributions of the SBML Test Suite are distributed under the terms of the LGPL v2.1; the overall SBML Test Suite (including the software components) are distributed under the LGPL but include components from other sources licensed under other open-source terms. (However, none of the terms are more restrictive than the LGPL.)
