A study that involves developing and using a mathematical model that imitates a real-world system's behavior, which often entails problem understanding, data collection, model development, verification, validation, design of experiments, data analysis, and implementation of results.
The standard applies to research studies that use simulation to understand, assess or improve a system or process and its behavior. Use this standard for in silico simulations, i.e., studies representing everything using computational models. For in virtuo simulations, i.e., human participants manipulating simulation models, use the Experiments (with human participants) Standard. For simulations used to assess a new or improved technological artifact, also consider the Engineering Research standard.
- justifies that simulation is a suitable method for investigating the problem (or research question, etc.)
- describes the simulation model (conceptual, implementation, or hybrid abstraction levels), including input parameters and response variables
- describes the underlying simulation approach1
- describes simulation packages or tools used to develop and run the simulation model, including version numbers, and computational environments
- describes the data used for model calibration, the calibration procedures, and contextual information
- describes how the simulation model was verified and validated at different abstraction levels2
- describes the study protocol, including independent variables, scenarios, number of runs per scenario (in case of using stochastic simulation), and steady-state or terminating conditions
- analyzes validity threats considering the supporting data and the simulation model3
- clearly explicates the assumptions of the simulation model
- provides supplementary materials including the raw data (for real data) or generation mechanism (for synthetic data) used for model calibration, all simulation models and source code, analysis scripts
- characterizes reference behaviors for the definition of simulation scenarios with representative and known values or probability distributions for input parameters4
- separates conceptual and implementation levels of the simulation model
- reports sensitivity analysis for input parameters or factors
- clearly distinguishes evidence-based results from interpretations and speculation5
- describes how stakeholders were involved in developing and validating the simulation model6
- provides a modular view of the simulation model, allowing reuse in different contexts7
Conclusion validity, construct validity, internal validity (if examining causal relationships), external validity, and reproducibility.
- Overfitting8 the simulation model to reproduce a reference behavior.
- Use of non-standard experimental designs9 without justification.
- Using a single run instead of multiple runs to experiment with stochastic models.
- If insufficient data is available (or too costly to collect) to calibrate the model, assumptions can be used to implement parts of the model. These assumptions, however, must be explained and justified.
- When the translation from a conceptual to implementation model is straightforward, authors may present them together.
- If the simulation approach used is very common in software engineering (e.g. discrete-event simulation, system dynamics), it is sufficient to indicate which approach is used, citing appropriate references, rather than explaining in full how the approach works.
- The mere presence of assumptions in the model is not a valid basis for criticism as long as the assumptions are documented and justified, and their implications for the validity of the simulation are sufficiently addressed. All models make assumptions.
- Claiming that the model is too abstact without explaining why the level of abstraction is inadequate for the purposes of the study.
- Claiming that the study is invalid because it uses generated data, secondary data or approximations based on expert opinion, when no appropriate primary data is available.
Nauman Bin Ali, Kai Petersen. A consolidated process for software process simulation: State of the art and industry experience. In: 38th Euromicro Conference on Software Engineering and Advanced Applications, 2012, IEEE, pp 327–336.
Nauman Bin Ali, Kai Petersen, Claes Wohlin. A systematic literature review on the industrial use of software process simulation. Journal of Systems and Software, 97, 2014, 65–85.
Dietmar Pfahl. Process Simulation: A Tool for Software Project Managers? In: Günther Ruhe, Claes Wohlin (Eds.) Software Project Management in a Changing World. Springer-Verlag Berlin Heidelberg, 2014, 425-446.
Ivo Babuska, and J. Tinsley Oden. Verification and validation in computational engineering and science: basic concepts. Computer methods in applied mechanics and engineering, 193, 36, 2004, 4057–4066.
Breno Bernard Nicolau de França, Nauman Bin Ali. The Role of Simulation-Based Studies in Software Engineering Research. In: Felderer M., Travassos G. (eds) Contemporary Empirical Methods in Software Engineering. Springer, Cham. 2020. https://doi.org/10.1007/978-3-030-32489-6\_10.
Breno Bernard Nicolau de França, Guilherme Horta Travassos. Experimentation with dynamic simulation models in software engineering: planning and reporting guidelines. Empirical Software Engineering, 21, 3, 2016, 1302–1345.
Breno Bernard Nicolau de França, Guilherme Horta Travassos. (2015). Simulation Based Studies in Software Engineering: A Matter of Validity. CLEI Electronic Journal, 18(1), 5.
Houston DX, Ferreira S, Collofello JS, Montgomery DC, Mackulak GT, Shunk DL. Behavioral characterization: finding and using the influential factors in software process simulation models. Journal of Systems and Software, 59, 3, 2001, 259– 270, DOI https://doi.org/10.1016/S0164-1212(01)00067-X.
Kleijnen JPC, Sanchez SM, Lucas TW, Cioppa TM. State-of-the-art review: A user's guide to the brave new world of designing simulation experiments. INFORMS Journal on Computing, 17, 3, 2005, 263–289. DOI 10.1287/ijoc.1050.0136.
Law AM. Simulation modeling and analysis. 5th ed., 2015, McGraw-Hill, New York.
Madachy RJ. Software Process Dynamics, 2008, Wiley-IEEE Press.
Ali NB, Petersen K, de França BBN (2015) Evaluation of simulation-assisted value stream mapping for software product development: Two industrial cases. Information & Software Technology 68:45–61 [an example of a simulation-based study in industrial settings].
Concas, Giulio, Maria Ilaria Lunesu, Michele Marchesi, and Hongyu Zhang. "Simulation of software maintenance process, with and without a work‐in‐process limit." Journal of software: Evolution and Process 25, no. 12 (2013): 1225-1248. [an example of model description and discussion of threats to validity].
Garousi V, Khosrovian K, Pfahl D (2009) A customizable pattern-based software process simulation model: design, calibration, and application. Software Process: Improvement and Practice 14(3):165–180, DOI 10.1002/spip.411. [an example of a complete report of a simulation-based study].
Smith, Neil, Andrea Capiluppi, and Juan F. Ramil. "A study of open source software evolution data using qualitative simulation." Software Process: Improvement and Practice 10, no. 3 (2005): 287-300. [an example of a simulation study using a unusual simulation approach: qualitative simulation].
1e.g. discrete-event simulation, system dynamics, agent-based simulation
2Some verification and validation procedures may be applied to the model at the conceptual level (e.g., validating variables and relationships) down to an implementation level (e.g., using tests, reproducing reference behaviors, or performing simulated experiments).
3Simulation studies are prone to several validity threats, including non-representative simulation scenarios, insufficient verification and validation, using different datasets (contexts) for model calibration and experimentation, and others (de França and Travassos, 2015).
4Reference behaviors represent a real-world model (often based on actual measurement of a system or process), which is characterized by data distribution or series of model variables. Usually, these models are used for validating simulation outcomes. For instance, an effort and schedule baseline for software project simulation.
5Simply separating results and discussion into different sections is typically sufficient. No speculation in the results section.
6Apart from the developers of the simulation model, stakeholders could be data providers (for model calibration), domain experts (who may have hypotheses about causal relationships between model variables) and users of the simulation model. All of these stakeholders could, for example, be involved in checking the plausibility of the model output (face validity check).
7Understanding simulation models as software, they may become too large and difficult to understand in a single view. So, the idea is to have a composite model, in which each module concerns a particular set of variables. The following book presents an entire model on software projects in a modular perspective: Abdel-Hamid, T. and Madnick, S.E., 1991. Software project dynamics: an integrated approach. Prentice-Hall, Inc.
8For instance, implementing a specific model capable of only producing desired outcomes.
9Houston et al. (2001) discusses some usual experimental design for software process simulation, such as (fractional) factorial designs. For a more general view on experimental designs for simulation, we suggest Kleijnen et al. (2005).