Home

Simulation-based analysis made easy and reliable

Summary

MultiVeStA is an efficient statistical analysis tool which

• can be easily integrated with existing discrete-event simulators and agent-based models
• enriching them with automated statistical analysis techniques from the family of Statistical Model Checking
• allows to distribute simulations in the cores of a machine or in a network for free

Please find details in our recent paper presenting the novel MultiVeStA developments (see replicability page on the left menu

Supported Analysis

1. Transient analysis: what is the expected value of a model's property at a given point in time/as time progresses?

2. Counterfactual analysis: do the model dynamics change significantly across different parametrizations?
   

   • We perform a transient analysis of the average number of bankruptcies in each of the first 400 time points (quarters) of a large macro ABM from Caiani et al. (2016)
   • We compare the results obtained for two analysis configurations:
     • Left column: analysis using the common choice of 100 simulations for all time points
     • Right column: analysis using the correct number of simulations for each time point
   • First row: confidence bands for the properties studied for 2 different parametrizations (green/red)
     • No significant differences in the two columns
   • Second row: T-test do the results significantly differ across the red/green parametrizations?
     • The use of a correct number of simulations allows to better distinguish the two parametrizations
   • Third row: Power of the T-tests
     • Using 100 simulations leads to power below acceptable thresholds (Secchi and Seri, 2017)
     • Using the correct number of simulations gives power above the commonly accepted threshold of 0.8 (Secchi and Seri, 2017)

3. Steady-state analysis: what is the expected value of a model's property on the long run / after it stabilizes?
   

   • We perform a steady-state analysis of the agents' wealth and market price in an ABM model of market selection with 3 agents (Kets et al. 2014)
   • We compare the results obtained for two analysis configurations, using as ground truth the analytical solutions to the model by Bottazzi and Giachini (2019)
     • Left column: a manual procedure followed by Kets et al. (2014) using an arbitrarily chosen warmup estimation
     • Right column: an automated analysis by MultiVeStA applying advanced statistical procedures to estimate the warmup period and the steady-state properties
   • First row: each bar denotes the agents' wealth share at steady state for different value of the parameter given in the x-axis
     • Kets et al. (2014) erroneously conclude that there exist configurations for which all three agents survive
     • MultiVeStA correctly shows that no model configuration leads to the survival of all agents
   • Second row: we study how the steady-state market price changes through different parametrizations, comparing it with the parameter
     • Kets et al. (2014) erroneously conclude that it is never possible to distinguish the market price from the parameter
     • MultiVeStA correctly estimates that the two quantities almost never coincide.

4. Ergodicity diagnosis: does the model actually has a steady-state? / does it make sense at all to perform a steady-state analysis?
   

   • Not all models and properties admit a steady-state analysis, simply because not all have a steady-state value.
   • MultiVeStA offers a methodology based on two complementary automated steady-state analysis techniques (autoRD and autoBM) to perform ergodicity diagnosis, informing the modeller on whether it makes sense at all to perform a steady-state analysis

History

Development

• MultiVeStA has been created in 2012, focusing on transient analysis find more info here
• It has been recently deeply refactored in 2021, adding a number of analysis techniques (as discussed in the draft available in the replicability page of our right-menu
• It supports any simulator written
  • in Java, C++, Python, R,
  • or developed within several frameworks, including NetLogo JMAB, DESMO-J, PMaude, DEUS, QFLAN, RisQFLan, ERODE, BioPEPA, ONE

Examples of Considered models/simulators

MultiVeStA has been successfully applied in a wide range of domains including, e.g.,

• economical agent-based models (see the replicability page in the right-menu),
• engineering of highly-configurable systems,
• public transportation systems in smart cities example 1, example 2, example 3
• biological systems
• robotic scenarios with planning capabilities example 1, example 2
• crowd steering scenarios
• middleware for reliable distributed systems

Contacts

Please contact us for information on

• how to get MultiVeStA and
• how to integrate it with your simulator
  • we added a page on how to integrated python simulators.

We will reply shortly

---

Replicability

Please use the menu on the right to access the replicability material for recent papers