reproducibility checks

[humnaawan]

We'll use this thread to compile and discuss details re reproducibility checks, making sure reproducibility is handled as the project matures.

[bnord]

Could you remind us what the current plan is for testing reproducibility/stability/consistency?

in particular, how will we know the amount variation with respect to stochasticity in the training/validation/test sets and the initial weight randomization of any neural networks that are used (e.g., the MAF in NPE)?

[bnord]

With all of our other projects on SBI, we've made many data sets, where we've specified random seeds. We also tracked the random seed for every computational/analysis experiment. This kind of protocol has helped us to catch bugs and identify missing things in our analysis.