What capabilities do we want to see in this package?

[beckynevin]

DeepDiagnostics is a package that will assist people in the group (and beyond!) with diagnosing the quality of the inference they are running. We're building it for both MCMC-type inference analyses and sbi analyses. The idea being, beginning with a already-trained posterior, you can then diagnose how it did.

Things that I'm already planning to implement:

• SBC (rank plots, rank statistics, coverage plots)
• better corner plots
• posterior predictive plots
• 1:1 parameter plots

Any other ideas for tools that you'd like to see or things that you've previously used in your inference analyses?

@gkhullar @Jasonpoh @jsv1206 @AleksCipri @bnord @kiryteo @samueldmcdermott @evavagiakis @apadhiUCLS @amanda-pagul @yyzhang

[samueldmcdermott]

Sorry if I missed discussions about this, but doesn't arviz (here and here) do these things?

[beckynevin]

I've used arviz in the past, it's what the numpyro tutorials use to display posteriors. I like but don't love their pairplot style: https://python.arviz.org/en/stable/examples/plot_pair_kde_hdi.html

[Jasonpoh]

Oh I've not used arviz. Are there any examples of them plotting multiple sets of data on a scatter plot?

[beckynevin]

Follow up question - do we want the functionality to plot multiple contours from different posteriors?, ie Gourav's paper or in Jason's paper; looks like both of you @Jasonpoh and @gkhullar are using different plotting tools?

[Jasonpoh]

I'm a bit out of the loop on the corner plots - are you planning to write your own? I think Gourav uses pyGTC while I'm using GetDist.

[bnord]

I think I prefer getdist for the cornerplots.

[beckynevin]

Another question I had is about the format people are using to save the training set for SBI and the posterior itself. I'm currently using .pkl format for both because that's one of the options I saw on your repo @Jasonpoh . I talked to @jsv1206 and it sounds like she's using .h5 and .npy to save the training set? Do we want to standardize this?

[Jasonpoh]

I think h5 is probably more standardized for large data sets?

[bnord]

I'd vote for pkl for the posterior model and h5 for the data.

[bnord]

@beckynevin Is it possible to implement GetDist for corner plots in v0?

[beckynevin]

Already did it