[PRE REVIEW]: calzone: A Python package for measuring calibration of probabilistic models for classification

[editorialbot]

Submitting author: @qiancao (Qian Cao)
Repository: https://github.com/DIDSR/calzone/
Branch with paper.md (empty if default branch): main
Version: 0.1.0
Editor: Pending
Reviewers: Pending
Managing EiC: Chris Vernon

Status

Status badge code:

HTML: <a href="https://joss.theoj.org/papers/fc9232bcdbc721957ec8564fdddb8671"><img src="https://joss.theoj.org/papers/fc9232bcdbc721957ec8564fdddb8671/status.svg"></a>
Markdown: [![status](https://joss.theoj.org/papers/fc9232bcdbc721957ec8564fdddb8671/status.svg)](https://joss.theoj.org/papers/fc9232bcdbc721957ec8564fdddb8671)

Author instructions

Thanks for submitting your paper to JOSS @qiancao. Currently, there isn't a JOSS editor assigned to your paper.

@qiancao if you have any suggestions for potential reviewers then please mention them here in this thread (without tagging them with an @). You can search the list of people that have already agreed to review and may be suitable for this submission.

Editor instructions

The JOSS submission bot @editorialbot is here to help you find and assign reviewers and start the main review. To find out what @editorialbot can do for you type:

@editorialbot commands

[editorialbot]

Hello human, I'm @editorialbot, a robot that can help you with some common editorial tasks.

For a list of things I can do to help you, just type:

@editorialbot commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@editorialbot generate pdf

[editorialbot]

Software report:

github.com/AlDanial/cloc v 1.90  T=0.14 s (812.8 files/s, 386204.5 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
CSV                              3              0              0          20003
TeX                             28           1561           2628           8983
HTML                            23            857             69           7308
SVG                              3              0              0           2689
Python                          10            354            638           1246
CSS                              6            223             57           1015
JavaScript                      12            138            228            912
Jupyter Notebook                14              0           3530            504
Markdown                         2             64              0            261
YAML                             3             12             20             66
make                             2             22             13             50
DOS Batch                        2             17              2             47
reStructuredText                 4             32             63             39
TOML                             1              5              0             38
-------------------------------------------------------------------------------
SUM:                           113           3285           7248          43161
-------------------------------------------------------------------------------

Commit count by author:

    90	jasonfan1997
    30	Qian Cao
    12	qiancao
     4	Kwok Lung(Jason) Fan
     1	Jason Fan

[editorialbot]

Checking the BibTeX entries failed with the following error:

Failed to parse BibTeX entry: cite-key missing

[editorialbot]

Paper file info:

📄 Wordcount for paper.md is 2737

✅ The paper includes a Statement of need section

[editorialbot]

License info:

✅ License found: MIT License (Valid open source OSI approved license)

[editorialbot]

👉📄 Download article proof 📄 View article proof on GitHub 📄 👈

[editorialbot]

Five most similar historical JOSS papers:

dsBinVal: Conducting distributed ROC analysis using DataSHIELD
Submitting author: @schalkdaniel
Handling editor: @csoneson (Active)
Reviewers: @patRyserWelch8, @brunomontezano, @AnthonyOfSeattle
Similarity score: 0.6381

metrica: an R package to evaluate prediction performance of regression and classification point-forecast models
Submitting author: @adriancorrendo
Handling editor: @osorensen (Active)
Reviewers: @wiljnich, @simonpcouch
Similarity score: 0.6351

modeLLtest: An R Package for Unbiased Model Comparison using Cross Validation
Submitting author: @ShanaScogin
Handling editor: @usethedata (Retired)
Reviewers: @andrewheiss, @usethedata
Similarity score: 0.6166

hopit: An R Package for Analysis of Reporting Behavior Using Generalized Ordered Probit Models
Submitting author: @MaciejDanko
Handling editor: @alexhanna (Retired)
Reviewers: @soodoku, @mkearney
Similarity score: 0.6097

Metrics As Scores: A Tool- and Analysis Suite and Interactive Application for Exploring Context-Dependent Distributions
Submitting author: @mrshoenel
Handling editor: @mikldk (Retired)
Reviewers: @mdhaber, @kostiantyn-kucher
Similarity score: 0.6088

⚠️ Note to editors: If these papers look like they might be a good match, click through to the review issue for that paper and invite one or more of the authors before considering asking the reviewers of these papers to review again for JOSS.

[jasonfan1997]

@editorialbot check references

[editorialbot]

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

✅ OK DOIs

- 10.1002/sim.5941 is OK
- 10.1609/aaai.v29i1.9602 is OK
- 10.1175/WAF993.1 is OK
- 10.1002/sim.8281 is OK
- 10.1016/j.acra.2008.04.022 is OK
- 10.1080/03610928008827941 is OK
- 10.1002/qj.456 is OK
- 10.1002/9781118445112.stat08078 is OK
- 10.1177/0962280216661371 is OK
- 10.1016/0895-4356(92)90192-P is OK
- 10.1093/biomet/45.3-4.562 is OK
- 10.1093/biostatistics/kxq045 is OK
- 10.1080/03610928008827941 is OK
- 10.1093/jamia/ocz228 is OK

🟡 SKIP DOIs

- No DOI given, and none found for title: Metrics of Calibration for Probabilistic Predictio...
- No DOI given, and none found for title: Applied logistic regression
- No DOI given, and none found for title: MAPIE: an open-source library for distribution-fre...
- No DOI given, and none found for title: Uncertainty Toolbox: an Open-Source Library for As...
- No DOI given, and none found for title: Scikit-learn: Machine learning in Python
- No DOI given, and none found for title: On Calibration of Modern Neural Networks
- No DOI given, and none found for title: Measuring Calibration in Deep Learning.
- No DOI given, and none found for title: Posterior Re-calibration for Imbalanced Datasets

❌ MISSING DOIs

- 10.2307/2276774 may be a valid DOI for title: Probable inference, the law of succession, and sta...
- 10.2307/2529131 may be a valid DOI for title: Maximum Likelihood Estimation for the Beta-Binomia...
- 10.2307/2346866 may be a valid DOI for title: Reliability of Subjective Probability Forecasts of...
- 10.21236/ada454828 may be a valid DOI for title: Strictly proper scoring rules, prediction, and est...
- 10.1201/9781439891148-8 may be a valid DOI for title: Generalized Linear Models
- 10.32614/cran.package.resourceselection may be a valid DOI for title: ResourceSelection: Resource Selection (Probability...
- 10.1109/cvprw50498.2020.00171 may be a valid DOI for title: Multivariate Confidence Calibration for Object Det...
- 10.1002/sim.4780050506 may be a valid DOI for title: Probabilistic prediction in patient management and...
- 10.1016/j.jbi.2017.10.008 may be a valid DOI for title: Beyond discrimination: a comparison of calibration...

❌ INVALID DOIs

- None

[jasonfan1997]

@editorialbot check references

[editorialbot]

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

✅ OK DOIs

- 10.1002/sim.5941 is OK
- 10.1609/aaai.v29i1.9602 is OK
- 10.2307/2276774 is OK
- 10.1175/WAF993.1 is OK
- 10.1002/sim.8281 is OK
- 10.2307/2529131 is OK
- 10.2307/2346866 is OK
- 10.1016/j.acra.2008.04.022 is OK
- 10.1080/03610928008827941 is OK
- 10.1002/qj.456 is OK
- 10.1198/016214506000001437 is OK
- 10.1201/9781439891148-8 is OK
- 10.1002/9781118445112.stat08078 is OK
- 10.1177/0962280216661371 is OK
- 10.1016/0895-4356(92)90192-P is OK
- 10.32614/cran.package.resourceselection is OK
- 10.1109/cvprw50498.2020.00171 is OK
- 10.1093/biomet/45.3-4.562 is OK
- 10.1093/biostatistics/kxq045 is OK
- 10.1080/03610928008827941 is OK
- 10.1093/jamia/ocz228 is OK
- 10.1002/sim.4780050506 is OK
- 10.1016/j.jbi.2017.10.008 is OK

🟡 SKIP DOIs

- No DOI given, and none found for title: Metrics of Calibration for Probabilistic Predictio...
- No DOI given, and none found for title: Applied logistic regression
- No DOI given, and none found for title: MAPIE: an open-source library for distribution-fre...
- No DOI given, and none found for title: Uncertainty Toolbox: an Open-Source Library for As...
- No DOI given, and none found for title: Scikit-learn: Machine learning in Python
- No DOI given, and none found for title: On Calibration of Modern Neural Networks
- No DOI given, and none found for title: Measuring Calibration in Deep Learning.
- No DOI given, and none found for title: Posterior Re-calibration for Imbalanced Datasets

❌ MISSING DOIs

- None

❌ INVALID DOIs

- None

[jasonfan1997]

@editorialbot generate pdf

[editorialbot]

👉📄 Download article proof 📄 View article proof on GitHub 📄 👈

[editorialbot]

Five most similar historical JOSS papers:

dsBinVal: Conducting distributed ROC analysis using DataSHIELD
Submitting author: @schalkdaniel
Handling editor: @csoneson (Active)
Reviewers: @patRyserWelch8, @brunomontezano, @AnthonyOfSeattle
Similarity score: 0.6389

metrica: an R package to evaluate prediction performance of regression and classification point-forecast models
Submitting author: @adriancorrendo
Handling editor: @osorensen (Active)
Reviewers: @wiljnich, @simonpcouch
Similarity score: 0.6364

modeLLtest: An R Package for Unbiased Model Comparison using Cross Validation
Submitting author: @ShanaScogin
Handling editor: @usethedata (Retired)
Reviewers: @andrewheiss, @usethedata
Similarity score: 0.6170

hopit: An R Package for Analysis of Reporting Behavior Using Generalized Ordered Probit Models
Submitting author: @MaciejDanko
Handling editor: @alexhanna (Retired)
Reviewers: @soodoku, @mkearney
Similarity score: 0.6101

Metrics As Scores: A Tool- and Analysis Suite and Interactive Application for Exploring Context-Dependent Distributions
Submitting author: @mrshoenel
Handling editor: @mikldk (Retired)
Reviewers: @mdhaber, @kostiantyn-kucher
Similarity score: 0.6100

⚠️ Note to editors: If these papers look like they might be a good match, click through to the review issue for that paper and invite one or more of the authors before considering asking the reviewers of these papers to review again for JOSS.