Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Outreachy applications] Calibration plot #5

Closed
dzeber opened this issue Mar 4, 2020 · 2 comments · Fixed by #69 or #110
Closed

[Outreachy applications] Calibration plot #5

dzeber opened this issue Mar 4, 2020 · 2 comments · Fixed by #69 or #110

Comments

@dzeber
Copy link
Contributor

dzeber commented Mar 4, 2020

A calibration plot assesses how well the predicted class probabilities match the actual rate of occurrence in the dataset.

Write a function to create a calibration plot given one or more binary classifiers and a test set. For each model it would compute predicted probabilities for each test datapoint. These probability values should then be binned according to a scheme (eg. intervals of 10%), and the observed occurrence rate can be computed as the proportion of true positive class samples out of all samples in each bin. The plot then displays the observed occurrence rates vs the bin midpoints for each model.
@Soniyanayak51
Copy link
Contributor

@dzeber I would like to try this.

KaairaGupta added a commit to KaairaGupta/PRESC that referenced this issue Mar 9, 2020
@KaairaGupta
For mozilla#5: Calibration plot
ec51b98
@KaairaGupta KaairaGupta mentioned this issue Mar 9, 2020
Merged
KaairaGupta added a commit to KaairaGupta/PRESC that referenced this issue Mar 10, 2020
@KaairaGupta
For mozilla#5: Calibration plot
faeab04
@Soniyanayak51 Soniyanayak51 mentioned this issue Mar 12, 2020
Merged
@Addi-11 Addi-11 mentioned this issue Mar 16, 2020
Merged
@dzekem
Copy link
Contributor

dzekem commented Mar 18, 2020

Also working on this issue.

@shashigharti shashigharti mentioned this issue Mar 20, 2020
Merged
mlopatka pushed a commit that referenced this issue Mar 20, 2020
@KaairaGupta
For #5 : Calibration plot (#35)
66f4c05 
* For #5: Calibration plot

* reformatted calibration_plot using black

* improve function code for better readability

* renamed function calibration_plot to plot_calibration_curves, added explanatory text and curve interpretation in example notebook, reformatted notebook
mlopatka pushed a commit that referenced this issue Mar 20, 2020
@Addi-11
#5 Calibration Plots (#69)
fb0c69c 
* visual for eeg

* code restructured

* #3 data-split space mapped

* fixes issue3

* studied data splits for all classifiers

* added graph in the loop

* docstrings added

* validation sets added

* formatting

* evaluated all classifiers

* compared models

* result added

* calibration plot added

* docstrings
@mlopatka mlopatka reopened this Mar 20, 2020
Sidrah-Madiha added a commit to Sidrah-Madiha/PRESC that referenced this issue Mar 22, 2020
@Sidrah-Madiha
fixes mozilla#5
f6f1fb3
@Sidrah-Madiha Sidrah-Madiha mentioned this issue Mar 22, 2020
Merged
mlopatka pushed a commit that referenced this issue Mar 25, 2020
@Sidrah-Madiha
Calibration plot [fixes # 5] (#110)
a186ae6 
* fixes #8

* fixes #4, attempt 1

* implemeneted all change requests

* formatted code for all helper files

* minor fix

* fixes #5

* fixing conflicts
@mlopatka mlopatka reopened this Mar 25, 2020
@namrathagopalabhatla namrathagopalabhatla mentioned this issue Apr 3, 2020
Merged
asthad16 referenced this issue in asthad16/PRESC Apr 7, 2020
@asthad16
issue#5comparison of calibration of classifiers
719aa3f 
this PR  fixes issue #5 by making calibration plot for 3 classifiers namely:
1-logistic regression
2-gaussian naive bayes(GNB)
3-support vector machine(SVM)

the evaluation is done using brier score loss.
also the other calibration methods are implemented to improve the brier score value of GNB and SVM namely:
1-isotonic calibration
2-sigmoid calibration
@asthad16 asthad16 mentioned this issue Apr 7, 2020
Merged
@dzeber dzeber changed the title Calibration plot [Outreachy applications] Calibration plot Jul 13, 2020
@dzeber dzeber closed this as completed Jul 14, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

#5 Calibration Plots Addi-11/PRESC
Calibration plot [fixes # 5] Sidrah-Madiha/PRESC
4 participants
@dzeber @mlopatka @dzekem @Soniyanayak51