Awesome Conformal Predictionl

A curated list of Awesome Conformal Prediction tutorials, videos, books, papers and open-source libraries in Python and R.

Table of Contents

• Books

• Video Tutorials, Videos and Lectures

• Papers

• Tutorials

• Researchers

• Websites

• Conferences

• Software in Python

• Software in R

• Software in Julia

• Software in C++

• Software in Scala and Spark

• Miscellaneous

• Contributing

Books

1. Algorithmic Learning in a Random World by Vladimir Vovk and Alex Gammerman, also Glenn Shafer (2005). Second edition in progress.
2. Conformal Prediciton for Reliable Machine Learning by Vineeth Balasubramanian, Shen-Shyang Ho, Vladimir Vovk (2014)

Tutorials

1. A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification by Anastasios N. Angelopoulos and Stephen Bates (2021)
2. A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification by Anastasios N. Angelopoulos and Stephen Bates (2021)
3. A Tutorial on Conformal Prediction by Anastasios Angelopoulos and Stephen Bates (2021)
4. A Tutorial on Conformal Predictive Distributions by Paolo Toccaceli (2020)
5. Conformal Prediciton Tutorial by Henrik Linusson (2021)
6. Henrik Linusson: Conformal Prediction by Henrik Linusson (2020)
7. Predicting with Confidence - Henrik Boström by Henrik Boström (2016)
8. Venn Predictors Tutorial by Ulf Johansson, Cecilia Sönströd, Tuve Löfström, and Henrik Boström (2021)
9. Ulf Johansson: Venn Predictors by Ulf Johansson (2020)
10. A Tutorial on Conformal Prediction by Glenn Shafer and Vladimir Vovk (2008)
11. Conformal Prediction: a Unified Review of Theory and New Challenges by Gianluca Zeni, Matteo Fontana1 and Simone Vantini (Politecnico di Milano, Italy, 2021)

Video Tutorials, Videos and Lectures

1. Treatment of Uncertainty in the Foundations of Probability by Vladimir Vovk (Royal Holloway, UK, 2017)
2. Large-Scale Probabilistic Prediction With and Without Validity Guarantees by Vladimir Vovk (Royal Holloway, UK, 2015)
3. Conformal testing in a binary model situation by Vladimir Vovk (Royal Holloway, UK, 2021)
4. Protected probabilistic classification by Vladimir Vovk (Royal Holloway, UK, 2021)
5. Retrain or not retrain: conformal test martingales for change-point detection by Vladimir Vovk (Royal Holloway, UK, 2021)
6. A Tutorial on Conformal Prediction by Anastasios Angelopoulos and Stephen Bates (Berkeley, 2021)
7. Steps Toward Trustworthy Machine Learning by Tom Dietterich (2021)
8. A Tutorial on Conformal Predictive Distributions by Paolo Toccaceli (Royal Holloway, UK, 2020)
9. Conformal Prediciton Tutorial by Henrik Linusson (2021)
10. Henrik Linusson: Conformal Prediction by Henrik Linusson (2020)
11. Predicting with Confidence - Henrik Boström by Henrik Boström (2016)
12. How to increase certainty in predictive modeling by Emmanuel Candes (Stanford, 2021)
13. Recent Progress in Predictive Inference by Emmanuel Candes (Stanford, 2020)
14. Some recent progress in predictive inference" (Stanford) @ MAD+ by Emmanuel Candes (Stanford, 2020)
15. Conformal Prediciton in 2020 by Emmanuel Candes (Stanford, 2020)
16. Assumption-free prediction intervals for black-box regression algorithms by Aaditya Ramdas (Carnegie Mellon, 2020)
17. Maria Navarro: Quantifying uncertainty in Machine Learning predictions | PyData London 2019 by Maria Navarro (2019)
18. Conformal Prediction: Enhanced Method for Understanding the Prediction Quality by Artem Ryasik and Greg Landrum
19. Venn Predictors Tutorial by Ulf Johansson, Cecilia Sönströd, Tuwe Löfström, and Henrik Boström (2021)
20. Mondrian conformal predictive distributions by Henrik Boström, Ulf Johansson, and Tuwe Löfström (2021)
21. Calibrating Multi-Class Models by Ulf Johansson, Tuwe Löfström, and Henrik Boström (2021)
22. Conformal testing in a binary model situation by Vladimir Vovk (Royal Holloway, UK, 2021)
23. Conformal prediction in Orange by Tomaž Hočevar and Blaž Zupan (2021)
24. Distribution-Free, Risk-Controlling Prediction Sets by Anastasios Angelopoulos (Stanford, 2021)
25. Conformal Prediction and Distribution-Free Calibration by Aaditya Ramdas (Carnegie Mellon, 2021)
26. Reliable Diagnostics by Conformal Predictors by Alexander Gammerman (Royal Holloway, UK, 2015)
27. Distribution-Free, Risk-Controlling Prediction Sets by Anastasios Angelopoulos (Stanford, 2021)
28. Conformal Inference of Counterfactuals and Time-to-event Outcomes by Lihua Lei (Stanford, 2021)
29. Algo Hour – Conformal Inference of Counterfactuals and Individual Treatment Effect by Lihua Lei (Stanford, 2021)
30. Conformal Inference of Counterfactuals and Individual Treatment effects(Stanford) by Lihua Lei (Stanford, 2021)
31. Approximation to object conditional validity with inductive conformal predictors by Anthony Bellotti (University of Nottingham Ningbo, China, 2021)
32. Ulf Johansson: Venn Predictors by Ulf Johansson (Jönköping University, Sweden, 2021)
33. Transformer-based conformal predictors for paraphrase detection by Patrizio Giavannotti and Prof. Alexander Gammerman (Royal Holloway, UK, 2021)
34. Conformal Inference of Counterfactuals and Individual Treatment Effects by Lihua Lei (Stanford, 2020)
35. Model-Free Predictive Inference by Larry Wasserman (Carnegie Mellon, 2020)
36. Shapley-value based inductive conformal prediction by William Lopez Jaramillo (2021)
37. Conformal Training: Learning Optimal Conformal Classifiers | DeepMind by David Stutz (2021)
38. Distribution-Free, Risk-Controlling Prediction Sets by Anastasios Angelopoulos (2021)
39. Assumption-Free, High-Dimensional Inference by Larry Wasserman (2016)
40. Neural Predictive Monitoring under Partial Observability by Francesca Cairolli (2021)
41. Conformalized Kernel Ridge Regression and Its Efficiency by Evgeny Burnaev (Skolkovo, Russia, 2015)
42. Fast conformal classification using influence functions by Giovanni Cherubin (Alan Turing Institute, UK, 2021)
43. Valid inferential models and conformal prediction by Ryan Martin (North Carolina State University, USA, 2021)
44. Mondrian conformal predictive distributions by Henrik Boström, Ulf Johansson and Tuwe Löfström (KTH Royal Institute of Technology, Sweden, 2021)
45. Evaluation of updating strategies for conformal predictive systems in the presence of extreme events by Hugo Werner, Lars Carlsson, Ernst Ahlberg and and Henrik Boström (KTH Royal Institute of Technology, Sweden, 2021)
46. Exact and Robust Conformal Inference Methods for Predictive Machine Learning With Dependent Data by Victor Chernozhukov (MIT, USA, 2019)
47. Ulf Johansson: Venn Predictors by Ulf Johansson (Jönköping University, Sweden, 2020)
48. Class-wise confidence for debt prediction in real estate management by Soundouss Messoudi (2021)
49. How Nonconformity Functions and Difficulty of Datasets Impact the Efficiency of Conformal Classifiers by Marharyta Aleksandrova (2021)
50. Nested conformal prediction and quantile out-of-bag ensemble methods by Chirag Gupta (Carnegie Mellon, 2020)

Papers

1. Introducing Conformal Prediction in Predictive Modeling. A Transparent and Flexible Alternative to Applicability Domain Determination by Ulf Norinder, Lars Carlsson, Scott Boyer, and Martin Eklund (2014)
2. Uncertainty Sets for Image Classifiers using Conformal Prediction by Anastasios N. Angelopoulos, Stephen Bates, Jitendra Malik, & Michael I. Jordan (Berkeley, 2021)
3. Conformal Prediction Under Covariate Shift by Ryan Tibshirani, Rina Foygel Barber, Emmanuel Candes, Aaditya Ramdas (Carnegie Mellon, Stanford, Chicago, 2019)
4. Regression Conformal Prediction with Nearest Neighbours by Harris Papadopoulos, Vladimir Vovk and Alex Gammerman (Royal Holloway, UK, 2014)
5. Nested conformal prediction and quantile out-of-bag ensemble methods by Chirag Gupta, Arun Kuchibhotla and Aaditya Ramdas (Carnegie Mellon, 2021)
6. Cross-conformal predictive distributions by Vladimir Vovk, Ilia Nouretdinov, Valery Manokhin and Alexander Gammerman (Royal Holloway, UK, 2018)
7. Criteria of Efficiency for Conformal Prediction by Vladimir Vovk, Ilia Nouretdinov, Valentina Fedorova, Ivan Petej, and Alex Gammerman ((Royal Holloway, UK, 2016)
8. Conformal Prediction for Simulation Models by Benjamin LeRoy and Chad Schafer (Carnegie Mellon, 2021)
9. Distribution-free, risk-controlling prediction sets Stephen Bates, Anastasios Angelopoulos, Lihua Lei, Jitendra Malik and Michael I Jordan (Berkeley, 2021)
10. Conditional calibration for false discovery rate control under dependence by William Fithian and Lihua Lei (Stanford, 2021)
11. Conformal Prediction: a Unified Review of Theory and New Challenges by Gianluca Zeni, Matteo Fontana and S. Vantini (2021)
12. Regression conformal prediction with random forests by Ulf Johansson, Henrik Boström, Tuve Löfström and Henrik Linusson (2014)
13. A conformal prediction approach to explore functional data by Jing Lei, Alessandro Rinaldo, Larry Wasserman (Carnegie Mellon, 2013)
14. An electronic nose-based assistive diagnostic prototype for lung cancer detection with conformal prediction by Xianghao Zhana,c, Zhan Wanga, Meng Yangb, Zhiyuan Luod, You Wanga, Guang Li (2020)
15. Predicting the Rate of Skin Penetration Using an Aggregated Conformal Prediction Framework by Martin Lindh, A. Karlén, Ulf Norinder (2017)
16. The application of conformal prediction to the drug discovery process by Martin Eklund, Ulf Norinder, Scott Boyer & Lars Carlsson (2014)
17. Distributional conformal prediction by Victor Chernozhukov, Kaspar Wüthrich, Yinchu Zhu (2021)
18. Anomaly Detection of Trajectories with Kernel Density Estimation by Conformal Prediction by James Smith, Ilia Nouretdinov, Rachel Craddock, Charles Offer, and Alexander Gammerman (2009)
19. Conformal prediction interval estimation and applications to day-ahead and intraday power markets by Christopher Kath, Florian Ziel (2019)
20. The application of conformal prediction to the drug discovery process by Martin Eklund, Ulf Norinder, Scott Boyer & Lars Carlsson (2013)
21. Anomaly Detection of Trajectories with Kernel Density Estimation by Conformal Prediction by James Smith, Ilia Nouretdinov, Rachel Craddock, Charles Offer, Alexander Gammerman (Royal Holloway, UK, 2014)
22. Conformal Prediction: a Unified Review of Theory and New Challenges by Gianluca Zeni, Matteo Fontana1 and Simone Vantini (Politecnico di Milano, Italy, 2021)
23. Exchangeability, Conformal Prediction, and Rank Tests by Arun Kuchibhotla (Carnegie Mellon, 2021)
24. Conformal prediction with localization by Leying Guan (Yale, 2020)
25. Predicting skin sensitizers with confidence - Using conformal prediction to determine applicability domain of GARD by Andy Forreryd, Ulf Norinder, Tim Lindberg, Malin Lindstedt (2018)
26. Binary classification of imbalanced datasets using conformal prediction by Ulf Norinder, Scott Boyer (2017)
27. Discretized conformal prediction for efficient distribution-free inference by Wenyu Chen, Kelli-Jean Chun, and Rina Foygel Barber (2017)
28. Validity, consonant plausibility measures, and conformal prediction by Leonardo Cella. and Ryan Martin (2021)
29. Conformal Prediction Classification of a Large Data Set of Environmental Chemicals from ToxCast and Tox21 Estrogen Receptor Assays by Ulf Norinder, Scott Boyer (2016)
30. Conformal prediction to define applicability domain – A case study on predicting ER and AR binding by U. Norinder, A. Rybacka, P.Andersson (2016)
31. Conformal prediction of biological activity of chemical compounds by Paolo Toccaceli, Ilia Nouretdinov, Alex Gammerman (Royal Holloway, UK, 2017)
32. Introducing conformal prediction in predictive modeling for regulatory purposes. A transparent and flexible alternative to applicability domain determination by Ulf Norinder, Lars Carlsson, Scott Boyer, Martin Eklund (2015)
33. Aggregated Conformal Prediction by Lars CarlssonMartin EklundUlf Norinder (2014)
34. Interpretation of Conformal Prediction Classification Models by Ernst Ahlberg, Ola Spjuth, Catrin Hasselgren, Lars Carlsson (2015)
35. Cross-Conformal Prediction with Ridge Regression by Harris Papadopoulos (2015)
36. Sparse conformal prediction for dissimilarity data by Frank-Michael Schleif, Xibin Zhu and Barbara Hammer (2015)
37. Effective utilization of data in inductive conformal prediction using ensembles of neural networks by Tuve Löfström, Ulf Johansson and Henrik Boström (2013)
38. Beyond the Basic Conformal Prediction Framework by Vladimir Vovk (2014)
39. An electronic nose-based assistive diagnostic prototype for lung cancer detection with conformal prediction by Xianghao Zhan, Zhan Wang, Meng Yang, Zhiyuan Luo, You Wang, Guang Li (Stanford, Royal Holloway, China University of Mining and Technology, 2020)
40. Predicting with confidence: Using conformal prediction in drug discovery by Jonathan Alvarsson, Staffan Arvidsson McShane, Ulf Norinder, Ola Spjuth (2017)
41. Inductive conformal prediction for silent speech recognition by Ming Zhang, You Wang, Zhang Wei, Meng Yang, Zhiyuan Luo, Guang Li (2020)
42. Large scale comparison of QSAR and conformal prediction methods and their applications in drug discovery by Nicolas Bosc, Francis Atkinson, Eloy Felix, Anna Gaulton, Anne Hersey and Andrew R. Leach (2019)
43. Deep Conformal Prediction for Robust Models by Soundouss Messoudi, Sylvain Rousseau and Sébastien Destercke (2020)
44. Strong validity, consonance, and conformal prediction by Leonardo Cella and Ryan Martin (2020)
45. Skin Doctor CP: Conformal Prediction of the Skin Sensitization Potential of Small Organic Molecules by Anke Wilm, U. Norinder, M. Agea, Christina de Bruyn Kops, Conrad Stork, J. Kühnl, J. Kirchmair (2020)
46. Conformal prediction based active learning by linear regression optimization by Sergio Matiz, Kenneth E.Barner (2020)
47. Conformal prediction intervals for the individual treatment effect by Danijel Kivaranovic, Robin Ristl, Martin Poschb, Hannes Leeb (2020)
48. Nearest neighbor based conformal prediction by László Györfi and Harro Walk (2020)
49. Concepts and Applications of Conformal Prediction in Computational Drug Discovery by Isidro Cortés-Ciriano and Andreas Bender (2019)
50. Predicting Ames Mutagenicity Using Conformal Prediction in the Ames/QSAR International Challenge Project by Ulf Norinder, Ernst Ahlberg, Lars Carlsson (2018)
51. Nested Conformal Prediction and the Generalized Jackknife by Arun Kuchibhotla and Aaditya Ramdas (Carnegie Mellon, 2019)
52. Predictive inference with the jackknife+ by Rina Foygel Barber, Emmanuel Candès, Aaditya Ramdas, and Ryan Tibshirani (2020)
53. Nonparametric predictive distributions based on conformal prediction by Vladimir Vovk, Jieli Shen, Valery Manokhin and Min-ge Xie (Royal Holloway, UK, Rutgers, USA, 2018)
54. A Distribution-Free Test of Covariate Shift Using Conformal Prediction by Xiaoyu Hu and Jing Lei (Peking Univerity, China and Carnegie Mellon, USA, 2020)
55. Exchangeability, Conformal Prediction, and Rank Tests by Arun Kuchibhotla (Carnegie Mellon, 2021)
56. Conformal prediction with localization by Leying Guan (2020)
57. Multitask Modeling with Confidence Using Matrix Factorization and Conformal Prediction by Ulf Norinder, Fredrik Svensson
58. Conformal prediction of HDAC inhibitors by U. Norinder, J.J.Navaka, E. Lopez-Lopez, D. Mucs & J.L. Medina-Franco (2019)
59. Computing Full Conformal Prediction Set with Approximate Homotopy by Eugene Ndiaye, Ichiro Takeuchi (2019)
60. Conformal Prediction Based on Raman Spectra for the Classification of Chinese Liquors by Jiao Gu, Huaibo Liu, Chaoqun Ma, Lei Li, Chun Zhu, Christ Glorieux, Guoqing Chen (2019)
61. Efficient and minimal length parametric conformal prediction regions by Daniel Eck and Forrest Crawford (2019)
62. Conformal Prediction for Students' Grades in a Course Recommender System by Raphael Morsomme and Evgueni Smirnov (2019)
63. Efficient iterative virtual screening with Apache Spark and conformal prediction by Laeeq Ahmed, Valentin Georgiev, Marco Capuccini, Salman Toor, Wesley Schaal, Erwin Laure and Ola Spjuth (2018)
64. Predicting Off-Target Binding Profiles With Confidence Using Conformal Prediction by Samuel Lampa, Jonathan Alvarsson, Staffan Arvidsson Mc Shane, Arvid Berg, Ernst Ahlberg, Ola Spjuth (2018)
65. Maximizing gain in high-throughput screening using conformal prediction by Fredrik Svensson, Avid M. Afzal1, Ulf Norinder and Andreas Bender (2018)
66. Conformalized Survival Analysis by Emmanuel Candès, Lihua Lei and Zhimei Ren (2021)
67. Random Forest Prediction Intervals by Haozhe Zhang†, Joshua Zimmerman†, Dan Nettleton† and Daniel J. Nordman† (Iowa State University, USA, 2019)
68. Conformal Training: Learning Optimal Conformal Classifiers | DeepMind by David Stutz (DeepMind), Krishnamurthy Dvijotham, Ali Taylan Cemgil and Arnaud Doucet (2021)

Researchers

1. Vladimir Vovk, Royal Holloway, United Kingdom
2. Alexander Gammerman, Royal Holloway, United Kingdom
3. Glenn Shafer, Rutgers University, USA
4. Emmanuel Candès, Stanford, USA
5. Ryan Tibshiriani, Carnegie Mellon, USA
6. Yaniv Romano, Technion—Israel Institute of Technology
7. Michael I. Jordan, Berkeley, USA
8. Jitendra Malik, Berkeley, USA
9. Anastasios Angelopoulos, Berkeley, USA
10. Lihua Lei, Stanford, USA
11. Henrik Boström, KTH, Sweden
12. Ulf Johansson, Jönköping University, Sweden
13. Henrik Linusson, University of Borås, Sweden
14. Harris Papadopoulos, Frederick University, Cyprus
15. Vladimir V'yugin, Institute for Information Transmission Problems (IITP), Russia
16. Evgeny Burnaev, Skoltech, Russia
17. Aaditya Ramdas, Carnegie Mellon, USA
18. Benjamin LeRoy, Carnegie Mellon, USA
19. Victor Chernozhukov, MIT, USA
20. Ulf Norinder, Stockholm University, Sweden
21. Ola Spjuth, Uppsala University, Sweden
22. Ilia Nouretdinov, Royal Holloway, United Kingdom

Websites

1. Main website with research from Prof. Vladimir (Volodya) Vovk
2. Conformal Prediction - Prediction with guaranteed performance Royal Holloway, United Kingdom
3. A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification by Anastasios N. Angelopoulos

Conferences

1. https://icml.cc/Conferences/2021/ScheduleMultitrack?event=8373
2. https://cml.rhul.ac.uk/copa2021/
3. https://cml.rhul.ac.uk/copa2020/
4. https://cml.rhul.ac.uk/copa2019/
5. https://cml.rhul.ac.uk/copa2018/
6. https://cml.rhul.ac.uk/copa2017/

Software Python

1. Nonconformist by Henrik Linusson (2015)
2. MAPIE - Model Agnostic Prediction Interval Estimator (2021)
3. 'Crêpes' - Conformal regressors and predictive systems by Henrik Boström (2021)
4. Conformalized Quantile Regression by Yaniv Romano (2019)
5. Conformal Classification by Anastasios N. Angelopoulos (2021)
6. Orange3 Conformal Prediction
7. EnbPI by Chen Xu (2021)
8. Conformal: an R package to calculate prediction errors in the conformal prediction framework by Isidro Cortes, 2019
9. Copula Conformal Multi Target Regression by Soundouss Messoudi (2021)

Software R

1. Conformal Inference R Project maintained by Ryan Tibshirani (2016)
2. Prediction Bands by Rafael Izbicki and Benjamin LeRoy (2019)
3. Conformal: an R package to calculate prediction errors in the conformal prediction framework by Isidro Cortes, 2019
4. Online Time Series Anomaly Detectors by Alaine Iturria, 2021
5. piRF - Prediction Intervals for Random Forests by Chancellor Johnstone and Haozhe Zhang (2019)

Software C++

1. LibCP -- A Library for Conformal Prediction
2. An Implementation of Venn-ABERS predictor
3. LibVM -- A Library for Venn Machine

Software Julia

1. RandomForest by Henrik Boström (2017)

Software Scala and Spark

Scala-CP by Marco Capuccini (2020)