Overview of Explainability (Draft)

[mauicv]

Progress:

Intro:

• What is explainability
• Applications
• Black-box vs White-box
• Global and local insights
• Biases

Types of insight

• Table

Global Feature Attribution

• Intro
• ALE

Local Necessary Features

• Intro
• Anchors
• Pertinent Positives

Local Feature Attribution

• Intro
• Integrated Gradients
• KernelSHAP
• Path dependent
• Interventional

Counter Factuals

• Intro
• Instances
• CEM
• Prototypes
• Reinforcement Learning

[codecov]

Codecov Report

Merging #510 (002d531) into master (0ea0dc7) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #510   +/-   ##
=======================================
  Coverage   82.03%   82.03%           
=======================================
  Files          77       77           
  Lines       10513    10513           
=======================================
  Hits         8624     8624           
  Misses       1889     1889           

[mauicv]

Discussion on Explainability insights and misuse (From discussion with @jklaise)

Point I want to make is that you can use explainability for testing. That given a model you can draw insights and if they conform to your expectations then that's evidence that the model has been trained properly. However this needs further discussion because:

1. You may have an insight that accurately reflects the model but not reality and yet also confirms some bias in which case you'll fall into the trap of assuming the insight is evidence for something it's not.
2. Vice versa you might see get an insight that properly reflects both the model and the data but doesn't reflect an expectation of yours in which case you might incorrectly classify a model as broken
3. Sometimes insights can be very complex/Not interpretable, for instance anchors close to the decision boundary. This can still be a correct reflection of the model however but might lead developers to assume errors.

[jklaise]

A few more to keep track of:

• Include a link to the white-box/black-box page from the section with the same name (this could go in the "Note" box).
• I think we should be more specific for the white-box frameworks in the summary table, e.g. explicitly say White-box (Tensorflow) or White-box (xgboost, catboost, lightbgbm ...)
• Add a link to the wine datset when first introduces. Potentially even add a sentence or two introducing it with the task being solved.
• Since the notebook is linked from the sidebar too, it probably makes sense to put it at the very top? Unless we feel it's a bit rough in which case we can leave it off the sidebar entirely and people can discover it when they read the overview.
• I think we should remove the link to the wiki page on "counterfactual thinking" as discussed previously.
• Should we be even more explicit re types of models supported? E.g. for CFRL it is truly black-box, but for other counterfactual ones it's "black-box" but expectation is that the model is differentiable. I know this is said in the text, but might be worth making it clear in the summary and method tables?
• Rephrase anchor discussion as "size" of anchor is confusing when coverage is meant, which has an inverse relationship to length of anchor.
• Is it possible in myst to create anchors/labels and link directly to sections? Specifically, linking to Note 4 from the IG cons (it currently says Note 5 because it's manual and doesn't have a link). Not crucial but nice to check.
• In the CFRL section I think we should add a comment that there are 2 explainers, CounterfactualRL and CounterfactualRlTabular, especially because we use the former for a tabular dataset which could be confusing. The Tabular one really is for supporting categorical features.
• Add a sentence or two for the CF example results with a few general statements, e.g. the basic CI method perturbs many features which may not be desirable.
• Can we add class labels to the wine instance wherever appropriate (e.g. as code comments: Instance prediction: 0 # "good") so it's always at hand for each example.
• As a continuation from the above one on adding labels, maybe add another sentence to put the ALE plots in context with the CF results?

[arnaudvl]

Left some very minor last comments. Otherwise good to go!