Chapter 32 — Interpretability

Model Interpretability & Explainability

A model you can't explain is one you can't trust or ship in regulated settings. Global vs local explanations, SHAP, permutation importance, PDP, and their pitfalls.

Stakeholders, regulators, and your future self all ask "why did the model decide that?" Interpretability turns a black box into something you can debug, defend, and deploy responsibly.

32.1 Global vs local

what question?

What do you need to explain?
│
├── "What drives the model overall?" ──► GLOBAL
│      ├── Permutation importance
│      ├── SHAP summary plot
│      └── Partial dependence (PDP)
│
└── "Why THIS prediction?" ────────────► LOCAL
       ├── SHAP force/waterfall
       └── LIME

32.2 Intrinsic vs post-hoc

Intrinsic (glass-box)

Linear/logistic coefficients, shallow decision trees, rule sets. Explainable by construction. Prefer when a simple model is accurate enough.

Post-hoc (explain a black box)

SHAP, LIME, PDP applied after the fact to RF/XGBoost/NN. Powerful but approximate — explanations can mislead if features are correlated.

32.3 The techniques

Method	Scope	Strength	Pitfall
Coefficients	Global	Exact, free (linear models)	Only for linear; scale matters
Permutation importance	Global	Model-agnostic, simple	Misleads with correlated features
SHAP	Both	Consistent, local + global	Slow on big data; assumes feature independence
PDP / ICE	Global	Shows effect shape	Hides interactions (PDP)
LIME	Local	Any model, intuitive	Unstable across runs

32.4 SHAP — the modern default

SHAP assigns each feature a contribution to each prediction, grounded in game theory. It unifies local ("why this row?") and global ("what matters overall?") views.

python

import shap
explainer = shap.TreeExplainer(model)        # fast for tree models
shap_values = explainer.shap_values(X_test)

shap.summary_plot(shap_values, X_test)        # global: ranked impact
shap.plots.waterfall(explainer(X_test)[0])  # local: one prediction

reading a SHAP waterfall

base value (avg prediction)
   + feature A pushed up   ▲ +0.12
   − feature B pushed down ▼ −0.05
   + feature C pushed up   ▲ +0.03
   ─────────────────────────────
   = this row's prediction

32.5 The correlated-features trap

When features are correlated, importance "splits" between them and permutation/SHAP can assign credit oddly — a key driver may look weak because a correlated twin shares the credit. Reduce redundancy first (Chapter 15.5) and interpret groups, not lone features.

Professional recommendation

Need trust + simpleLogistic/linear, read coefficients

Tree modelSHAP TreeExplainer

Effect shapePDP + ICE

RegulatedPrefer glass-box models

Common mistakes to avoid

Reading feature importance as causal effect
Trusting importances when features are highly correlated
Using PDP on interacting features (it averages the interaction away)
Explaining a needlessly complex model when a glass-box would do
Comparing unscaled linear coefficients as if they're importances

Quick cheatsheet

shap.TreeExplainer(model) -> fast SHAP for trees

shap.summary_plot() -> global importance

permutation_importance() -> model-agnostic global

PartialDependenceDisplay -> effect shape

glass-box first -> when explanation matters