Diagnostics and model criticism: posterior checks, SBC, ESS, divergences, and predictive evaluation
- distinguish computation diagnostics from model diagnostics;
- derive posterior predictive checking logic;
- explain simulation-based calibration and what rank uniformity means;
- interpret ESS, \(\hat{R}\), divergences, and E-BFMI together;
- compare WAIC, PSIS-LOO, Bayes factors, and stacking at a high level.
1 Two diagnostic questions
Bayesian workflow requires two different diagnostic questions:
- Did the algorithm approximate the posterior reliably?
- Is the posterior model itself adequate for the scientific question?
These are not the same question. Good chains can target a bad model, and a good model can still be approximated badly.
2 Posterior predictive checks
The posterior predictive distribution is
\[ p(\tilde y \mid y) = \int p(\tilde y \mid \theta)p(\theta \mid y)\, d\theta. \]
Posterior predictive checking compares the observed data to replicated data drawn from this distribution.
2.1 Discrepancy statistics
Choose a statistic \(T(y)\) sensitive to the aspect of fit you care about:
- mean;
- variance;
- tail behavior;
- zero inflation;
- residual asymmetry;
- group-level dispersion.
Then compare \(T(y)\) with the distribution of \(T(\tilde y)\) under replicated data.
2.2 Interpretation
If the observed statistic is extreme relative to the replicated distribution, the model struggles to reproduce an important data feature.
Posterior predictive checks do not prove a model correct. They reveal where it fails to mimic data features that matter.
3 Prior predictive checks
Posterior predictive checks happen after seeing the data. Prior predictive checks happen before fitting:
\[ p(y)=\int p(y \mid \theta)\pi(\theta)\, d\theta. \]
These are indispensable because many computational and inferential failures begin with absurd priors that already imply absurd data.
4 Effective sample size
For an MCMC estimator based on autocorrelated draws,
\[ \operatorname{ESS} = \frac{S}{1 + 2\sum_{k=1}^\infty \rho_k}, \]
where \(\rho_k\) is lag-\(k\) autocorrelation.
This quantifies how much independent-sample information the chain contains for a given functional of interest.
4.1 Bulk and tail ESS
Modern workflows distinguish:
- bulk ESS for central posterior summaries;
- tail ESS for quantiles and extreme posterior behavior.
Both matter.
5 \(\hat{R}\)
The modern rank-normalized \(\hat{R}\) compares within-chain and between-chain variation across multiple chains. Values near 1 indicate the chains mix consistently with one another (Vehtari et al. 2021).
5.1 Interpretation
- large \(\hat{R}\) signals nonconvergence or poor mixing;
- good \(\hat{R}\) is necessary but not sufficient;
- one must still inspect ESS, trace plots, and model-specific diagnostics.
6 Divergences and e-BFMI
For HMC-based workflows:
6.1 Divergences
Divergences indicate the numerical integrator struggled with posterior geometry. They are computational warnings with inferential consequences.
6.2 E-BFMI
Low E-BFMI indicates poor exploration of the energy distribution and often suggests momentum resampling does not match target geometry well.
These are geometry diagnostics, not generic MCMC diagnostics. They should be treated as signals of model-parameterization problems unless mild and clearly resolved.
7 Simulation-based calibration
SBC tests end-to-end inferential calibration under the assumed model.
7.1 Procedure
- Draw parameters \(\theta^{(s)}\) from the prior.
- Simulate data \(y^{(s)}\) from the likelihood.
- Fit the model to \(y^{(s)}\).
- Compare the true simulated parameter to its posterior rank among posterior draws.
If the full workflow is calibrated under the model, those ranks should be approximately uniform.
7.2 What SBC diagnoses
- coding mistakes;
- algorithmic bias;
- systematic under- or over-dispersion in approximate inference.
SBC is one of the strongest tests of a Bayesian workflow because it validates the entire inferential pipeline, not just one run on one data set (Rubin 1984).
8 Predictive model comparison
8.1 WAIC
WAIC approximates expected out-of-sample predictive fit using posterior draws and pointwise log-likelihood terms.
8.2 PSIS-LOO
PSIS-LOO approximates leave-one-out cross-validation through importance reweighting and Pareto smoothing (Vehtari et al. 2017).
8.3 Bayes factors
Bayes factors compare marginal likelihoods:
\[ \operatorname{BF}_{12}=\frac{p(y \mid M_1)}{p(y \mid M_2)}. \]
They are coherent but often highly sensitive to prior mass in low-likelihood regions.
8.4 Stacking
Stacking combines models by optimizing predictive performance of an ensemble rather than selecting a single model.
The key distinction is:
- Bayes factors emphasize model evidence under the prior;
- LOO and stacking emphasize predictive adequacy.
9 Sensitivity analysis
A posterior conclusion should be stress-tested against:
- prior choices;
- likelihood assumptions;
- outliers;
- parameterization choices;
- alternative plausible model structures.
Sensitivity analysis belongs to diagnostics because robustness is itself a property to be evaluated, not assumed.
10 R workflow
The companion diagnostics labs are:
Together they illustrate:
- replicated-data diagnostics;
- SBC rank histograms;
- observation-level predictive reasoning.
11 Diagnostics and pitfalls
11.1 Trusting one scalar summary
No single diagnostic certifies correctness.
11.2 Confusing model fit with causal validity
Posterior predictive success does not prove the model answers the substantive question of interest.
11.3 Ignoring observation-wise predictive failures
Aggregate metrics can hide localized bad fit.
11.4 Treating Bayes factors as default
They require strong prior care and are often not the most practical model comparison tool.
12 Exercises
- Explain why posterior predictive checking requires simulated replicated data rather than only residuals.
- Derive the ESS formula conceptually from autocorrelation inflation of Monte Carlo variance.
- Describe the SBC workflow and explain what uniform ranks mean.
- Compare the goals of PSIS-LOO and Bayes factors.
- Give an example where the algorithm is well behaved but the model is still poor.
13 Summary
Bayesian diagnostics are multi-layered:
- chain diagnostics assess computation;
- predictive checks assess model-data mismatch;
- SBC assesses full-workflow calibration;
- sensitivity analysis assesses robustness;
- predictive comparison assesses practical out-of-sample utility.
14 Further reading
Gelman and Rubin (1992) introduced the multiple-chain convergence diagnostic, and Vehtari et al. (2021) is its modern replacement: rank-normalised \(\widehat{R}\), improved effective sample size estimators, and a clear account of why the older versions could pass a chain that had not converged. Use the latter.
Rubin (1984) is the source for posterior predictive checking, and remains the best statement of the logic — simulate replicated data from the fitted model and ask whether the observed data look like a plausible draw. Gelman and Rubin (1992) and Gabry et al. (2019) develop the graphical side; Gabry et al. (2019) in particular is organised around the workflow rather than around individual diagnostics.
Vehtari et al. (2017) covers approximate leave-one-out cross-validation with Pareto-smoothed importance sampling, including the \(\hat{k}\) diagnostic that tells you when the approximation has failed. Talts et al. (2018) gives simulation-based calibration, which validates an inference algorithm rather than a model and is the right tool when you have implemented a sampler yourself. Gelman et al. (2020) places all of it in a single workflow.