Part IV. Bayesian models

Part IV

This part is the modeling core of the course: regression, multilevel structure, latent variables, time dependence, nonparametrics, spatial structure, and causal perspectives.

Part overview

Part IV is the modeling core of the course. The goal is not to present disconnected model families, but to show how Bayesian modeling principles recur across regression, multilevel structure, latent variables, temporal dependence, nonparametrics, and causal reasoning.

Part learning goals

By the end of Part IV, learners should be able to:

  • formulate generative models for structured data;
  • choose priors that regularize realistically;
  • interpret posterior structure in terms of shrinkage, dependence, and latent heterogeneity;
  • anticipate computational and identifiability issues for each model family.

Fully written chapters in this repository

Reading path

Recommended order:

  1. Bayesian linear models
  2. Generalized linear models
  3. Hierarchical and multilevel models
  4. Mixture models and latent variable models
  5. Time series and state-space models
  6. Nonparametric Bayesian methods
  7. Spatial and graphical Bayesian models
  8. Bayesian causal inference

Chapter 13. Bayesian linear models

Core question

How does Bayesian regression extend classical least squares into a full posterior and predictive framework?

Section blueprint

  • Gaussian linear regression model.
  • Priors on regression coefficients and variance.
  • Conjugate and semi-conjugate analyses.
  • \(g\)-priors and shrinkage interpretations.
  • Bayesian analogues of ridge and lasso regularization.
  • Posterior predictive regression.

Derivations that must appear

  • Posterior for coefficients under Gaussian prior and known variance.
  • Normal-Inverse-Gamma posterior for unknown variance.
  • Posterior predictive distribution for a new covariate vector.
  • Ridge connection from Gaussian priors and MAP estimation.

Computational lab

  • Simulate linear regression with known signal-to-noise ratio.
  • Compare flat, weakly informative, and shrinkage priors.
  • Fit the same model in closed form and in Stan.

Common pitfalls

  • Using diffuse priors on poorly scaled predictors.
  • Ignoring posterior predictive uncertainty in forecasts.
  • Treating shrinkage as purely computational rather than inferential.

Exercises

  • Derive the posterior mean and covariance for Bayesian linear regression.
  • Show the penalized least-squares form of the MAP estimator under a Gaussian prior.

Core references

Chapter 14. Generalized linear models

Core question

How do Bayesian methods generalize regression when responses are binary, counts, ordinal, or multinomial?

Section blueprint

  • Logistic and probit regression.
  • Poisson and negative binomial regression.
  • Ordinal models.
  • Multinomial models.
  • Hierarchical GLMs.
  • Prior scaling and regularization in nonlinear link models.

Derivations that must appear

  • Likelihood and log-posterior for logistic regression.
  • Posterior predictive probabilities for binary and count outcomes.
  • Negative binomial as a gamma-mixed Poisson intuition.
  • Latent-variable view of probit and ordinal models.

Computational lab

  • Prior sensitivity for logistic regression.
  • Stan implementation of Poisson and negative binomial models.
  • Posterior predictive checks for overdispersion and zero inflation.

Common pitfalls

  • Separation in logistic regression.
  • Using weak priors on coefficients with unscaled predictors.
  • Overinterpreting coefficients without translating to predictive probabilities.

Exercises

  • Derive gradient and Hessian of a logistic regression log posterior.
  • Compare Poisson and negative binomial predictive dispersion.

Core references

Chapter 15. Hierarchical and multilevel models

Core question

How does partial pooling emerge from exchangeability assumptions and hierarchical priors?

Section blueprint

  • Complete pooling, no pooling, and partial pooling as points on one spectrum.
  • Exchangeability and de Finetti’s representation theorem.
  • The normal-normal model and the shrinkage factor.
  • Stein’s inadmissibility result and the James-Stein estimator.
  • The marginal posterior of the group-level scale.
  • Neal’s funnel, and centred versus non-centred parameterisation.
  • Random intercepts, random slopes, and group-level predictors.
  • Hyperpriors for variance components.
  • Prediction for observed and for new groups.

Derivations that must appear

  • Hierarchical factorisation of the joint posterior.
  • The normal-normal posterior by completing the square.
  • The shrinkage decomposition, and its four limiting cases.
  • The marginal likelihood for \(\tau\) with the group effects integrated out.
  • The posterior predictive distribution for a new group.
  • The non-centring map and its effect on posterior geometry.

Computational lab

  • Eight schools in Stan under both parameterisations.
  • Plot the marginal posterior of \(\tau\) and the eight shrinkage paths.
  • Diagnose funnel pathologies and locate divergences in \((\theta_j, \log\tau)\) space.
  • Simulate implied prior pooling under competing hyperpriors.

Common pitfalls

  • Reading shrinkage as distortion rather than as variance reduction.
  • Inverse-gamma hyperpriors on variance components.
  • Raising adapt_delta instead of reparameterising.
  • Assuming non-centred is always the right choice.
  • Confusing observed-group with new-group prediction.

Exercises

  • Derive the posterior mean in the normal-normal model and interpret the shrinkage weight.
  • Derive the marginal posterior of \(\tau\) and plot it for the eight schools data.
  • Compare centred and non-centred parameterisations analytically and empirically.
  • Show why the better parameterisation reverses as the groups become data-rich.

Core references

  • Gelman et al. (2013)
  • Gelman (2006)
  • Stein (1956)
  • Efron and Morris (1975)
  • Papaspiliopoulos et al. (2007)
  • Betancourt and Girolami (2015)

Full chapter

Chapter 16. Mixture models and latent variable models

Core question

How do latent allocations create flexible models, and what new inferential pathologies do they introduce?

Section blueprint

  • Finite mixtures.
  • Latent class models.
  • Gaussian mixtures.
  • Switching regressions.
  • Hidden Markov models.
  • Label switching and identifiability.
  • Mixture-of-experts overview.

Derivations that must appear

  • Complete-data likelihood factorization.
  • Gibbs updates for latent allocations in simple mixture settings.
  • Marginalization versus latent-state sampling comparison.
  • Forward-backward recursions for discrete latent-state models.

Computational lab

  • Fit a Gaussian mixture in Stan or with custom Gibbs sampling.
  • Visualize label switching.
  • Compare identifiable summaries based on predictive density rather than component labels.

Common pitfalls

  • Interpreting raw component labels as meaningful without identifiability constraints.
  • Overstating certainty in the number of mixture components.
  • Ignoring multimodality in posterior summaries.

Exercises

  • Write the full conditional for mixture allocations.
  • Explain why symmetric priors induce label-switching invariance.

Core references

Chapter 17. Time series and state-space models

Core question

How can Bayesian models represent evolving latent states and temporal dependence?

Section blueprint

  • Dynamic linear models.
  • Bayesian AR and ARIMA ideas.
  • Local level and local trend models.
  • Stochastic volatility overview.
  • Hidden Markov models.
  • Kalman filtering and smoothing.

Derivations that must appear

  • State-space factorization.
  • Kalman filter prediction and update equations.
  • Rauch-Tung-Striebel smoothing overview.
  • Bayesian forecast distribution under linear Gaussian assumptions.

Computational lab

  • Implement a local-level model in Stan or R.
  • Compare filtering and smoothing estimates.
  • Explore posterior uncertainty under missing observations.

Common pitfalls

  • Confusing latent state uncertainty with observation noise.
  • Ignoring initialization sensitivity in short time series.
  • Overfitting with highly flexible latent dynamics.

Exercises

  • Derive Kalman filter recursions for a scalar local-level model.
  • Compare AR(1) and local-level formulations conceptually.

Core references

  • Gelman et al. (2013)
  • Doucet et al. (2001)

Chapter 18. Nonparametric Bayesian methods

Core question

How can Bayesian models increase flexibility without fixing parameter dimension in advance?

Section blueprint

  • Dirichlet process and Ferguson construction.
  • Chinese restaurant process intuition.
  • Stick-breaking construction.
  • DP mixtures.
  • Hierarchical Dirichlet process overview.
  • Gaussian processes.
  • Bayesian additive regression trees overview.

Derivations that must appear

  • Basic DP expectation and clustering intuition.
  • Predictive scheme underlying the Chinese restaurant process.
  • Stick-breaking representation.
  • GP posterior mean and covariance formulas.

Computational lab

  • GP regression on a small one-dimensional data set.
  • Visualize prior and posterior functions under different kernels.
  • Simulate cluster allocations under a DP-style predictive process.

Common pitfalls

  • Thinking “nonparametric” means assumption-free.
  • Ignoring kernel prior implications in GPs.
  • Assuming DP mixtures automatically identify the true number of clusters.

Exercises

  • Derive the GP posterior conditional distribution.
  • Explain how DP concentration affects expected clustering.

Core references

  • Ferguson (1973)
  • Rasmussen and Williams (2006)
  • Murphy (2023)

Chapter 19. Spatial and graphical Bayesian models

Core question

How do conditional independence structures encode spatial and network dependence efficiently?

Section blueprint

  • Probabilistic graphical models.
  • Bayesian networks and directed factorization.
  • Gaussian Markov random fields.
  • CAR and SAR model overview.
  • Message passing overview.
  • Conditional independence and sparse precision matrices.

Derivations that must appear

  • Factorization from DAG structure.
  • Precision-based conditional independence in Gaussian models.
  • CAR full conditional structure.

Computational lab

  • Build a simple Bayesian network factorization.
  • Explore a lattice-based CAR prior conceptually or computationally.
  • Compare covariance and precision views in Gaussian models.

Common pitfalls

  • Confusing sparse covariance with sparse precision.
  • Ignoring identifiability constraints in intrinsic CAR models.
  • Treating graphical structure as purely computational rather than substantive.

Exercises

  • Factorize a joint distribution from a DAG.
  • Show conditional independence relations implied by a GMRF precision matrix.

Core references

Chapter 20. Causal inference and Bayesian perspectives

Core question

What does a Bayesian workflow add to causal inference problems formulated through potential outcomes or structural models?

Section blueprint

  • Potential outcomes and estimands.
  • Bayesian propensity-score-style approaches overview.
  • Bayesian mediation overview.
  • Structural model overview.
  • Sensitivity analysis for unmeasured confounding.
  • Posterior predictive thinking in causal contexts.

Derivations that must appear

  • Factorization of causal assumptions into model components.
  • Posterior for causal estimands under a simple outcome model.
  • Sensitivity parameter incorporation in a Bayesian model.

Computational lab

  • Fit a Bayesian treatment-effect model under ignorability.
  • Perform sensitivity analysis to hidden bias.
  • Compare causal estimands under different prior assumptions.

Common pitfalls

  • Confusing predictive fit with causal identification.
  • Allowing flexible models to obscure identifying assumptions.
  • Treating posterior probability of an effect as proof of causality.

Exercises

  • Write a Bayesian model for treatment and outcome under ignorability.
  • Explain how sensitivity parameters can be treated probabilistically.

Core references

Back to top

References

Berger, James O. 1985. Statistical Decision Theory and Bayesian Analysis. 2nd ed. Springer.
Betancourt, Michael, and Mark Girolami. 2015. “Hamiltonian Monte Carlo for Hierarchical Models.” Current Trends in Bayesian Methodology with Applications, 79–101.
Doucet, Arnaud, Nando de Freitas, and Neil Gordon. 2001. Sequential Monte Carlo Methods in Practice. Springer.
Efron, Bradley, and Carl Morris. 1975. “Data Analysis Using Stein’s Estimator and Its Generalizations.” Journal of the American Statistical Association 70 (350): 311–19.
Ferguson, Thomas S. 1973. “A Bayesian Analysis of Some Nonparametric Problems.” The Annals of Statistics 1 (2): 209–30.
Gelman, Andrew. 2006. “Prior Distributions for Variance Parameters in Hierarchical Models.” Bayesian Analysis 1 (3): 515–34.
Gelman, Andrew, John B. Carlin, Hal S. Stern, David B. Dunson, Aki Vehtari, and Donald B. Rubin. 2013. Bayesian Data Analysis. 3rd ed. CRC Press.
Hoff, Peter D. 2009. A First Course in Bayesian Statistical Methods. Springer.
MacKay, David J. C. 2003. Information Theory, Inference, and Learning Algorithms. Cambridge University Press.
McElreath, Richard. 2020. Statistical Rethinking. 2nd ed. CRC Press.
Murphy, Kevin P. 2023. Probabilistic Machine Learning: Advanced Topics. MIT Press.
Neal, Radford M. 2000. “Markov Chain Sampling Methods for Dirichlet Process Mixture Models.” Journal of Computational and Graphical Statistics 9 (2): 249–65.
Papaspiliopoulos, Omiros, Gareth O. Roberts, and Martin Sköld. 2007. “A General Framework for the Parametrization of Hierarchical Models.” Statistical Science 22 (1): 59–73.
Rasmussen, Carl Edward, and Christopher K. I. Williams. 2006. Gaussian Processes for Machine Learning. MIT Press.
Stein, Charles. 1956. “Inadmissibility of the Usual Estimator for the Mean of a Multivariate Normal Distribution.” In Proceedings of the Third Berkeley Symposium on Mathematical Statistics and Probability, vol. 1. University of California Press.