Bayesian inference and Bayesian statistics
A long-form course in probability, inference, computation, modelling, and workflow
Bayesian inference and Bayesian statistics
A full university course, written for advanced undergraduates, MSc and PhD students, and anyone with the background to work through it alone. Thirty-two chapters run from measure-theoretic foundations to modern probabilistic workflow, with derivations worked in full, runnable labs in R, Stan, and JAX, and exercises throughout.
The premise: Bayesian statistics is a workflow, not a formula. It starts with a probability model, passes through prior specification and posterior computation, and ends in decision-relevant summaries, predictive checks, and criticism of the model you just fitted (Bernardo and Smith 1994; Gelman et al. 2013, 2020).
The course
I. Foundations
Probability spaces, conditional expectation, limit theorems, likelihood theory, Bayes’ theorem, exchangeability.
II. Core inference
Conjugate analysis, marginal likelihoods, decision theory, prior design and sensitivity.
III. Computation
Laplace approximation, Monte Carlo, MCMC, HMC and NUTS, variational inference, particle methods, the telescoping sampler.
IV. Models
Linear models, GLMs, hierarchical models, mixtures, state-space models, Gaussian processes, spatial and causal models.
V. Diagnostics
MAP estimation, tuning and calibration, convergence diagnostics, posterior predictive checking.
VI. Software
End-to-end workflows in R, Stan, and JAX, with reproducibility treated as part of the analysis.
VII. Advanced
Model selection and stacking, robustness, experimental design, communication, capstone work.
Where to start
Skim Part I and begin at conjugate analysis. Come back to the measure-theoretic material when a proof depends on it.
Then follow the main line: Bayesian linear models → hierarchical models → MCMC → diagnostics.
Start with conjugate analysis and Bayesian linear models, then go straight to the workflow chapters and fit something.
Return to Part I once the practice has given the theory something to attach to. Most people find it easier in that order.
Pair Part III with the JAX chapter, writing each sampler by hand before using a library version.
The telescoping sampler and the funnel geometry are the two places where implementation detail genuinely changes the statistics.
Read hierarchical and multilevel models. It has the derivations, the decision-theoretic argument, the geometry, and a worked example, so it shows the intended depth better than any description would.
What you need
Multivariable calculus, linear algebra to the level of eigenvalues and quadratic forms, undergraduate probability and statistics, and basic R or Python. No prior exposure to Bayesian methods.
The appendices cover notation, a measure-theoretic refresher, and installation.
Three tools, three purposes
The same model, three ways — this is the eight schools hierarchical model that runs through the course.
Simulation, conjugate calculations, custom samplers, and posterior summaries (R Core Team 2025; Bürkner et al. 2024).
# Partial pooling by hand: the shrinkage factor is the whole idea.
B <- sigma^2 / (sigma^2 + tau^2)
theta_hat <- (1 - B) * y + B * muA mature HMC and NUTS implementation, for hierarchical and nonlinear models (Carpenter et al. 2017; Hoffman and Gelman 2014).
parameters {
real mu;
real<lower=0> tau;
vector[J] z; // standardised group effects
}
transformed parameters {
vector[J] theta = mu + tau * z; // non-centring: no funnel here
}
model {
z ~ std_normal();
y ~ normal(theta, sigma);
}Automatic differentiation and JIT, for chapters where the point is to build the algorithm rather than call it (Bradbury et al. 2018).
# Writing HMC by hand is the fastest way to see what Stan does for you.
grad_logp = jax.grad(log_posterior)
p -= 0.5 * eps * -grad_logp(q) # leapfrog half-step
q += eps * pEvery lab in the repository runs: R in labs/r/, Stan in labs/stan/, Python in labs/jax/.
Licence
Content is CC BY-NC-SA 4.0; code is MIT. Structure follows David (2023); presentation owes much to Kosmidis (2024). Full credits on the instructor page.