Part I. Foundations of probability and Bayesian thinking

Part i

This part establishes the mathematical language used everywhere else in the course: probability spaces, expectations, likelihoods, Bayes’ theorem, exchangeability, and sequential learning.

Part overview

Part I establishes the language in which the rest of the course is written. The purpose is not to turn the site into a full measure-theory text, but to make the probabilistic objects used throughout Bayesian statistics precise: sample spaces, conditional distributions, random variables, likelihoods, priors, posteriors, and predictive distributions.

Part learning goals

By the end of Part I, learners should be able to:

distinguish probability, likelihood, and posterior statements;
write probability models with explicit conditioning structure;
derive posterior distributions from first principles;
explain exchangeability as a Bayesian structural assumption;
connect sequential updating to recursive learning.

Fully written chapters in this repository

Reading path

Recommended order:

Chapter 1. Probability foundations

Core question

What probabilistic language is required to express Bayesian models rigorously and usefully?

Section blueprint

Sample spaces, events, $\sigma$-algebras, and probability measures.
Random variables as measurable functions.
Distributions, densities, and cumulative distribution functions.
Expectation, variance, covariance, and conditional expectation.
Independence, conditional independence, and factorization.
Laws of large numbers and the central limit theorem.
Transformations of variables and Jacobians.
Common discrete and continuous distributions.

Derivations that must appear

Law of the unconscious statistician.
Tower property of conditional expectation.
Variance decomposition: \[ \operatorname{Var}(X)=\mathbb{E}\!\left[\operatorname{Var}(X\mid Y)\right]+\operatorname{Var}\!\left(\mathbb{E}[X\mid Y]\right). \]
Change-of-variables formula in one and several dimensions.
LLN and CLT illustrations with Bernoulli and Gaussian examples.

Computational lab

Simulate LLN and CLT behavior in R.
Empirically compare independent and merely uncorrelated variables.
Visualize transformed distributions and Jacobian effects.

Common pitfalls

Treating densities as probabilities.
Confusing independence with zero covariance.
Ignoring support transformations when writing priors on constrained parameters.
Using conditional expectation as a pointwise substitution rather than a random variable.

Exercises

Prove the tower property in a finite discrete setting.
Derive the expectation and variance of a transformed random variable.
Simulate the CLT for skewed distributions and compare finite-sample behavior.

Core references

Kolmogorov (1956)
Durrett (2019)
Casella and Berger (2002)

Full chapter

Probability foundations

Chapter 2. Likelihood theory

Core question

How does the likelihood summarize information in the sampling model, and how does it differ conceptually from probability and posterior inference?

Section blueprint

Definition of the likelihood function.
Sufficiency and identifiability.
Score function and Fisher information.
Maximum likelihood estimation and invariance.
Profile likelihood and nuisance parameters.
Likelihood asymptotics and asymptotic normality.
Comparison between likelihood-based and posterior-based uncertainty.

Derivations that must appear

Score and observed information for Bernoulli, Poisson, and Gaussian examples.
MLE derivations for normal mean/variance and Poisson rate.
Second-order Taylor expansion leading to asymptotic normality.
Connection between curvature of the log-likelihood and information.

Computational lab

Plot a likelihood surface and log-likelihood curvature in R.
Compare profile likelihood intervals with Bayesian credible intervals in a one-parameter model.
Explore identifiability failures in a simple mixture model.

Common pitfalls

Treating the likelihood as a probability distribution over $\theta$.
Forgetting that different models can imply the same likelihood shape locally.
Overinterpreting asymptotic normality in weakly identified or small-sample settings.

Exercises

Derive Fisher information for a Binomial model.
Show why reparameterization preserves MLEs but changes curvature.
Compare posterior mode and MLE under weakly informative priors.

Core references

Casella and Berger (2002)
Vaart (1998)
Berger (1985)

Full chapter

Likelihood theory

Chapter 3. Bayes’ theorem and the Bayesian paradigm

Core question

How do prior information, sampling assumptions, and observed data combine into posterior and predictive inference?

Section blueprint

Discrete and continuous forms of Bayes’ theorem.
Prior, likelihood, posterior, and evidence.
Posterior predictive distribution.
Sequential updating.
Exchangeability and de Finetti intuition.
Subjective, objective, and pragmatic Bayesian positions.
Bayes factors and the role of marginal likelihood.

Derivations that must appear

Posterior factorization from the product rule.
Sequential identity \[ p(\theta \mid y_{1:t}) \propto p(y_t \mid \theta)\, p(\theta \mid y_{1:t-1}). \]
Posterior predictive integral \[ p(\tilde y \mid y)=\int p(\tilde y\mid \theta)p(\theta\mid y)\, d\theta. \]
Beta-Binomial worked example as the first full posterior derivation.

Computational lab

Conjugate updating in R.
Grid approximation for a one-parameter posterior.
Sequential updating with one observation at a time.

Common pitfalls

Confusing evidence with posterior probability of a model.
Using improper priors in settings where Bayes factors are required.
Treating exchangeability as literal independence.

Exercises

Derive posterior and predictive forms for a Bernoulli-Beta model.
Compare two priors with the same likelihood and explain the posterior differences.
Show algebraically that batch updating and sequential updating agree.

Core references

Bernardo and Smith (1994)
Jaynes (2003)
Gelman et al. (2013)
Finetti (1974)

Full sample chapter

The fully written version of this chapter is available here:

Bayes’ theorem, sequential updating, and posterior prediction

References

Berger, James O. 1985. Statistical Decision Theory and Bayesian Analysis. 2nd ed. Springer.

Bernardo, José M., and Adrian F. M. Smith. 1994. Bayesian Theory. Wiley.

Casella, George, and Roger L. Berger. 2002. Statistical Inference. 2nd ed. Duxbury.

Durrett, Rick. 2019. Probability: Theory and Examples. 5th ed. Cambridge University Press.

Finetti, Bruno de. 1974. “Theory of Probability: A Critical Introductory Treatment.” Wiley Series in Probability and Mathematical Statistics 1.

Gelman, Andrew, John B. Carlin, Hal S. Stern, David B. Dunson, Aki Vehtari, and Donald B. Rubin. 2013. Bayesian Data Analysis. 3rd ed. CRC Press.

Jaynes, E. T. 2003. Probability Theory: The Logic of Science. Cambridge University Press.

Kolmogorov, A. N. 1956. Foundations of the Theory of Probability. Chelsea Publishing Company.

Vaart, A. W. van der. 1998. Asymptotic Statistics. Cambridge University Press.

--- title: "Part I. Foundations of probability and Bayesian thinking" number-sections: false --- ::: {.part-hero} ## Part i This part establishes the mathematical language used everywhere else in the course: probability spaces, expectations, likelihoods, Bayes' theorem, exchangeability, and sequential learning. ::: ## Part overview Part I establishes the language in which the rest of the course is written. The purpose is not to turn the site into a full measure-theory text, but to make the probabilistic objects used throughout Bayesian statistics precise: sample spaces, conditional distributions, random variables, likelihoods, priors, posteriors, and predictive distributions. ## Part learning goals By the end of Part I, learners should be able to: - distinguish probability, likelihood, and posterior statements; - write probability models with explicit conditioning structure; - derive posterior distributions from first principles; - explain exchangeability as a Bayesian structural assumption; - connect sequential updating to recursive learning. ## Fully written chapters in this repository - [Probability foundations](../chapters/foundations/probability-foundations.qmd) - [Likelihood theory](../chapters/foundations/likelihood-theory.qmd) - [Bayes' theorem, sequential updating, and posterior prediction](../chapters/foundations/bayes-theorem-and-updating.qmd) ## Reading path Recommended order: 1. [Probability foundations](../chapters/foundations/probability-foundations.qmd) 2. [Likelihood theory](../chapters/foundations/likelihood-theory.qmd) 3. [Bayes' theorem, sequential updating, and posterior prediction](../chapters/foundations/bayes-theorem-and-updating.qmd) ## Chapter 1. Probability foundations ### Core question What probabilistic language is required to express Bayesian models rigorously and usefully? ### Section blueprint - Sample spaces, events, $\sigma$-algebras, and probability measures. - Random variables as measurable functions. - Distributions, densities, and cumulative distribution functions. - Expectation, variance, covariance, and conditional expectation. - Independence, conditional independence, and factorization. - Laws of large numbers and the central limit theorem. - Transformations of variables and Jacobians. - Common discrete and continuous distributions. ### Derivations that must appear - Law of the unconscious statistician. - Tower property of conditional expectation. - Variance decomposition: $$ \operatorname{Var}(X)=\mathbb{E}\!\left[\operatorname{Var}(X\mid Y)\right]+\operatorname{Var}\!\left(\mathbb{E}[X\mid Y]\right). $$ - Change-of-variables formula in one and several dimensions. - LLN and CLT illustrations with Bernoulli and Gaussian examples. ### Computational lab - Simulate LLN and CLT behavior in `R`. - Empirically compare independent and merely uncorrelated variables. - Visualize transformed distributions and Jacobian effects. ### Common pitfalls - Treating densities as probabilities. - Confusing independence with zero covariance. - Ignoring support transformations when writing priors on constrained parameters. - Using conditional expectation as a pointwise substitution rather than a random variable. ### Exercises - Prove the tower property in a finite discrete setting. - Derive the expectation and variance of a transformed random variable. - Simulate the CLT for skewed distributions and compare finite-sample behavior. ### Core references - @kolmogorov1956 - @durrett2019 - @casella2002 ### Full chapter - [Probability foundations](../chapters/foundations/probability-foundations.qmd) ## Chapter 2. Likelihood theory ### Core question How does the likelihood summarize information in the sampling model, and how does it differ conceptually from probability and posterior inference? ### Section blueprint - Definition of the likelihood function. - Sufficiency and identifiability. - Score function and Fisher information. - Maximum likelihood estimation and invariance. - Profile likelihood and nuisance parameters. - Likelihood asymptotics and asymptotic normality. - Comparison between likelihood-based and posterior-based uncertainty. ### Derivations that must appear - Score and observed information for Bernoulli, Poisson, and Gaussian examples. - MLE derivations for normal mean/variance and Poisson rate. - Second-order Taylor expansion leading to asymptotic normality. - Connection between curvature of the log-likelihood and information. ### Computational lab - Plot a likelihood surface and log-likelihood curvature in `R`. - Compare profile likelihood intervals with Bayesian credible intervals in a one-parameter model. - Explore identifiability failures in a simple mixture model. ### Common pitfalls - Treating the likelihood as a probability distribution over $\theta$. - Forgetting that different models can imply the same likelihood shape locally. - Overinterpreting asymptotic normality in weakly identified or small-sample settings. ### Exercises - Derive Fisher information for a Binomial model. - Show why reparameterization preserves MLEs but changes curvature. - Compare posterior mode and MLE under weakly informative priors. ### Core references - @casella2002 - @vandervaart1998 - @berger1985 ### Full chapter - [Likelihood theory](../chapters/foundations/likelihood-theory.qmd) ## Chapter 3. Bayes' theorem and the Bayesian paradigm ### Core question How do prior information, sampling assumptions, and observed data combine into posterior and predictive inference? ### Section blueprint - Discrete and continuous forms of Bayes' theorem. - Prior, likelihood, posterior, and evidence. - Posterior predictive distribution. - Sequential updating. - Exchangeability and de Finetti intuition. - Subjective, objective, and pragmatic Bayesian positions. - Bayes factors and the role of marginal likelihood. ### Derivations that must appear - Posterior factorization from the product rule. - Sequential identity $$ p(\theta \mid y_{1:t}) \propto p(y_t \mid \theta)\, p(\theta \mid y_{1:t-1}). $$ - Posterior predictive integral $$ p(\tilde y \mid y)=\int p(\tilde y\mid \theta)p(\theta\mid y)\, d\theta. $$ - Beta-Binomial worked example as the first full posterior derivation. ### Computational lab - Conjugate updating in `R`. - Grid approximation for a one-parameter posterior. - Sequential updating with one observation at a time. ### Common pitfalls - Confusing evidence with posterior probability of a model. - Using improper priors in settings where Bayes factors are required. - Treating exchangeability as literal independence. ### Exercises - Derive posterior and predictive forms for a Bernoulli-Beta model. - Compare two priors with the same likelihood and explain the posterior differences. - Show algebraically that batch updating and sequential updating agree. ### Core references - @bernardo1994 - @jaynes2003 - @gelman2013 - @definetti1974 ### Full sample chapter The fully written version of this chapter is available here: - [Bayes' theorem, sequential updating, and posterior prediction](../chapters/foundations/bayes-theorem-and-updating.qmd)