Module 1: Foundations of Bayesian Inference

This module develops the measure-theoretic foundations of Bayesian inference.

We treat priors, likelihoods, and posteriors as measures, and formulate Bayes’ theorem as a Radon–Nikodym identity. We then introduce Bayesian decision theory, Bayes estimators, and the Bernstein–von Mises theorem.

1. Probability Spaces and Conditional Expectation

Definition 1.1 (Probability Space)

A probability space is a triple ((, , )) where:

() is a sample space,
() is a ()-algebra of subsets of (),
(: ) is a probability measure with (()=1).

Definition 1.2 (Random Variable)

Let ((, , )) be a probability space and ((S, )) a measurable space. A map \[ X: (\Omega, \mathcal{F}) \to (S, \mathcal{S}) \] is a random variable if it is measurable: for all (A ), (X^{-1}(A) ).

Definition 1.3 (Conditional Expectation)

Let (X L^1(, , )) and ( ) be a sub-()-algebra. A random variable (Y) is called the conditional expectation of (X) given (), written (Y = [X ]), if:

(Y) is ()-measurable, and
For all (G ), \[ \int_G Y \, d\mathbb{P} = \int_G X \, d\mathbb{P}. \]

Theorem 1.1 (Existence and Uniqueness of Conditional Expectation)

Let (X L^1(, , )) and () a sub-()-algebra. Then there exists a ()-measurable random variable (Y) satisfying the defining property of conditional expectation. Moreover, (Y) is unique up to ()-almost sure equality.

Proof sketch: Use the Radon–Nikodym theorem applied to the finite signed measure ((G) = _G X , d) on ().

2. Conditional Probabilities and Regular Conditional Distributions

Definition 1.4 (Regular Conditional Probability)

Let (X: S), (Y: T) be random variables with (S, T) standard Borel spaces. A regular conditional distribution of (Y) given (X) is a Markov kernel \[ K: S \times \mathcal{T} \to [0,1] \] such that for all (B ), \[ \mathbb{P}(Y \in B \mid X)(\omega) = K(X(\omega), B) \quad \text{a.s.} \]

Theorem 1.2 (Existence of Regular Conditional Distributions)

If (S) and (T) are standard Borel spaces, there exists a regular conditional distribution (K(, )) of (Y) given (X).

Proof idea: Use disintegration of measures on product spaces and the existence of regular conditional probabilities on standard Borel spaces.

3. Bayes’ Theorem as Radon–Nikodym Identity

3.1 Setup

Parameter space: ((, )), data space: ((, )), both standard Borel.
Prior: a probability measure () on ((, )).
Likelihood: a Markov kernel (P()) from () to ((, )).

Define the joint measure on () by \[ \mathbb{P}(A \times B) = \int_A P(B \mid \theta)\, \Pi(d\theta), \quad A \in \mathcal{T}, B \in \mathcal{B}. \]

The marginal law of (X) is (_X(B) = (B)).

Definition 1.5 (Posterior as Conditional Measure)

A posterior is a Markov kernel ((x)) from () to ((, )) such that for all (A ) and (B ), \[ \mathbb{P}(A \times B) = \int_B \Pi(A \mid x)\, \mathbb{P}_X(dx). \]

3.2 Absolute Continuity and Densities

Assume that for each (), (P()) is absolutely continuous with respect to a ()-finite measure () on ((, )): \[ P(B \mid \theta) = \int_B p(x \mid \theta)\, \lambda(dx), \] for some nonnegative measurable density (p(x )).

Define the marginal density \[ m(x) = \int_\Theta p(x \mid \theta)\, \Pi(d\theta). \]

Theorem 1.3 (Bayes’ Theorem via Radon–Nikodym)

Assume (0 < m(x) < ) for ()-almost all (x). Then the posterior measure ((x)) exists and is given for ()-a.e. (x) by \[ \Pi(A \mid x) = \frac{\int_A p(x \mid \theta)\, \Pi(d\theta)}{m(x)}, \quad A \in \mathcal{T}. \]

Proof sketch: Consider the joint density (f(,x) = p(x )) with respect to (). The marginal of (X) has density (m(x)). For fixed (x), define a finite measure (_x(A) = _A f(,x), (d)). Then (_x) is absolutely continuous with respect to (), with total mass (m(x)). The Radon–Nikodym derivative (d_x/d) yields the posterior kernel.

Definition 1.6 (Posterior Propriety)

The posterior ((x)) is proper if, for (_X)-almost all (x), ((x)) is a probability measure (total mass 1). This requires (0 < m(x) < ) almost everywhere.

4. Bayesian Decision Theory

4.1 Basic Setup

Parameter space: ().
Action (decision) space: ().
Loss function: (L: [0, )).
Decision rule: measurable map (: ).

Definition 1.7 (Bayes Risk and Bayes Estimator)

Given prior (), the Bayes risk of () is \[ r(\Pi, \delta) = \int_\Theta \int_\mathcal{X} L(\theta, \delta(x))\, P(dx \mid \theta)\, \Pi(d\theta). \]

A decision rule (^*) is a Bayes estimator (w.r.t. ()) if \[ r(\Pi, \delta^*) = \inf_\delta r(\Pi, \delta). \]

Theorem 1.4 (Posterior Expected Loss Representation)

Under integrability conditions permitting Fubini’s theorem, \[ r(\Pi, \delta) = \int_\mathcal{X} \left( \int_\Theta L(\theta, \delta(x))\, \Pi(d\theta \mid x) \right) \mathbb{P}_X(dx). \]

In particular, any Bayes rule (^*) satisfies \[ \delta^*(x) \in \arg\min_{a \in \mathcal{A}} \int_\Theta L(\theta, a)\, \Pi(d\theta \mid x) \quad \text{for } \mathbb{P}_X\text{-almost all } x. \]

Proof: Express (r(,)) using the joint measure on (), then condition on (X) using the posterior kernel.

Corollary 1.5 (Quadratic Loss)

For scalar () and quadratic loss (L(,a)=(-a)^2), any Bayes estimator satisfies \[ \delta^*(x) = \mathbb{E}[\theta \mid x], \] provided the posterior mean exists.

5. Credible Intervals, Confidence Intervals, and Bernstein–von Mises

5.1 Credible vs Confidence Intervals

A ((1-))-credible set (C_(x)) satisfies \[ \Pi(C_\alpha(x) \mid x) \ge 1 - \alpha. \]
A ((1-))-confidence set (C_(X)) satisfies \[ \inf_{\theta \in \Theta} \mathbb{P}_\theta(\theta \in C_\alpha(X)) \ge 1 - \alpha. \]

These notions generally differ for finite samples; they are connected asymptotically by the Bernstein–von Mises theorem.

5.2 Bernstein–von Mises Theorem (Informal Statement)

Consider i.i.d. data (X_1, , X_n) from a parametric family ({P_: ^d}) with true parameter (_0). Under standard regularity conditions (identifiability, smoothness, non-degenerate Fisher information, prior density positive and continuous near (_0)), the posterior distribution of \[ \sqrt{n}(\theta - \hat{\theta}_n) \] converges in total variation to a multivariate normal distribution (N(0, I(_0)^{-1})), where (_n) is the MLE and (I(_0)) is the Fisher information.

Consequence: Bayesian credible sets asymptotically coincide with frequentist confidence sets and have asymptotically correct coverage.

Proof sketch: Use local asymptotic normality of the log-likelihood (LAN) and Laplace approximation for the posterior, combining the likelihood expansion with the prior density near (_0).

6. Subjective vs Objective Bayes

Subjective Bayes: Priors represent personal or expert beliefs about parameters; coherence arguments (e.g. de Finetti) justify Bayesian updating as the unique coherent updating rule.
Objective Bayes: Priors chosen to satisfy formal criteria (invariance, reference priors, Jeffreys priors), aiming to reduce subjectivity while retaining Bayesian coherence.

The course emphasizes the formal measure-theoretic and decision-theoretic structure, applicable in both interpretations.

7. Problem Set 1 (Representative Problems)

Bayes via Radon–Nikodym. Prove Theorem ?@thm-bayes-rn under the given assumptions, explicitly invoking the Radon–Nikodym theorem and checking all measurability requirements.
Posterior Propriety Counterexample. Construct an example of an improper prior such that the resulting posterior is improper. Show that the marginal density (m(x)) is infinite on a set of positive ()-measure.
Bayes Estimators under General Loss. For a general convex loss function (L(,a)), prove that any Bayes estimator must minimize the posterior expected loss (cf. Theorem ?@thm-posterior-loss) and analyze conditions for uniqueness.
Credible vs Confidence Intervals in Normal–Normal Model. In the normal–normal model with known variance, show that the 95% equal-tailed credible interval coincides with the classical 95% confidence interval. Explain why this equivalence is special to the conjugate Gaussian setting.
LAN and BvM in Exponential Families. For a regular exponential family, verify the LAN expansion and outline how it leads to the Bernstein–von Mises conclusion.