Module 2: Conjugate Models and Exact Inference

This module develops the theory of conjugate priors, with a focus on exponential family models and exact posterior calculations. We treat conjugacy as a structural property of prior–likelihood pairs and derive posterior and predictive distributions, as well as marginal likelihoods and Bayes factors.

1. Exponential Family Models

Definition 2.1 (Canonical Exponential Family)

Let ((, , )) be a measure space. A family of densities ({p_: }) with respect to () is a (canonical) exponential family if there exist:

a measurable statistic (T: ^k),
a natural parameter space (^k),
a log-partition function (A: ),
a base density (h: [0,)),

such that \[ p_\theta(x) = h(x) \exp\{ \theta^\top T(x) - A(\theta) \}, \quad \theta \in \Theta, \; x \in \mathcal{X}. \]

We assume (A() < ) for () and that () is open.

Proposition 2.2 (Moments via Log-Partition Function)

Assume differentiability of (A) and sufficient regularity to justify differentiating under the integral sign (e.g., dominated convergence conditions). Then \[ \nabla A(\theta) = \mathbb{E}_\theta[T(X)], \quad \nabla^2 A(\theta) = \operatorname{Var}_\theta(T(X)). \]

Proof: By definition of (A), the normalizing identity is \[ \int_\mathcal{X} h(x) \exp\{ \theta^\top T(x) - A(\theta) \}\, \lambda(dx) = 1, \quad \theta \in \Theta. \] Differentiate both sides with respect to (j), the (j)-th coordinate of (). The left-hand side gives \[ \int_\mathcal{X} h(x) \exp\{ \theta^\top T(x) - A(\theta) \}\, (T_j(x) - \partial_{\theta_j} A(\theta))\, \lambda(dx) = 0, \] since the derivative of (-A()) contributes (-{j} A()). Recognizing the density (p(x) = h(x) { ^T(x) - A() }), we rewrite this as \[ \int_\mathcal{X} (T_j(x) - \partial_{\theta_j} A(\theta))\, p_\theta(x)\, \lambda(dx) = 0, \] so \[ \partial_{\theta_j} A(\theta) = \int_\mathcal{X} T_j(x) p_\theta(x)\, \lambda(dx) = \mathbb{E}_\theta[T_j(X)]. \] This yields (A() = ). Differentiating once more gives \[ \partial_{\theta_k} \partial_{\theta_j} A(\theta) = \partial_{\theta_k} \mathbb{E}_\theta[T_j(X)] = \int_\mathcal{X} T_j(x) (T_k(x) - \partial_{\theta_k} A(\theta))\, p_\theta(x)\, \lambda(dx). \] Substituting ({k} A() = ) yields \[ \partial_{\theta_k} \partial_{\theta_j} A(\theta) = \mathbb{E}_\theta[T_j(X) T_k(X)] - \mathbb{E}_\theta[T_j(X)]\, \mathbb{E}_\theta[T_k(X)], \] which is the ((j,k))-entry of the covariance matrix of (T(X)). Hence (^2 A() = _(T(X))). ()

2. Conjugate Priors in Exponential Families

Definition 2.3 (Conjugate Prior)

Let ({P_: }) be a parametric family of distributions on ((, )). A family of priors ({{}: H}) on ((, )) is conjugate for ({P}) if for any prior ({}) and observed data (x), the posterior ((x)) is again in the same family, i.e. equals ({‘}) for some updated hyperparameter (’).

Theorem 2.4 (General Conjugate Prior for Exponential Families)

Consider an exponential family (p_(x) = h(x){^T(x) - A()}). For hyperparameters (_0 ^k) and (_0 > 0), define a prior density on () with respect to some reference measure () by \[ \pi(\theta \mid \eta_0, \tau_0) \propto \exp\{ \eta_0^\top \theta - \tau_0 A(\theta) \}. \]

Let (X_1, , X_n) be conditionally i.i.d. given () from (p_). Then the posterior density is \[ \pi(\theta \mid x_{1:n}) \propto \exp\big\{ (\eta_0 + \textstyle\sum_{i=1}^n T(x_i))^\top \theta - (\tau_0 + n) A(\theta) \big\}, \] i.e. the same functional form with updated hyperparameters \[ \eta_n = \eta_0 + \sum_{i=1}^n T(x_i), \quad \tau_n = \tau_0 + n. \]

Proof: The prior density is \[ \pi(\theta \mid \eta_0, \tau_0) \propto \exp\{ \eta_0^\top \theta - \tau_0 A(\theta) \}. \] The likelihood for conditionally i.i.d. observations is \[ \prod_{i=1}^n p_\theta(x_i) = \prod_{i=1}^n h(x_i) \exp\{ \theta^\top T(x_i) - A(\theta) \} = \Bigl( \prod_{i=1}^n h(x_i) \Bigr) \exp\Bigl\{ \theta^\top \sum_{i=1}^n T(x_i) - n A(\theta) \Bigr\}. \] The joint density of ((, x_{1:n})), up to a factor depending only on (x_{1:n}), is \[ \pi(\theta \mid \eta_0, \tau_0) \prod_{i=1}^n p_\theta(x_i) \propto \exp\Bigl\{ \eta_0^\top \theta - \tau_0 A(\theta) + \theta^\top \sum_{i=1}^n T(x_i) - n A(\theta) \Bigr\}. \] Grouping terms in () and (A()) gives \[ \eta_0^\top \theta + \theta^\top \sum_{i=1}^n T(x_i) = (\eta_0 + \textstyle\sum_{i=1}^n T(x_i))^\top \theta, \] and \[ -\tau_0 A(\theta) - n A(\theta) = - (\tau_0 + n) A(\theta). \] Thus the posterior kernel has the claimed exponential family form with updated hyperparameters \[ \eta_n = \eta_0 + \sum_{i=1}^n T(x_i), \qquad \tau_n = \tau_0 + n. \] Finiteness of the normalizing constant (posterior propriety) follows from suitable integrability conditions on (A), (_0), and (_0). ()

Assumptions

() open and (A) finite on ().
Prior normalizing constant finite (posterior propriety requires further integrability conditions).
Identifiability of the parameter () from (P_) for interpretability of posterior updates.

3. Classical Conjugate Pairs

3.1 Beta–Binomial Model

Likelihood: (X p (n,p)), \[ \mathbb{P}(X=x \mid p) = \binom{n}{x} p^x(1-p)^{n-x}, \quad x=0,1,\dots,n. \]
Prior: (p (, )), density \[ \pi(p) = \frac{1}{B(\alpha, \beta)} p^{\alpha-1}(1-p)^{\beta-1}, \quad 0<p<1. \]

Theorem 2.5 (Beta–Binomial Conjugacy). {#thm-beta-binomial}

The posterior is \[ p \mid x \sim \operatorname{Beta}(\alpha + x, \beta + n - x). \]

Proof: The likelihood is \[ \mathbb{P}(X=x \mid p) = \binom{n}{x} p^x (1-p)^{n-x}, \] and the prior density is \[ \pi(p) = \frac{1}{B(\alpha, \beta)} p^{\alpha-1}(1-p)^{\beta-1}, \quad 0<p<1. \] Bayes’ theorem gives the posterior density (up to a normalizing constant) as \[ \pi(p \mid x) \propto \mathbb{P}(X=x \mid p) \, \pi(p) \propto p^x (1-p)^{n-x} \, p^{\alpha-1} (1-p)^{\beta-1} = p^{\alpha + x - 1} (1-p)^{\beta + n - x - 1}. \] This is exactly the kernel of a ((+ x, + n - x)) density, so after normalization we obtain the stated result. ()

Posterior Predictive Distribution. For a new count (X_{} p (m,p)), the posterior predictive mass function is \[ \mathbb{P}(X_{\text{new}} = k \mid x) = \int_0^1 \binom{m}{k} p^k(1-p)^{m-k} \, \operatorname{Beta}(p \mid \alpha+x, \beta + n - x)\, dp. \] This integral evaluates to a Beta–Binomial distribution; compute explicitly using Beta function identities.

3.2 Gamma–Poisson Model

Likelihood: (X_i ()) i.i.d., ((X_i=k ) = e^{-} ^k / k!).
Prior: ((, )) (shape–rate), density \[ \pi(\lambda) = \frac{\beta^{\alpha}}{\Gamma(\alpha)} \lambda^{\alpha-1} e^{-\beta \lambda}, \quad \lambda>0. \]

Theorem 2.6 (Gamma–Poisson Conjugacy). {#thm-gamma-poisson}

If (X_1,,X_n ) are i.i.d. Poisson, the posterior is \[ \lambda \mid x_{1:n} \sim \operatorname{Gamma}\Big(\alpha + \sum_{i=1}^n x_i,\; \beta + n\Big). \]

Proof: The joint likelihood is \[ \prod_{i=1}^n \mathbb{P}(X_i = x_i \mid \lambda) = \prod_{i=1}^n e^{-\lambda} \frac{\lambda^{x_i}}{x_i!} = e^{-n\lambda} \lambda^{\sum_{i=1}^n x_i} \prod_{i=1}^n \frac{1}{x_i!}. \] The prior density is \[ \pi(\lambda) = \frac{\beta^{\alpha}}{\Gamma(\alpha)} \lambda^{\alpha-1} e^{-\beta \lambda}, \quad \lambda>0. \] Thus the posterior kernel is \[ \pi(\lambda \mid x_{1:n}) \propto e^{-n\lambda} \lambda^{\sum x_i} \, \lambda^{\alpha-1} e^{-\beta \lambda} = \lambda^{\alpha + \sum x_i - 1} e^{-(\beta + n) \lambda}, \quad \lambda>0, \] which is exactly the kernel of a ((+ x_i, + n)) density. Normalization gives the stated posterior. ()

Posterior Predictive. For a future count (X_{} ()), the posterior predictive mass function is \[ \mathbb{P}(X_{\text{new}} = k \mid x_{1:n}) = \int_0^\infty \mathbb{P}(X_{\text{new}} = k \mid \lambda) \, \pi(\lambda \mid x_{1:n})\, d\lambda, \] which can be evaluated in closed form to yield a Negative Binomial distribution by using the Gamma function identity \[ \int_0^\infty \lambda^{a-1} e^{-b\lambda} d\lambda = \frac{\Gamma(a)}{b^a}. \]

3.3 Normal–Normal Models

We treat two cases:

Known variance (^2).
Unknown variance with inverse-gamma prior.

Known Variance

Likelihood: (Y_i N(, ^2)) i.i.d.
Prior: (N(m_0, v_0)).

Theorem 2.7 (Normal–Normal Conjugacy, Known Variance). {#thm-normal-normal-knownvar}

The posterior for () is \[ \mu \mid y_{1:n} \sim N(m_n, v_n), \] where \[ v_n^{-1} = v_0^{-1} + \frac{n}{\sigma^2}, \quad m_n = v_n \Big( v_0^{-1} m_0 + \frac{n \bar{y}}{\sigma^2} \Big). \]

Proof: The likelihood is \[ p(y_{1:n} \mid \mu) = \prod_{i=1}^n \frac{1}{\sqrt{2\pi \sigma^2}} \exp\Bigl\{ -\frac{1}{2\sigma^2} (y_i - \mu)^2 \Bigr\} \propto \exp\Bigl\{ -\frac{1}{2\sigma^2} \sum_{i=1}^n (y_i - \mu)^2 \Bigr\}. \] The prior density is \[ \pi(\mu) \propto \exp\Bigl\{ -\frac{1}{2 v_0} (\mu - m_0)^2 \Bigr\}. \] The posterior kernel is therefore \[ \pi(\mu \mid y_{1:n}) \propto \exp\Bigl\{ -\frac{1}{2\sigma^2} \sum_{i=1}^n (y_i - \mu)^2 - \frac{1}{2 v_0} (\mu - m_0)^2 \Bigr\}. \] Expand the quadratic terms in (). We have \[ \sum_{i=1}^n (y_i - \mu)^2 = \sum_{i=1}^n (y_i^2 - 2 y_i \mu + \mu^2) = \sum_{i=1}^n y_i^2 - 2 \mu \sum_{i=1}^n y_i + n \mu^2. \] Similarly, \[ (\mu - m_0)^2 = \mu^2 - 2 m_0 \mu + m_0^2. \] Collecting coefficients of (^2) and () in the exponent, the posterior kernel is proportional to \[ \exp\Bigl\{ -\tfrac{1}{2} \Bigl( \Bigl( \tfrac{n}{\sigma^2} + \tfrac{1}{v_0} \Bigr) \mu^2 - 2 \Bigl( \tfrac{\sum y_i}{\sigma^2} + \tfrac{m_0}{v_0} \Bigr) \mu + \text{const} \Bigr) \Bigr\}. \] Completing the square, \[ \Bigl( \tfrac{n}{\sigma^2} + \tfrac{1}{v_0} \Bigr) \mu^2 - 2 \Bigl( \tfrac{\sum y_i}{\sigma^2} + \tfrac{m_0}{v_0} \Bigr) \mu = v_n^{-1} (\mu - m_n)^2 + \text{const}, \] where \[ v_n^{-1} = v_0^{-1} + \frac{n}{\sigma^2}, \qquad m_n = v_n \Bigl( v_0^{-1} m_0 + \frac{n \bar{y}}{\sigma^2} \Bigr), \] with ({y} = n^{-1} y_i). Thus the posterior is Normal with mean (m_n) and variance (v_n). ()

4. Marginal Likelihood and Bayes Factors

Definition 2.8 (Marginal Likelihood)

Given prior () and likelihood (p(x )), the marginal likelihood (or evidence) is \[ m(x) = \int_\Theta p(x \mid \theta)\, \Pi(d\theta). \]

This normalizing constant appears in Bayes’ theorem and is central to model comparison.

Definition 2.9 (Bayes Factor)

For two models (M_1, M_2) with priors (_1, _2) and marginal likelihoods (m_1(x), m_2(x)), the Bayes factor in favor of (M_1) over (M_2) is \[ BF_{12}(x) = \frac{m_1(x)}{m_2(x)}. \]

Example 2.10 (Beta–Binomial Model Comparison)

Consider two Beta priors on the same Binomial likelihood, ((_1, _1)) and ((_2, _2)). Compute the Bayes factor between the corresponding models by integrating the Binomial likelihood under each prior, using Beta function identities.

5. Assumptions and Discussion

Proper priors and propriety of posteriors: Conjugate forms give closed-form kernels, but proper posteriors require integrability conditions on hyperparameters (e.g., (,>0) in Beta–Binomial).
Identifiability: Exponential family structure typically yields identifiability, but degenerate designs or constraints can break it.
Interpretability of hyperparameters: Conjugate priors often admit clear interpretations (e.g., Beta parameters as pseudo-counts), useful for elicitation and sensitivity analysis.

6. Problem Set 2 (Representative Problems)

General Conjugacy in Exponential Families. Prove Theorem ?@thm-expfam-conjugate in full detail, including the case of multiple observations and verifying all measurability conditions.
Beta–Binomial Predictive Distribution. Starting from Theorem ?@thm-beta-binomial, derive the Beta–Binomial posterior predictive distribution explicitly and compute its mean and variance. Compare these to the plug-in predictive distribution using the posterior mean of (p).
Gamma–Poisson Predictive and Overdispersion. Using Theorem ?@thm-gamma-poisson, show that the Gamma–Poisson predictive distribution is Negative Binomial and analyze its overdispersion relative to a Poisson with parameter equal to the posterior mean of ().
Normal–Normal with Unknown Variance. For the Normal model with unknown variance and conjugate Normal–Inverse-Gamma prior, derive the joint posterior of ((,^2)) and the marginal posterior of (). Relate your expression for the marginal posterior of () to the known-variance result in Theorem ?@thm-normal-normal-knownvar, and identify the Student-t predictive distribution for future observations.
Bayes Factors with Conjugate Priors. In a normal mean testing problem with known variance, compute analytically the Bayes factor for testing (H_0: ) vs (H_1: ) using a Normal prior under (H_1). Compare the resulting evidence measure to the classical z-test p-value.