Module 4: Bayesian Generalized Linear Models

This module introduces Bayesian generalized linear models (GLMs), focusing on logistic regression as a canonical example. We study non-conjugacy, Laplace approximations, identifiability and separation, and the geometry of the posterior.

1. Logistic Regression Model

Let \(Y_i \in \{0,1\}\) and covariates \(x_i \in \mathbb{R}^p\), collected in design matrix \(X \in \mathbb{R}^{n \times p}\).

1.1 Likelihood

Assume conditionally independent responses with \[ \mathbb{P}(Y_i = 1 \mid x_i, \beta) = \sigma(x_i^\top \beta) := \frac{1}{1 + e^{-x_i^\top \beta}}, \] \[ \mathbb{P}(Y_i = 0 \mid x_i, \beta) = 1 - \sigma(x_i^\top \beta). \]

The log-likelihood is \[ \ell(\beta) = \sum_{i=1}^n \big( y_i x_i^\top \beta - \log(1 + e^{x_i^\top \beta}) \big). \]

1.2 Prior on \(\beta\)

Consider a Gaussian prior \[ \beta \sim N_p(m_0, V_0). \]

The posterior has density \[ \pi(\beta \mid y) \propto \exp\{ \ell(\beta) \} \cdot \exp\Big\{ -\tfrac12 (\beta - m_0)^\top V_0^{-1} (\beta - m_0) \Big\}. \]

2. Non-Conjugacy

Unlike Gaussian linear regression, no finite-dimensional parametric family of priors on \(\beta\) is conjugate to the logistic likelihood.

Proposition 4.1 (Non-Existence of Finite-Dimensional Conjugate Family)

There is no finite-dimensional parametric family of priors \(\{\Pi_\eta\}\) on \(\mathbb{R}^p\) such that for all data \(y\), the posterior \(\Pi(\cdot \mid y)\) remains in \(\{\Pi_\eta\}\).

Proof sketch: Suppose, for contradiction, that such a conjugate family exists with parameter \(\eta \in \mathbb{R}^k\), and that updating from prior \(\eta\) with data \((x_i, y_i)_{i=1}^n\) produces an updated parameter \(T(\eta; x_{1:n}, y_{1:n})\) in the same \(k\)-dimensional space. The (log-)posterior density is \[ \log \pi(\beta \mid y) = \sum_{i=1}^n \big( y_i x_i^\top \beta - \log(1 + e^{x_i^\top \beta}) \big) + \log \pi_\eta(\beta) + C. \] Conjugacy would require that, as a function of \(\beta\), this log-density can be written in the same functional form as \(\log \pi_\eta(\beta)\) with updated hyperparameters depending on the sufficient statistics of the data. In exponential families this means that the dependence on \(\beta\) must be linear in a fixed, finite set of functions of \(\beta\).

However, the logistic log-likelihood introduces the term \[ - \sum_{i=1}^n \log(1 + e^{x_i^\top \beta}), \] whose dependence on \(\beta\) cannot in general be expressed using a fixed finite collection of basis functions independent of the sample size and design (except in trivial cases where the design takes only finitely many patterns and the prior is allowed to depend on them). As \(n\) grows and the \(x_i\) explore more directions, the map \[ \beta \mapsto \big( \log(1 + e^{x_1^\top \beta}), \dots, \log(1 + e^{x_n^\top \beta}) \big) \] generates increasingly rich nonlinear structure, which cannot be encoded by a fixed-dimensional parameter \(\eta\). Thus no finite-dimensional family can absorb the effect of arbitrary data through a finite-dimensional hyperparameter update, and a genuinely conjugate family would have to be infinite-dimensional. \(\square\)

This non-conjugacy motivates approximation methods such as Laplace approximations, variational inference, or MCMC.

3. Laplace Approximation for the Posterior

3.1 General Setup

Let \[ \log \pi(\beta \mid y) = f(\beta) + C, \] where \[ f(\beta) = \ell(\beta) - \tfrac12 (\beta - m_0)^\top V_0^{-1} (\beta - m_0) \] and \(C\) is a constant independent of \(\beta\).

Let \(\hat{\beta}\) denote a maximizer of \(f(\beta)\) (the MAP estimator). Let \[ H(\hat{\beta}) = -\nabla^2 f(\hat{\beta}) \] be the negative Hessian at the mode.

Theorem 4.2 (Laplace Approximation of the Posterior)

Under regularity conditions (twice differentiability of \(f\), strict local maximum at \(\hat{\beta}\), positive-definite Hessian), the posterior can be approximated by \[ \pi(\beta \mid y) \approx \tilde{\pi}(\beta \mid y) := N_p(\hat{\beta}, H(\hat{\beta})^{-1}). \]

Moreover, the marginal likelihood admits the approximation \[ \log m(y) \approx f(\hat{\beta}) + C + \frac{p}{2}\log(2\pi) - \frac12 \log |H(\hat{\beta})|. \]

Proof sketch: Write the (unnormalized) posterior density as \[ \pi(\beta \mid y) \propto \exp\{ f(\beta) \}, \] where \(f\) attains a strict local maximum at \(\hat{\beta}\) and is twice continuously differentiable. A second-order Taylor expansion around \(\hat{\beta}\) yields \[ f(\beta) \approx f(\hat{\beta}) - \tfrac12 (\beta - \hat{\beta})^\top H(\hat{\beta}) (\beta - \hat{\beta}), \] since \(\nabla f(\hat{\beta}) = 0\) and \(H(\hat{\beta}) = -\nabla^2 f(\hat{\beta})\) is positive-definite. Substituting into the exponential gives the Gaussian approximation \[ \pi(\beta \mid y) \approx \exp\{ f(\hat{\beta}) \} \exp\Bigl\{ -\tfrac12 (\beta - \hat{\beta})^\top H(\hat{\beta}) (\beta - \hat{\beta}) \Bigr\}, \] which, after normalization, corresponds to a Normal distribution with mean \(\hat{\beta}\) and covariance \(H(\hat{\beta})^{-1}\).

For the marginal likelihood, \[ m(y) = \int \exp\{ f(\beta) + C \} d\beta, \] Laplace’s method applied to this integral gives \[ \log m(y) \approx f(\hat{\beta}) + C + \frac{p}{2}\log(2\pi) - \frac12 \log |H(\hat{\beta})|, \] where the \(\log |H(\hat{\beta})|\) term arises from the determinant of the quadratic form in the Gaussian integral. \(\square\)

3.2 Hessian of the Logistic Log-Posterior

For logistic regression, the negative Hessian of the log-likelihood is \[ -\nabla^2 \ell(\beta) = X^\top W(\beta) X, \] where \(W(\beta)\) is the diagonal matrix with entries \(w_i(\beta) = \sigma(x_i^\top \beta) (1 - \sigma(x_i^\top \beta))\). Including the Gaussian prior, the posterior negative Hessian is \[ H(\beta) = X^\top W(\beta) X + V_0^{-1}. \]

4. Identifiability and Separation

4.1 Identifiability

The logistic regression model is identifiable under standard conditions when the design matrix \(X\) has full column rank and the covariates span a sufficiently rich subspace.

Definition 4.3 (Complete and Quasi-Complete Separation)

Complete separation: There exists \(\beta\) such that \[ x_i^\top \beta > 0 \quad \text{for all } i \text{ with } y_i=1, \quad x_i^\top \beta < 0 \quad \text{for all } i \text{ with } y_i=0. \]
Quasi-complete separation: Inequalities hold with \(\ge 0\) and \(\le 0\), with at least one equality.

Proposition 4.4 (Separation and MLE)

In the presence of complete or quasi-complete separation, the logistic regression MLE does not exist as a finite vector; some components diverge to \(\pm \infty\).

Proof sketch: Under complete separation, there exists \(\beta\) such that \(x_i^\top \beta > 0\) for all \(y_i = 1\) and \(x_i^\top \beta < 0\) for all \(y_i = 0\). Consider the ray \(t \mapsto t \beta\). Then for \(y_i = 1\), \(x_i^\top (t\beta) \to +\infty\) as \(t \to +\infty\), so \[ \log(1 + e^{x_i^\top (t\beta)}) \sim x_i^\top (t\beta), \] and the contribution \(y_i x_i^\top (t\beta) - \log(1 + e^{x_i^\top (t\beta)})\) tends to 0. For \(y_i = 0\), \(x_i^\top (t\beta) \to -\infty\), so \(\log(1 + e^{x_i^\top (t\beta)}) \to 0\) and the contribution also tends to 0. By perturbing along suitable directions one can construct sequences \(\beta_t\) for which the log-likelihood increases without bound, showing that no finite maximizer exists. Quasi-complete separation is handled by similar arguments, with some components diverging while leaving certain linear predictors approximately constant. \(\square\)

4.2 Impact on the Bayesian Posterior

With a proper prior (e.g., Gaussian), the posterior remains proper even under separation, but may place most mass on large-magnitude coefficients, reflecting near-deterministic classification.

5. Posterior Geometry and Curvature

The geometry of the posterior in logistic regression is governed by the curvature of \(f(\beta)\):

The Hessian \(H(\beta)\) reflects local certainty about \(\beta\).
Near separation, \(W(\beta)\) becomes nearly singular, leading to flat directions and heavy posterior tails.
Laplace approximations may be inaccurate in such regimes, motivating MCMC or more robust approximations.

6. Problem Set 4 (Representative Problems)

Non-Conjugacy. Provide a rigorous argument that no finite-dimensional parametric family of priors on \(\beta\) is conjugate to the logistic likelihood (cf. Proposition @prop-logistic-nonconjugate), except in trivial degeneracies.
Laplace Approximation Derivation. Derive the Laplace approximation for the posterior of \(\beta\) in logistic regression, starting from the second-order Taylor expansion of \(f(\beta)\) and justifying each step in Theorem ?@thm-laplace-logistic. Write down the corresponding approximate marginal likelihood.
Hessian and Curvature. Derive the explicit form of \(H(\hat{\beta})\) for logistic regression with Gaussian prior and discuss how it depends on the data and prior. Show that \(H(\hat{\beta})\) is positive-definite when \(V_0\) is, and interpret this in terms of local posterior geometry.
Separation Example. Construct a small dataset exhibiting complete separation (Definition ?@def-separation). Show that the MLE does not exist, and analyze the form of the posterior under a Gaussian prior, connecting to Proposition @prop-separation-mle. Comment on the accuracy of Laplace approximations in this regime.
Posterior Geometry and Identifiability. Give an example where collinearity in the covariates leads to near non-identifiability of certain directions in \(\beta\). Analyze the posterior covariance structure and discuss how prior information can restore effective identifiability.