Mixture models and latent variable models: heterogeneity, allocation, and label switching

What you should be able to do after this chapter

write finite mixture models as latent allocation models;
derive complete-data likelihoods and conditional updates for simple mixtures;
explain label switching and why naive posterior summaries fail;
distinguish identifiable predictive summaries from component-labeled summaries;
connect HMMs and switching regressions to the broader latent-variable framework.

1 Why mixtures

Mixture models represent heterogeneity by assuming observations arise from one of several latent subpopulations. They are useful for:

clustering;
density estimation;
latent classes;
regime switching;
outlier accommodation;
hidden-state dynamics.

Their flexibility comes with inferential complications: multimodality, weak identifiability, and label switching (Neal 2000; Murphy 2023).

2 Finite mixture model

For $K$ components,

\[ p(y_i \mid \Theta,\pi) = \sum_{k=1}^K \pi_k f(y_i \mid \theta_k), \]

where:

$\pi_k \ge 0$ and $\sum_k \pi_k=1$;
$\theta_k$ are component parameters.

Introduce latent allocations $z_i \in \{1,\dots,K\}$ with

\[ \Pr(z_i=k \mid \pi)=\pi_k, \]

and

\[ y_i \mid z_i=k,\Theta \sim f(y_i \mid \theta_k). \]

Then the observed-data likelihood is recovered by marginalizing over $z_i$.

3 Complete-data likelihood

With latent indicators,

\[ p(y,z \mid \Theta,\pi) = \prod_{i=1}^n \prod_{k=1}^K \left[\pi_k f(y_i \mid \theta_k)\right]^{\mathbf{1}(z_i=k)}. \]

This factorization is the basis of Gibbs sampling, EM algorithms, and many variational approximations.

4 Gaussian mixture example

Suppose

\[ y_i \mid z_i=k,\mu_k,\sigma_k^2 \sim \mathcal{N}(\mu_k,\sigma_k^2). \]

Take priors

\[ \pi \sim \operatorname{Dirichlet}(\alpha_1,\dots,\alpha_K), \]

and conditionally conjugate priors on $(\mu_k,\sigma_k^2)$.

4.1 Allocation conditional

The full conditional for $z_i$ is

\[ \Pr(z_i=k \mid y_i,\Theta,\pi) \propto \pi_k f(y_i \mid \theta_k). \]

Normalized,

\[ \Pr(z_i=k \mid y_i,\Theta,\pi) = \frac{\pi_k f(y_i \mid \theta_k)} {\sum_{\ell=1}^K \pi_\ell f(y_i \mid \theta_\ell)}. \]

This is the Bayesian soft assignment rule.

4.2 Mixing weights conditional

If $n_k=\sum_i \mathbf{1}(z_i=k)$, then

\[ \pi \mid z \sim \operatorname{Dirichlet}(\alpha_1+n_1,\dots,\alpha_K+n_K). \]

4.3 Component parameter conditionals

Under conjugate normal-inverse-gamma priors within each component, the conditional updates reduce to the same formulas seen in ordinary Gaussian conjugate analysis, but now computed separately within each cluster using the observations currently assigned to that component.

5 Label switching

If the prior is symmetric across components, then the posterior is invariant to relabeling:

\[ (\theta_1,\dots,\theta_K,\pi_1,\dots,\pi_K) \mapsto (\theta_{\sigma(1)},\dots,\theta_{\sigma(K)},\pi_{\sigma(1)},\dots,\pi_{\sigma(K)}). \]

This implies the posterior usually has $K!$ symmetric modes.

5.1 Consequence

Naively averaging component-specific parameters across MCMC draws can produce nonsense. For example, posterior means of ordered component means may collapse toward one another even when the clusters are well separated.

5.2 What remains identifiable

Predictive densities,

\[ p(\tilde y \mid y), \]

mixture distribution summaries, and cluster-coassignment probabilities can remain meaningful even when component labels themselves are not.

6 Identifiability strategies

Several strategies exist, each with tradeoffs:

impose ordering constraints such as $\mu_1 < \mu_2 < \cdots < \mu_K$;
use asymmetric priors;
relabel MCMC draws post hoc;
focus on label-invariant summaries.

In teaching and practice, label-invariant summaries are often the safest default because they respect the actual symmetry of the model.

7 Hidden Markov models and switching regressions

Mixture ideas extend naturally to dependent latent states.

7.1 Hidden Markov models

The component indicator becomes a Markov chain:

\[ \Pr(z_t=j \mid z_{t-1}=i)=\Gamma_{ij}. \]

Conditional on the latent state, the observation follows a state-specific emission model.

7.2 Switching regression

A regime-switching model may have

\[ y_t \mid z_t=k \sim \mathcal{N}(x_t^\top \beta_k,\sigma_k^2). \]

Thus each latent state carries its own regression surface. This is a dynamic extension of finite mixtures.

8 Mixture of experts

A mixture-of-experts model replaces constant mixing weights with covariate-dependent gating probabilities:

\[ \Pr(z_i=k \mid x_i)=g_k(x_i). \]

This is powerful, but it magnifies identifiability and computational complexity because both the expert models and the gating network are learned jointly.

9 `R` implementation

The companion lab is mixture_label_switching.R.

It is deliberately simple: a two-component Gaussian mixture with fixed variances and a Gibbs update for allocations. The point is to show how easily label switching appears, even in tiny examples.

10 Diagnostics and pitfalls

10.1 Mistaking multimodality for nonconvergence

Sometimes a sampler is mixing correctly across symmetric modes. The issue is interpretation, not failure.

10.2 Interpreting labels literally

Component labels are often arbitrary without explicit identifying structure.

10.3 Overfitting component count

More mixture components can fit noise, not just structure. Everything in this chapter has assumed $K$ is known, which it rarely is. Treating $K$ as a parameter rather than a setting requires a sampler that can change the length of the parameter vector; the telescoping sampler develops that case in full, along with the distinction between the number of components and the number of components the data actually occupy.

10.4 Ignoring predictive goals

For density estimation or clustering, the most meaningful posterior summaries are often predictive or partition-based rather than parameter-label based.

11 Exercises

Derive the allocation full conditional for a finite mixture.
Show why a symmetric prior implies label-switching invariance.
Explain why posterior means of component parameters can be misleading under label switching.
Compare identifiable and non-identifiable summaries in a two-component Gaussian mixture.
Describe how an HMM differs from an i.i.d. mixture in conditional independence structure.

12 Summary

Mixture and latent-variable models are flexible but subtle:

latent allocations create powerful factorization structure;
complete-data representations simplify computation;
symmetry leads naturally to multimodality and label switching;
predictive and partition summaries are often more meaningful than naive component averages.

13 Further reading

Frühwirth-Schnatter (2006) is the standard book-length treatment of finite mixtures and Markov switching models, and covers identifiability, label switching, and prior choice more thoroughly than any general text.

Gelman et al. (2013) chapter 22 gives a compact account of mixture modelling within a Bayesian workflow, including the practical advice on identifiability constraints. Murphy (2023) chapters 21 and 22 place mixtures in the wider latent-variable setting alongside factor models and EM.

On label switching specifically, Stephens (2000) is the reference for principled relabelling, and Wade and Ghahramani (2018) for summarising the posterior over partitions without relabelling at all — usually the better approach.

When the number of components is itself unknown, the model becomes a mixture of finite mixtures; Neal (2000) covers the Dirichlet process route and the telescoping sampler develops the finite-$K$ alternative in full.

References

Frühwirth-Schnatter, Sylvia. 2006. Finite Mixture and Markov Switching Models. Springer.

Gelman, Andrew, John B. Carlin, Hal S. Stern, David B. Dunson, Aki Vehtari, and Donald B. Rubin. 2013. Bayesian Data Analysis. 3rd ed. CRC Press.

Murphy, Kevin P. 2023. Probabilistic Machine Learning: Advanced Topics. MIT Press.

Neal, Radford M. 2000. “Markov Chain Sampling Methods for Dirichlet Process Mixture Models.” Journal of Computational and Graphical Statistics 9 (2): 249–65.

Stephens, Matthew. 2000. “Dealing with Label Switching in Mixture Models.” Journal of the Royal Statistical Society: Series B 62 (4): 795–809.

Wade, Sara, and Zoubin Ghahramani. 2018. “Bayesian Cluster Analysis: Point Estimation and Credible Balls.” Bayesian Analysis 13 (2): 559–626.

--- title: "Mixture models and latent variable models: heterogeneity, allocation, and label switching" --- ::: {.callout-note} ## What you should be able to do after this chapter - write finite mixture models as latent allocation models; - derive complete-data likelihoods and conditional updates for simple mixtures; - explain label switching and why naive posterior summaries fail; - distinguish identifiable predictive summaries from component-labeled summaries; - connect HMMs and switching regressions to the broader latent-variable framework. ::: ## Why mixtures Mixture models represent heterogeneity by assuming observations arise from one of several latent subpopulations. They are useful for: - clustering; - density estimation; - latent classes; - regime switching; - outlier accommodation; - hidden-state dynamics. Their flexibility comes with inferential complications: multimodality, weak identifiability, and label switching [@neal2000; @murphy2023]. ## Finite mixture model For $K$ components, $$ p(y_i \mid \Theta,\pi) = \sum_{k=1}^K \pi_k f(y_i \mid \theta_k), $$ where: - $\pi_k \ge 0$ and $\sum_k \pi_k=1$; - $\theta_k$ are component parameters. Introduce latent allocations $z_i \in \{1,\dots,K\}$ with $$ \Pr(z_i=k \mid \pi)=\pi_k, $$ and $$ y_i \mid z_i=k,\Theta \sim f(y_i \mid \theta_k). $$ Then the observed-data likelihood is recovered by marginalizing over $z_i$. ## Complete-data likelihood With latent indicators, $$ p(y,z \mid \Theta,\pi) = \prod_{i=1}^n \prod_{k=1}^K \left[\pi_k f(y_i \mid \theta_k)\right]^{\mathbf{1}(z_i=k)}. $$ This factorization is the basis of Gibbs sampling, EM algorithms, and many variational approximations. ## Gaussian mixture example Suppose $$ y_i \mid z_i=k,\mu_k,\sigma_k^2 \sim \mathcal{N}(\mu_k,\sigma_k^2). $$ Take priors $$ \pi \sim \operatorname{Dirichlet}(\alpha_1,\dots,\alpha_K), $$ and conditionally conjugate priors on $(\mu_k,\sigma_k^2)$. ### Allocation conditional The full conditional for $z_i$ is $$ \Pr(z_i=k \mid y_i,\Theta,\pi) \propto \pi_k f(y_i \mid \theta_k). $$ Normalized, $$ \Pr(z_i=k \mid y_i,\Theta,\pi) = \frac{\pi_k f(y_i \mid \theta_k)} {\sum_{\ell=1}^K \pi_\ell f(y_i \mid \theta_\ell)}. $$ This is the Bayesian soft assignment rule. ### Mixing weights conditional If $n_k=\sum_i \mathbf{1}(z_i=k)$, then $$ \pi \mid z \sim \operatorname{Dirichlet}(\alpha_1+n_1,\dots,\alpha_K+n_K). $$ ### Component parameter conditionals Under conjugate normal-inverse-gamma priors within each component, the conditional updates reduce to the same formulas seen in ordinary Gaussian conjugate analysis, but now computed separately within each cluster using the observations currently assigned to that component. ## Label switching If the prior is symmetric across components, then the posterior is invariant to relabeling: $$ (\theta_1,\dots,\theta_K,\pi_1,\dots,\pi_K) \mapsto (\theta_{\sigma(1)},\dots,\theta_{\sigma(K)},\pi_{\sigma(1)},\dots,\pi_{\sigma(K)}). $$ This implies the posterior usually has $K!$ symmetric modes. ### Consequence Naively averaging component-specific parameters across MCMC draws can produce nonsense. For example, posterior means of ordered component means may collapse toward one another even when the clusters are well separated. ### What remains identifiable Predictive densities, $$ p(\tilde y \mid y), $$ mixture distribution summaries, and cluster-coassignment probabilities can remain meaningful even when component labels themselves are not. ## Identifiability strategies Several strategies exist, each with tradeoffs: - impose ordering constraints such as $\mu_1 < \mu_2 < \cdots < \mu_K$; - use asymmetric priors; - relabel MCMC draws post hoc; - focus on label-invariant summaries. In teaching and practice, label-invariant summaries are often the safest default because they respect the actual symmetry of the model. ## Hidden Markov models and switching regressions Mixture ideas extend naturally to dependent latent states. ### Hidden Markov models The component indicator becomes a Markov chain: $$ \Pr(z_t=j \mid z_{t-1}=i)=\Gamma_{ij}. $$ Conditional on the latent state, the observation follows a state-specific emission model. ### Switching regression A regime-switching model may have $$ y_t \mid z_t=k \sim \mathcal{N}(x_t^\top \beta_k,\sigma_k^2). $$ Thus each latent state carries its own regression surface. This is a dynamic extension of finite mixtures. ## Mixture of experts A mixture-of-experts model replaces constant mixing weights with covariate-dependent gating probabilities: $$ \Pr(z_i=k \mid x_i)=g_k(x_i). $$ This is powerful, but it magnifies identifiability and computational complexity because both the expert models and the gating network are learned jointly. ## `R` implementation The companion lab is [mixture_label_switching.R](../../labs/r/mixture_label_switching.R). It is deliberately simple: a two-component Gaussian mixture with fixed variances and a Gibbs update for allocations. The point is to show how easily label switching appears, even in tiny examples. ## Diagnostics and pitfalls ### Mistaking multimodality for nonconvergence Sometimes a sampler is mixing correctly across symmetric modes. The issue is interpretation, not failure. ### Interpreting labels literally Component labels are often arbitrary without explicit identifying structure. ### Overfitting component count More mixture components can fit noise, not just structure. Everything in this chapter has assumed $K$ is known, which it rarely is. Treating $K$ as a parameter rather than a setting requires a sampler that can change the length of the parameter vector; [the telescoping sampler](../computation/telescoping-sampler.qmd) develops that case in full, along with the distinction between the number of components and the number of components the data actually occupy. ### Ignoring predictive goals For density estimation or clustering, the most meaningful posterior summaries are often predictive or partition-based rather than parameter-label based. ## Exercises 1. Derive the allocation full conditional for a finite mixture. 2. Show why a symmetric prior implies label-switching invariance. 3. Explain why posterior means of component parameters can be misleading under label switching. 4. Compare identifiable and non-identifiable summaries in a two-component Gaussian mixture. 5. Describe how an HMM differs from an i.i.d. mixture in conditional independence structure. ## Summary Mixture and latent-variable models are flexible but subtle: - latent allocations create powerful factorization structure; - complete-data representations simplify computation; - symmetry leads naturally to multimodality and label switching; - predictive and partition summaries are often more meaningful than naive component averages. ## Further reading @fruhwirth2006 is the standard book-length treatment of finite mixtures and Markov switching models, and covers identifiability, label switching, and prior choice more thoroughly than any general text. @gelman2013 chapter 22 gives a compact account of mixture modelling within a Bayesian workflow, including the practical advice on identifiability constraints. @murphy2023 chapters 21 and 22 place mixtures in the wider latent-variable setting alongside factor models and EM. On label switching specifically, @stephens2000 is the reference for principled relabelling, and @wade2018 for summarising the posterior over partitions without relabelling at all — usually the better approach. When the number of components is itself unknown, the model becomes a mixture of finite mixtures; @neal2000 covers the Dirichlet process route and [the telescoping sampler](../computation/telescoping-sampler.qmd) develops the finite-$K$ alternative in full.