Bayes’ theorem, sequential updating, and posterior prediction

1 Chapter position

This chapter is the fully written exemplar for Part I. It shows the intended style for the rest of the course: intuition, formal structure, derivations, worked examples, code, diagnostics, and exercises in one integrated chapter.

What you should be able to do after this chapter

After working through this chapter, you should be able to:

derive Bayes’ theorem in both discrete and density-based settings;
explain the roles of prior, likelihood, posterior, and marginal likelihood;
derive posterior predictive distributions explicitly;
show why sequential updating gives the same result as batch updating;
interpret exchangeability as a structural justification for Bayesian modeling.

2 Why Bayes’ theorem matters

At a high level, Bayesian inference answers the following question:

Given a probabilistic model for data and a prior distribution over unknown parameters, how should uncertainty about the parameters change after observing data?

Bayes’ theorem is the algebraic identity that answers this question. But in serious statistical work, it is more than an identity. It is the organizing rule that connects:

scientific assumptions encoded in the prior;
data-generating assumptions encoded in the likelihood;
inferential uncertainty encoded in the posterior;
out-of-sample reasoning encoded in the posterior predictive distribution.

The theorem itself is elementary. The difficulty lies in understanding what its pieces mean and how they behave in realistic models (Bernardo and Smith 1994; Gelman et al. 2013).

3 Formal setup

Suppose:

$\theta \in \Theta$ denotes an unknown parameter;
$y \in \mathcal{Y}$ denotes observed data;
$\pi(\theta)$ is a prior density or mass function for $\theta$;
$p(y \mid \theta)$ is the sampling model or likelihood;
$p(\theta \mid y)$ is the posterior.

The joint distribution factors as

\[ p(\theta, y) = p(y \mid \theta)\pi(\theta). \]

Marginalizing over $\theta$ yields the evidence or marginal likelihood

\[ p(y) = \int_{\Theta} p(y \mid \theta)\pi(\theta)\, d\theta, \]

assuming the integral is finite.

4 Bayes’ theorem

Theorem 1 If $p(y) > 0$, then the posterior density or mass function is

\[ p(\theta \mid y) = \frac{p(y \mid \theta)\pi(\theta)}{p(y)} = \frac{p(y \mid \theta)\pi(\theta)}{\int_{\Theta} p(y \mid \vartheta)\pi(\vartheta)\, d\vartheta}. \]

4.1 Derivation

Start from the product rule:

\[ p(\theta, y) = p(\theta \mid y)p(y). \]

But the same joint distribution can also be written as

\[ p(\theta, y) = p(y \mid \theta)\pi(\theta). \]

Equating the two factorizations gives

\[ p(\theta \mid y)p(y) = p(y \mid \theta)\pi(\theta), \]

and therefore

\[ p(\theta \mid y) = \frac{p(y \mid \theta)\pi(\theta)}{p(y)}. \]

Finally, since

\[ p(y)=\int_{\Theta} p(y \mid \vartheta)\pi(\vartheta)\, d\vartheta, \]

we obtain the usual normalized form.

4.2 Why the denominator matters

Many computational methods work with the posterior only up to proportionality:

\[ p(\theta \mid y) \propto p(y \mid \theta)\pi(\theta). \]

That simplification is useful for MCMC and optimization, but it can hide an important conceptual fact: the denominator is not decorative. It is the model-implied probability of the observed data under the prior and plays a central role in model comparison, Bayes factors, and predictive reasoning (Robert 2007).

5 Likelihood versus posterior

The likelihood $p(y \mid \theta)$ is a function of $\theta$ with $y$ fixed. It is not generally a probability distribution over $\theta$ because it need not integrate to one over the parameter space.

The posterior $p(\theta \mid y)$ is a probability distribution over $\theta$. It combines:

the data through the likelihood;
prior information or regularization through $\pi(\theta)$;
normalization through the marginal likelihood.

This distinction is fundamental. In Bayesian inference, the posterior is the inferential object; the likelihood is one component used to build it.

6 Posterior prediction

Inference about parameters is rarely the end of the story. Usually we care about future or replicated data $\tilde{y}$.

Proposition 1 The posterior predictive distribution is

\[ p(\tilde{y} \mid y) = \int p(\tilde{y} \mid \theta)\, p(\theta \mid y)\, d\theta. \]

6.1 Derivation

Condition on $\theta$ and integrate it out:

\[ p(\tilde{y} \mid y) = \int p(\tilde{y}, \theta \mid y)\, d\theta = \int p(\tilde{y} \mid \theta, y)\, p(\theta \mid y)\, d\theta. \]

If the future data are conditionally independent of the observed data given the parameter, then

\[ p(\tilde{y} \mid \theta, y) = p(\tilde{y} \mid \theta), \]

\[ p(\tilde{y} \mid y)=\int p(\tilde{y} \mid \theta)p(\theta \mid y)\, d\theta. \]

This integral shows that Bayesian prediction averages over parameter uncertainty rather than plugging in a single estimate.

7 Sequential updating

One of the cleanest features of Bayesian inference is that learning is recursive. Suppose the data arrive in a stream $y_{1:t} = (y_1, \dots, y_t)$. Then

\[ p(\theta \mid y_{1:t}) \propto p(y_t \mid \theta, y_{1:t-1})\, p(\theta \mid y_{1:t-1}). \]

If the data are conditionally independent given $\theta$, this simplifies to

\[ p(\theta \mid y_{1:t}) \propto p(y_t \mid \theta)\, p(\theta \mid y_{1:t-1}). \]

The posterior at time $t-1$ becomes the prior at time $t$.

7.1 Why batch and sequential updating agree

For conditionally independent data,

\[ p(\theta \mid y_{1:n}) \propto \left(\prod_{i=1}^n p(y_i \mid \theta)\right)\pi(\theta). \]

Sequentially applying the recursion multiplies the same likelihood factors in the same product, so batch and sequential analysis yield the same posterior.

This is one reason Bayesian methods are natural for online and streaming settings.

8 Exchangeability and de Finetti intuition

Independent and identically distributed observations are often introduced first in statistics, but Bayesian modeling typically begins one level deeper: with exchangeability.

A sequence $(Y_1, Y_2, \dots)$ is exchangeable if for every finite $n$ and every permutation $\sigma$ of $\{1,\dots,n\}$,

\[ p(y_1,\dots,y_n)=p(y_{\sigma(1)},\dots,y_{\sigma(n)}). \]

Exchangeability says the joint distribution does not depend on labeling order. De Finetti’s representation theorem gives the intuition that an exchangeable sequence can often be understood as conditionally i.i.d. given some latent quantity (Finetti 1974).

This matters because it explains why hierarchical and latent-variable models are so natural in Bayesian statistics. Exchangeability motivates modeling units as similar but not identical, which becomes partial pooling later in the course.

9 Worked example: Beta-Binomial updating

Suppose

\[ Y \mid \theta \sim \operatorname{Binomial}(n, \theta), \qquad \theta \sim \operatorname{Beta}(\alpha, \beta), \]

with prior density

\[ \pi(\theta) = \frac{\Gamma(\alpha + \beta)}{\Gamma(\alpha)\Gamma(\beta)} \theta^{\alpha-1}(1-\theta)^{\beta-1}, \qquad 0 < \theta < 1. \]

The likelihood for observing $Y=y$ is

\[ p(y \mid \theta) = \binom{n}{y}\theta^y(1-\theta)^{n-y}. \]

9.1 Posterior derivation

Multiply prior and likelihood:

\[ p(\theta \mid y) \propto \theta^y(1-\theta)^{n-y} \theta^{\alpha-1}(1-\theta)^{\beta-1}. \]

Collect exponents:

\[ p(\theta \mid y) \propto \theta^{\alpha+y-1}(1-\theta)^{\beta+n-y-1}. \]

Therefore

\[ \theta \mid y \sim \operatorname{Beta}(\alpha + y, \beta + n - y). \]

This is the simplest example of conjugacy.

9.2 Posterior mean and variance

For a $\operatorname{Beta}(a,b)$ random variable,

\[ \mathbb{E}[\theta] = \frac{a}{a+b}, \qquad \operatorname{Var}(\theta) = \frac{ab}{(a+b)^2(a+b+1)}. \]

Hence,

\[ \mathbb{E}[\theta \mid y] = \frac{\alpha+y}{\alpha+\beta+n}. \]

This can be rewritten as

\[ \mathbb{E}[\theta \mid y] = \frac{\alpha+\beta}{\alpha+\beta+n}\cdot \frac{\alpha}{\alpha+\beta} \;+\; \frac{n}{\alpha+\beta+n}\cdot \frac{y}{n}, \]

which shows the posterior mean as a weighted average of the prior mean and the sample proportion.

9.3 Posterior predictive

For one future Bernoulli observation $\tilde{Y}$,

\[ \Pr(\tilde{Y}=1 \mid y) = \int_0^1 \theta\, p(\theta \mid y)\, d\theta = \mathbb{E}[\theta \mid y] = \frac{\alpha+y}{\alpha+\beta+n}. \]

For $m$ future trials, the posterior predictive count is Beta-Binomial:

\[ \Pr(\tilde{Y}=k \mid y) = \binom{m}{k} \frac{B(\alpha+y+k,\ \beta+n-y+m-k)} {B(\alpha+y,\ \beta+n-y)}. \]

9.4 Marginal likelihood

The evidence is

\[ p(y) = \binom{n}{y} \frac{B(\alpha+y,\ \beta+n-y)}{B(\alpha,\beta)}. \]

This quantity is useful for model comparison but highly sensitive to prior choice.

10 `R` implementation

The following code reproduces the analytic posterior, posterior predictive, and a simple visualization. A runnable version also lives in beta_binomial.R.

beta_binomial_update <- function(y, n, alpha, beta) {
  list(
    alpha_post = alpha + y,
    beta_post = beta + n - y,
    post_mean = (alpha + y) / (alpha + beta + n),
    post_var = ((alpha + y) * (beta + n - y)) /
      (((alpha + beta + n)^2) * (alpha + beta + n + 1))
  )
}

beta_binomial_predictive <- function(k, m, y, n, alpha, beta) {
  choose(m, k) *
    beta(alpha + y + k, beta + n - y + m - k) /
    beta(alpha + y, beta + n - y)
}

grid <- seq(0.001, 0.999, length.out = 500)
prior <- dbeta(grid, 2, 2)
post  <- dbeta(grid, 2 + 14, 2 + 20 - 14)

plot(grid, prior, type = "l", lwd = 2, col = "steelblue",
     ylab = "Density", xlab = expression(theta))
lines(grid, post, lwd = 2, col = "firebrick")
legend("topright", legend = c("Prior", "Posterior"),
       col = c("steelblue", "firebrick"), lwd = 2, bty = "n")

11 Diagnostics, pitfalls, and conceptual checks

11.1 Prior sensitivity

Even in the Beta-Binomial model, different prior concentrations can materially change the posterior in small samples. This is a feature, not a bug. Priors encode information, and small data sets cannot erase strong prior assumptions automatically.

11.2 Posterior propriety

Improper priors can be useful in some settings, but one must check whether the resulting posterior is proper. A posterior that does not integrate to one is not a valid inferential distribution.

11.3 Evidence is not “the probability the model is true”

The marginal likelihood $p(y)$ is the probability of the observed data under the model and prior, not the posterior probability of the model itself. Posterior model probabilities require prior probabilities on models as well.

11.4 Posterior prediction is not plug-in prediction

Bayesian predictive distributions integrate over $\theta$. Replacing $\theta$ by a point estimate underestimates predictive uncertainty whenever parameter uncertainty matters.

12 Exercises

Derive the posterior for a Gamma-Poisson model and show the predictive count is negative binomial.
Show algebraically that sequential updating and batch updating are identical for the Beta-Binomial model.
For a Beta$(\alpha,\beta)$ prior, derive the prior predictive distribution of $Y \sim \text{Binomial}(n,\theta)$.
Compare the posteriors from Beta$(1,1)$, Beta$(2,2)$, and Beta$(20,20)$ priors after observing $y=3$ successes in $n=5$ trials. Explain the differences.
Explain why exchangeability is weaker than independence but still enough to motivate latent-variable Bayesian models.

13 Summary

Bayes’ theorem turns prior information and sampling assumptions into posterior inference. Its real statistical value appears in four facts:

the posterior is built from prior and likelihood;
the posterior predictive integrates over parameter uncertainty;
updating is sequential and recursive;
exchangeability provides a conceptual bridge from symmetry assumptions to latent Bayesian structure.

These ideas will recur in every later chapter, from conjugate models to HMC, hierarchical modeling, and Bayesian workflow.

14 Further reading

Finetti (1974) is the source for exchangeability and the representation theorem, and it remains the clearest statement of the subjective position on what a probability is. Bernardo and Smith (1994) is the systematic modern development of that programme, and the place to go for the axiomatic derivation of the Bayesian apparatus from coherence requirements.

Jaynes (2003) argues the case from a different direction — probability as extended logic, derived from desiderata about consistent reasoning. It is opinionated and worth reading for that reason.

For the working mechanics of updating and prediction, Gelman et al. (2013) chapters 1 to 3 and Hoff (2009) chapters 1 to 3 cover the same ground as this chapter with different examples. Hoff (2009) is the more compact of the two and moves faster to the conjugate calculations taken up in conjugate analysis.

References

Bernardo, José M., and Adrian F. M. Smith. 1994. Bayesian Theory. Wiley.

Finetti, Bruno de. 1974. “Theory of Probability: A Critical Introductory Treatment.” Wiley Series in Probability and Mathematical Statistics 1.

Gelman, Andrew, John B. Carlin, Hal S. Stern, David B. Dunson, Aki Vehtari, and Donald B. Rubin. 2013. Bayesian Data Analysis. 3rd ed. CRC Press.

Hoff, Peter D. 2009. A First Course in Bayesian Statistical Methods. Springer.

Jaynes, E. T. 2003. Probability Theory: The Logic of Science. Cambridge University Press.

Robert, Christian P. 2007. The Bayesian Choice. 2nd ed. Springer.

--- title: "Bayes' theorem, sequential updating, and posterior prediction" --- ## Chapter position This chapter is the fully written exemplar for Part I. It shows the intended style for the rest of the course: intuition, formal structure, derivations, worked examples, code, diagnostics, and exercises in one integrated chapter. ::: {.callout-note} ## What you should be able to do after this chapter After working through this chapter, you should be able to: - derive Bayes' theorem in both discrete and density-based settings; - explain the roles of prior, likelihood, posterior, and marginal likelihood; - derive posterior predictive distributions explicitly; - show why sequential updating gives the same result as batch updating; - interpret exchangeability as a structural justification for Bayesian modeling. ::: ## Why Bayes' theorem matters At a high level, Bayesian inference answers the following question: > Given a probabilistic model for data and a prior distribution over unknown parameters, how should uncertainty about the parameters change after observing data? Bayes' theorem is the algebraic identity that answers this question. But in serious statistical work, it is more than an identity. It is the organizing rule that connects: - scientific assumptions encoded in the prior; - data-generating assumptions encoded in the likelihood; - inferential uncertainty encoded in the posterior; - out-of-sample reasoning encoded in the posterior predictive distribution. The theorem itself is elementary. The difficulty lies in understanding what its pieces mean and how they behave in realistic models [@bernardo1994; @gelman2013]. ## Formal setup Suppose: - $\theta \in \Theta$ denotes an unknown parameter; - $y \in \mathcal{Y}$ denotes observed data; - $\pi(\theta)$ is a prior density or mass function for $\theta$; - $p(y \mid \theta)$ is the sampling model or likelihood; - $p(\theta \mid y)$ is the posterior. The joint distribution factors as $$ p(\theta, y) = p(y \mid \theta)\pi(\theta). $$ Marginalizing over $\theta$ yields the evidence or marginal likelihood $$ p(y) = \int_{\Theta} p(y \mid \theta)\pi(\theta)\, d\theta, $$ assuming the integral is finite. ## Bayes' theorem :::{.theorem #thm-bayes} If $p(y) > 0$, then the posterior density or mass function is $$ p(\theta \mid y) = \frac{p(y \mid \theta)\pi(\theta)}{p(y)} = \frac{p(y \mid \theta)\pi(\theta)}{\int_{\Theta} p(y \mid \vartheta)\pi(\vartheta)\, d\vartheta}. $$ ::: ### Derivation Start from the product rule: $$ p(\theta, y) = p(\theta \mid y)p(y). $$ But the same joint distribution can also be written as $$ p(\theta, y) = p(y \mid \theta)\pi(\theta). $$ Equating the two factorizations gives $$ p(\theta \mid y)p(y) = p(y \mid \theta)\pi(\theta), $$ and therefore $$ p(\theta \mid y) = \frac{p(y \mid \theta)\pi(\theta)}{p(y)}. $$ Finally, since $$ p(y)=\int_{\Theta} p(y \mid \vartheta)\pi(\vartheta)\, d\vartheta, $$ we obtain the usual normalized form. ### Why the denominator matters Many computational methods work with the posterior only up to proportionality: $$ p(\theta \mid y) \propto p(y \mid \theta)\pi(\theta). $$ That simplification is useful for MCMC and optimization, but it can hide an important conceptual fact: the denominator is not decorative. It is the model-implied probability of the observed data under the prior and plays a central role in model comparison, Bayes factors, and predictive reasoning [@robert2007]. ## Likelihood versus posterior The likelihood $p(y \mid \theta)$ is a function of $\theta$ with $y$ fixed. It is not generally a probability distribution over $\theta$ because it need not integrate to one over the parameter space. The posterior $p(\theta \mid y)$ is a probability distribution over $\theta$. It combines: - the data through the likelihood; - prior information or regularization through $\pi(\theta)$; - normalization through the marginal likelihood. This distinction is fundamental. In Bayesian inference, the posterior is the inferential object; the likelihood is one component used to build it. ## Posterior prediction Inference about parameters is rarely the end of the story. Usually we care about future or replicated data $\tilde{y}$. :::{.proposition #prp-posterior-predictive} The posterior predictive distribution is $$ p(\tilde{y} \mid y) = \int p(\tilde{y} \mid \theta)\, p(\theta \mid y)\, d\theta. $$ ::: ### Derivation Condition on $\theta$ and integrate it out: $$ p(\tilde{y} \mid y) = \int p(\tilde{y}, \theta \mid y)\, d\theta = \int p(\tilde{y} \mid \theta, y)\, p(\theta \mid y)\, d\theta. $$ If the future data are conditionally independent of the observed data given the parameter, then $$ p(\tilde{y} \mid \theta, y) = p(\tilde{y} \mid \theta), $$ so $$ p(\tilde{y} \mid y)=\int p(\tilde{y} \mid \theta)p(\theta \mid y)\, d\theta. $$ This integral shows that Bayesian prediction averages over parameter uncertainty rather than plugging in a single estimate. ## Sequential updating One of the cleanest features of Bayesian inference is that learning is recursive. Suppose the data arrive in a stream $y_{1:t} = (y_1, \dots, y_t)$. Then $$ p(\theta \mid y_{1:t}) \propto p(y_t \mid \theta, y_{1:t-1})\, p(\theta \mid y_{1:t-1}). $$ If the data are conditionally independent given $\theta$, this simplifies to $$ p(\theta \mid y_{1:t}) \propto p(y_t \mid \theta)\, p(\theta \mid y_{1:t-1}). $$ The posterior at time $t-1$ becomes the prior at time $t$. ### Why batch and sequential updating agree For conditionally independent data, $$ p(\theta \mid y_{1:n}) \propto \left(\prod_{i=1}^n p(y_i \mid \theta)\right)\pi(\theta). $$ Sequentially applying the recursion multiplies the same likelihood factors in the same product, so batch and sequential analysis yield the same posterior. This is one reason Bayesian methods are natural for online and streaming settings. ## Exchangeability and de Finetti intuition Independent and identically distributed observations are often introduced first in statistics, but Bayesian modeling typically begins one level deeper: with **exchangeability**. A sequence $(Y_1, Y_2, \dots)$ is exchangeable if for every finite $n$ and every permutation $\sigma$ of $\{1,\dots,n\}$, $$ p(y_1,\dots,y_n)=p(y_{\sigma(1)},\dots,y_{\sigma(n)}). $$ Exchangeability says the joint distribution does not depend on labeling order. De Finetti's representation theorem gives the intuition that an exchangeable sequence can often be understood as conditionally i.i.d. given some latent quantity [@definetti1974]. This matters because it explains why hierarchical and latent-variable models are so natural in Bayesian statistics. Exchangeability motivates modeling units as similar but not identical, which becomes partial pooling later in the course. ## Worked example: Beta-Binomial updating Suppose $$ Y \mid \theta \sim \operatorname{Binomial}(n, \theta), \qquad \theta \sim \operatorname{Beta}(\alpha, \beta), $$ with prior density $$ \pi(\theta) = \frac{\Gamma(\alpha + \beta)}{\Gamma(\alpha)\Gamma(\beta)} \theta^{\alpha-1}(1-\theta)^{\beta-1}, \qquad 0 < \theta < 1. $$ The likelihood for observing $Y=y$ is $$ p(y \mid \theta) = \binom{n}{y}\theta^y(1-\theta)^{n-y}. $$ ### Posterior derivation Multiply prior and likelihood: $$ p(\theta \mid y) \propto \theta^y(1-\theta)^{n-y} \theta^{\alpha-1}(1-\theta)^{\beta-1}. $$ Collect exponents: $$ p(\theta \mid y) \propto \theta^{\alpha+y-1}(1-\theta)^{\beta+n-y-1}. $$ Therefore $$ \theta \mid y \sim \operatorname{Beta}(\alpha + y, \beta + n - y). $$ This is the simplest example of conjugacy. ### Posterior mean and variance For a $\operatorname{Beta}(a,b)$ random variable, $$ \mathbb{E}[\theta] = \frac{a}{a+b}, \qquad \operatorname{Var}(\theta) = \frac{ab}{(a+b)^2(a+b+1)}. $$ Hence, $$ \mathbb{E}[\theta \mid y] = \frac{\alpha+y}{\alpha+\beta+n}. $$ This can be rewritten as $$ \mathbb{E}[\theta \mid y] = \frac{\alpha+\beta}{\alpha+\beta+n}\cdot \frac{\alpha}{\alpha+\beta} \;+\; \frac{n}{\alpha+\beta+n}\cdot \frac{y}{n}, $$ which shows the posterior mean as a weighted average of the prior mean and the sample proportion. ### Posterior predictive For one future Bernoulli observation $\tilde{Y}$, $$ \Pr(\tilde{Y}=1 \mid y) = \int_0^1 \theta\, p(\theta \mid y)\, d\theta = \mathbb{E}[\theta \mid y] = \frac{\alpha+y}{\alpha+\beta+n}. $$ For $m$ future trials, the posterior predictive count is Beta-Binomial: $$ \Pr(\tilde{Y}=k \mid y) = \binom{m}{k} \frac{B(\alpha+y+k,\ \beta+n-y+m-k)} {B(\alpha+y,\ \beta+n-y)}. $$ ### Marginal likelihood The evidence is $$ p(y) = \binom{n}{y} \frac{B(\alpha+y,\ \beta+n-y)}{B(\alpha,\beta)}. $$ This quantity is useful for model comparison but highly sensitive to prior choice. ## `R` implementation The following code reproduces the analytic posterior, posterior predictive, and a simple visualization. A runnable version also lives in [beta_binomial.R](../../labs/r/beta_binomial.R). ```r beta_binomial_update <- function(y, n, alpha, beta) { list( alpha_post = alpha + y, beta_post = beta + n - y, post_mean = (alpha + y) / (alpha + beta + n), post_var = ((alpha + y) * (beta + n - y)) / (((alpha + beta + n)^2) * (alpha + beta + n + 1)) ) } beta_binomial_predictive <- function(k, m, y, n, alpha, beta) { choose(m, k) * beta(alpha + y + k, beta + n - y + m - k) / beta(alpha + y, beta + n - y) } grid <- seq(0.001, 0.999, length.out = 500) prior <- dbeta(grid, 2, 2) post <- dbeta(grid, 2 + 14, 2 + 20 - 14) plot(grid, prior, type = "l", lwd = 2, col = "steelblue", ylab = "Density", xlab = expression(theta)) lines(grid, post, lwd = 2, col = "firebrick") legend("topright", legend = c("Prior", "Posterior"), col = c("steelblue", "firebrick"), lwd = 2, bty = "n") ``` ## Diagnostics, pitfalls, and conceptual checks ### Prior sensitivity Even in the Beta-Binomial model, different prior concentrations can materially change the posterior in small samples. This is a feature, not a bug. Priors encode information, and small data sets cannot erase strong prior assumptions automatically. ### Posterior propriety Improper priors can be useful in some settings, but one must check whether the resulting posterior is proper. A posterior that does not integrate to one is not a valid inferential distribution. ### Evidence is not “the probability the model is true” The marginal likelihood $p(y)$ is the probability of the observed data under the model and prior, not the posterior probability of the model itself. Posterior model probabilities require prior probabilities on models as well. ### Posterior prediction is not plug-in prediction Bayesian predictive distributions integrate over $\theta$. Replacing $\theta$ by a point estimate underestimates predictive uncertainty whenever parameter uncertainty matters. ## Exercises 1. Derive the posterior for a Gamma-Poisson model and show the predictive count is negative binomial. 2. Show algebraically that sequential updating and batch updating are identical for the Beta-Binomial model. 3. For a Beta$(\alpha,\beta)$ prior, derive the prior predictive distribution of $Y \sim \text{Binomial}(n,\theta)$. 4. Compare the posteriors from Beta$(1,1)$, Beta$(2,2)$, and Beta$(20,20)$ priors after observing $y=3$ successes in $n=5$ trials. Explain the differences. 5. Explain why exchangeability is weaker than independence but still enough to motivate latent-variable Bayesian models. ## Summary Bayes' theorem turns prior information and sampling assumptions into posterior inference. Its real statistical value appears in four facts: - the posterior is built from prior and likelihood; - the posterior predictive integrates over parameter uncertainty; - updating is sequential and recursive; - exchangeability provides a conceptual bridge from symmetry assumptions to latent Bayesian structure. These ideas will recur in every later chapter, from conjugate models to HMC, hierarchical modeling, and Bayesian workflow. ## Further reading @definetti1974 is the source for exchangeability and the representation theorem, and it remains the clearest statement of the subjective position on what a probability is. @bernardo1994 is the systematic modern development of that programme, and the place to go for the axiomatic derivation of the Bayesian apparatus from coherence requirements. @jaynes2003 argues the case from a different direction — probability as extended logic, derived from desiderata about consistent reasoning. It is opinionated and worth reading for that reason. For the working mechanics of updating and prediction, @gelman2013 chapters 1 to 3 and @hoff2009 chapters 1 to 3 cover the same ground as this chapter with different examples. @hoff2009 is the more compact of the two and moves faster to the conjugate calculations taken up in [conjugate analysis](../core/conjugate-analysis.qmd).