Capstone projects: end-to-end Bayesian workflow, project architecture, and assessment

What you should be able to do after this chapter

structure a serious Bayesian project from scientific question to final report;
define the minimum evidentiary standard for a graduate-level Bayesian analysis;
build a model-development ladder instead of jumping directly to the most complex model;
document diagnostics, sensitivity analysis, and limitations as core outputs;
use the provided proposal and report templates to organize a reproducible capstone.

1 What a Bayesian capstone must demonstrate

A capstone is not a code dump and not a single fitted model. It is an integrated demonstration of:

scientific framing;
probabilistic modeling;
prior design;
posterior computation;
diagnostic criticism;
predictive or decision evaluation;
uncertainty communication.

The standard should be closer to a small research report than to a homework submission.

2 The project as a workflow object

Every serious project should make explicit the tuple

\[ (\mathcal{Q}, \mathcal{D}, M, \Pi, A, C, R), \]

where:

$\mathcal{Q}$ is the substantive question;
$\mathcal{D}$ is the observed data and data-generating context;
$M$ is the model family;
$\Pi$ is the prior specification;
$A$ is the computation algorithm;
$C$ is the criticism and sensitivity plan;
$R$ is the reporting target or decision target.

Weak projects leave several of these implicit.

3 Project architecture

3.1 Stage 1: scientific question

State the question in a way that determines the estimand. “Fit a Bayesian model” is not a question.

3.2 Stage 2: data audit

Document:

units of observation;
missingness;
measurement scale;
grouping or dependence;
likely sources of bias or contamination.

3.3 Stage 3: generative story

Write down the joint factorization

\[ p(y,\theta)=p(y \mid \theta)\pi(\theta) \]

or its hierarchical extension. If the factorization cannot be written clearly, the model is not yet ready.

3.4 Stage 4: model ladder

A good project usually fits several models:

baseline model;
minimally enriched model;
scientifically serious reference model;
sensitivity or robustness alternatives.

This ladder is essential. Without it, the analyst cannot distinguish genuine scientific structure from overfitting or computational artifact.

4 Minimum evidence package

At a minimum, the final project should contain:

prior predictive checks;
posterior computation diagnostics;
posterior predictive checks;
sensitivity analysis;
predictive evaluation or decision analysis;
discussion of limitations.

Any project lacking several of these may still be interesting, but it is not a complete Bayesian workflow.

5 Proposal requirements

The proposal should include:

the scientific question and estimand;
the candidate model family;
key priors and why they are plausible;
the intended computation engine;
the planned diagnostics;
the evaluation criterion;
a risk register describing expected failure points.

The companion proposal template is project-proposal-template.qmd.

6 Final report requirements

The final report should do more than show that a model ran. It should provide:

the final model specification;
why simpler alternatives were insufficient;
diagnostics and what they imply;
predictive or decision summaries;
sensitivity to prior and likelihood choices;
a transparent statement of residual uncertainty.

The companion report template is final-project-report-template.qmd.

7 Computational standards

Each project should make the computational target explicit:

posterior draws from MCMC;
an approximation from variational inference;
a MAP plus Laplace approximation;
filtering or sequential approximation;
a custom algorithm for a structured latent model.

The choice is acceptable only if the project states:

why the algorithm matches the model geometry;
what tuning choices were made;
how convergence or approximation quality was checked.

8 Diagnostic review checklist

Before interpretation, a project should be able to answer:

Did prior predictive simulation reveal absurd implications?
Did the algorithm mix or converge adequately?
Did predictive checks reveal structural mismatch?
Were influential observations or outliers assessed?
Were key conclusions stable to plausible alternative priors?
Was predictive utility or decision utility evaluated on a clear scale?

If several answers are “no,” the project is not yet mature.

9 Recommended repository structure

For a reproducible project, a clean structure is:

project/
  data/
  notebooks/
  model/
  scripts/
  reports/
  figures/
  README.md
  _quarto.yml

The point is not rigid directory aesthetics. The point is separation of data, model code, diagnostics, and narrative.

10 Project bank

The canonical project bank is projects.qmd. It includes:

hierarchical regression with partial pooling;
hidden Markov modeling of longitudinal behavior;
Dirichlet-multinomial compositional analysis;
prior-sensitive Bayesian logistic regression;
Gaussian process regression;
finite-mixture label-switching study;
HMC diagnostics case study.

Each of these can be expanded to doctoral-level depth if the proposal is framed carefully.

11 Assessment rubric

The grading logic is aligned with the rubric on projects.qmd, but the hierarchy is:

first: scientific coherence and workflow completeness;
second: mathematical and computational rigor;
third: communication quality and reproducibility.

A mathematically sophisticated project with weak diagnostics is weaker than a slightly simpler project with full workflow discipline.

12 Common failure modes

12.1 Starting from a maximal model

This hides whether the final complexity is necessary.

12.2 Using diagnostics as decoration

Plots and scalar summaries must change modeling decisions, not merely appear in an appendix.

12.3 Underreporting computational failures

Divergences, unstable VI, or label switching are results. They are not embarrassments to be hidden.

12.4 Omitting sensitivity analysis

Without sensitivity analysis, posterior confidence often exceeds what the workflow has earned.

13 Suggested timeline

Week 1: proposal, literature review, and data audit.
Week 2: prior predictive exploration and baseline model.
Week 3: serious model fitting and computational debugging.
Week 4: diagnostics, criticism, and sensitivity.
Week 5: communication, revision, and final report.

Longer schedules should still preserve this order.

14 Exercises

Draft a project proposal that includes question, estimand, prior strategy, computation plan, and diagnostics plan.
Write a model ladder for one project in the project bank, from simplest defensible model to reference model.
Identify three realistic failure modes for a hierarchical or latent-variable project and explain how you would detect them.
Explain why prior predictive checks belong in the proposal stage rather than only in the final report.
Describe what evidence would justify using VI rather than MCMC in a capstone.

15 Summary

The capstone chapter closes the course by treating workflow as the real object of mastery:

the scientific question determines the estimand;
the model must be stated as a joint probability structure;
diagnostics and sensitivity are mandatory, not optional;
reproducible computation is part of the evidence;
communication should report what the model supports, and no more.

16 Further reading

Gelman et al. (2020) is the single most useful reference for a capstone: it sets out the full workflow — prior predictive checks, fitting, diagnostics, posterior predictive checks, model comparison, and iteration — and is explicit that the process is cyclical rather than linear.

Gabry et al. (2019) covers the graphical side of that workflow, and Vehtari et al. (2017) the predictive evaluation you will need to justify a model choice. Gelman et al. (2013) remains the reference for the modelling questions themselves.

McElreath (2020) is the best guide to writing the results up in a way a reader can follow. For projects with a substantial computational component, Murphy (2023) is the most convenient single survey of the modern methods.

References

Gabry, Jonah, Daniel Simpson, Aki Vehtari, Michael Betancourt, and Andrew Gelman. 2019. “Visualization in Bayesian Workflow.” Journal of the Royal Statistical Society: Series A 182 (2): 389–402.

Gelman, Andrew, John B. Carlin, Hal S. Stern, David B. Dunson, Aki Vehtari, and Donald B. Rubin. 2013. Bayesian Data Analysis. 3rd ed. CRC Press.

Gelman, Andrew, Aki Vehtari, Daniel Simpson, et al. 2020. “Bayesian Workflow.” arXiv Preprint arXiv:2011.01808.

McElreath, Richard. 2020. Statistical Rethinking. 2nd ed. CRC Press.

Murphy, Kevin P. 2023. Probabilistic Machine Learning: Advanced Topics. MIT Press.

Vehtari, Aki, Andrew Gelman, and Jonah Gabry. 2017. “Practical Bayesian Model Evaluation Using Leave-One-Out Cross-Validation and WAIC.” Statistics and Computing 27 (5): 1413–32.

--- title: "Capstone projects: end-to-end Bayesian workflow, project architecture, and assessment" --- ::: {.callout-note} ## What you should be able to do after this chapter - structure a serious Bayesian project from scientific question to final report; - define the minimum evidentiary standard for a graduate-level Bayesian analysis; - build a model-development ladder instead of jumping directly to the most complex model; - document diagnostics, sensitivity analysis, and limitations as core outputs; - use the provided proposal and report templates to organize a reproducible capstone. ::: ## What a Bayesian capstone must demonstrate A capstone is not a code dump and not a single fitted model. It is an integrated demonstration of: - scientific framing; - probabilistic modeling; - prior design; - posterior computation; - diagnostic criticism; - predictive or decision evaluation; - uncertainty communication. The standard should be closer to a small research report than to a homework submission. ## The project as a workflow object Every serious project should make explicit the tuple $$ (\mathcal{Q}, \mathcal{D}, M, \Pi, A, C, R), $$ where: - $\mathcal{Q}$ is the substantive question; - $\mathcal{D}$ is the observed data and data-generating context; - $M$ is the model family; - $\Pi$ is the prior specification; - $A$ is the computation algorithm; - $C$ is the criticism and sensitivity plan; - $R$ is the reporting target or decision target. Weak projects leave several of these implicit. ## Project architecture ### Stage 1: scientific question State the question in a way that determines the estimand. “Fit a Bayesian model” is not a question. ### Stage 2: data audit Document: - units of observation; - missingness; - measurement scale; - grouping or dependence; - likely sources of bias or contamination. ### Stage 3: generative story Write down the joint factorization $$ p(y,\theta)=p(y \mid \theta)\pi(\theta) $$ or its hierarchical extension. If the factorization cannot be written clearly, the model is not yet ready. ### Stage 4: model ladder A good project usually fits several models: 1. baseline model; 2. minimally enriched model; 3. scientifically serious reference model; 4. sensitivity or robustness alternatives. This ladder is essential. Without it, the analyst cannot distinguish genuine scientific structure from overfitting or computational artifact. ## Minimum evidence package At a minimum, the final project should contain: 1. prior predictive checks; 2. posterior computation diagnostics; 3. posterior predictive checks; 4. sensitivity analysis; 5. predictive evaluation or decision analysis; 6. discussion of limitations. Any project lacking several of these may still be interesting, but it is not a complete Bayesian workflow. ## Proposal requirements The proposal should include: - the scientific question and estimand; - the candidate model family; - key priors and why they are plausible; - the intended computation engine; - the planned diagnostics; - the evaluation criterion; - a risk register describing expected failure points. The companion proposal template is [project-proposal-template.qmd](../../templates/project-proposal-template.qmd). ## Final report requirements The final report should do more than show that a model ran. It should provide: - the final model specification; - why simpler alternatives were insufficient; - diagnostics and what they imply; - predictive or decision summaries; - sensitivity to prior and likelihood choices; - a transparent statement of residual uncertainty. The companion report template is [final-project-report-template.qmd](../../templates/final-project-report-template.qmd). ## Computational standards Each project should make the computational target explicit: - posterior draws from MCMC; - an approximation from variational inference; - a MAP plus Laplace approximation; - filtering or sequential approximation; - a custom algorithm for a structured latent model. The choice is acceptable only if the project states: - why the algorithm matches the model geometry; - what tuning choices were made; - how convergence or approximation quality was checked. ## Diagnostic review checklist Before interpretation, a project should be able to answer: 1. Did prior predictive simulation reveal absurd implications? 2. Did the algorithm mix or converge adequately? 3. Did predictive checks reveal structural mismatch? 4. Were influential observations or outliers assessed? 5. Were key conclusions stable to plausible alternative priors? 6. Was predictive utility or decision utility evaluated on a clear scale? If several answers are “no,” the project is not yet mature. ## Recommended repository structure For a reproducible project, a clean structure is: ```text project/ data/ notebooks/ model/ scripts/ reports/ figures/ README.md _quarto.yml ``` The point is not rigid directory aesthetics. The point is separation of data, model code, diagnostics, and narrative. ## Project bank The canonical project bank is [projects.qmd](../../projects.qmd). It includes: - hierarchical regression with partial pooling; - hidden Markov modeling of longitudinal behavior; - Dirichlet-multinomial compositional analysis; - prior-sensitive Bayesian logistic regression; - Gaussian process regression; - finite-mixture label-switching study; - HMC diagnostics case study. Each of these can be expanded to doctoral-level depth if the proposal is framed carefully. ## Assessment rubric The grading logic is aligned with the rubric on [projects.qmd](../../projects.qmd), but the hierarchy is: - first: scientific coherence and workflow completeness; - second: mathematical and computational rigor; - third: communication quality and reproducibility. A mathematically sophisticated project with weak diagnostics is weaker than a slightly simpler project with full workflow discipline. ## Common failure modes ### Starting from a maximal model This hides whether the final complexity is necessary. ### Using diagnostics as decoration Plots and scalar summaries must change modeling decisions, not merely appear in an appendix. ### Underreporting computational failures Divergences, unstable VI, or label switching are results. They are not embarrassments to be hidden. ### Omitting sensitivity analysis Without sensitivity analysis, posterior confidence often exceeds what the workflow has earned. ## Suggested timeline 1. Week 1: proposal, literature review, and data audit. 2. Week 2: prior predictive exploration and baseline model. 3. Week 3: serious model fitting and computational debugging. 4. Week 4: diagnostics, criticism, and sensitivity. 5. Week 5: communication, revision, and final report. Longer schedules should still preserve this order. ## Exercises 1. Draft a project proposal that includes question, estimand, prior strategy, computation plan, and diagnostics plan. 2. Write a model ladder for one project in the project bank, from simplest defensible model to reference model. 3. Identify three realistic failure modes for a hierarchical or latent-variable project and explain how you would detect them. 4. Explain why prior predictive checks belong in the proposal stage rather than only in the final report. 5. Describe what evidence would justify using VI rather than MCMC in a capstone. ## Summary The capstone chapter closes the course by treating workflow as the real object of mastery: - the scientific question determines the estimand; - the model must be stated as a joint probability structure; - diagnostics and sensitivity are mandatory, not optional; - reproducible computation is part of the evidence; - communication should report what the model supports, and no more. ## Further reading @gelman2020 is the single most useful reference for a capstone: it sets out the full workflow — prior predictive checks, fitting, diagnostics, posterior predictive checks, model comparison, and iteration — and is explicit that the process is cyclical rather than linear. @gabry2019 covers the graphical side of that workflow, and @vehtari2017 the predictive evaluation you will need to justify a model choice. @gelman2013 remains the reference for the modelling questions themselves. @mcelreath2020 is the best guide to writing the results up in a way a reader can follow. For projects with a substantial computational component, @murphy2023 is the most convenient single survey of the modern methods.