Bayesian regression
Estimate regression relationships and predictive uncertainty using explicit priors and posterior sampling.
Concepts and reasoning
Regression with explicit parameter uncertainty
Bayesian regression combines a likelihood for the observed outcomes with prior distributions for unknown parameters. The posterior describes which parameter values are plausible after observing the data, conditional on those choices. Predictions then average over posterior uncertainty rather than relying only on a single fitted coefficient vector.
For a continuous outcome, a Gaussian regression models a conditional mean and residual SD. For a binary outcome, logistic regression models the log odds of an event. The likelihood should match the outcome and its dependence structure; using a Bayesian algorithm does not make an inappropriate likelihood suitable.
Priors express plausible scales and relationships before the current observations. A weakly informative prior can regularize implausibly large coefficients while allowing meaningful effects. Its meaning depends on units: a coefficient of 2 per kilogram differs from 2 per standard deviation.
From a model to a computation
Many regression posteriors cannot be summarized analytically. Markov chain Monte Carlo, MCMC, generates dependent draws intended to explore the posterior. Modern samplers can be efficient, but a completed run does not establish that the draws represent the target distribution.
The Stan regression guide provides model examples, while the posterior analysis reference explains diagnostics. These sources teach general Bayesian ideas; the software used for an individual analysis may differ.
Mathematical foundations
Posterior and predictive distributions
Bayes' rule gives
For Gaussian regression, one possible model is
A prior such as βⱼ ∼ N(0, s²) must have a justified scale s. For binary outcomes, replace the Gaussian mean model with
Then exp(βⱼ) is a conditional odds ratio for a one-unit predictor increase. It is not a risk ratio.
For a future outcome ynew, posterior prediction integrates both parameter uncertainty and the observation model:
A credible interval for a mean response therefore differs from a predictive interval for one future observation. The latter also includes residual or outcome variation.
Worked example
Worked example: coefficient scale changes interpretation
Suppose a logistic model has intercept −1.386, giving a baseline probability of approximately 0.20. A coefficient of 0.693 for a standardized predictor corresponds to an odds ratio of exp(0.693) ≈ 2 per one-SD increase, holding other predictors fixed.
Baseline odds are 0.20/0.80 = 0.25. Doubling those odds gives 0.50, which corresponds to probability 0.50/(1 + 0.50) = 0.333. Doubling odds has not doubled probability to 0.40.
Now consider a Gaussian model in which both outcome and predictor were standardized. Let the fitted slope be 0.4, outcome SD be 10, and predictor SD be 5. The slope in original units is 0.4 × 10/5 = 0.8 outcome units per predictor unit. Standardization must be reversed to make that claim.
For an illustrative normal posterior slope, mean 0.4 and SD 0.1 give a central 95% interval near 0.204 to 0.596. This is posterior uncertainty about the slope, not a range containing 95% of individual outcomes.
Suppose 4,000 retained MCMC draws yield an effective sample size of only 400 for that slope. An approximate Monte Carlo SE of its posterior mean is 0.1/√400 = 0.005, not 0.1/√4000. Autocorrelation reduces computational information without changing the scientific sample size.
Interpreting the results
A posterior is conditional on the model and prior
Posterior coefficient summaries describe uncertainty after combining the prior with the observed-data likelihood. A credible interval has a probability interpretation under that joint model; it does not average over every unmodeled explanation or establish causality for an observational association.
For Gaussian regression, a slope describes a conditional mean change on the model's scale. For Bernoulli logistic regression, it describes a log-odds change; exponentiating gives an odds ratio, not a risk ratio. Standardization changes what a one-unit predictor contrast means and must be reported.
Posterior predictive intervals include variation in future outcomes as well as parameter uncertainty, so they differ from intervals for a mean response. A probability of a positive coefficient answers a direction question, not whether the effect is large enough to matter. Interpret magnitude, uncertainty, and predictive behavior together.
Assumptions and limitations
Priors express assumptions on a particular scale
A prior standard deviation of one has different implications for raw income, a standardized continuous outcome, and a log-odds coefficient. Inspect the outcomes implied by priors before treating them as weakly informative. Extremely broad priors can imply implausible predictions and make computation difficult.
The likelihood should reflect the endpoint: a Gaussian model assumes a suitable continuous error distribution, while Bernoulli models binary outcomes. Conditional independence, covariate form, missingness, and measurement quality remain important. Bayesian computation does not automatically correct a misspecified likelihood or an inappropriate adjustment set.
Posterior sampling requires the algorithm to explore the relevant distribution. Correlated draws are not independent replications, and additional draws do not guarantee exploration of missed modes. Predictions beyond the observed covariate range depend strongly on model form and prior assumptions.
Diagnostics and sensitivity analysis
Separate computational checks from scientific checks
Review chain agreement, effective sample sizes, Monte Carlo errors, and divergent transitions before interpreting posterior summaries. A near-one R-hat and adequate effective sizes support numerical reliability but do not establish that the model explains the data. The Stan diagnostic reference explains these quantities.
Then compare prior and posterior predictive distributions with meaningful features of observed outcomes: center, spread, tails, subgroup behavior, and impossible values. Agreement in an overall mean can conceal severe local mismatch. In-sample predictive agreement is not external validation.
Repeat with scientifically defensible priors and covariate forms. Report changes in decision-relevant quantities rather than only whether a credible interval crosses zero. More warm-up or draws can reduce sampling problems in some cases, but divergences may require revising priors, scaling, or model structure.
References and other material
Read the methodological references and supporting tutorials below. The examples in this lesson are original teaching calculations, not findings from these sources.
- Stan User’s Guide: regression models — Regression, priors and hierarchical modeling concepts; examples use Stan.
- Stan Reference Manual: posterior analysis — How to interpret effective sample size and convergence diagnostics.