Bayesian trial simulation
Evaluate success and futility rules for a single-arm trial with a binary response endpoint.
Concepts and reasoning
Updating uncertainty about a response rate
In a single-arm binary trial, each participant either responds or does not. Let p be the unknown response probability in the trial population. Bayesian analysis represents uncertainty about p with a prior distribution and updates it using the observed responses.
A beta distribution is convenient because it combines analytically with a binomial likelihood. Its two positive shape parameters control location and concentration. A prior concentrated around a historical rate supplies more information than a diffuse prior, but historical evidence may not be transportable to current patients.
A decision rule might declare success when the posterior probability that p exceeds a benchmark p₀ is sufficiently high. It might stop for futility when that probability is very low. The thresholds and analysis times define a trial procedure that can be evaluated through simulation.
Three different probabilities
A posterior probability describes uncertainty about p after the data, conditional on the model and prior. A predictive probability describes future data or future trial success, averaging over that uncertainty. Frequentist power describes the probability that the procedure declares success across repeated trials at a fixed true p.
These quantities answer different questions. A success threshold of 0.95 does not imply that the design's false-positive rate is 0.05, especially with repeated looks or informative priors. The FDA Bayesian guidance for medical device trials discusses prior information, modeling, and operating characteristics within that specific regulatory context.
Mathematical foundations
Beta–binomial updating
If the prior is Beta(a, b), and r of n independent participants respond, then
The posterior mean is (a + r)/(a + b + n). It can be expressed as a weighted average of the prior mean a/(a + b) and observed proportion r/n, with weights proportional to a + b and n. The concentration a + b is often treated as a prior effective sample-size convention in this model; it does not by itself establish that a prior is equivalent to real exchangeable participants.
The posterior probability of exceeding a benchmark is
For m future participants, the predictive distribution of future responders K is beta-binomial:
This integrates over uncertainty in p. Substituting only the posterior mean into a binomial model generally understates predictive variability.
Worked example
Worked example: eight responses in ten participants
Start with a uniform Beta(1, 1) prior. After 8 responses and 2 nonresponses, the posterior is Beta(9, 3). Its mean is 9/12 = 0.750, lower than the observed proportion 0.800 because the prior pulls toward 0.5.
For a benchmark p₀ = 0.5, the posterior probability P(p > 0.5) is approximately 0.9673. A single-look rule requiring a probability above 0.95 would declare success; a threshold above 0.99 would not. This is a property of the stated posterior and rule, not a universal recommendation for trial design.
Consider the same observations under Beta(10, 10). The posterior becomes Beta(18, 12), with mean 0.600. That prior has much more influence. The difference illustrates why prior choice should be evaluated before outcomes are known.
| Prior | Posterior | Posterior mean |
|---|---|---|
| Beta(1, 1) | Beta(9, 3) | 0.750 |
| Beta(10, 10) | Beta(18, 12) | 0.600 |
The predictive chance that the next participant responds under Beta(9, 3) is 0.750. The probability that the trial will eventually cross a success threshold requires summing over possible future response counts and the entire stopping schedule.
Suppose 500 of 10,000 simulated trials succeed when the true response rate equals the null benchmark. The estimated null success rate is 0.05, with MCSE approximately 0.00218. It must be measured from the complete procedure rather than inferred from the posterior threshold alone.
Interpreting the results
Posterior certainty and design error rates are different
At each look, a beta-binomial model updates uncertainty about a response probability. A posterior success probability concerns that parameter given the observed responses and prior. Across repeated simulated trials, the frequency of meeting the success rule is an operating characteristic of the entire design.
Under the null response probability, that frequency is a false-success rate for the specified rule. Under an alternative, it is the design's chance of success. A posterior threshold of 0.95 does not automatically imply a frequentist false-positive rate of 0.05 when there are multiple looks or an informative prior.
Expected sample size reflects early stopping as well as maximum enrollment. A futility stop means the posterior evidence failed the continuation rule at that look; it is not proof that the response probability is below the null. Report the prior, thresholds, look schedule, and simulated operating characteristics together.
Assumptions and limitations
The prior and likelihood both need justification
The beta-binomial model assumes independent binary responses with a common response probability in the modeled population. Delayed outcomes, dropout, subgroup heterogeneity, or changing eligibility can violate that simple setup. A single-arm historical benchmark also requires a credible comparison population and consistent endpoint definition.
A beta prior is determined by its mean and concentration. A seemingly mild prior mean can still exert strong influence if its concentration is large. Historical information may not be exchangeable with current patients, so borrowing needs a substantive rationale rather than automatic reuse.
Posterior thresholds, maximum size, and interim looks should be chosen before observing trial outcomes. Success and futility are decision rules, not universal probability conventions. Regulatory or scientific acceptability depends on the design's behavior and context, not merely on labeling it Bayesian.
Diagnostics and sensitivity analysis
Evaluate prior sensitivity and the full response-rate range
Simulate at the null and several clinically meaningful response probabilities. Check false success, power, expected N, and where stops occur. A design may perform well at an optimistic alternative but poorly for a modest improvement.
Vary prior mean and concentration to see how much they drive early success or futility. Examine scenarios in which historical information conflicts with current data. Report Monte Carlo uncertainty, particularly for small false-success probabilities where many replications may be needed.
Check how changing the number of looks alters behavior even when posterior thresholds are unchanged. The FDA Bayesian guidance discusses operating-characteristic evaluation and prior sensitivity. These checks assess the proposed design; they do not validate the historical benchmark itself.
References and other material
Read the methodological references and supporting tutorials below. The examples in this lesson are original teaching calculations, not findings from these sources.
- FDA: Bayesian statistics in medical device clinical trials — Bayesian trial principles; this guidance specifically concerns medical devices.