Clinical prediction model development
How can information available now predict an individual clinical outcome?
When to use this project
Develop a prediction model when existing models do not adequately serve the intended population, outcome or decision. Include internal validation of the complete development procedure.
How it differs
Development estimates a new model; external validation tests an existing frozen model elsewhere. Predictive associations do not establish causal effects of changing the predictors.
Decisions before you start
- Define prediction time and horizon before selecting variables; exclude information unavailable at the intended decision.
- Justify sample size using outcome frequency, candidate parameters and anticipated performance rather than a universal events-per-variable rule.
- Keep imputation, transformations, selection and tuning inside resampling. Release the complete model and specify future external validation.
How the flows work together
Each flow below links to its diagram, steps and tables. Flow order organizes the project; it does not imply that all work is strictly sequential. Data assessment, analysis planning and protocol refinement may inform one another.
1. Prediction study overview
Overview · Intended use through model reporting
Use this overview to state the purpose and link to detailed decisions. Keep a versioned protocol and decision register, and add evidence references for the rationale.
2. Population and data suitability
Selection · Define development population and sources
Prediction time is the moment the model would be used. Sample-size planning depends on outcome frequency, candidate parameter count and expected performance; a universal events-per-variable rule is insufficient.
3. Predictors and outcome construction
Data · Align measurements without information leakage
Information recorded after prediction time can leak the outcome into the model. Define measurement availability and a feasible missing-input strategy at deployment. All learned preprocessing belongs inside resampling.
4. Model development plan
Statistics · Specify fitting tuning and complexity
Predictors serve prediction and need not represent causal effects. Tune and select models within the development procedure, then evaluate that procedure using data not used for the corresponding fitting decisions. Retain the full executable specification.
5. Internal validation plan
Statistics · Estimate optimism using appropriate resampling
Repeat imputation, transformations, feature selection and tuning in the appropriate resampling loops. Respect repeated patients, hospitals and chronology. Internal validation estimates optimism; it does not demonstrate performance in a new setting.
6. Performance and usefulness assessment
Statistics · Assess calibration discrimination and utility
Calibration concerns agreement between predicted probabilities and observed frequencies; discrimination concerns ranking. Report both with uncertainty. Decision utility depends on plausible thresholds and consequences, not only a high area under the curve.
Equations and statistical interpretation
These explanations describe notation and assumptions. Flow does not calculate these values.
Brier score for a binary outcome
yᵢ is the observed binary outcome and p̂ᵢ its predicted probability for participant i; n is the number evaluated. Lower scores indicate less squared prediction error.
Evaluate predictions held out from their fitting procedure or use optimism correction. The score depends on outcome frequency and is not a substitute for calibration plots. Censored outcomes require an appropriate extension.
Published example
Hippisley-Cox et al. (2017), Development and validation of QRISK3 risk prediction algorithms
Inspect predictor definitions, model specification and how development and validation data were separated. The published development study is an example, not a prescription to use its predictor set for other outcomes.
This publication illustrates a study design, not an endorsement of every method or a reproduction of its results. Follow its protocol, supplement and update history when studying the example.
Hypothetical scenario
A hospital network wants to predict 30-day readmission using information available at discharge.
- Prediction study overview: Define the prediction decision and explain why existing scores are insufficient.
- Population and data suitability: Identify development hospitals and align eligibility, discharge time and outcome capture.
- Predictors and outcome construction: Specify predictors available at discharge, readmission ascertainment and missing-input handling at deployment.
- Model development plan: Fit prespecified candidate procedures with controlled complexity and separate tuning from evaluation.
- Internal validation plan: Resample the full procedure, respecting repeated patients and hospitals, to estimate optimism.
- Performance and usefulness assessment: Report calibration, discrimination and clinically relevant decision performance, with an external validation plan.
Deliver a populated protocol, decision log, data specification, and planned outputs. Enter numerical findings only after analysis and verification. Keep decisions made before seeing results distinguishable from later amendments.
Sources
- TRIPOD+AI 2024: reporting prediction models using regression or machine learning
- Riley et al. (2020): sample size for prediction model development
- Riley et al. (2024): how to undertake an external validation study
Source check: 2026-09-17. Record the version of each reporting or appraisal tool used in your actual protocol.
Compare all Diagnostic and prediction studies project types · Open editor in a new tab