External prediction model validation
Does an existing model give reliable predictions in an independent population?
When to use this project
Evaluate a fully specified, frozen prediction model in independent data representing a proposed use. Assess calibration, discrimination and usefulness with adequate precision.
How it differs
External validation evaluates an existing model without refitting it first. Internal validation resamples development data. Recalibration or revision creates an updated model that needs its own evaluation.
Decisions before you start
- Obtain the exact model, intercept or baseline survival, preprocessing and input definitions.
- Define independence from development data and compare setting, case mix, outcome ascertainment and horizon.
- Plan precision for calibration and discrimination, preserve original-model results and evaluate any updating separately.
How the flows work together
Each flow below links to its diagram, steps and tables. Flow order organizes the project; it does not imply that all work is strictly sequential. Data assessment, analysis planning and protocol refinement may inform one another.
1. Validation study overview
Overview · Existing model through independent evaluation
Use this overview to state the purpose and link to detailed decisions. Keep a versioned protocol and decision register, and add evidence references for the rationale.
2. Model and population specification
Design · Define frozen model and applicability
A frozen model includes coefficients, intercept or baseline survival, preprocessing and input definitions. Independence from development data must be explicit. Plan precision for several performance domains rather than relying on a minimum event count alone.
3. Data mapping and prediction
Data · Reproduce predictors and generate predictions
Generate predictions from the original specification before any local updating. Missing required inputs and altered measurements are applicability problems to document, not permission to quietly refit the model.
4. Performance and precision plan
Statistics · Estimate validation metrics and uncertainty
Assess calibration across the risk range, discrimination and relevant utility. Case-mix changes can affect performance summaries. For time-to-event outcomes, align the prediction horizon and use methods that address censoring.
5. Transportability and updating decisions
Statistics · Separate evaluation from model modification
Preserve evaluation of the original model. Recalibration, coefficient revision and extension are distinct changes; all produce an updated model whose performance needs a separate validation strategy.
Equations and statistical interpretation
These explanations describe notation and assumptions. Flow does not calculate these values.
Calibration intercept and slope
For a fully observed binary outcome, probabilities p̂ come from the frozen model. Ideal calibration intercept a is 0 and slope b is 1.
Fit this evaluation relationship without replacing the original predictions. A calibration curve can reveal departures hidden by two summary values. Survival outcomes require horizon-specific methods that address censoring.
Published example
Examine the independent cohort, differences from development and calibration assessment. QRISK2 is the model evaluated here; it is not the later QRISK3 model.
This publication illustrates a study design, not an endorsement of every method or a reproduction of its results. Follow its protocol, supplement and update history when studying the example.
Hypothetical scenario
A regional health network wants to assess an existing 10-year cardiovascular risk score before adopting it.
- Validation study overview: Specify the intended decision and identify the released model version.
- Model and population specification: Select an independent cohort and plan sample size for performance precision.
- Data mapping and prediction: Map predictors and generate predictions using the original model without refitting coefficients.
- Performance and precision plan: Compare predicted and observed risks, discrimination and utility with appropriate handling of censoring.
- Transportability and updating decisions: If calibration is poor, assess updating as a separate model and validate it independently.
Deliver a populated protocol, decision log, data specification, and planned outputs. Enter numerical findings only after analysis and verification. Keep decisions made before seeing results distinguishable from later amendments.
Sources
- TRIPOD+AI 2024: reporting prediction models using regression or machine learning
- Riley et al. (2024): how to undertake an external validation study
- Riley et al. (2024): sample size for external validation
Source check: 2026-09-17. Record the version of each reporting or appraisal tool used in your actual protocol.
Compare all Diagnostic and prediction studies project types · Open editor in a new tab