Diagnostic and prediction studies
Choose by the question: evaluate a test, assess an existing prediction model, or develop a new model. Start with the intended clinical decision and population. The order below moves from focused evaluation toward model development; practical complexity also depends on the data and methods.
Which study should I use?
| Study type | When to use it | How it differs |
|---|---|---|
| Diagnostic accuracy study | Evaluate an index test against a defined reference standard in people representing the intended clinical pathway. Specify the target condition, test role and decision threshold. | Accuracy concerns classification against a reference standard. A prediction model combines inputs to estimate an individual probability; an accuracy study alone does not establish that using a test improves patient outcomes. |
| External prediction model validation | Evaluate a fully specified, frozen prediction model in independent data representing a proposed use. Assess calibration, discrimination and usefulness with adequate precision. | External validation evaluates an existing model without refitting it first. Internal validation resamples development data. Recalibration or revision creates an updated model that needs its own evaluation. |
| Clinical prediction model development | Develop a prediction model when existing models do not adequately serve the intended population, outcome or decision. Include internal validation of the complete development procedure. | Development estimates a new model; external validation tests an existing frozen model elsewhere. Predictive associations do not establish causal effects of changing the predictors. |
Accuracy, prediction and clinical benefit
An accuracy study compares an index test with a reference standard for a target condition. A prediction model estimates an individual outcome probability from multiple inputs; the outcome may be present now or occur in the future. Neither type of evaluation alone shows that introducing the test or model improves patient outcomes. That requires an appropriate impact evaluation.
Terms used in these guides
- Index test and reference standard
- The index test is being evaluated; the reference standard defines the target-condition classification against which it is assessed. Limitations of the reference standard affect interpretation.
- Prediction time and horizon
- Prediction time is when the model would be used. The horizon specifies when the outcome is assessed. Predictors must be available at the intended decision time.
- Calibration and discrimination
- Calibration assesses agreement between predicted probabilities and observed frequencies. Discrimination assesses how well predictions rank people with different outcomes. Both need evaluation.
- Data leakage
- Information from evaluation outcomes or the future enters fitting, preprocessing or selection, making reported performance misleadingly favorable.
- Internal and external validation
- Internal validation evaluates the development procedure within its data, usually through resampling. External validation evaluates a frozen model in independent data. Updating changes the model and requires separate evaluation.
Protocol and reporting
Keep a dated protocol and amendments, specifying sampling, measurements and analysis before evaluation. Identify any applicable registry explicitly. A diagnostic or prediction study is not automatically an interventional clinical trial. Use STARD for diagnostic accuracy and TRIPOD+AI for prediction model development or validation using regression or machine learning; TRIPOD+AI is not limited to artificial intelligence models.
Sources
- STARD 2015: diagnostic accuracy reporting checklist
- TRIPOD+AI 2024: reporting prediction models using regression or machine learning
- Riley et al. (2020): sample size for prediction model development
- Riley et al. (2024): how to undertake an external validation study
- Riley et al. (2024): sample size for external validation