Generalizability of a Randomized Clinical Trial-Based Prediction Model in a Real-World Population
Author(s)
Wen Ye, PhD1, Jing Li, MS2.
1Research Scientist, University of Michigan, Ann Arbor, MI, USA, 2University of Michigan, Ann Arbor, MI, USA.
1Research Scientist, University of Michigan, Ann Arbor, MI, USA, 2University of Michigan, Ann Arbor, MI, USA.
OBJECTIVES: Studies evaluating randomized controlled trial (RCT) based prognostic models in real-world population are lacking. This study aims to externally evaluate an RCT-based mortality risk equation using a real-world population specifically for people with type 2 diabetes (T2D).
METHODS: We utilized data from the National Health and Nutrition Examination Survey (NHANES) participants with T2D from 2001 to 2018, along with their matched National Death Index dataset. The analysis focused on the published total mortality risk equation from the RECODe models (the Risk Equations for Complications of Type 2 Diabetes). The study assessed discrimination using the concordance index (C index) and calibration through observed-to-expected event rates (OEER) at 10 years. Analyses were conducted for the entire participant pool and stratified by race, sex, age, and income groups.
RESULTS: The analysis included 4,989 NHANES participants with T2D (mean age 63 years, 48% female, 65% non-White, and 48% with an income poverty ratio [IPR] ≤ 2). The overall OEER for the RECODe model was 0.57 (95% CI: 0.54, 0.60) and the C index was 0.77 (SE: 0.007). While the RECODe model's performance was consistent across sexes and races, it was notably less accurate for patients aged 75 and older (OEER = 0.49, C index = 0.63) and those with an IPR ≤ 2 (OEER = 0.53, C index = 0.74).
CONCLUSIONS: This study demonstrates that an RCT-based all-cause mortality model for T2D patients considerably underestimates the mortality risk in real-world populations, particularly among patients over 75 years old and those with low income. The results highlight the need for careful evaluation of RCT data when developing and validating prognostic models, emphasizing the importance of considering when and to whom these models should be applied.
METHODS: We utilized data from the National Health and Nutrition Examination Survey (NHANES) participants with T2D from 2001 to 2018, along with their matched National Death Index dataset. The analysis focused on the published total mortality risk equation from the RECODe models (the Risk Equations for Complications of Type 2 Diabetes). The study assessed discrimination using the concordance index (C index) and calibration through observed-to-expected event rates (OEER) at 10 years. Analyses were conducted for the entire participant pool and stratified by race, sex, age, and income groups.
RESULTS: The analysis included 4,989 NHANES participants with T2D (mean age 63 years, 48% female, 65% non-White, and 48% with an income poverty ratio [IPR] ≤ 2). The overall OEER for the RECODe model was 0.57 (95% CI: 0.54, 0.60) and the C index was 0.77 (SE: 0.007). While the RECODe model's performance was consistent across sexes and races, it was notably less accurate for patients aged 75 and older (OEER = 0.49, C index = 0.63) and those with an IPR ≤ 2 (OEER = 0.53, C index = 0.74).
CONCLUSIONS: This study demonstrates that an RCT-based all-cause mortality model for T2D patients considerably underestimates the mortality risk in real-world populations, particularly among patients over 75 years old and those with low income. The results highlight the need for careful evaluation of RCT data when developing and validating prognostic models, emphasizing the importance of considering when and to whom these models should be applied.
Conference/Value in Health Info
2025-11, ISPOR Europe 2025, Glasgow, Scotland
Value in Health, Volume 28, Issue S2
Code
MSR114
Topic
Methodological & Statistical Research
Topic Subcategory
Artificial Intelligence, Machine Learning, Predictive Analytics
Disease
Diabetes/Endocrine/Metabolic Disorders (including obesity)