EVALUATING MODEL PERFORMANCE FOR BETWEEN-COUNTRY SURVIVAL TRANSPORTABILITY
Author(s)
Mohamed S. Ali, PharmD, SM1, Harlan Pittell, PhD1, Elsie Horne, PhD2, Philani Mpofu, PhD1, Qianyi Zhang, MS1, Blythe Adamson, PhD, MPH1;
1Flatiron Health, New York, NY, USA, 2Flatiron Health UK, London, United Kingdom
1Flatiron Health, New York, NY, USA, 2Flatiron Health UK, London, United Kingdom
OBJECTIVES: Transportability analyses evaluate whether real-world evidence from one country can inform decisions in another. Evidence concerning overall survival (OS) typically relies on outcome regression models. Good practice requires assessing model performance using internal validation within the data used to estimate the model, before applying predicted OS for transportability. We aimed to demonstrate the value of evaluating model performance in an independent sample before using modeled OS for transportability, using a United States (US) cohort of patients with multiple myeloma (MM) as a case study.
METHODS: We used the US Flatiron Health Research Database to identify patients with MM diagnosed between 2015 and 2021, defined OS from first-line treatment to death or censoring, and randomly split the cohort into training (70%) and test (30%) sets. In the training set, OS was modeled using pooled logistic regression (PLR) with baseline age, sex, CD38 inhibitor use, chemotherapy, immunomodulatory drug use, proteasome inhibitor use, stem-cell transplantation within one year, time, and covariate-time interactions. The fitted PLR model was applied to the test set to generate survival predictions marginalized over the test covariate distribution. Model performance in the test set was evaluated by estimating Harrell’s C-index (discrimination) and comparing PLR-predicted and Kaplan-Meier (KM) survival using the mean absolute difference (MAD) over 0-60 months (calibration).
RESULTS: The cohort included 9,937 patients (7,004 training; 2,933 test). In the test set, the PLR model showed good discrimination (C-index: 0.72), and the marginalized PLR curve closely aligned with the KM curve, with a MAD of 0.0138 over 0-60 months (≈1.4 percentage points).
CONCLUSIONS: The PLR model demonstrated good discrimination and close agreement between predicted and observed survival in the test cohort, indicating strong model performance. Model performance should be assessed before using modeled OS in transportability analyses to avoid inaccurately attributing modeling misspecifications to between-country differences.
METHODS: We used the US Flatiron Health Research Database to identify patients with MM diagnosed between 2015 and 2021, defined OS from first-line treatment to death or censoring, and randomly split the cohort into training (70%) and test (30%) sets. In the training set, OS was modeled using pooled logistic regression (PLR) with baseline age, sex, CD38 inhibitor use, chemotherapy, immunomodulatory drug use, proteasome inhibitor use, stem-cell transplantation within one year, time, and covariate-time interactions. The fitted PLR model was applied to the test set to generate survival predictions marginalized over the test covariate distribution. Model performance in the test set was evaluated by estimating Harrell’s C-index (discrimination) and comparing PLR-predicted and Kaplan-Meier (KM) survival using the mean absolute difference (MAD) over 0-60 months (calibration).
RESULTS: The cohort included 9,937 patients (7,004 training; 2,933 test). In the test set, the PLR model showed good discrimination (C-index: 0.72), and the marginalized PLR curve closely aligned with the KM curve, with a MAD of 0.0138 over 0-60 months (≈1.4 percentage points).
CONCLUSIONS: The PLR model demonstrated good discrimination and close agreement between predicted and observed survival in the test cohort, indicating strong model performance. Model performance should be assessed before using modeled OS in transportability analyses to avoid inaccurately attributing modeling misspecifications to between-country differences.
Conference/Value in Health Info
2026-05, ISPOR 2026, Philadelphia, PA, USA
Value in Health, Volume 29, Issue S6
Code
MSR169
Topic
Methodological & Statistical Research
Disease
No Additional Disease & Conditions/Specialized Treatment Areas, SDC: Oncology