Development and Assessment of a Portability Strategy for Computable Operational Definitions
Author(s)
James R. Rogers, PhD1, Amanda Shields, BA2, Vladimir Turzhitsky, MS, PhD1, Shefali Haldar, PhD1, Aimee Harrison, MFA2, Aaron Kamauu, MD, MS, MPH2.
1Merck & Co., Inc., Rahway, NJ, USA, 2Navidence, Aurora, CO, USA.
1Merck & Co., Inc., Rahway, NJ, USA, 2Navidence, Aurora, CO, USA.
OBJECTIVES: Portability, defined as applying computable definitions to different data sources, is important for establishing consistency in medical concept representations for real-world evidence generation. The aim of this study is to compare differing validated literature review computable definitions by using weakly supervised machine learning as a possible benchmark for assessing portability for type 2 diabetes mellitus (T2DM).
METHODS: Validated T2DM literature review definitions were found via a targeted literature review, with cohorts built using a claims database data-cut (Optum’s de-identified Clinformatics® Data Mart Database). To create a portability benchmark for these cohorts, a random forest model predicting T2DM was trained using a silver standard label, where T2DM cases were defined as at least 5 T2DM dx present while non-cases were patients with no T2DM dx. Model inputs were relevant demographics, comorbidities, medications, and healthcare utilization derived from T2DM published work. Final model predictions served as the benchmark for the T2DM cohorts. Performance was measured using F1-score.
RESULTS: The silver standard testing set contained 32,334 cases and 301,006 non-cases. The final model developed performed with F1-score of 0.79, leading to 27,162 cases for benchmarking. A total of 10 validated literature review definitions were analyzed, with all validated using chart review in their respective studies. When benchmarking against the final model, F1-scores ranged from 0.27 (at least 1 inpatient T2DM dx) to 0.94 (at least 1 T2DM dx and at least 1 antiDM medication). The next highest F1-scores were 0.93 (at least 1 antiDM medication) and 0.78 (at least 1 T2DM dx or at least 1 antiDM; at least 1 T2DM dx).
CONCLUSIONS: Among the derived literature review algorithms, those involving antiDM use performed best at identifying T2DM when benchmarked against the developed machine learning model. This portability strategy can provide a shared benchmark when choosing among different computable definitions.
METHODS: Validated T2DM literature review definitions were found via a targeted literature review, with cohorts built using a claims database data-cut (Optum’s de-identified Clinformatics® Data Mart Database). To create a portability benchmark for these cohorts, a random forest model predicting T2DM was trained using a silver standard label, where T2DM cases were defined as at least 5 T2DM dx present while non-cases were patients with no T2DM dx. Model inputs were relevant demographics, comorbidities, medications, and healthcare utilization derived from T2DM published work. Final model predictions served as the benchmark for the T2DM cohorts. Performance was measured using F1-score.
RESULTS: The silver standard testing set contained 32,334 cases and 301,006 non-cases. The final model developed performed with F1-score of 0.79, leading to 27,162 cases for benchmarking. A total of 10 validated literature review definitions were analyzed, with all validated using chart review in their respective studies. When benchmarking against the final model, F1-scores ranged from 0.27 (at least 1 inpatient T2DM dx) to 0.94 (at least 1 T2DM dx and at least 1 antiDM medication). The next highest F1-scores were 0.93 (at least 1 antiDM medication) and 0.78 (at least 1 T2DM dx or at least 1 antiDM; at least 1 T2DM dx).
CONCLUSIONS: Among the derived literature review algorithms, those involving antiDM use performed best at identifying T2DM when benchmarked against the developed machine learning model. This portability strategy can provide a shared benchmark when choosing among different computable definitions.
Conference/Value in Health Info
2025-11, ISPOR Europe 2025, Glasgow, Scotland
Value in Health, Volume 28, Issue S2
Code
MSR70
Topic
Methodological & Statistical Research, Real World Data & Information Systems
Topic Subcategory
Artificial Intelligence, Machine Learning, Predictive Analytics
Disease
Diabetes/Endocrine/Metabolic Disorders (including obesity), No Additional Disease & Conditions/Specialized Treatment Areas