AI-DRIVEN ISPOR-SMDM VALIDATION OF AN EXCEL-BASED COST-EFFECTIVENESS MODEL
Author(s)
Jag Chhatwal, PhD1, Sumeyye Samur, PhD2, Ismail Fatih Yildirim, MSc2, Mine Tekman, PhD2, Turgay Ayer, PhD2.
1Associate Professor and Director of the Institute for Technology Assessment, Harvard Medical School / Massachusetts General Hospital, Boston, MA, USA, 2Value Analytics Labs, Boston, MA, USA.
1Associate Professor and Director of the Institute for Technology Assessment, Harvard Medical School / Massachusetts General Hospital, Boston, MA, USA, 2Value Analytics Labs, Boston, MA, USA.
OBJECTIVES: To evaluate whether an AI system can autonomously perform a comprehensive ISPOR-SMDM five-dimension validation of an Excel-based health economic model.
METHODS: We provided an AI-generated Excel-based Markov model of cost-effectiveness of semaglutide for metabolic dysfunction-associated steatohepatitis (MASH) to HEOR-specific agentic platform, ValueGen.AI. The model consisted of 7 health states, annual cycle, a 40-year lifetime horizon, 3% discounting, and a U.S. healthcare perspective. Validation encompassed all five ISPOR-SMDM dimensions: (1) conceptual validation via AI-driven structural review; (2) internal verification; (3) external validation against published cost-effectiveness estimates; (4) cross-validation assessment; and (5) predictive validation feasibility assessment. Dynamic tests included setting discount rates to 0% and 5%, zeroing all costs and utilities, equalizing treatment efficacy, eliminating mortality, and varying treatment effect magnitude.
RESULTS: The entire model validation was conducted in 65 minutes using 3.7 million tokens. The model contained 1,360 formulas across 13 worksheets with zero formula execution errors. All 25 embedded validation checks passed, and all 11 dynamic extreme-value tests produced expected directional outcomes (e.g., zero utilities yielded zero QALYs). Conceptual validation identified four structural limitations: (1) an unused 12% annual discontinuation rate parameter that was defined but not implemented; (2) constant background mortality (1.5%/year) independent of age; (3) drug costs incorrectly applied to all alive patients, instead of limiting to F2 and F3 fibrosis states; (4) absence of a liver transplantation health state. External validation showed that the model estimated an ICER of $10,228/QALY compared with $42,200-$44,138/QALY reported by Njei et al. (2026), attributable primarily to MFN versus wholesale acquisition cost drug pricing. Cross-validation and predictive validation could not be performed due to absence of independent long-term outcome data.
CONCLUSIONS: Our HEOR-specific agentic AI framework successfully conducted systematic validation of Excel-based model in one hour. Results suggest that AI can meaningfully augment human model review and successfully conduct full model validation per ISPOR-SMDM recommendations.
METHODS: We provided an AI-generated Excel-based Markov model of cost-effectiveness of semaglutide for metabolic dysfunction-associated steatohepatitis (MASH) to HEOR-specific agentic platform, ValueGen.AI. The model consisted of 7 health states, annual cycle, a 40-year lifetime horizon, 3% discounting, and a U.S. healthcare perspective. Validation encompassed all five ISPOR-SMDM dimensions: (1) conceptual validation via AI-driven structural review; (2) internal verification; (3) external validation against published cost-effectiveness estimates; (4) cross-validation assessment; and (5) predictive validation feasibility assessment. Dynamic tests included setting discount rates to 0% and 5%, zeroing all costs and utilities, equalizing treatment efficacy, eliminating mortality, and varying treatment effect magnitude.
RESULTS: The entire model validation was conducted in 65 minutes using 3.7 million tokens. The model contained 1,360 formulas across 13 worksheets with zero formula execution errors. All 25 embedded validation checks passed, and all 11 dynamic extreme-value tests produced expected directional outcomes (e.g., zero utilities yielded zero QALYs). Conceptual validation identified four structural limitations: (1) an unused 12% annual discontinuation rate parameter that was defined but not implemented; (2) constant background mortality (1.5%/year) independent of age; (3) drug costs incorrectly applied to all alive patients, instead of limiting to F2 and F3 fibrosis states; (4) absence of a liver transplantation health state. External validation showed that the model estimated an ICER of $10,228/QALY compared with $42,200-$44,138/QALY reported by Njei et al. (2026), attributable primarily to MFN versus wholesale acquisition cost drug pricing. Cross-validation and predictive validation could not be performed due to absence of independent long-term outcome data.
CONCLUSIONS: Our HEOR-specific agentic AI framework successfully conducted systematic validation of Excel-based model in one hour. Results suggest that AI can meaningfully augment human model review and successfully conduct full model validation per ISPOR-SMDM recommendations.
Conference/Value in Health Info
2026-11, ISPOR Europe 2026, Vienna, Austria
Value in Health, Volume 29, Issue 12S
Code
MSR129
Topic
Methodological & Statistical Research
Topic Subcategory
Artificial Intelligence, Machine Learning, Predictive Analytics
Disease
Diabetes/Endocrine/Metabolic Disorders (including obesity), Gastrointestinal Disorders