ISPOR – SMDM Joint Model Validation II Task Force: Making the Black Box Transparent - New Tools for Quality Assurance in Health Economic Modeling

Ensuring that a health economic (HE) model once implemented aligns with its conceptual framework and is “error-free” are essential elements of model validation, next to conceptual model validation and validation of model outputs.

This open meeting emphasizes discussion with the audience to receive input to update guidance on model validation. After a brief introduction, there will be 3 short presentations followed by a longer than usual overarching Q&A session. First, we address available guidance, like the 2012 ISPOR good practice guidelines and various national guidelines. For instance, the Canada Drug Agency (CDA) guidelines emphasize verification testing. The recent Dutch guidelines refer to dedicated tools. How specific should guidelines be and how to account for different model structures? Second, lessons learned from using the TECH-VER model verification instrument for 3rd party quality assessment tasks on different models will be shared, including the role of checklists, scoring results, and black box versus white box testing. Third, depending on the HTA agency, validation is done internally or by independent researchers, with varying levels of time and access to “turn a model upside down”. We will discuss the practical advantages and drawbacks of these approaches and their implications for assigning responsibility in model validation. In addition, we will present empirical experience with the CDA model validation tool, highlighting its application for model code verification under real-world conditions.  

At this meeting, there will be Q&A with the audience on further challenges in model validation, including the potential role of AI and the choice of software.