OBJECTIVES:

Validation of budget impact models is not commonly reported. This study evaluates three types of validity for a budget impact model built to predict the financial consequences of a payer’s formulary and utilization management changes for antidiabetic medications.

METHODS:

Three types of validation for a budget impact model were evaluated: i) face validity, ii) internal verification, and iii) predictive validity. Face validity was assessed using a questionnaire given to internal and external experts. Internal verification was checked through structured walk-throughs, unit tests, extreme condition tests, traces, replication tests, and double programming. Predictive validity was determined using actual claims data by comparing one-year empirical results to model predictions.

RESULTS:

Face validity assessment identified model edits such as adding an adjustment for patients with other health insurance and a modification for medications entering the market. Errors identified by internal verification checks were corrected. Predictive validity checks revealed that the model did not perform well based on an assumption regarding utilization growth, resulting in removal of this assumption in future models.

CONCLUSIONS:

A rigorous validation process improved our model performance and results. Future reports should state what face validity, internal verification, and predictive validity checks were performed to increase credibility and usefulness of budget impact models.

SPONSORSHIP:

This project was supported by grant number F32HS024857 from the Agency for Healthcare Research and Quality (AHRQ). The content is solely the responsibility of the authors and does not necessarily represent the official views of the AHRQ.