OBJECTIVES

: Decision-analytic models require a calibration step when model parameter values are not directly observable but can be fitted to external data. Our objectives were to (1) assess the performance of different optimization algorithms to calibrate a simple Markov model with transition probabilities to match observed cohort proportions at three different times, (2) compare the run times and achieved goodness of fit metrics, and (3) inform best strategies for using optimization algorithms to calibrate models.

METHODS

: A 3-state (Markov) state-transition cohort model describing simple disease progression served as reference. Transition probabilities are described by Weibull distributions defined by rate and shape parameters. The model is calibrated to observed cohort proportions generated by hypothetical “given” Weibull parameters. Calibration was applied using Scientific Python package with 11 different optimization algorithms (including hill-climbers, stochastic and hybrid types). 50 different initial sets of values of the 6 parameters where used to calculate the goodness of fit.

RESULTS

: The hill-climbing types tended to be 2 orders of magnitude faster (several minutes) than stochastic types which are fast, but are prone to identify local minima far away from actual solution (GoF on the order of 1E-03). The stochastic algorithms are more robust in finding global minima, however they require on average >1hr of run time to find a solution in order to reach globally lowest GoF (GoF on the order of 1E-08). The only 2 algorithms that consistently found global solutions are the stochastic types: Basinhopping (mean run time >1hr and GoF 1E-08 and Differential Evolution (run time >3hrs and an impressive GoF of 3E-19).

CONCLUSIONS

: In our simple modeling example, we calibrated Weibull parameters within a Markov cohort model simultaneously adjusting 6 parameters. This approach allows an assessment of performance of different optimization strategies and also to develop hybrid approach of using combination of algorithms.