OBJECTIVES

Patient-level simulation (PLS) models can be built using various software packages; these have been compared in simplified cases, but not using a complex model developed for national decision-making. We replicated an existing PLS model of this type to evaluate the advantages and disadvantages of different software.

METHODS

The model was originally created for a NICE clinical guideline on hyperphosphataemia in chronic kidney disease (CG157). It was built in visual basic for applications (VBA). PLS was chosen because it allowed the model to translate continuous biochemical measures from trials into effects on patient-relevant outcomes. We replicated the VBA model in Simul8^®, a dedicated simulation environment, to explore differences in build-time, results, run-time and usability.

RESULTS

The Simul8^® implementation was slower than the original VBA model when simulating 8 cohorts of 10,000 patients (approximately 12 minutes versus 7.5 minutes). The model structure required the use of multiple patient-level variables (‘labels’) in Simul8^®, which are known to affect run-time. When event-logs were recorded for debugging, both VBA and Simul8^® took considerably longer and Simul8^® crashed frequently. Model validation was challenging due to the stochastic nature of PLS. When potentially non-trivial differences emerged between implementations, it was difficult to determine whether these were due to technical errors, Monte-Carlo error, or bias from different pseudo-random number generators, an area in which VBA is known to perform poorly. Object-oriented VBA was difficult for new users to grasp and coding errors were difficult to identify. The Simul8^® interface allowed for patients’ routes through the model to be easily visualised and tracked; however, some simple operations (e.g. lookups across categories) required convoluted coding.

CONCLUSIONS

It is not clear that dedicated simulation software provides meaningful benefits over a generic scripting language for PLS. We are extending this project to encompass further replications in R and discretely integrated condition event simulation (DICE).