The Decision Uncertainty Toolkit: Risk Measures and Visual Outputs to Support Decision Making during Public Health Crisis

BACKGROUND: Decision-makers have relied on infectious disease (ID) models to predict and estimate the impact of various policy alternatives throughout the COVID-19 pandemic. While the outputs of most decision analytic models follow probabilistic distributions, decisions are discrete, and ID modelling outputs have not always met decision-makers’ needs to weigh uncertainty and risk as they decide which policies to implement. ID models use different approaches to parameter uncertainty than health economic models, which complicates the measurement and communication of risk. Furthermore, the outputs of these models are not always easily interpreted by decision-makers. The difficulties associated with communicating uncertainty have meant that decision-making has occurred with a high risk of making decisions that are misaligned to policy objectives.

OBJECTIVES: To adapt and extend health economic methods for the visualization, measurement, and communication of uncertainty to ID modeling.

METHODS: In consultation with decision-makers and ID modelling experts, we develop the ‘Decision Uncertainty Toolkit’ with guidance on methods, risk visualization and measurement, as well as communication with decision-makers. We develop methods to integrate traditional ID approaches to uncertainty (which focus on calibration parameters) with probabilistic sensitivity analysis (which typically applies to a broader parameter set). Visualizations are developed to quantify risk probabilistically. Quantitative measures of downside risk for policy alternatives are specified to capture both the probability and magnitude of losses relative to policy targets. To better communicate with decision makers, we embed decision thresholds within visualizations, aligning outputs more directly with policy objectives.

RESULTS: We develop the toolkit visuals and risk measures through a series of workshops with ID modellers and decision-makers in early 2023. Adoption of the toolkit will support decision-making by quantifying outcome uncertainty and the risks associated with policy alternatives.

CONCLUSIONS: The toolkit is designed to improve decision-maker understanding of decision risk and could improve outcomes during future health shocks.