Presentation (CTI)

OBJECTIVES: When comparing statistical models in for analysis of real-world data (RWD), researchers routinely face a choice between precise but biased estimators and less biased but less precise estimators. For example, choosing between ordinary least squares and instrumental variable methods, or selecting between simple models and flexible approaches that capture effect heterogeneity but risk overfitting. While the machine learning literature commonly utilizes mean squared error (MSE) to guide estimator choices, MSE lacks a direct link to decision-making payoffs or expected outcomes. This study aims to develop a methodology for the evaluation of statistical estimators based on expected health gain; this requires explicitly recognizing the bias-variance trade-off.
METHODS: INB represents the difference between the additional health generated by a technology and the health displaced. Under finite sample sizes, decision-makers face a risk of making false-positive or false-negative decision errors. An increase in estimator variance makes the observed INB more likely to cross decision thresholds due to chance, while introducing bias alters error risks depending on its direction. For a given estimator, our methodology computes the expected health losses resulting from both bias and variance.
RESULTS: The framework offers a novel methodological contribution that connects causal inference, cost-effectiveness analysis, decision theory and machine learning. It provides a principled, approach to trade off bias and variance based directly on expected health implications rather than statistical metrics. To explore the practical implications, the methodology will be applied to inform the choice of statistical estimator a RWD analysis in the UK.
CONCLUSIONS: The bias-variance trade-off has significant implications for expected population health, yet it is rarely considered in medical decision-making. This framework provides a principled and practical approach for applied researchers to select statistical estimators based on expected health impact.