OBJECTIVES : Predictive modeling has a wide range of applications in HEOR. When the number of predictors is large in comparison to the number of observations, conventional generalized linear regression models can yield poor predictions. Machine Learning (ML) methods are increasingly being used in HEOR as an alternative framework to increase prediction accuracy. Commonly used ML models include Logistic-LASSO, tree based models (CART, Random Forest), and Neural Networks. This research aimed to explore the value of ML for predictive modeling in HEOR, using high dimensional datasets like genomic or claims, and the tradeoffs between model performance and interpretability. Although ML models often outperform regression-based approaches in prediction quality, questions remain about their interpretability.

METHODS : Comparisons between ML and conventional regression models were performed in three different settings: High Dimensional Propensity Score (HDPS) to estimate treatment effect, predicting treatment resistance using genomic data, and classifying patients for risk of an undiagnosed condition. Predictive performance of ML models including Logistic-LASSO, Random Forests, Neural Networks and a conventional logistic regression model were evaluated both in-sample and out-of-sample. Prediction accuracy (i.e. ROC-AUC), and bias (through simulations) were assessed for the different models. In each application, cross validation was used to minimize overfitting risk.

RESULTS : ML models outperformed regression-based approaches in terms of prediction accuracy (increased ROC-AUC by ≥10%) and significantly decreased estimation bias (15% lower than regression in the HDPS application). However, there was a significant loss of interpretability, especially with Deep Neural Networks and Random Forests. Quantifying the impact of specific predictors on outcome is virtually impossible in these instances.

CONCLUSIONS : ML models exhibited better predictive performance measured by ROC-AUC and bias than conventional logistic regression models at the expense of interpretability. Conventional regression models can provide a better framework if interpretability is central to the prediction analysis.