OBJECTIVES : Electronic health/medical records (EHR/EMR) and administrative claims databases contain large amounts of data and highly resourceful real world evidence on patients’ medical history. Often associated with high processing speeds, machine learning (ML) methods are gaining traction in these data sources. The objective of this study was to compare the performance of existing ML methods to standard regression models.

METHODS : A targeted review of the literature was conducted using PubMed and EMBASE. Studies of interest were observational studies that applied ML methods to EHR, EMR, or claims databases in the US. Only studies that compared ML methods to conventional regression models were included in the final review. Information extracted pertained to the healthcare application area, ML and traditional regression methods, and statistical measures of their performances (area under the receiver operating characteristic curve (AUC), specificity, sensitivity, and accuracy).

RESULTS : Out of 1014 identified studies, 18 met the inclusion criteria. Most of the included studies applied ML predictive methods to improve clinical decision-making (n=10). All of the included articles used classification algorithms, with support vector machines (n=6) and random forests (n=6) being the most common methods. Most of the studies reported the AUC (n=14) and/or the sensitivity (n=5) to compare predictive abilities. Classification algorithms showed higher sensitivity than traditional regression models in 9/13 comparisons, with values ranging from 0.57 to 0.71. According to the AUC, artificial neural networks and AdaBoost had the highest performance when compared to logistic regression, performing better in 100% (n=2/2) and 75% (n=3/4) of the studies, respectively.

CONCLUSIONS : The results from this study show that ML methods have the potential to improve the predictive capability of healthcare outcomes. Given the small number of studies directly comparing ML methods to standard regression models, further evidence is required to guide healthcare data scientists in their methodological choices.