Utilizing Machine Learning to Identify Prediction Factors for Positive Medical Device HTA Recommendations

Author(s)

Tong C1, Huo Y1, Gupta A1, Rossi A2
1Johnson & Johnson Medical, Somerville, NJ, USA, 2Johnson& Johnson Medical, Williamstown, VIC, Australia

Objectives: Multiple factors and criteria are considered by health technology assessment (HTA) agencies, resulting in discordant recommendations; limited analyses exist on what drives differences in HTA recommendations for Medical Device technologies. Using machine learning, we explored 1) if available factors can predict the appraisal decision and 2) if factors that contribute most to differences in appraisal decisions can be identified. Methods: Medical device HTA appraisals with a decision date between 01 December 2003 and 08 October 2019 from the IQVIATM HTA accelerator were analyzed. Natural language processing was first used to extract keywords as features from data elements for clinical outcomes and conclusions. A machine learning algorithm, XGBoost, was then used to predict positive or negative HTA recommendations. Ten-fold cross-validation was used for training models and testing performance. To make the data more balanced, synthetic minority over-sampling was used to adjust the ratio between positive and negative recommendations. Results: 1,430 HTA submissions were included, representing 25 therapeutic areas, 10 HTA agencies, and 9 countries globally. The top 5 predictors were: primary therapeutic area, HTA agency, level of evidence (e.g. systematic review being the most important and expert opinion the least important), HTA type (procedure/intervention or medical device assessments), and the appearance of key string: “statistically significant difference”. Prediction performance, based on area under the receiver operating characteristic (ROC) curve, was 0.80. Conclusion: Machine learning could help predict HTA appraisal decisions and provides direct insight into relative contributions to prediction accuracy. Models of this nature may be useful for early screening of medical devices and interventions to predict device recommendations. Though therapeutic areas and HTA agency were important features of the model, explanatory research is required to better understand interactions among these and other model inputs.

Conference/Value in Health Info

2020-11, ISPOR Europe 2020, Milan, Italy

Value in Health, Volume 23, Issue S2 (December 2020)

Code

PMD23

Topic

Health Technology Assessment

Topic Subcategory

Decision & Deliberative Processes, Systems & Structure

Disease

Medical Devices

Explore Related HEOR by Topic


Your browser is out-of-date

ISPOR recommends that you update your browser for more security, speed and the best experience on ispor.org. Update my browser now

×