Decision-Making in Ireland: Utilizing Machine Learning to Build Predictive Algorithms That Assess Words Contributing to NCPE Decision Making on Reimbursement

OBJECTIVES: In Ireland, the National Centre for Pharmacoeconomics (NCPE) conduct Health Technology Assessments (HTAs) on certain new medicines on behalf of the national payer, the Health Service Executive (HSE), issuing reimbursement recommendations and a short technical summary report (TSR). This research aims to build predictive algorithms that produce the outcome of HTAs and assess the feature importance of phrases within TSRs.

METHODS: TSRs were collated in Excel® from the NCPE website. A sample of 84 recently assessed HTA submissions between 2021 and 2024 were selected to train algorithms that predict the four possible NCPE recommendations:

1. Considered for reimbursement
2. Considered for reimbursement if cost-effectiveness is improved
3. Not considered for reimbursement unless cost-effectiveness can be improved
4. Not considered for reimbursement

After text preprocessing in Python®, a text matrix was produced for exploration. Utilising machine learning, four model predictions were created with the weighted text matrix:

• Gaussian Naïve Bayes (GNB)
• Random Forest Classifier (RFC)
• Logistic regression (LR)
• Linear Support Vector Classifier (SVC).

The initial algorithms presented a class imbalance. Therefore, an oversampling method (Adaptive Synthetic Sampling) was implemented to improve the models’ accuracies.

RESULTS: The number of assessments for each recommendation were 3, 28, 49, and 4, respectively. Oversampling significantly improved the accuracy of each model:

• GNB: 0.816
• RFC: 0.794
• LR: 0.816
• SVC: 0.824

Furthermore, the precision and recall for each recommendation across all algorithms improved with oversampling techniques. Utilising the SVC algorithm, important phrases affecting recommendation included trial designed, case estimated, bias applicant, treatment comparison, and likelihood uncertainty.

CONCLUSIONS: Most HTAs in Ireland require improvement of cost-effectiveness results. Trial design, bias, treatment comparison and uncertainties are likely to influence outcomes, ultimately impacting the final reimbursement decision and acceptable price to the payer. Further analyses will include increasing the sample size using real-world data and altering the text matrix to improve the algorithms.