OBJECTIVES:

In addition to model performance, the interpretability of machine learning (ML) models remains a critical factor in their adoption in healthcare. Previously, ML models in multiple sclerosis (MS) have primarily focused on predicting disease onset and progression, and there is limited application of ML models for treatment-related outcomes. Hence, this study compared the interpretability and predictive accuracy of rule-based and Random Forests (RF) models in predicting treatment switching among MS patients.

METHODS:

This retrospective cohort study used the TriNetX data from a federated electronic medical records network. The earliest disease-modifying agents (DMA) prescription (index date) was identified among adults aged ≥18 MS patients (September 2010-May 2017). Patients who had a different DMA than their index DMA were classified as the switching cohort. The rule-based RF models were used to predict treatment switching with 72 pre-index variables. Models were trained on up-sampled 70% randomly partitioned data and evaluated on 30%. The model's performance was assessed using Area Under the Curves (AUC), accuracy, recall, and F-1 score.

RESULTS:

During the 24-month post-index period, 16% of the 7,258 eligible MS patients switched to a different DMA. Compared to the rule-based model, the RF model had slightly better prediction performance on test data (AUC: 66% vs. 63%, Accuracy: 61% vs. 55%, Recall: 62% vs. 57%, and F-1 score: 73% vs. 67%). Both models identified age as the most important factor associated with DMA switching. Additionally, the rule-based model identified DMA patients with “age<52, oral DMA at the index, and ≤4 outpatient visits” have less likelihood to switch their DMA.

CONCLUSIONS:

Although the RF model had better prediction performance, the rule-based model provided more clinically applicable factors to predict treatment switching in MS patients. Future studies on clinical outcome prediction should consider both performance and interpretability to improve clinical decision-making.