Physician-Augmented AI: Improving Machine Learning-Based Detection of NSCLC and Associated Clinical Indicators
Author(s)
Abhinav Nayyar, MBBS, MBA1, Mohini Rastogi, Msc2, Sudhanshu Chawla, B.Tech2, Aditi Paul, MBA1, Abhimanyu Roy, MBA1, Shailaja Daral, MBA, MD1, Shashi B. Khan, Msc2, Anshul Sethi, B.Tech1, Vikash K. Verma, MBA, PharmD1, Marissa Seligman, BS Pharma3, Arunima Sachdev, MA1, Ina Kukreja, MBA, PT1, Rahul Goyal, BS Tech4, Louis Brooks Jr, MA5;
1Optum, Gurgaon, India, 2Optum, Noida, India, 3Optum, Boston, MA, USA, 4Optum, Phoenix, AZ, USA, 5Optum, Bloomsbury, NJ, USA
1Optum, Gurgaon, India, 2Optum, Noida, India, 3Optum, Boston, MA, USA, 4Optum, Phoenix, AZ, USA, 5Optum, Bloomsbury, NJ, USA
Presentation Documents
OBJECTIVES: This analysis evaluates the impact of physician inputs on the performance of AI/ML models in identifying NSCLC and related clinical variables from unstructured data and compared it with models trained solely on pre-annotated data.
METHODS: We utilized Optum’s de-identified clinical notes for the timeframe 2007 to 2023. The diagnosis of NSCLC was confirmed through a combination of ICD-10 codes and mentions of NSCLC and its variations within the clinical notes. Two AI/ML models were developed, one trained on pre-annotated data and another on the same dataset with additional physician review and inputs. The NER model employed a Char CNNs - BiLSTM - CRF architecture, trained to detect medical entities specific to NSCLC and related variables. Both models aimed to identify NSCLC, risk factors, family history, and signs and symptoms. We then compared the precision, recall, and F1-scores of the two models to objectively evaluate the impact of physician input on model performance.
RESULTS: The findings on the validation dataset suggest that the model incorporating physician inputs significantly outperformed the model without such inputs, with higher precision, recall, and F1-score values. The physician-augmented model demonstrated an increase in precision from 36% to 75%, recall from 55% to 67%, and F1-score from 43% to 71% for the identification of NSCLC signs and symptoms. Similarly, the precision, recall, and F1-score improved from 52% to 86%, 63% to 81% and 57% to 84% respectively for the risk factors.
CONCLUSIONS: Incorporating physician expertise into AI/ML model training significantly enhances identification accuracy of NSCLC and related clinical variables within unstructured clinical data. This highlights the value of merging clinical knowledge with analytics in healthcare AI. Further research is needed to scale these models and assess their effect on patient outcomes and clinical processes.
METHODS: We utilized Optum’s de-identified clinical notes for the timeframe 2007 to 2023. The diagnosis of NSCLC was confirmed through a combination of ICD-10 codes and mentions of NSCLC and its variations within the clinical notes. Two AI/ML models were developed, one trained on pre-annotated data and another on the same dataset with additional physician review and inputs. The NER model employed a Char CNNs - BiLSTM - CRF architecture, trained to detect medical entities specific to NSCLC and related variables. Both models aimed to identify NSCLC, risk factors, family history, and signs and symptoms. We then compared the precision, recall, and F1-scores of the two models to objectively evaluate the impact of physician input on model performance.
RESULTS: The findings on the validation dataset suggest that the model incorporating physician inputs significantly outperformed the model without such inputs, with higher precision, recall, and F1-score values. The physician-augmented model demonstrated an increase in precision from 36% to 75%, recall from 55% to 67%, and F1-score from 43% to 71% for the identification of NSCLC signs and symptoms. Similarly, the precision, recall, and F1-score improved from 52% to 86%, 63% to 81% and 57% to 84% respectively for the risk factors.
CONCLUSIONS: Incorporating physician expertise into AI/ML model training significantly enhances identification accuracy of NSCLC and related clinical variables within unstructured clinical data. This highlights the value of merging clinical knowledge with analytics in healthcare AI. Further research is needed to scale these models and assess their effect on patient outcomes and clinical processes.
Conference/Value in Health Info
2025-05, ISPOR 2025, Montréal, Quebec, CA
Value in Health, Volume 28, Issue S1
Code
MSR144
Topic
Methodological & Statistical Research
Topic Subcategory
Artificial Intelligence, Machine Learning, Predictive Analytics
Disease
SDC: Oncology