Analyzing Generative AI Proficiency for ICD-10 Code Generation and Interpretation

OBJECTIVES: This study aimed to compare the efficacy of common, accessible Generative AI models in identifying ICD-10 codes.

METHODS: Researchers compiled a list of 12 rare and 1 common diseases and collected the associated ICD-10 codes. A second researcher validated each code. Researchers asked ChatGPT 4.0, Google Gemini, and Bing CoPilot to define what was the ICD-10 code for a specific disease as well as the reverse question of what disease was associated with a specified ICD-10 code. All responses were recorded and accuracy was calculated for each model. A second researcher tested model consistency. Analyses were run in May 2024

RESULTS: For rare diseases, the models were able to give the correct disease 75%, 92%, and 100% of the time respectively when given an ICD code. Gemini and Copilot performed similarly in providing the ICD code associated with the disease, but ChatGPT performed worse (right only 50% of the time). ChatGPT performed the worst on newer ICD codes introduced in 2023. For rare diseases with no specific ICD-10 code, all models were able to give the proper 3-digit code. In terms of common diseases, all models were able to accurately generate the disease associated with an ICD-10 code but struggled with supplying a complete list of codes and could only produce a limited list (for example only E11 not all the sub-codes for diabetes). Google Gemini gave the most information about the disease in addition to the codes, CoPilot was the only model to give consistent citations and told the user to speak with a healthcare provider.

CONCLUSIONS: Generative AI models are a good starting point for searching and identifying ICD codes. However, researchers need to recognize which version of the ICD book a model was trained on and the specificity of the target ICD code.