OBJECTIVES: ChatGPT is a natural language processing tool. Since ChatGPT is trained on published materials such as coding guidebooks and blogs, ChatGPT may be able to write code. This project aims to assess the success, accuracy, and effectiveness of ChatGPT to code statistical programming for public health research in Stata, R, and Visual Basic for Applications (VBA).

METHODS: Researchers handwrote code in Stata, R, and Visual Basic for VBA to conduct a Difference-in-Differences and Event Study to understand the impact of Medicaid expansion using tax and American Hospital Association data. Using a pre-written, standardized script, researchers asked ChatGPT to code for data cleaning, set-up and analysis. If the given code did not work, researchers told GPT which error they were experiencing until the code ran properly. These steps were completed in GPT4 and GPT3.5.

RESULTS: For Stata, ChatGPT completed all the analysis in 36 compared to 48 lines of code when done by hand - eight changes needed to be made to ChatGPT’s code. It was most efficient at an event study but was not able to make the proper table. For R, ChatGPT completed all four steps in 98 lines of code compared to 137 lines of code, although specificity was added to each prompt. When coding VBA, ChatGPT accurately coded the initial summary tables, but recommended using more advanced statistical software for further steps. Code in VBA was significantly more efficient than hand-written code data preparation and cleaning - 43 lines vs 70 lines. Results were consistent across the hand coded methods and all 3 softwares using ChatGPT. ChatGPT3.5 had more accurate code than ChatGPT4 – although the paid version gave better explanations.

CONCLUSIONS: ChatGPT works well within a statistical platform’s capabilities. It is essential that researchers learn coding languages to create specific prompts and edit code.