The Use of Code Llama, a Large Language Model, as Programming Assistant in Real-World Research

OBJECTIVES: This study aims to 1) build a programming tool based on Large Language Mode (LLM) to improve coding efficiency of researchers without exposing confidential company information 2) assess usability of the LLM in different programming languages (Python, R, SAS) that are commonly used in observational research.

METHODS: Code Buddy was built using Code Llama as the underlying LLM engine, with a prompt interface built using streamlit. For this study, an interface for collecting users’ feedback was built. To test the performance of the tool, various coding questions were asked, output was generated in Python, SAS and R languages. Same questions were also asked of ChatGPT 3.5 turbo, results obtained were used as reference.

RESULTS: Code Buddy and ChatGPT generated the most helpful code in python, due to the abundant high-quality python code used to train Code Llama. The tools were also helpful in generating code in R, however, both made more mistakes in R than in Python. The tools’ performance in generating code in SAS is less ideal, potentially because of the limited availability of SAS code to train the LLM model with.

CONCLUSIONS: Leveraging Code Llama and ChatGPT to aid programming is most successful in python language, succeeded by R. However, their ability to generate SAS code is limited, and more debugging effort is needed. This may limit their usability among researchers depending on SAS. Further improvement on the tool in generating SAS code depends either on release of a more capable version of LLM, or fine-tuning the LLM with large quantity of high-quality SAS code, which involves significant amount of data collecting and processing. This finding has provided critical insight to researchers who are interested in deploying their own LLM model to aid SAS programming, that strategic planning is needed for its success.