Presentation (CTI)

OBJECTIVES: The automation of network meta-analyses (NMAs) through the use of large language models (LLMs) provides a significant opportunity for healthcare technology developers to optimize workflows critical for health technology assessment (HTA). This is particularly pertinent to ensure requirements of the joint clinical assessment (JCA) are fulfilled. The objective was to develop a system leveraging LLMs to automate substantial elements involved with conducting de novo NMAs.
METHODS: An automated system, utilising Claude 3.5 Sonnet [V2] LLM, was designed to process analysis-ready datasets. Based on each dataset, the LLM was prompted to select a suitable statistical model, write code and execute analyses, evaluate outputs, and interpret results. The automated results were validated by: 1) replicating examples from the National Institute for Health and Care Excellence (NICE) Technical Support Document (TSD 2); 2) reproducing results from two non-DSU published NMAs; and 3) generating and assessing comprehensive outputs, including heterogeneity, inconsistency, and convergence.
RESULTS: For all 14 TSD 2 examples (seven models, fixed and random effects), the LLM produced executable code without the need for human correction or intervention. The mean values produced were within 0.02 of the published results and credible interval limits were within the expected range. For non-DSU published NMA examples, the automated process replicated consistent results. Each model (DSU and non-DSU) was run five times and the LLM consistently selected the appropriate statistical model and generated correct results. Finally, the LLM successfully generated and interpreted heterogeneity, inconsistency, and convergence.
CONCLUSIONS: Starting with analysis-ready datasets, this study has demonstrated LLMs can execute essential steps required for de novo NMAs, paving the way for an automated NMA system. Automation of NMAs can streamline workflows and significantly reduce the amount of time and resource required. This is especially relevant when considering the extensive analyses to be completed within tight timeframes to comply with JCA requirements.