OBJECTIVES: Systematic literature reviews (SLR) are essential for informing regulatory and health technology assessments (HTA) but are resource intensive and time consuming. LiveRef™ is a novel tool that uses Artificial Intelligence (AI) to extract summary data from publications with an accuracy of 0.76 and has been estimated to reduce the time required for a global value dossier (GVD) update by 99.8%. To improve the performance of LiveRef™ and increase confidence in its predictions, we re-trained the model using a new curated dataset.

METHODS: We collected a dataset of 1000 congress abstracts and 1000 curated references from Ovid searches across 12 indications. Two independent and experienced reviewers manually extracted and annotated summary data that would typically be collected as part of a GVD (indication, sub-population, category of evidence, study design, products, regimens, data sources, sponsor, and country) plus subjective interpretation of the main message and summary of results.

Of the 2000 records, 75% were used for training and 25% for testing. SciFive, a generative biomedical large language model (LLM), was fine-tuned on the dataset to extract the summary data. The accuracy of the updated model in predicting summary data was assessed, and its performance in subjective interpretation was evaluated by a Senior Medical Writer and Value Communication Specialist.

RESULTS: The updated LiveRef^™ model showed a 21% increase in accuracy of predicting summary data compared to the original results published in May 2023 (0.92 vs 0.76). Additionally, LiveRef^™ demonstrated excellent grammar, syntax, and logical editing for subjective interpretations. GVD update by the new LiveRef^TM model demonstrated a 280-hour reduction (98%) compared with manual extraction.

CONCLUSIONS: The accuracy and performance of the LiveRef^™ AI tool were substantially improved through controlled data collection and annotation, and the use of a biomedical LLM for supervised training. This improvement could yield considerable time and resource savings in SLR processes.