Re-Training of the Artificial Intelligence Tool LiveRefTM: Improved Accuracy and Performance in Data Extraction

Author(s)

Zhang M1, Jafar R2, Liu R3, Rizzo M4, Lucas S5, Young V6
1Cytel Inc, Ancaster, ON, Canada, 2Cytel Inc., Vancouver, BC, Canada, 3Cytel Inc, Toronto, ON, Canada, 4Cytel Inc., Kent, KEN, UK, 5Cytel Ltd, London, LON, UK, 6Cytel Inc., London, LON, UK

OBJECTIVES: Systematic literature reviews (SLR) are essential for informing regulatory and health technology assessments (HTA) but are resource intensive and time consuming. LiveRef™ is a novel tool that uses Artificial Intelligence (AI) to extract summary data from publications with an accuracy of 0.76 and has been estimated to reduce the time required for a global value dossier (GVD) update by 99.8%. To improve the performance of LiveRef™ and increase confidence in its predictions, we re-trained the model using a new curated dataset.

METHODS: We collected a dataset of 1000 congress abstracts and 1000 curated references from Ovid searches across 12 indications. Two independent and experienced reviewers manually extracted and annotated summary data that would typically be collected as part of a GVD (indication, sub-population, category of evidence, study design, products, regimens, data sources, sponsor, and country) plus subjective interpretation of the main message and summary of results.

Of the 2000 records, 75% were used for training and 25% for testing. SciFive, a generative biomedical large language model (LLM), was fine-tuned on the dataset to extract the summary data. The accuracy of the updated model in predicting summary data was assessed, and its performance in subjective interpretation was evaluated by a Senior Medical Writer and Value Communication Specialist.

RESULTS: The updated LiveRef model showed a 21% increase in accuracy of predicting summary data compared to the original results published in May 2023 (0.92 vs 0.76). Additionally, LiveRefdemonstrated excellent grammar, syntax, and logical editing for subjective interpretations. GVD update by the new LiveRefTM model demonstrated a 280-hour reduction (98%) compared with manual extraction.

CONCLUSIONS: The accuracy and performance of the LiveRef AI tool were substantially improved through controlled data collection and annotation, and the use of a biomedical LLM for supervised training. This improvement could yield considerable time and resource savings in SLR processes.

Conference/Value in Health Info

2024-11, ISPOR Europe 2024, Barcelona, Spain

Value in Health, Volume 27, Issue 12, S2 (December 2024)

Code

MSR70

Topic

Methodological & Statistical Research

Topic Subcategory

Artificial Intelligence, Machine Learning, Predictive Analytics

Disease

Drugs, No Additional Disease & Conditions/Specialized Treatment Areas

Explore Related HEOR by Topic


Your browser is out-of-date

ISPOR recommends that you update your browser for more security, speed and the best experience on ispor.org. Update my browser now

×