Presentation (CTI)

OBJECTIVES: The first artificial intelligence (AI)-assisted health technology assessment (HTA) submission, accepted by the NICE Evidence Assessment Group, demonstrated that using AI as a second reviewer for title and abstract screening can reduce systematic literature review (SLR) time and cost by approximately 50%. The objective of this study is to assess whether additional efficiencies can be achieved in an oncology indication through fully automated SLR screening using multiple large language models (LLMs).
METHODS: Pharmacoevidence AI-based SLR tool (metaSLR) was used to facilitate automated title/abstract and full text screening using multiple LLMs, guided by predefined inclusion and exclusion criteria. The records with low confidence or conflicts were escalated for manual review. A subject matter expert (SME) optimized, fine-tuned the final prompt, and conducted quality control on the records excluded by AI.
RESULTS: Compared to the semi-automated benchmark by Makhija et. al. (AI as a second reviewer), which provided approximately 50%-time savings, the multi-LLM screening approach substantially enhanced efficiency, reducing screening time by approximately 90%. Of approximately 1,300 citations and 100 full texts screened, fewer than 5% of screened citations were flagged for human review. Moreover, SME assessment of all the AI-excluded citations confirmed that LLMs did not exclude any relevant citation. The models showed a modest over-inclusion rate of approximately 1-2%.
CONCLUSIONS: This study shows that a fully automated multi‑LLM approach can generate high‑quality oncology SLRs up to ten times faster than conventional methods. The approach achieves ~90% efficiency, reduces costs, while maintaining human oversight in line with Cochrane and HTA guidelines. It is also scalable and practical, supporting faster evidence synthesis and healthcare decision‑making.