Transparent, Traceable, and Reproducible Evidence Generation for HTA Using Generative AI and Retrieval-Augmented Generation (RAG)
Author(s)
Rajdeep Kaur, PhD, Vedant Soni, B.Tech, Ankita Sood, MPH, PharmD, Mrinal Mayank, B.Tech, Barinder Singh, RPh, Shubhram Pandey, MSc.
Pharmacoevidence Pvt. Ltd., Mohali, India.
Pharmacoevidence Pvt. Ltd., Mohali, India.
OBJECTIVES: Health technology assessment (HTA) bodies such as the National Institute for Health and Care Excellence (NICE) and the Canadian Drug Agency (CDA) acknowledge the growing potential of artificial intelligence (AI) to enhance the decision-making process. This study aimed to explore the application of RAG, a generative AI approach, to enable transparent, traceable, and reproducible evidence generation in HTA submissions.
METHODS: A proof-of-concept study was conducted to evaluate a RAG based framework designed to enhance transparency, traceability, and reproducibility in evidence generation for HTA submissions. The system was developed in Python and integrated with Claude Sonnet 3.7. Overall, 30 source documents related to neurodegenerative disorders, including clinical trials, economic models, and regulatory summaries, were uploaded into the RAG pipeline. To ensure source traceability, metadata such as document identifiers and section references were captured alongside embedded content. The tool was tested using 40 scenarios developed by subject matter experts (SMEs) across clinical, economic, and regulatory domains. Each output was independently evaluated by two SMEs for factual accuracy, completeness, and traceability to the original source. Discrepancies were documented, categorized, and reviewed for potential improvements.
RESULTS: The RAG-based system effectively retrieved and synthesized content from the embedded source documents. Across 40 SME-designed scenarios, 36 (90%) outputs were rated as factually accurate, complete, and fully traceable and included correct sources. In 3 scenarios (7.5%), the factual content was accurate; however, minor issues such as missing citations required SME intervention to restore explicit source traceability. In 1 scenario (2.5%), the output included only partial paraphrasing of the supporting data source.
CONCLUSIONS: The study demonstrated that a RAG-enabled generative AI framework can reliably support transparent, traceable, and reproducible evidence generation. By significantly reducing manual efforts and enabling structured SME validation, this approach facilitates efficient content generation, thereby advancing responsible AI integration in HTA submissions.
METHODS: A proof-of-concept study was conducted to evaluate a RAG based framework designed to enhance transparency, traceability, and reproducibility in evidence generation for HTA submissions. The system was developed in Python and integrated with Claude Sonnet 3.7. Overall, 30 source documents related to neurodegenerative disorders, including clinical trials, economic models, and regulatory summaries, were uploaded into the RAG pipeline. To ensure source traceability, metadata such as document identifiers and section references were captured alongside embedded content. The tool was tested using 40 scenarios developed by subject matter experts (SMEs) across clinical, economic, and regulatory domains. Each output was independently evaluated by two SMEs for factual accuracy, completeness, and traceability to the original source. Discrepancies were documented, categorized, and reviewed for potential improvements.
RESULTS: The RAG-based system effectively retrieved and synthesized content from the embedded source documents. Across 40 SME-designed scenarios, 36 (90%) outputs were rated as factually accurate, complete, and fully traceable and included correct sources. In 3 scenarios (7.5%), the factual content was accurate; however, minor issues such as missing citations required SME intervention to restore explicit source traceability. In 1 scenario (2.5%), the output included only partial paraphrasing of the supporting data source.
CONCLUSIONS: The study demonstrated that a RAG-enabled generative AI framework can reliably support transparent, traceable, and reproducible evidence generation. By significantly reducing manual efforts and enabling structured SME validation, this approach facilitates efficient content generation, thereby advancing responsible AI integration in HTA submissions.
Conference/Value in Health Info
2025-11, ISPOR Europe 2025, Glasgow, Scotland
Value in Health, Volume 28, Issue S2
Code
MSR204
Topic
Methodological & Statistical Research
Topic Subcategory
Artificial Intelligence, Machine Learning, Predictive Analytics
Disease
No Additional Disease & Conditions/Specialized Treatment Areas