AI Assistant Meets Human: Refining Methods for Research-Tailored AI Assistants to Support Global Product Strategy, Decision Making, and HEOR Impact
Author(s)
Katelyn Keyloun, PharmD, MS1, Justin Yu, PharmD MS2, Tyler Reinsch, PharmD3.
1Director, Product Innovation & Development, Arysana, Carson City, NV, USA, 2Arysana, Jersey City, NJ, USA, 3Arysana, Springfield, MO, USA.
1Director, Product Innovation & Development, Arysana, Carson City, NV, USA, 2Arysana, Jersey City, NJ, USA, 3Arysana, Springfield, MO, USA.
OBJECTIVES: Integrated evidence generation and dissemination plans are important tools that inform strategic decision-making throughout the product lifecycle. Reliable retrieval and summarization enabled through AI may save time and potentially support robust global product strategies. Yet, the methods to develop research-tailored AI to support access and reimbursement remains underexplored; therefore, the objective was to develop a retrieval-augmented generation (RAG)-based system that grounds its responses in an organization’s proprietary knowledge to facilitate generation and communication of HEOR.
METHODS: RAG was used to generate answers to queries related to key strategic documents (e.g., value messages, key evidence statements, corresponding publications) for a hypothetical product. 19 queries relevant to global integrated evidence planning were developed a priori and used to evaluate the accuracy of the LLM-generated responses. Responses were initially evaluated by HEOR researchers (on metrics such as accuracy and relevance), then data pre-processing, post-processing, and prompt optimization approaches were applied to improve performance of the RAG pipeline. Performance was evaluated as a passing score of 90% or greater, or at least 17 queries correct out of 19.
RESULTS: Initial LLM performance was 58% (11/19 correct). After rewriting test queries and updating the RAG pipeline, the final LLM performance was 95% (18/19 correct). Prompting had the largest impact on performance, followed by use case-specific pre-processing methods, and then specific keyword searches, which improved specificity of the responses.
CONCLUSIONS: In developing research-tailored AI Assistants for complex queries related to global product strategy, multiple factors influence the success of RAG. Naive RAG approaches can create value in select instances, but often fall short for specific research-tailored tasks. By optimizing each aspect of the RAG pipeline per use case, limitations in LLM-generated responses were sufficiently overcome with performance of 95%; findings support the potential for faster decision-making for global product strategy and evidence planning using research-tailored RAG.
METHODS: RAG was used to generate answers to queries related to key strategic documents (e.g., value messages, key evidence statements, corresponding publications) for a hypothetical product. 19 queries relevant to global integrated evidence planning were developed a priori and used to evaluate the accuracy of the LLM-generated responses. Responses were initially evaluated by HEOR researchers (on metrics such as accuracy and relevance), then data pre-processing, post-processing, and prompt optimization approaches were applied to improve performance of the RAG pipeline. Performance was evaluated as a passing score of 90% or greater, or at least 17 queries correct out of 19.
RESULTS: Initial LLM performance was 58% (11/19 correct). After rewriting test queries and updating the RAG pipeline, the final LLM performance was 95% (18/19 correct). Prompting had the largest impact on performance, followed by use case-specific pre-processing methods, and then specific keyword searches, which improved specificity of the responses.
CONCLUSIONS: In developing research-tailored AI Assistants for complex queries related to global product strategy, multiple factors influence the success of RAG. Naive RAG approaches can create value in select instances, but often fall short for specific research-tailored tasks. By optimizing each aspect of the RAG pipeline per use case, limitations in LLM-generated responses were sufficiently overcome with performance of 95%; findings support the potential for faster decision-making for global product strategy and evidence planning using research-tailored RAG.
Conference/Value in Health Info
2025-11, ISPOR Europe 2025, Glasgow, Scotland
Value in Health, Volume 28, Issue S2
Code
MSR18
Topic
Methodological & Statistical Research, Organizational Practices, Real World Data & Information Systems
Topic Subcategory
Artificial Intelligence, Machine Learning, Predictive Analytics
Disease
No Additional Disease & Conditions/Specialized Treatment Areas