Superior Performance of Generative AI After Application of Mixture-of-Agents LLMs in Outcomes Research
Author(s)
Achilleas Livieratos, PhD1, Maria Kudela, PhD2, Yuxi Zhao, PhD2, Cynthia Basu, PhD2, All-shine Chen, PhD2, Junjing Lin, PhD3, Di Zhang, PhD4, Sai Hurrish Dharmarajan, PhD5, Margaret Gamalo, PhD2.
1SPAIML Scientific Working Group, New York, NY, USA, 2SPAIML Scientific Working Group/Pfizer, New York, NY, USA, 3SPAIML Scientific Working Group/Takeda Pharmaceuticals, New York, NY, USA, 4SPAIML Scientific Working Group/Teva Pharmaceuticals, New York, NY, USA, 5SPAIML Scientific Working Group/Sarepta Therapeutics, New York, NY, USA.
1SPAIML Scientific Working Group, New York, NY, USA, 2SPAIML Scientific Working Group/Pfizer, New York, NY, USA, 3SPAIML Scientific Working Group/Takeda Pharmaceuticals, New York, NY, USA, 4SPAIML Scientific Working Group/Teva Pharmaceuticals, New York, NY, USA, 5SPAIML Scientific Working Group/Sarepta Therapeutics, New York, NY, USA.
Presentation Documents
OBJECTIVES: This work addresses the critical need to improve data extraction accuracy and efficiency in Health Economics and Outcomes Research (HEOR) by introducing a novel approach—Mixture-of-Agents (MoA) architecture. Traditional methods using single Large Language Models (LLMs) like GPT-4 face challenges with variability and inconsistencies, particularly in the context of complex, large-scale HEOR tasks such as network meta-analyses and systematic reviews. The healthcare decision under focus is how to enhance these capabilities for better decision-making in market access and real-world evidence generation.
METHODS: The need for this innovative approach arose from the limitations of existing LLM models in handling the growing complexity of medical data. MoA offers a unique solution by employing multiple LLMs that work collaboratively to refine each other's outputs. Unlike single-model systems, MoA assigns specialized tasks to each agent model, improving accuracy, robustness, and flexibility. This architecture was tested on PubMed immunology abstracts, where it significantly outperformed single models like GPT-4. Using a judge model (Claude 3.5 Sonnet), the system was evaluated across multiple metrics, including PICO parameters (Population, Intervention, Comparator, Outcomes) and additional criteria like logical reasoning, insights generation, and quantitative analysis.
RESULTS: The MoA system showed a 7.33% improvement in overall performance, demonstrating its potential to transform data extraction in HEOR by producing more accurate, nuanced insights with minimal human intervention. The key takeaway is that the MoA architecture, with its layered, multi-agent approach, substantially improves data extraction quality. This novel method not only enhances the precision of insights but also enables more scalable and robust decision-making, especially in data-heavy HEOR domains.
CONCLUSIONS: From the perspective of the pharmaceutical industry, this breakthrough offers a transformative solution for accelerating regulatory approvals and optimizing market access strategies. The MoA system paves the way for more efficient, data-driven decisions, ultimately benefiting healthcare innovation and patient access.
METHODS: The need for this innovative approach arose from the limitations of existing LLM models in handling the growing complexity of medical data. MoA offers a unique solution by employing multiple LLMs that work collaboratively to refine each other's outputs. Unlike single-model systems, MoA assigns specialized tasks to each agent model, improving accuracy, robustness, and flexibility. This architecture was tested on PubMed immunology abstracts, where it significantly outperformed single models like GPT-4. Using a judge model (Claude 3.5 Sonnet), the system was evaluated across multiple metrics, including PICO parameters (Population, Intervention, Comparator, Outcomes) and additional criteria like logical reasoning, insights generation, and quantitative analysis.
RESULTS: The MoA system showed a 7.33% improvement in overall performance, demonstrating its potential to transform data extraction in HEOR by producing more accurate, nuanced insights with minimal human intervention. The key takeaway is that the MoA architecture, with its layered, multi-agent approach, substantially improves data extraction quality. This novel method not only enhances the precision of insights but also enables more scalable and robust decision-making, especially in data-heavy HEOR domains.
CONCLUSIONS: From the perspective of the pharmaceutical industry, this breakthrough offers a transformative solution for accelerating regulatory approvals and optimizing market access strategies. The MoA system paves the way for more efficient, data-driven decisions, ultimately benefiting healthcare innovation and patient access.
Conference/Value in Health Info
2025-05, ISPOR 2025, Montréal, Quebec, CA
Value in Health, Volume 28, Issue S1
Code
SA16
Topic
Study Approaches
Disease
SDC: Systemic Disorders/Conditions (Anesthesia, Auto-Immune Disorders (n.e.c.), Hematological Disorders (non-oncologic), Pain)