Presentation (CTI)

OBJECTIVES: Evidence gap analysis forms the basis of evidence planning and generation, and requires assessment of factors such as HTA (Health Technology Assessment) requirements/expectations as well as review of existing literature. A key element of evidence gap analysis is the understanding of HTA decision drivers - evidence submitted for past submissions for a certain indication, its appraisal and impact on the final HTA decisions. We aimed to analyse HTA decisions for PCSK-9 inhibitors using AI to conduct evidence gap analysis.
METHODS: A Large Language Model (LLM) assisted platform was used to extract and analyse past PCSK-9 inhibitor assessments made by 3 European HTAs - NICE (UK), HAS (France) and IQWiG (Germany). Three levels of prompt engineering were applied - simple prompts, series of conversational prompts and structured output framework based prompts.
RESULTS: Across all approaches, AI was able to extract/structure the response well and provide accurate factual summaries such as assessments timelines, reimbursement decision details and general archetyping of HTA decision-making processes. It also provided intuitive headlines for the key gaps - need for longer term outcomes, comparator selection and evidence in special population groups. However, with more complex analyses requiring reasoning, the simple prompt and conversational prompt approaches yielded a large volume of minor to major hallucinations and conflated ideas/information. Examples of the hallucination were reporting and analysing evidence that was not submitted or generalising patient sub-populations evaluated by individual HTAs. The structured framework-based prompt engineering approach almost eliminated these hallucinations/conflations but required higher level of involvement and subject matter expertise from the user. Even though this approach was the most time consuming, it still represents >95% time saving over a manual evidence gap analysis approach.
CONCLUSIONS: AI is a useful tool to perform steps for a gap analysis but with significant levels of prompt engineering, can provide excellent evidence gap analysis results.