Evaluating Generative Artificial Intelligence (GenAI) in Health Technology Assessment (HTA) Content Generation: A Proof-of-Concept Using Canadian Agency for Drugs and Technologies in Health (CADTH) Reimbursement Dossier Forms

OBJECTIVES: GenAI has the potential to assist in HTA content generation, ultimately saving human time. This proof-of-concept study explored the use of GenAI in creating initial drafts of reimbursement dossier forms submitted to CADTH and investigated the potential for reducing time spent on submissions.

METHODS: For three separate drugs, sections of CADTH pre-submission templates were drafted using GenAI (specifically OpenAI’s GPT-4). Sections were generated based on source documents, which included the respective global value dossiers for the drugs and Canada-specific documents to supplement evidence gaps. ‘One-shot’ or ‘few-shot’ (depending on the complexity of the section) examples of pre-submission form answers were utilised to guide the model’s style and information retrieval from the source documents, facilitating retrieval-augmented generation (RAG). All outputs were reviewed by AI engineers and subject matter experts (SMEs) in HTA to improve performance.

RESULTS: Findings revealed that GPT-4, when provided with the most relevant Canada-specific documentation and ‘few-shot’ examples of required outputs, produced high-quality drafts of CADTH pre-submission forms. Section answers were generated accurately in terms of correct information, accepted tone, and aligned format. Complexity of source documents and pre-submission form sections were found to impact output quality (e.g., multiple indications, large number of clinical trials, long section answer required), and feeding GPT-4 irrelevant documents diluted the quality of responses.

CONCLUSIONS: The findings from this proof-of-concept study demonstrate that when used optimally and with compulsory SME review, GenAI can be used to create acceptable drafts of CADTH reimbursement dossier forms. There is high potential that its use will reduce human time spent on submissions. However, it is essential that documentation required for the forms is optimised with the most relevant data in advance of use by GenAI. Improving prompting and increasing ‘few-shot’ examples for more complex sections of the forms will further help to improve time savings.