A Review of Guidelines and Checklists for the Use of Artificial Intelligence/Machine Learning (AI/ML) in Evidence Generation: Current Landscape and Recommendations
Author(s)
Raju Gautam, PhD1, Saeed Anwar, MSPharm2, Radha Sharma, PhD3, Tushar Srivastava, MSc1.
1ConnectHEOR, London, United Kingdom, 2ConnectHEOR, Delhi, India, 3ConnectHEOR, Edmonton, AB, Canada.
1ConnectHEOR, London, United Kingdom, 2ConnectHEOR, Delhi, India, 3ConnectHEOR, Edmonton, AB, Canada.
OBJECTIVES: The use of artificial intelligence/machine learning (AI/ML) for evidence generation (EG) has increased recently due to its ability to streamline the resource-intensive EG process. Nevertheless, the guidelines on AI/ML use in EG are unclear and have not been systematically evaluated. This study aims to review the guidelines and checklists available on the use of AI/ML in EG and provide recommendations.
METHODS: A pragmatic literature review was conducted using PubMed and Google Scholar to identify guidelines and checklists reporting the use of AI/ML in EG. The websites of key health technology assessment (HTA) agencies (i.e., NICE, SMC, HAS, NCPE, CADTH, IQWiG, TLV, and PBAC) were also searched.
RESULTS: Four guidelines and seven checklists were identified on AL/ML use in EG. The guidelines comprised NICE Position Statement, NICE Real World Evidence Framework, Cochrane 2024 guidance on AI in evidence synthesis, and US FDA guidelines on AI/ML in drug development. The checklists included CHEERS-AI, CONSORT-AI, PALISADE, TRIPOD-AI, SPIRIT-AI, STARD-AI, and DECIDE-AI. These guidelines recommend using AI/ML models, such as Large Language Models, for study selection, automated data extraction, economic model construction, and using ML-based Cochrane RCT classifier to identify RCTs from titles/abstracts. Additionally, Natural Language Processing can convert unstructured real-world data into structured formats, and it can mine electronic health records and support identification of eligible trial participants or reporting of adverse events. Among HTA agencies, only NICE (UK) has developed AI/ML EG guidelines. The IQWiG (Germany) stated the use of AI/ML in EG in their methodological guidelines.
CONCLUSIONS: The emergence of guidelines and checklists on AI/ML use in EG is the right step in standardizing and ensuring quality in use of AI/ML. However, these guidelines should also address key issues like transparency and responsibility to ensure that AI/ML enhances the robustness and fairness of EG in the HTA process.
METHODS: A pragmatic literature review was conducted using PubMed and Google Scholar to identify guidelines and checklists reporting the use of AI/ML in EG. The websites of key health technology assessment (HTA) agencies (i.e., NICE, SMC, HAS, NCPE, CADTH, IQWiG, TLV, and PBAC) were also searched.
RESULTS: Four guidelines and seven checklists were identified on AL/ML use in EG. The guidelines comprised NICE Position Statement, NICE Real World Evidence Framework, Cochrane 2024 guidance on AI in evidence synthesis, and US FDA guidelines on AI/ML in drug development. The checklists included CHEERS-AI, CONSORT-AI, PALISADE, TRIPOD-AI, SPIRIT-AI, STARD-AI, and DECIDE-AI. These guidelines recommend using AI/ML models, such as Large Language Models, for study selection, automated data extraction, economic model construction, and using ML-based Cochrane RCT classifier to identify RCTs from titles/abstracts. Additionally, Natural Language Processing can convert unstructured real-world data into structured formats, and it can mine electronic health records and support identification of eligible trial participants or reporting of adverse events. Among HTA agencies, only NICE (UK) has developed AI/ML EG guidelines. The IQWiG (Germany) stated the use of AI/ML in EG in their methodological guidelines.
CONCLUSIONS: The emergence of guidelines and checklists on AI/ML use in EG is the right step in standardizing and ensuring quality in use of AI/ML. However, these guidelines should also address key issues like transparency and responsibility to ensure that AI/ML enhances the robustness and fairness of EG in the HTA process.
Conference/Value in Health Info
2025-05, ISPOR 2025, Montréal, Quebec, CA
Value in Health, Volume 28, Issue S1
Code
SA27
Topic
Study Approaches
Topic Subcategory
Literature Review & Synthesis
Disease
No Additional Disease & Conditions/Specialized Treatment Areas