Qualitative Literature Reviews: A Comparison of Researcher and AI Screening of Articles to Inform Conceptual Model Development

OBJECTIVES: Artificial intelligence (AI) models are being used in systematic literature reviews, reducing researcher burden and improving efficiency. However, in structured (not fully systematic) literature reviews when terminology and reporting is not formal or standardized within the literature, as in reviews to identify qualitative research and insights exploring the patient lived experience, it can be challenging to develop focused yet comprehensive search strategies and screening criteria without compromising results. A novel AI model is being developed and trained specifically to facilitate the expert researcher in screening literature for qualitative reviews.

METHODS: Data from 27 medical literature database reviews (5671 citations overall, ranging from 20 to 942 per review) was used to compare researcher and AI title/abstract screening decisions. The reviews were previously conducted to identify qualitative research across a number of conditions to inform the development of conceptual models of the patient experience. Screening decisions were annotated using researcher developed eligibility screening criteria based on PICO principles adapted for the specific context of a qualitative review: Population (search dependent), study design (qualitative research), and outcomes/concepts of interest (patient experience).

RESULTS: Level of agreement between researcher and AI screening decisions was 86% overall, ranging from 44% to 100% across individual screening criteria. For all but 4 searches, agreement was 75% or above on all screening criteria. The main criteria on which there was discrepancy was population, which comprised individual of interest and condition. Time to screen reduced from a skilled researcher screening approximately 30-40 citations/hour to the AI software screening approximately 1000 citations/hour.

CONCLUSIONS: There is a high level of agreement between expert researcher and the AI model in title/abstract screening, highlighting the potential of AI to facilitate the researcher in efficient screening for qualitative literature reviews, supporting research informing the development of conceptual models of the patient experience in the context of COA research.