OBJECTIVES: Systematic literature reviews (SLRs) are essential for synthesizing research evidence and guiding informed decision-making. Artificial intelligence (AI) can reduce significant resources and substantial effort needed for SLRs. It is unclear what SLR researchers expect from AI used for this type of research. Thus, we investigated the preferences in SLR screening regarding various AI tools and their associated trade-offs.

METHODS: A discrete choice experiment was performed among professionals performing SLRs. We identified attributes and their levels through a literature search and expert consultations. Final AI tool attributes evaluated included the tool’s specific role in the screening process, the required proficiency of the user, estimated sensitivity, workload reduction, and investment needed (i.e. training an AI tool). Data were collected via an online survey, with participants providing background information on their education and experience and completing two sets of discrete choice tasks (13 and 14 tasks, respectively) involving tools with varying levels of attributes. Statistical analysis was performed using a conditional multinomial logit model.

RESULTS: Preliminary results were based on responses received from 173 participants (as of 24 June 2024) with varied backgrounds and experience in performing SLRs and AI use. Most respondents (63.4%) primarily work in academia. While respondents had a high willingness to use AI in SLRs (83.1%), 56.4% considered themselves unfamiliar with AI tools. The most important attributes with a positive impact on choice preference were estimated sensitivity and workload reduction (coefficients:5.3 and 1.3, p-value:<0.05). Conversely, investment needed to train the AI tool had the highest negative impact on choice preferences (coefficient: -3.2, p-value: 0.2).

CONCLUSIONS: The results show that SLR researchers are more concerned about the accuracy of AI tools compared to other attributes such as workload reduction. Future AI tool development for SLRs should prioritize end-user preferences, focusing on the AI performance and usability while keeping high accuracy of the tool.