OBJECTIVES: Systematic reviews (SR) require both comprehensive and efficient methodology and automation helps address these competing conditions. Natural language processors (NLP) may decrease screening time (e.g. prioritized screening); however, their benefits/risks as a second or autonomous reviewer require more investigation.This study assessed the performance of NLPs to exclude records from SRs and compared their performance to human reviewers.

METHODS: Using data from 8 completed SRs conducted by a US Evidence-Based Practice Center, we randomly selected 10% of references from each SR and trained two NLP classifiers (one support vector machine (SVM) and one Naïve Bayes (NB)) on inclusion/exclusion decisions. The classifiers screened the remaining references, leaving records unreviewed in the absence of consensus. Our primary outcome was NLP false negative (FN) rate (FN/screened records) compared to dual human-screened results. We also estimated single-human FN rates, workload savings and the potential impact of NLP FN on review results/conclusions.

RESULTS: Including 33,191 total screened records, the SRs spanned diverse topic domains (e.g. metabolic, neoplasms, respiratory), review types (e.g. interventional, umbrella, qualitative) and proportions of included studies. NLP FN rates ranged from 0 to 0.04%; no missed studies in 6 reviews and 1 in each of 2 reviews (0.3 and 1.5% eligible studies). Estimated human FN rates were 0.11 - 0.94%. These NLPs achieved consensus exclusion for a median of 26% of records (range 3-57%). Descriptive assessment of NLP FNs suggested no or little impact on the reviews’ results/conclusions.

CONCLUSIONS: In these reviews, NB/SVM classifiers, running independently, incorrectly excluded fewer records than individual human reviewers. Even in this conservative scenario, these tools decreased workload to identify relevant literature. This adds to the growing body of evidence to help review teams considering such tools. Additional research spanning review topics and types will inform the use of this continually evolving technology.