Validating the AI Assisted Abstract Screening Feature of Nested Knowledge Platform

OBJECTIVES: The use of AI platforms such as Nested Knowledge (NK) has become more popular due to potential efficiencies while conducting large and time consuming literature reviews.

METHODS: We aimed to assess the accuracy of one of NK AI’s key features, AI assisted screening. We compared all cross validations when utilizing the AI-assisted abstracts screening. These include different measures for accuracy, AUC (area under the curve), and recall.

NK recommends 50 expert-screened abstracts as a minimum training requirement for the AI model. Given that our literature review had different population subgroups and several outcomes to be addressed, we compared cross validation-based accuracy after screening 225 and 450 abstracts, as well as after screening all abstracts that were to be screened (1550 abstracts).

RESULTS: After screening 225 and 450 abstracts, the accuracy increased minimally from 0.71 to 0.72. After screening 1550 abstracts, the accuracy reached 0.89.

The AUC, which reflects how well the model discerns between included and excluded records. It increase very slightly from 0.83 to 0.84 after screening 225 and 450 abstracts. After completing the 1550 screenings, it changed slightly to 0.89. (0.80+ indicates a high AUC).

The recall was the only measured aspect that might have changed after screening all 1550 abstracts, indicating that the model will less frequently lean towards exclusion of relevant records. The recall was 0.89, 0.91 and 0.76 after screening the 225, 450 and 1550 abstracts, respectively. NK recommends minimum acceptable recall of >70%, which was met for the three different thresholds of abstracts screened.

CONCLUSIONS: The AI screening feature offered by NK can be effectively utilized after training the model with a sufficient number of abstracts. Our exercise proved the great AUC, recall, and accuracy of the AI assisted abstract screening offered as an important method for providing screening assistance