OBJECTIVES. Natural Language processing (NLP) techniques are used in Electronic Medical Records (EMR) in order to improve patient diagnosis and management. The use of NLP has been increasing in the last few years. It is not clear to what extent these applications undergo a robust clinical validation process. We conducted a review of published validation studies of NLP techniques, to describe the systems validated and the quality of validation.

METHODS. A PubMed search was conducted for the terms “natural language processing” and “medicine”. Data extracted by three reviewers were: year of publication, country of study, application area, disease area, type of AI technique and method and sample size of validation study.

RESULTS. Only 2.13% of all published studies focused on NLP and Medicine (N=1,546) were included in PubMed as validation studies and almost 50% were published in the last 5 years. Limiting the search to the last 5 years, 33 papers were retrieved and 14 included in this review which were truly validation studies of NLP systems: 10 (71%) from the US; 9 (64.3%) diagnosis systems and 2 (14.3%) classification systems; 5 (35.7%) NLP systems were used in EMR from cancer patients and 4 (28.6%) from patients with cardiovascular disease; most NLP systems (11/14) underwent an external validation process and 100% of them were considered successfully validated; studies were carried out with very different sample sizes (from 46 to 50,669 subjects).

CONCLUSIONS. Although studies focusing on validation of NLP techniques in EMR are scarce, most of published studies show an adequately robust clinical validation process. Efforts are needed to increase the number of validation studies in this field.