OBJECTIVES: In-hospital complications are a key performance indicator in the assessment of healthcare results due to their close relationship with the quality of care. Their comparison between hospitals needs to consider many factors such as hospital characteristics or inter-patient variation. The objective was to develop a between-hospital comparable Risk-Adjusted In-hospital Complication Rate (RAICR) using machine learning (ML) methods with hospitalization episodes from more than 150 hospitals.

METHODS: The dataset comprised 5 million hospitalization episodes extracted from the In-patient Database proprietary of IQVIA (ICD-10 codes). Each episode was categorized as “complicated” or “not-complicated” according to Agency for Healthcare Research and Quality (AHRQ) criteria. ML techniques (ensemble algorithms complemented by modelling calibration methods) were used to develop a ML model to predict complication probabilities, considering patient and hospital characteristics, diagnoses, comorbidities, and information relative to the medical and surgical procedures associated to each episode. Four different models were developed: general, obstetric, pediatric, and neonate. The presented RAICR methodology calculates RAICR scores as the ratio between observed and expected (predicted by the model) number of complications (applicable at any level of granularity, e.g. hospital, procedure category, hospital department, etc.).

RESULTS: Key variables helping to identify risk of complication were different for each of the four models, allowing to identify diagnoses and surgical procedures connected to complication risk under each of the four scenarios. Other variables such as age group, sex, type of hospital, COVID, and AHRQ comorbidities played a key role as well. Average sensitivity for the models behind the RAICR was 0.87 (max 0.98; min 0.70) and average specificity was 0.89 (max 0.99; min 0.75).

CONCLUSIONS: A new methodology supported by ML techniques was developed for the definition of four different RAICR (general, obstetric, pediatric and neonate) allowing inter-hospital comparisons and supported by high sensitivity and specificity in the models behind it.