IMPACT OF MEASUREMENT ERROR AND DIFFERENTIAL MISCLASSIFICATION ON ASSESSMENTS OF HOSPITAL-BASED SURGICAL QUALITY
Author(s)
David Etzioni, MD1, Ian Elliot, (none)2, Adam Schwartz, MD3, Nabil Wasif, MD, MS2, Michelle C. Nguyen, MD, MS2, Mark Tyson, MD, MS2, Robert Cima, MD2, Elizabeth Habermann, PhD4, YuHui Chang, PhD2.
1Mayo Clinic, Scottsdale, AZ, USA, 2Mayo Clinic, Phoenix, AZ, USA, 3Tenet Health, Phoenix, AZ, USA, 4Mayo Clinic, Rochester, MN, USA.
1Mayo Clinic, Scottsdale, AZ, USA, 2Mayo Clinic, Phoenix, AZ, USA, 3Tenet Health, Phoenix, AZ, USA, 4Mayo Clinic, Rochester, MN, USA.
OBJECTIVES: To characterize the impact of measurement error on the assessment of hospital surgical quality using a simulation model grounded in the structure of the Centers for Medicare and Medicaid Services (CMS) Patient Safety Indicator-90 (PSI-90) composite measure.
METHODS: We developed a Monte Carlo simulation model to analyze the impact of variations in measurement sensitivity and specificity on assessments of hospital quality. Variable “true” hospital quality was modeled as a multiplier applied to each of the 10 component PSI rates drawn from national benchmarks: high-performing (0.66x), average (1.00x), and low-performing (1.50x). The model additionally incorporated differential misclassification, in which hospitals had varying sensitivity (levels of 0.7, 0.8, and 0.9) and specificity (calibrated to positive predictive value tiers of 75%, 85%, and 95%) for the determination of PSI occurrences. With this approach, the primary outcome was the model-estimated likelihood of a hospital being identified as a 5% outlier for quality (high or low).
RESULTS: Within the parameters of this model, measurement error was as consequential as true hospital performance. A high-performing hospital (0.66x multiplier) had a 64.8% likelihood of being identified as a high-quality outlier under favorable measurement conditions (sensitivity 0.7, high specificity), compared to zero likelihood under unfavorable measurement conditions (sensitivity 0.9, low specificity). A low-performing hospital (1.50x multiplier) had a 67.0% likelihood of being identified as a low-quality outlier under unfavorable measurement conditions (sensitivity 0.9, low specificity), compared to zero likelihood under favorable measurement conditions (sensitivity 0.7, high specificity).
CONCLUSIONS: Hospital-level differences in outcome ascertainment may be as important as underlying clinical performance in assessments of surgical quality. For a hospital to be identified as a high-performance outlier, it must be both clinically excellent and possess a favorable profile of measurement error relative to other hospitals. Systems that rely on administrative data for quality reports need to account for measurement error.
METHODS: We developed a Monte Carlo simulation model to analyze the impact of variations in measurement sensitivity and specificity on assessments of hospital quality. Variable “true” hospital quality was modeled as a multiplier applied to each of the 10 component PSI rates drawn from national benchmarks: high-performing (0.66x), average (1.00x), and low-performing (1.50x). The model additionally incorporated differential misclassification, in which hospitals had varying sensitivity (levels of 0.7, 0.8, and 0.9) and specificity (calibrated to positive predictive value tiers of 75%, 85%, and 95%) for the determination of PSI occurrences. With this approach, the primary outcome was the model-estimated likelihood of a hospital being identified as a 5% outlier for quality (high or low).
RESULTS: Within the parameters of this model, measurement error was as consequential as true hospital performance. A high-performing hospital (0.66x multiplier) had a 64.8% likelihood of being identified as a high-quality outlier under favorable measurement conditions (sensitivity 0.7, high specificity), compared to zero likelihood under unfavorable measurement conditions (sensitivity 0.9, low specificity). A low-performing hospital (1.50x multiplier) had a 67.0% likelihood of being identified as a low-quality outlier under unfavorable measurement conditions (sensitivity 0.9, low specificity), compared to zero likelihood under favorable measurement conditions (sensitivity 0.7, high specificity).
CONCLUSIONS: Hospital-level differences in outcome ascertainment may be as important as underlying clinical performance in assessments of surgical quality. For a hospital to be identified as a high-performance outlier, it must be both clinically excellent and possess a favorable profile of measurement error relative to other hospitals. Systems that rely on administrative data for quality reports need to account for measurement error.
Conference/Value in Health Info
2026-11, ISPOR Europe 2026, Vienna, Austria
Value in Health, Volume 29, Issue 12S
Code
HPR145
Topic
Clinical Outcomes, Health Policy & Regulatory, Methodological & Statistical Research
Disease
No Additional Disease & Conditions/Specialized Treatment Areas