Meaningful Difference in Performance (MDP): A Proposed Method to Improve Evaluation of Meaningful Variation Healthcare Quality Measures

OBJECTIVES:: A National Quality Forum’s “must pass” criterion for endorsing healthcare quality measures is the existence of a quality gap - “considerable variation, or overall less-than-optimal performance, in the quality of care across providers”. However, methods to determine if gaps exist are either informal, arbitrary, or based on statistical tests of differences from mean performance or other benchmarks. The objective of our work is to develop a conceptual and statistical foundation for incorporating patient and stakeholder perspectives into operationalizing a new concept: Meaningful Difference in Performance (MDP). If a MDP between providers does not exist, then patients, clinicians, and healthcare systems might be saved tremendous effort and resources needed to implement measures that will not be valuable in driving improvements.

METHODS: MDP is defined as the smallest difference from average (or other benchmark) hospital/provider performance that is considered meaningful in a multi-stakeholder consensus process weighing the perspectives of patients, systems, and payors. The MDP can then be used as an equivalence threshold to evaluate which providers are meaningfully (not just statistically) different from average performance. We use patient reported outcome performance measures for carpal tunnel release and knee/hip replacement surgeries to develop and illustrate these concepts.

RESULTS: Using MDPs as equivalence thresholds identifies a different subset of providers as being meaningfully different rather than sole reliance on statistical difference from the mean.

CONCLUSIONS: The methods illustrated here show promise for the evaluation of meaningful quality gaps so that attention and effort can be reserved for measures focused on the most important problems. More conceptual and empirical work is needed to refine the process for incorporating patient and stakeholder perspectives into choosing the MDP for different measure types (i.e., structure, process, outcome), quality domains (e.g., effectiveness, safety, equity), and diverse clinical contexts.