OBJECTIVES: To evaluate the sensitivity and specificity of opioid overutilization quality measures based on the Centers for Medicare and Medicaid Services (CMS) Controlled Substance Overutilization Monitoring System (OMS) using proxy indicators of potential overutilization.  METHODS: Based on the CMS measure (proportion of patients with opioid prescriptions from ≥4 prescribers and ≥4 pharmacies among patients with ≥2 opioid prescriptions), claims from the 2012 IMS LRx database were used to evaluate the metric at different prescriber and pharmacy thresholds. Cash payment of ≥1 opioid prescription was used as a proxy for potential overutilization and set as the dependent variable in logistic regression models to generate separate receiver operating characteristic (ROC) curves for thresholds of 2-8 prescribers and 2-8 pharmacies. Optimal cutpoints for number of physicians and pharmacies were selected as the value on the curve with the shortest distance to perfect prediction. Sensitivity, specificity and positive predictive value (PPV) for every combination were calculated. As a sensitivity analysis, the process was repeated using ≥1 opioid-abuse related diagnosis as a proxy for potential overutilization.   RESULTS: Of the 1,213,909 qualified patients, 5.9% met the CMS criteria for overutilization, while 13.8% met the alternative ROC optimal criteria (≥4 prescribers and ≥3 pharmacies).  Defining cash payment patients (12.3% of total) as overutilizers, the CMS definition had sensitivity (15.0%), specificity (95.3%) and PPV (31.0%) The ROC alternative had sensitivity (28.0%), specificity (88.2%) and PPV (24.9%).  For patients with opioid abuse-related claims defined as overutilizers (2.6% of total), the CMS criteria had sensitivity (16.1%), specificity (94.3%) and PPV (7.0%). The ROC alternative had sensitivity (28.5%), specificity (86.6%) and PPV (5.3%).   CONCLUSIONS: Using two proxies for opioid overutilization, analysis of ROC curves suggested optimal criteria similar to the CMS criteria. Quality organizations can use the range of results and their preferences for sensitivity and specificity tradeoffs to develop needed quality measures.