OBJECTIVES: The purpose of this study was to evaluate the effectiveness of a program aimed at changing physician behavior in accordance with evidence-based guidelines. The program goals were to: improve accuracy of otitis diagnosis; prescribe antibiotics less frequently; and prescribe analgesic drops more frequently. We examined outcome differences for patients treated by intervention clinic providers vs. by control clinic providers. METHODS: Ten primary care clinics in a southwestern integrated delivery system were organized into matched pairs and assigned to either control status (mailed guidelines only) or intervention status (provider exposure to formal education, guideline tools, reinforcement messages, and feedback). The impact of the intervention on providers who participated in the intervention was measured using claims data for patients diagnosed with otitis media and a post-intervention provider survey. The outcome measures were: percentage of episodes with a primary diagnosis of either acute otitis media (AOM), or otitis media with effusion (OME) and/or otalgia, rate of antibiotic fills for the two diagnosis categories; and analgesic drop fill rate. RESULTS: Claims data analysis showed significant improvement for intervention clinic providers over control group providers in decreased diagnosis of AOM (94.0% pre- vs. 91.2% post-intervention, p=0.0231) and increased analgesic drop fills (7.6% pre- vs. 10.4% post-intervention for AOM, 2.4% to 11.5% for OME/otalgia, p=0.0250), but no statistical significance for reduction in antibiotic fills. The provider survey showed statistically significant self-reported differences between the intervention and control clinic providers in the hypothesized direction for all three outcome measures. Significant differences in outcomes were seen by provider type (family practice, urgent care, or pediatrician). CONCLUSIONS: Analyses of claims data showed a reduction in the diagnosis of AOM and increased fills of analgesic drops for pain, as hypothesized. Improvement was more marked in the self-reports than in the claims data. Outcome differences were mediated by provider specialty.