OBJECTIVES: Systematic reviews and meta-analyses are valuable tools when researchers require a synthesis of the results of different studies. However, between-study heterogeneity (e.g., differences in population, methodology, or results) can limit the reliability of results from meta-analyses. This study applied current methods which have been proposed to address the difficulties presented by heterogeneity to recent systematic reviews, to assess robustness of the reviews’ findings.

METHODS: Recently published Cochrane systematic reviews that contained five or more trials investigating the effects of atypical antipsychotics in schizophrenia patients were identified. Positive and Negative Syndrome Scale (PANSS) total score endpoint data up to week 12 was extracted for seven direct treatment comparisons (aripiprazole vs. clozapine, quetiapine, risperidone, olanzapine, or ziprasidone; risperidone vs. quetiapine or olanzapine). Relative efficacy was measured using mean difference (MD) in average PANSS total score. For each direct treatment comparison, the results from the original, conventional meta-analysis were compared to that for (1) the single most precise trial in the comparison, (2) a meta-analysis restricted to the largest trials, (3) a limit meta-analysis, and (4) a meta-analysis restricted to trials at a low risk of attrition bias, using the difference in mean difference (∆MD).

RESULTS: Where considerable differences were observed, MD in PANSS total score was smaller in the conventional meta-analyses than that for three of the four analytic strategies. Such differences were notably small, with all ∆MD less than 4, which is unlikely to correspond to a clinically significant difference. No considerable differences in effect size were observed in the comparison with a limit meta-analysis (|∆MD| < 1).

CONCLUSIONS: The analyses validated the systematic review results and demonstrate the value of confirmatory sensitivity analysis. Routine sensitivity analyses are recommended to assess the robustness of findings and validate conclusions in the presence of between-study heterogeneity.