OBJECTIVES : To evaluate the degree of agreement between findings from randomized controlled trials (RCTs) and observational studies evaluating the effect of drugs, and to explore attributes of observational studies that lead to higher levels of agreement.

METHODS : We performed a systematic search of meta-analyses reporting the effect of drugs or drug classes where both RCTs and observational evidence were reported, using PubMed and Google Scholar. Studies included in the review had to report a common outcome measure and drug treatment and had to report pooled evidence separately for observational and RCTs. We then extracted pooled RCT estimates, pooled observational estimate and individual observational estimates from forest plots or abstracting individual studies included in the review. Agreement between pooled RCT and observational estimates was defined using 3 metrics: overlap between confidence intervals (metric 1); point estimate of observational evidence falling within 95%CI of RCTs (metric 2); and agreement between the statistical conclusions (metric 3).

RESULTS

We identified 1182 studies from PubMed and Google Scholar. After title and abstract review, we excluded 1085 studies. Full-text review was performed for the remaining 97 studies, of which 34 studies met inclusion criteria resulting in 81 comparisons between pooled RCT estimate and pooled observational estimates. Agreement was 88.9 %, 81.5%, and 58.0% using metrics 1, 2 and 3. 490 comparisons were observed from 420 individual studies. The agreement was found to be respectively 86.1%, 53.5% and 61.6% with metric 1, 2 and 3. When treatment effects were studied, the statistical conclusions agreed 61.5% compared to 57.3% when adverse effects were studied.

CONCLUSIONS : Observational evidence generally agrees with evidence obtained from RCTs. The level of agreement did not meaningfully vary if treatment effect or adverse effects were studied and it will be important to identify the influence of other study attributes on agreement.