OBJECTIVE: A systematic review is the preferred approach for assembling clinical evidence. The gold standard for literature searching comprises sensitive search strategies applied to multiple literature databases and hand-searching of journals and conference abstracts. As a follow-up to a previous ISPOR poster, we sought to evaluate the impact of different search approaches on the pooled statistical results from systematic reviews, rather than simply numbers of studies included. METHODS: Previously, we selected a series of published Cochrane systematic reviews and compared the effect of more limited search strategies (a search of multiple literature databases without grey literature, and a Medline keyword search) on the number of included studies. We extended this work to compare the pooled effect sizes resulting from meta-analyses of key outcomes from the studies included in each search strategy. The reviews covered five different areas: smoking cessation, non-small cell lung cancer, neuropathic pain, acupuncture and Crohn's disease. All meta-analyses where studies were missed by lower level searches were re-run including only the studies retrieved by these searches. This allowed the impact of missing studies on the meta-analysis result to be assessed. RESULTS: Differences between meta-analysis results were generally fairly minimal, although in some cases missing studies changed the result of a meta-analysis from a significant to a non-significant result. In several cases lower level searches resulted in there being no studies at all looking at particular outcomes. For example, in a review of neuropathic pain treatments, which showed a significant effect of antidepressants versus placebo on atypical facial pain relief (RR = 1.67), both studies included in the meta-analysis were absent when searching with a Medline keyword search only. CONCLUSIONS: A comprehensive search strategy is needed to retrieve all relevant studies in a systematic review. Less comprehensive searches impact results of meta-analyses and can distort the evidence base.