BACKGROUND: Evidence synthesis involves a critical analysis of collected research to identify patterns or trends in the data from which conclusions or inferences can be drawn. Often, a quantitative evaluation of the data is used to draw conclusions. Applications such as comparative effectiveness analyses for HTA submissions require meta-analyses. Other research questions, however, may not always be illuminated by a quantitative meta-analysis.

OBJECTIVES: To present a thought framework to evaluate when a quantitative meta-analysis can be useful and when it may not.

METHODS: Methods for evidence synthesis beyond meta-analysis and their value within the decision-making and inference process are illustrated with case studies where meta-analyses may be neither helpful nor possible.

RESULTS: When evaluating a new objective, it is common to ask whether a meta-analysis is feasible to assess the objective. I argue that more appropriate questions to ask are: 1) What data address the research question?; 2) What does a qualitative assessment of those data tell us? Are there any controversies or challenges in interpreting or comparing them?; and finally 3) Can a quantitative meta-analysis mitigate these difficulties? In many cases, a proper and thorough “visual analysis” of the data is an effective and sufficient process to generate compelling evidence, draw inferences, and support decision-making.

CONCLUSIONS: Meta-analysis is a powerful tool for comparative effectiveness analyses and one method of evidence synthesis. Qualitative evidence synthesis is often undervalued but can also be a powerful tool in making sense of diverse data.