OBJECTIVES: To determine the impact of publication bias on the magnitude of the treatment effect in clinical studies evaluating hormone therapy (HT) on the frequency of hot flashes among menopausal women. METHODS: We searched computerized databases for English language articles from 1966 to April, 2004. Two reviewers evaluated 5840 published titles, identified placebo-controlled observations that met explicit inclusion criteria, and extracted data pertaining to study characteristics, interventions used, and outcomes measured. Studies evaluating the effects of HT in cancer patients were excluded. The impact of HT on the frequency of hot flashes was calculated using a random-effects model. The effects of publication bias were assessed using the trim and fill method. RESULTS: Based on our criteria, a total of 29 studies were included in the meta-analysis. A funnel plot examining the presence of publication bias was asymmetric, suggesting that small non-significant studies were missing. The trim and fill method suggested that four studies were missing. The standardized mean difference between unadjusted and adjusted point estimates was 0.8 (CI 0.3-1.2). CONCLUSIONS: When evaluating the efficacy and safety of therapeutic interventions, the validity of findings from a meta-analysis is questionable if publication bias is present. Given that restricting a meta-analysis to published literature can distort the effects under investigation by as much as 30%, researchers should try to detect and correct for publication bias when synthesizing the evidence. Our meta-analysis confirmed that HT is effective in relieving menopause-related hot flashes but less than originally suggested. For therapies with significant risk-benefit tradeoffs, the clinical decision could vary upon a complete and unbiased assessment.