OBJECTIVES: To be able to identify observational studies of good enough quality for decision support by validating a set of screening questions to qualify studies likely to produce reasonably accurate and unbiased estimates of comparative effectiveness (CE).    METHODS: An 11-item checklist was developed through literature review and consultation with experts from ISPOR, ISPE, payer groups, private sector and academia.  Item content covers four quality domains: comparability of subjects, information about the exposure or intervention, outcome measurement, and statistical analysis, which are metrics similar to those used in assessing observational study quality for systematic reviews.  Checklist items were tested using studies of drugs, medical devices and medical procedures.   We focused on research quality, not applicability to any decision.  A fundamental challenge was to find a gold standard against which to test checklist items.  113 volunteers from 5 continents each rated  >3 articles (N=280 assessments) from three validation sets of studies that 1) had quality assessments published in systematic reviews;  2) were assessed for quality by one of nine advisors from academic and payer groups;  or 3) were assessed for quality by two of the nine advisors.    RESULTS: Expert reviews uncovered an unsettling lack of agreement about what “good” looks like, especially in situations that lacked context, with 52% concordance (5 experts, 23 assessments.)   The single best performing checklist item, data quality for the primary outcome(s), scored >= 0.67 for positive predictive value in 4 of 6 samples and > = 0.67 for negative predictive values in all 6 samples.  Another high scoring question, sensitivity analyses, had a positive predictive value >= 0.69 for in all 6 samples.  CONCLUSIONS: This quantitative study shows that many content items recommended by experts do not consistently distinguish high quality observational CE studies.