OBJECTIVES: Observational studies are often necessary for assessing the comparative effectiveness of therapeutics in “real-world” settings. The strength of evidence generated by individual studies, however, varies. We describe the development of the GRACE Checklist, a tool for rating the quality of observational CER and assessing whether studies merit consideration for decision making.  METHODS: A checklist was developed based on existing guidelines for the conduct and reporting of observational studies, including the GRACE Principles, and existing scales for the inclusion of observational studies in systematic reviews.  An external advisory board reviewed the checklist content and scoring options; majority opinion of the advisors was used to refine the question items and scoring.  The construct validity of the checklist was measured using two rounds of testing where over 100 volunteer testers rated articles, to determine if the checklist can distinguish studies of known quality.  Articles of “known quality” were first extracted from systematic reviews and then, in the second round, were based on reviews of observational CER studies by recognized experts.  RESULTS: Two domains, internal validity and applicability, were identified for inclusion with a total of 15 questions.  Results and proposed scoring algorithms will be presented for a categorical assessment of study quality to determine if studies are 1) of sufficient quality for decision support; 2) sufficiently flawed to make interpretation unreliable; or 3) require additional consideration.  First round testing results showed that subsets of item responses could yield positive predictive values for identifying high quality studies as high as 0.86, and negative predictive values as high as 0.91. CONCLUSIONS: A validated checklist to assess the quality of observational CER can help decision makers recognize strong evidence without substantial advanced training.