OBJECTIVES: To determine the best algorithm for using the GRACE checklist to rate the quality of observational comparative effectiveness research (CER) studies METHODS: An 11-item checklist about data and methods was developed through literature review and consultation with experts. The checklist was applied to 88 articles by 113 raters from 5 continents, and their feedback helped shape questions and user guidance. Positive and negative predictive values (PPV and NPV, respectively) were calculated based on a volunteer rater’s assessment of a published article compared to a “gold standard” as determined through an article’s inclusion in a systematic review or through expert assessment. Multivariate regression analysis and Classification and Regression Trees (CART) analysis will be conducted based on data collected from 22 volunteer raters’ assessment of 28 articles of comparative effectiveness using a revised version of the checklist. RESULTS: Checklist items pertaining to data have relatively strong NPV, meaning the checklist is effective in identifying CER articles of insufficient quality. The single best performing item was the validity of the primary outcome, which achieved NPV and PPV ≥ 0.67 in four rounds of testing. Question items pertaining to whether the primary outcome was measured objectively and adequately recorded also achieved high NPV (≥ 0.67 in 4 and 5 rounds of testing, respectively). Other indicators of quality include the use of concurrent comparators (NPV ≥ 0.67 in 5 rounds of testing). CONCLUSIONS: Univariate analyses have demonstrated the usefulness of the checklist items to screen out articles of insufficient quality for decision support. Results of multivariate analyses as used to develop 1) a parsimonious model to identify which checklist items can best be used as a quick screening tool and 2) a regression tree that maximizes the NPV of the checklist will be presented.