An Evaluation of the Impact of Evidence Grouping on Certainty Rating When Performing a Grade Assessment Without a Meta-Analysis

OBJECTIVES: Grading of Recommendations Assessment, Development, and Evaluation (GRADE) provides a reproducible, transparent framework for rating the certainty of a body of evidence identified systematically. Meta-analysis is often used to inform GRADE assessment but may not always be feasible, especially where the evidence is heterogenous. We analyzed the impact of evidence grouping on certainty rating as per the GRADE framework in the absence of a meta-analysis.

METHODS: We applied the GRADE framework to systematically identified randomized controlled trials (RCTs) comparing the patient-important outcome, rate of annual exacerbations (moderate or severe), in patients with chronic obstructive pulmonary disease (COPD) following triple or dual therapy. One assessment considered all evidence identified on the endpoint of interest, whereas another split evidence by measurement criteria; the evidence certainty rating was compared between assessments.

RESULTS: Five RCTs were identified that reported on rate of annual COPD exacerbations; all studies reported a statistically significant lower annual rate of exacerbations with triple vs dual therapy. When all identified evidence was assessed as a single group, the certainty of evidence was rated “very low”. However, when the evidence was grouped by measurement criteria, the three outcomes measured using the same criteria had a “moderate” certainty rating. The two remaining outcomes, in which the measurement criteria were not reported, were rated as “very low” certainty owing to downgrading in the indirectness and imprecision domains (non-reporting of outcome measurement methodology), differences in follow-up periods and lack of reporting on statistical power.

CONCLUSIONS: We illustrate how evidence grouping can affect certainty ratings as per the GRADE framework, and highlight the need for careful consideration of potentially confounding factors even in the absence of meta-analysis. This is especially pertinent when considering the growing body of literature including real-world evidence and the inevitable source of heterogeneity associated with different study designs.