OBJECTIVES: Quality assessment (QA) is an important part of a well-designed SLR, yet finding an appropriate, valid, and easy-to-use QAT for each study design of interest can be a challenge. Therefore, we conducted an SLR to identify published evaluations of QATs.

METHODS: We searched MEDLINE (via PubMed) from database inception to October 18, 2017 for English language publications evaluating QATs. Included publications assessed agreement, validity and/or reliability of QATs between reviewers or compared with another QAT.

RESULTS: Thirty-one studies reporting on 34 scales were included in the SLR. The identified QATs most frequently assessed randomized controlled trials (RCTs; 25 scales; 15 publications), non-randomized studies (5 scales; 4 publications), and SLRs (3 scales; 4 publications), but tools assessing case-control and cohort studies (1 scale; 2 publications), guidelines (1 scale; 1 publication), diagnostic studies (2 scales; 2 publications), health economic studies (1 scale; 1 publication), and meta-analyses (MA; 1 scale; 1 publication) were also identified. Inter-rater reliability was most commonly reported and was measured with the kappa statistic or intraclass correlation coefficient (ICC). QAT reliability was highest for SLRs/MAs (ICC 0.91; kappa 0.84) followed by RCTs (ICC 0.55-0.92; kappa 0.02-0.94), guidelines (ICC 0.755), non-randomized trials (ICC 0.595-0.66), diagnostic studies (kappa 0.18), and case-control and cohort studies (kappa -0.004-0.29). Additionally, included studies reported on the agreement between tools and ease of use.

CONCLUSIONS: Evaluating study quality is essential for correctly interpreting the results of SLRs. While QATs for SLRs and RCTs are plentiful and reliable, tools for other study designs are limited and lack in overall reliability.