OBJECTIVES: Considerable confusion exists around the nomenclature, interpretation, and calculation of intra-class correlation coefficients (ICCs) for measuring test-retest reliability (TRT) of patient-reported outcome assessments (PROs). This literature review describes the most common ICC forms used in computing TRTs.

METHODS: A systematic search using OvidSP (Embase, MEDLINE®, PsycINFO®) identified publications that reported use of ICCs in their various forms in psychometric evaluation studies. Articles were included for full-text review if they reported on TRT as determined by one of 6 ICC forms by Shrout and Fleiss or 10 versions by McGraw and Wong and excluded if they reported data without TRT or ICC results. Full-text articles that met inclusion criteria were reviewed, and data (e.g., ICC type and rationale, study design/model) were extracted and summarized.

RESULTS: A total of 216 abstracts from the initial literature search were reviewed, and eight publications spanning therapeutic areas were selected for full-text review. Five full-text articles were identified by Google search. A majority of the studies employed two-way random or mixed-effects model with absolute agreement definition (i.e., use of ICC[2,1], ICC[2,2] and ICC[3,1]). Two articles employed a one-way random effects model to compute ICC(1,1). One article compared results for ICC(1,1) versus ICC(3,1) and found minimal differences, thereby showing no systematic error in measurement errors associated with patient ratings. The most commonly cited rationale for employing the two-way mixed-effects model was the assumption of patients being random and the raters (i.e., items) being fixed.

CONCLUSIONS: Given that there are several forms of ICCs, psychometric evaluation studies should identify ICC type and justify its use in establishing TRT of PROs, a practice that has not been consistently implemented. This literature search aims to bridge that gap by focusing on the few recent articles that do provide rationales, helping inform specific ICC types for use in future studies.