OBJECTIVES: Evaluation of item comprehension is a key component of cognitive interviewing to support the content validity of a clinical outcome assessment (COA) for medical product labeling. Currently there is no standardized methodology for the assessment of item comprehension. In this study a novel and replicable approach for assessing item comprehension in cognitive interviews is presented.

METHODS: A pragmatic literature review of cognitive interviewing techniques and the evaluation of comprehension was conducted. The findings were used to construct a novel framework for the evaluation of comprehension; this framework was then employed in a qualitative cognitive interview study. Comprehension of individual items and instrument instructions, was evaluated in three components: literal comprehension (ability to understand the item as written), inferential comprehension (ability to explain the item), and evaluative comprehension (ability to provide an answer to the item). Literal comprehension was assessed based on interviewer judgement during the interview, while inferential and evaluative comprehension were assessed during the coding of transcripts, where evidentiary quotes were identified and tabled.

RESULTS: Two thresholds for comprehension were established a priori: 1) an individual participant was determined to have shown comprehension of an item or instrument instructions if they demonstrated two of three components of comprehension; 2) an item was considered to have been comprehended if ≥80% of participants demonstrated comprehension for each of the three components of comprehension. For the study in which this method was employed, both individual participant comprehension and item comprehension were demonstrated for all components and items.

CONCLUSIONS: A novel three-component approach to evaluating item comprehension in cognitive interviews has been developed. This framework provides clear and replicable assessment guidelines for future cognitive interview studies.