OBJECTIVES: Researchers conducting Discrete Choice Experiments (DCE) frequently use qualitative research methods whilst developing surveys, particularly for elicitation of attributes. We conducted cognitive debriefing interviews to evaluate survey comprehension, in particular of choice tasks, in 3 online DCE studies.

METHODS: Three online DCEs were conducted: DCE 1 - drivers of patient preference regarding endometriosis treatment characteristics, DCE 2 - drivers of patient preference regarding characteristics of inhalers for chronic obstructive pulmonary disease and DCE 3 – understanding the preference of patients with schizophrenia regarding antipsychotic tolerability profiles. All surveys included screening and socio-demographic questions, followed by 12 DCE choice sets. Pilot tests in 2 phases were conducted: a subset of respondents took part in telephone cognitive debriefing interviews. Interviews were recorded and detailed notes were taken. Changes to surveys were made accordingly after phase 1; these changes were adjusted or confirmed in phase 2.

RESULTS: A total of 10 cognitive debriefing interviews were conducted for DCE 1, 5 for DCE 2 and 8 for DCE 3. Most issues identified during interviews were related to misunderstanding vocabulary in the screening questions or choice sets and misunderstanding the task for the choice sets. Some issues were common to all surveys, notably the hypothetical nature of scenarios, where respondents often attempted to identify the scenario closest to their current situation. Changes made after the first pilot phase according to respondents’ feedback generally resolved comprehension issues. Some additional changes were made after phase 2 and all issues were resolved before the full surveys were launched.

CONCLUSIONS: Carrying out cognitive debriefing in two phases for DCE pilot studies highlights the difficulties encountered by respondents with this type of exercise. Pilot studies in two phases including two phases of cognitive debriefing interviews are recommended for DCEs to ensure that respondents correctly understand choice tasks.