OBJECTIVES: Online survey targeting general population without interviewer to measure utility using composite time trade-off (cTTO) may improve generalization of results and reduce survey costs and time. However, there are concerns about respondent engagement and validation issues with online surveys. We examine the feasibility and challenges of using online survey by comparing results of cTTO-based utilities measured by both face-to-face (F2F) and online methods.

METHODS: We assessed utilities of 11 health status of metastatic pancreatic cancer, developed in our previous study, using cTTO targeting to general Japanese population through online survey method without interviewers and F2F survey method with interviewers. We used the cTTO online response system to remove bias between online without interviewers and F2F with interviewers in survey procedure. The primary endpoint was utility for each scenario. Secondary endpoints were the consistency of cTTO exercises, the comprehension questionnaire, the number of tasks, the time per task to assess respondents' understanding and engagement.

RESULTS: 319 participants in the online method and 201 participants in the F2F method were selected as analysis population. Comparison of utility for each scenario showed that generally lower values were observed in the online compared with those of the F2F. For example, the scenario describing “stable disease+ no adverse event” was 0.385 in online and 0.634 in F2F. For comprehension and engagement, a higher proportion of respondents in online (6.3%) compared to F2F (0.2%), (P<0.005) indicated that a scenario had the same utility as full health status. The average response time (IQR) per task was also shorter in online, 47(31-47) seconds compared to F2F, 94(74-125) seconds, (P<0.005).

CONCLUSIONS: This finding suggested that poor understanding of cTTO and unreliable responses among the general population may cause differences between two methods. Further studies are required to improve quality and reliability of online survey.