An Adaptation of the RAND/UCLA Modified Delphi Panel Method in the Time of COVID-19


Broder M, Gibbs SN, Yermilov I
Partnership for Health Analytic Research (PHAR), LLC, Beverly Hills, CA, USA

OBJECTIVES: The RAND/UCLA modified Delphi panel method is a formal group process that systematically and quantitatively combines expert opinion and evidence to arrive at consensus, which traditionally includes an in-person meeting. Experts (physicians, advocates) meet in-person to discuss results of a first-round survey before repeating the survey. The COVID-19 pandemic made in-person meetings impossible. In this study, we examine the impact on achieving consensus when moving from in-person to virtual panel meetings.

METHODS: We conducted 5 virtual panels over 13 months and compared them to 4 pre-pandemic in-person panels. We report the number of panelists, meeting duration, items rated, and percent disagreement in first- and second-round surveys.

RESULTS: Both the in-person and virtual panels included a mean of 11 panelists. Panelists joined virtual meetings for 6-7 hours across 2-4 hour sessions. In-person meetings lasted 6-9 hours plus up to 10 hours of travel. Panelists rated a mean of 488 and 453 items in the virtual and in-person panels, respectively. Disagreement was higher in first-round surveys (range 13-67% virtual, 34-67% in-person) than in second-round surveys (range 1-32% virtual, 10-43% in-person). Mean decreases in disagreement were 19% (virtual) and 27% (in-person).

CONCLUSION: We maintained certain aspects of the panel method (e.g., review of existing evidence, number of panelists, number of survey items) and found similar decreases in disagreement between first- and second-round surveys. We engaged a diverse group of experts, including those with busy clinic schedules who may not have traveled to an in-person meeting. While we completed panel discussions in less time virtually, we were unable to recreate the social interactions that built rapport among panelists during in-person meetings. Transitioning from in-person to virtual meetings was not without challenges, but there were also unexpected advantages. This virtual Delphi panel method can be an effective and efficient alternative for researchers and clinicians.

Conference/Value in Health Info

2022-05, ISPOR 2022, Washington, DC, USA




Medical Technologies, Methodological & Statistical Research

Topic Subcategory

Implementation Science, Survey Methods


No Additional Disease & Conditions/Specialized Treatment Areas

Your browser is out-of-date

ISPOR recommends that you update your browser for more security, speed and the best experience on Update my browser now