Improving Design Choices in Delphi Studies for Clinical Consensus Recommendations: Results of a Pilot Approach to Rapid, High-Quality Engagement
Author(s)
Janu A. Barhate, MBA1, Jyoti Sahota, MSc1, Rachel Chu, MSc2, Liga Bennetts, PhD3;
1Amaris Consulting, Toronto, ON, Canada, 2Amaris Consulting, Montreal, QC, Canada, 3Amaris Consulting, Principal Consultant, Montreal, QC, Canada
1Amaris Consulting, Toronto, ON, Canada, 2Amaris Consulting, Montreal, QC, Canada, 3Amaris Consulting, Principal Consultant, Montreal, QC, Canada
Presentation Documents
OBJECTIVES: The Delphi technique is a structured, systematic method to obtain knowledge and perspectives of a group of experts in an area of uncertainty. However, methodological choices can influence whether valid research results are obtained or whether the process can be executed within required timeframes. We assessed the success of a pilot approach to build consensus among clinical experts.
METHODS: We formulated an approach to conduct a complete, 3-round modified Delphi study, including preparation, and an in-person debrief meeting, within 3 months. The goal was to build consensus on optimal management of an oncology indication in Canada. We addressed potential methodological shortcomings in design, including panel recruitment bias, attrition, clear question specification, and panelist engagement. We placed emphasis on the preparatory phase, collaborating closely with the lead moderator, expert chairs and advisor to formulate the study plan, questionnaire format, and draft statements. Diverse panelist profiles were recruited, and full commitment from panelists was sought. Content was validated by an expert advisor before each round. Consensus was clearly defined and explained to panelists (5-point Likert scale; consensus set at >80% respondents agreeing with a statement).
RESULTS: Three Delphi rounds were completed in <4 weeks. Response rates consistently exceeded 80%. Consensus was achieved for 68% (14/22 statements) in the first round, 86% (19/22 statements) in the second, reaching 95% (21/22 statements) in the third round. Following a debrief meeting, consensus was achieved across all statements.
CONCLUSIONS: We advise researchers considering a Delphi study not to overlook establishing a thorough, clearly communicated plan, to involve experts in planning process and validation of the questionnaire at each round, pre-specify consensus definition, recruit varied panelist profiles, request personal commitment of panelists, and clearly communicate requirements and reminders. Ensuring attention to these aspects can improve the rigor and quality of the Delphi process, and improve participant engagement and timely completion.
METHODS: We formulated an approach to conduct a complete, 3-round modified Delphi study, including preparation, and an in-person debrief meeting, within 3 months. The goal was to build consensus on optimal management of an oncology indication in Canada. We addressed potential methodological shortcomings in design, including panel recruitment bias, attrition, clear question specification, and panelist engagement. We placed emphasis on the preparatory phase, collaborating closely with the lead moderator, expert chairs and advisor to formulate the study plan, questionnaire format, and draft statements. Diverse panelist profiles were recruited, and full commitment from panelists was sought. Content was validated by an expert advisor before each round. Consensus was clearly defined and explained to panelists (5-point Likert scale; consensus set at >80% respondents agreeing with a statement).
RESULTS: Three Delphi rounds were completed in <4 weeks. Response rates consistently exceeded 80%. Consensus was achieved for 68% (14/22 statements) in the first round, 86% (19/22 statements) in the second, reaching 95% (21/22 statements) in the third round. Following a debrief meeting, consensus was achieved across all statements.
CONCLUSIONS: We advise researchers considering a Delphi study not to overlook establishing a thorough, clearly communicated plan, to involve experts in planning process and validation of the questionnaire at each round, pre-specify consensus definition, recruit varied panelist profiles, request personal commitment of panelists, and clearly communicate requirements and reminders. Ensuring attention to these aspects can improve the rigor and quality of the Delphi process, and improve participant engagement and timely completion.
Conference/Value in Health Info
2025-05, ISPOR 2025, Montréal, Quebec, CA
Value in Health, Volume 28, Issue S1
Code
SA74
Topic
Study Approaches
Topic Subcategory
Surveys & Expert Panels
Disease
SDC: Oncology