TEST-RETEST RELIABILITY OF CASE 1 BEST-WORST SCALING: EVIDENCE FROM RELATIVE ATTRIBUTE IMPORTANCE OF PREFERENCES FOR AI USE IN HEALTHCARE
Author(s)
Vinh Vo, MEPP, Maame E. Woode, PhD, Gang Chen, MSc, PhD;
Monash University, Melbourne, Australia
Monash University, Melbourne, Australia
OBJECTIVES: To examine test-retest reliability of Case 1 BWS in investigating public preferences toward the use of AI in healthcare and relative attribute importance (RAI) of preferences for the use of AI in healthcare.
METHODS: Data were collected from members of the Australian general public, on 20 key attributes related to the use of AI in healthcare. Surveys were administered twice, with an interval of 6 months between tests. Only respondents whose reported attitudes towards AI remained unchanged between waves were included in the analysis. The BWS data was modelled using a panel-data mixed logit model to account for unobserved preference heterogeneity. The relative importance share of attributes were calculated. Test-retest reliability was analysed by using intra-class correlation coefficient (ICC) to determine absolute agreement.
RESULTS: Across 316 respondents who reported that their attitudes towards AI remained unchanged, similar RAI results were found across two waves for the most important attributes: “AI not causing harm”, “AI systems being accurate”, and “Ensuring privacy, data security, and confidentiality in AI systems” and for the least important attributes: “Engaging the patients/public in AI development”, “Allowing industry to self-regulate”, “Knowing who funded AI systems”, and “Promoting innovations in AI”. Attributes in the Social group varied much more than attributes in the Legal and Ethical group. Both male and female groups prioritise safety and privacy but diverge in their concerns about governance and transparency. High-income respondents placed greater emphasis on individual autonomy and formal regulation while low-income respondents focused more on accountability and protection. The ICC indicated that the BWS method demonstrates good test-retest reliability over time for full sample and subsamples.
CONCLUSIONS: This study makes an important contribution to the emerging literature on the test-retest reliability using relative attribute importance derived from BWS in the context of AI in healthcare.
METHODS: Data were collected from members of the Australian general public, on 20 key attributes related to the use of AI in healthcare. Surveys were administered twice, with an interval of 6 months between tests. Only respondents whose reported attitudes towards AI remained unchanged between waves were included in the analysis. The BWS data was modelled using a panel-data mixed logit model to account for unobserved preference heterogeneity. The relative importance share of attributes were calculated. Test-retest reliability was analysed by using intra-class correlation coefficient (ICC) to determine absolute agreement.
RESULTS: Across 316 respondents who reported that their attitudes towards AI remained unchanged, similar RAI results were found across two waves for the most important attributes: “AI not causing harm”, “AI systems being accurate”, and “Ensuring privacy, data security, and confidentiality in AI systems” and for the least important attributes: “Engaging the patients/public in AI development”, “Allowing industry to self-regulate”, “Knowing who funded AI systems”, and “Promoting innovations in AI”. Attributes in the Social group varied much more than attributes in the Legal and Ethical group. Both male and female groups prioritise safety and privacy but diverge in their concerns about governance and transparency. High-income respondents placed greater emphasis on individual autonomy and formal regulation while low-income respondents focused more on accountability and protection. The ICC indicated that the BWS method demonstrates good test-retest reliability over time for full sample and subsamples.
CONCLUSIONS: This study makes an important contribution to the emerging literature on the test-retest reliability using relative attribute importance derived from BWS in the context of AI in healthcare.
Conference/Value in Health Info
2026-05, ISPOR 2026, Philadelphia, PA, USA
Value in Health, Volume 29, Issue S6
Code
PCR162
Topic
Patient-Centered Research
Disease
No Additional Disease & Conditions/Specialized Treatment Areas