Content Validation of Survey Instruments for Identifying User Preferences in Grey Literature Review Automation
Author(s)
ZEN YANG ANG, RPh1, Shaun WH Lee, PhD2.
1Student, Monash University Malaysia, Subang Jaya, Malaysia, 2Monash University, Subang, Malaysia.
1Student, Monash University Malaysia, Subang Jaya, Malaysia, 2Monash University, Subang, Malaysia.
OBJECTIVES: Conducting systematic grey literature reviews(SGLRs) is arduous. The recent introduction of large language models has potentiated the automation of SGLRs. We have examined users' preferred features of these automation tools based on the Unified Theory of Acceptance and Use of Technology(UTAUT) framework.
METHODS: A draft survey containing nine attributes with 3 levels was developed.. Six experts were invited to rate(1 or 0) the content validity rating in terms of simplicity, relevancy, and provide comments. Five experts commented on the survey's usability. Item-level content validity index(I-CVI), Scale-level CVI based on average method(S-CVI/Ave) and Scale-level CVI based on universal agreement(S-CVI/UA) were calculated.
RESULTS: The content validity was acceptable, with S-CVI/Ave and S-CVI/UA scores of 96.9% and 85.2%. Experts agreed that the UTAUT‐based attributes related to facilitating conditions and performance expectancy were relevant, but not those related to social influence and effort expectancy. Qualitative responses found that the survey was time consuming and complex to administer.
CONCLUSIONS: While the survey had acceptable content validity, usability testing revealed that the survey was complex and would need to be redesigned.
METHODS: A draft survey containing nine attributes with 3 levels was developed.. Six experts were invited to rate(1 or 0) the content validity rating in terms of simplicity, relevancy, and provide comments. Five experts commented on the survey's usability. Item-level content validity index(I-CVI), Scale-level CVI based on average method(S-CVI/Ave) and Scale-level CVI based on universal agreement(S-CVI/UA) were calculated.
RESULTS: The content validity was acceptable, with S-CVI/Ave and S-CVI/UA scores of 96.9% and 85.2%. Experts agreed that the UTAUT‐based attributes related to facilitating conditions and performance expectancy were relevant, but not those related to social influence and effort expectancy. Qualitative responses found that the survey was time consuming and complex to administer.
CONCLUSIONS: While the survey had acceptable content validity, usability testing revealed that the survey was complex and would need to be redesigned.
Conference/Value in Health Info
2025-11, ISPOR Europe 2025, Glasgow, Scotland
Value in Health, Volume 28, Issue S2
Code
MSR61
Topic
Methodological & Statistical Research, Organizational Practices, Study Approaches
Topic Subcategory
Artificial Intelligence, Machine Learning, Predictive Analytics
Disease
No Additional Disease & Conditions/Specialized Treatment Areas