Application of Sentiment Analysis to Aid in Evaluation of Patient Reported Outcome Measures: Analysis of Qualitative Cognitive Debriefing Interviews
Author(s)
Christine Bradshaw, MA1, Kristina Davis, PhD2, Elizabeth Merikle, PhD2;
1Fortrea, Senior Research Associate, Durham, NC, USA, 2Fortrea, Durham, NC, USA
1Fortrea, Senior Research Associate, Durham, NC, USA, 2Fortrea, Durham, NC, USA
Presentation Documents
OBJECTIVES: Respondent understanding of item content of patient-reported outcome measures (PROMs) is typically assessed via qualitative methods such as cognitive debriefing interviews. This research aimed to examine a quantitative method (sentiment analysis) for determining item comprehension. We compared methods and identified key considerations and limitations.
METHODS: Sixteen transcripts from a cognitive debriefing study assessing the respondent understanding and relevance of items in an electronic daily symptom diary were analyzed. Each base word in the diary was matched against the sentiment lexicon in the QDAP library of positive (e.g. straightforward, clear, good) and negative (e.g. difficult, hard, and problem) words (key.pol dictionary) with R Studio. Three iterations were run: 1) the default, 2) adjusted dictionary [to remove irrelevant terms], and 3) adjusted size of context window. Results of the quantitative analysis were compared to the qualitative results.
RESULTS: The average sentiment score for the first run was negatively valent (-0.47) as the key.pol dictionary skewed results negative because item content aligned with negative runs. The interviews indicated that the items were generally well understood, relevant, and easy to complete, with clear response options. However, one-third of participants found some items difficult, suggesting areas for improvement. The second and third runs showed mildly positive scores of 0.25 and 0.24, consistent with the qualitative feedback.
CONCLUSIONS: The positively valent average sentiment is consistent with the qualitative results and confirms that participants generally understood the items and their response options. Sentiment analysis provides additional quantitative data that can help inform decisions to remove or retain items during the development of new or modification of existing PROMs. Consideration should be given to creating a dictionary tailored to the sentiment of instrument and item comprehension to enhance the utility of this quantitative approach.
METHODS: Sixteen transcripts from a cognitive debriefing study assessing the respondent understanding and relevance of items in an electronic daily symptom diary were analyzed. Each base word in the diary was matched against the sentiment lexicon in the QDAP library of positive (e.g. straightforward, clear, good) and negative (e.g. difficult, hard, and problem) words (key.pol dictionary) with R Studio. Three iterations were run: 1) the default, 2) adjusted dictionary [to remove irrelevant terms], and 3) adjusted size of context window. Results of the quantitative analysis were compared to the qualitative results.
RESULTS: The average sentiment score for the first run was negatively valent (-0.47) as the key.pol dictionary skewed results negative because item content aligned with negative runs. The interviews indicated that the items were generally well understood, relevant, and easy to complete, with clear response options. However, one-third of participants found some items difficult, suggesting areas for improvement. The second and third runs showed mildly positive scores of 0.25 and 0.24, consistent with the qualitative feedback.
CONCLUSIONS: The positively valent average sentiment is consistent with the qualitative results and confirms that participants generally understood the items and their response options. Sentiment analysis provides additional quantitative data that can help inform decisions to remove or retain items during the development of new or modification of existing PROMs. Consideration should be given to creating a dictionary tailored to the sentiment of instrument and item comprehension to enhance the utility of this quantitative approach.
Conference/Value in Health Info
2025-05, ISPOR 2025, Montréal, Quebec, CA
Value in Health, Volume 28, Issue S1
Code
PCR8
Topic
Patient-Centered Research
Topic Subcategory
Instrument Development, Validation, & Translation
Disease
SDC: Infectious Disease (non-vaccine)