OBJECTIVES: The Quality of Health Economic Studies (QHES) instrument is a standardized tool designed to evaluate the quality of health economic studies. Due to the diverse nature of study designs, assessing the quality of economic evaluations of clinical pharmacy services (EECPS) using the QHES has posed unique challenges. The purpose of this assessment was to determine the usability, reliability, content validity, and criterion validity of the QHES for evaluating EECPS.

METHODS: After conducting a systematic review of EECPS using the QHES, a survey with four open-ended questions was developed. This survey aimed to assess the clarity and interpretation of the QHES questions, its effectiveness in generating consistent scores, its ability to capture essential study quality constructs, and its overall capability to measure the quality of economic evaluations. Thematic analysis was performed on responses to identify key themes and insights. Interrater reliability was assessed using Cohen’s Kappa and score consistency using Intraclass Correlation Coefficient (ICC).

RESULTS: Two faculty, three graduate students, and four pharmacy doctorate students evaluated 47 EECPS. Thematic analysis identified several challenges with the QHES, such as unclear questions and a lack of context, which contributed to answering difficulty. The question’s binary response options and subjective nature led to concerns about score consistency, as interpretations varied based on the reviewer’s experience. Cohen's Kappa values ranged from -0.1115 to 0.6205, indicating no strong agreement for any question, with only one question showing a moderate Kappa statistic. The ICC (2,k) revealed poor agreement between senior and junior reviewers (ICC = 0.14811).

CONCLUSIONS: This study highlights significant challenges in using the QHES for evaluating its use on clarity, consistency, and the appropriateness of the tool for different study designs. These findings suggest the need for revisions to the QHES or the development of new tools better suited to the diverse nature of EECPS and non-model-based studies.