From Workshops to a Dashboard Prototype: Evaluating a Checklist Approach and User Design for Digital Health Technology Assessment
Author(s)
Gurmit Sandhu, B Pharm (Hons)| MSc Medical Informatics | MPH |MBM1, Emanuele Laurenzi, PhD, MSc BIS2, Izzuna MM Ghazali, MBBS, MPH3, Enkelejda Miho, PhD, MSc4.
1AI Practitioner in Healthcare, Gurmit Sandhu Consulting GmbH, Basel, Switzerland, 2FHNW University of Applied Sciences and Arts Northwestern Switzerland, Olten, Switzerland, 3Malaysian Health Technology Assessment Section (MaHTAS), Ministry of Health Malaysia, Putrajaya, Malaysia, 4FHNW University of Applied Sciences and Arts Northwestern Switzerland, Muttenz, Switzerland.
1AI Practitioner in Healthcare, Gurmit Sandhu Consulting GmbH, Basel, Switzerland, 2FHNW University of Applied Sciences and Arts Northwestern Switzerland, Olten, Switzerland, 3Malaysian Health Technology Assessment Section (MaHTAS), Ministry of Health Malaysia, Putrajaya, Malaysia, 4FHNW University of Applied Sciences and Arts Northwestern Switzerland, Muttenz, Switzerland.
OBJECTIVES: To evaluate user satisfaction and perceived usefulness of a novel checklist developed to support digital health technology (DHT) value assessment, based on feedback from two online workshops. Additionally, to collect post-workshop feedback on a dashboard prototype from a broader group of stakeholders.
METHODS: Participants (n1=13, n2=9) self-rated their satisfaction with the checklist and its utility in DHT value assessment before and after the 2 workshops, following a review of an AI-based DHT use case. A 5-point Likert scale assessed agreement with statements on clarity of instructions, usability, comprehensiveness, accountability, transparency, and self-learning. Participants also identified which domains were most negatively affected. An online dashboard prototype was developed based on self-ratings of information completeness for each value domain. Malaysian workshop participants: representatives from the HTA agency, Ministry of Health, and a patient advocacy leader. Post-workshop, Likert-based feedback (n3=22) on the dashboard’s user design (UX) and likelihood of use was also collected from international stakeholders, including startups, ethicists, HTA agencies, clinicians, and an insurer.
RESULTS: Before the 1st workshop, 28% of participants were unfamiliar with the checklist; 14% were above-average familiar. After the workshops, the highest agreement was for applicability (4.1/5) and self-learning (4.0/5), with the lowest for transparency (3.6/5) and accountability (3.4/5). Despite average ratings, improvements were noted in clarity of instructions (+15%), accountability (+13%), and transparency (+9%). Safety and ethical aspects domains were negatively affected. 85% of participants reported they were likely to use or recommend the dashboard. Positive feedback emphasized usability, usefulness, credibility, and visualization; customization was noted as an area for improvement.
CONCLUSIONS: Checklist was perceived as a valuable tool for DHT value assessment. Feedback confirmed a well-received dashboard prototype. Stakeholder engagement on UX supports usefulness and adoption of digital health evaluation tools. Methodology guidelines on how to evaluate cost impact and effectiveness may be needed.
METHODS: Participants (n1=13, n2=9) self-rated their satisfaction with the checklist and its utility in DHT value assessment before and after the 2 workshops, following a review of an AI-based DHT use case. A 5-point Likert scale assessed agreement with statements on clarity of instructions, usability, comprehensiveness, accountability, transparency, and self-learning. Participants also identified which domains were most negatively affected. An online dashboard prototype was developed based on self-ratings of information completeness for each value domain. Malaysian workshop participants: representatives from the HTA agency, Ministry of Health, and a patient advocacy leader. Post-workshop, Likert-based feedback (n3=22) on the dashboard’s user design (UX) and likelihood of use was also collected from international stakeholders, including startups, ethicists, HTA agencies, clinicians, and an insurer.
RESULTS: Before the 1st workshop, 28% of participants were unfamiliar with the checklist; 14% were above-average familiar. After the workshops, the highest agreement was for applicability (4.1/5) and self-learning (4.0/5), with the lowest for transparency (3.6/5) and accountability (3.4/5). Despite average ratings, improvements were noted in clarity of instructions (+15%), accountability (+13%), and transparency (+9%). Safety and ethical aspects domains were negatively affected. 85% of participants reported they were likely to use or recommend the dashboard. Positive feedback emphasized usability, usefulness, credibility, and visualization; customization was noted as an area for improvement.
CONCLUSIONS: Checklist was perceived as a valuable tool for DHT value assessment. Feedback confirmed a well-received dashboard prototype. Stakeholder engagement on UX supports usefulness and adoption of digital health evaluation tools. Methodology guidelines on how to evaluate cost impact and effectiveness may be needed.
Conference/Value in Health Info
2025-11, ISPOR Europe 2025, Glasgow, Scotland
Value in Health, Volume 28, Issue S2
Code
MT19
Topic
Health Policy & Regulatory, Health Technology Assessment, Medical Technologies
Topic Subcategory
Digital Health
Disease
No Additional Disease & Conditions/Specialized Treatment Areas, Oncology, Respiratory-Related Disorders (Allergy, Asthma, Smoking, Other Respiratory)