Enhancing Non-Experts' Understanding of Uncertainty in Psychometric Results through Effective Visual Probabilistic Representations


Willis A1, Jewett A2, Foster B2
1Lumanity, Cambridge, MA, USA, 2Lumanity, Boston, MA, USA

Presentation Documents

OBJECTIVES: Psychometric analysis is vital in drug development to obtain reliable and valid clinical outcomes assessments (COAs) for evaluating drug effects, understanding patient experiences, aiding diagnosis, and supporting clinical decision-making. However, communicating results to non-technical decision-makers presents challenges. Interpretation can be difficult, and success criteria are often applied without considering uncertainty in the psychometric results, which can create false certainty regarding the performance of COAs and lead to underpowered trials with misleading results or ineffective COAs. New research has shown that data visualizations can effectively illustrate uncertainty in statistical estimates. This poster presents a method called the quantile dot plot, which uses a frequency-framing visualization to convey the uncertainty of common psychometric results to non-experts.

METHODS: Common psychometric analyses (e.g., factor analysis, reliability, and validity analyses) were performed on a simulated dataset containing a COA, co-validating COAs, and global impression measures. Analyses were conducted using a Bayesian framework, allowing for the visualization of posterior distributions of each result using quantile dot plots, which were color-coded based on general guidelines for interpreting performance. The probability of a specific result is determined by counting the dots in the plot that meet or exceed the result.

RESULTS: The quantile dot plots effectively displayed plausible estimates and levels of uncertainty for each psychometric result. Some plots indicated that while the modal estimate met the performance criterion, there was a higher likelihood of not meeting the criterion than expected, which was determined by counting the dots.

CONCLUSIONS: Quantile dot plots offer an accessible visualization method for communicating complex continuous probabilistic distributions to non-experts and decision-makers, and clearly illustrates the uncertainty of results, which is critical to consider when developing endpoint strategies. By using this visualization method, decision-makers can better understand the reliability and validity of COAs, leading to improved patient care and more accurate clinical decision-making.

Conference/Value in Health Info

2024-05, ISPOR 2024, Atlanta, GA, USA

Value in Health, Volume 27, Issue 6, S1 (June 2024)




Clinical Outcomes, Methodological & Statistical Research, Patient-Centered Research

Topic Subcategory

Clinical Outcomes Assessment, Instrument Development, Validation, & Translation, Patient-reported Outcomes & Quality of Life Outcomes, PRO & Related Methods


No Additional Disease & Conditions/Specialized Treatment Areas

Your browser is out-of-date

ISPOR recommends that you update your browser for more security, speed and the best experience on ispor.org. Update my browser now