Does Our Instrument Include the Right Response Options? Empirical Evidence to Evaluate Floor and Ceiling Effects for Multi-Item Ordinal Instruments
Author(s)
Nicolai D. Ayasse, PhD, Fraser D. Bocell, MEd, PhD, Cheryl D. Coon, PhD.
Clinical Outcome Assessment Program, Critical Path Institute, Tucson, AZ, USA.
Clinical Outcome Assessment Program, Critical Path Institute, Tucson, AZ, USA.
Presentation Documents
OBJECTIVES: Floor or ceiling (collectively “scale attenuation”) effects in an ordinal item or instrument are marked by a large percentage of participants endorsing the lowest or highest response option or score. They are problematic when observed due to response options not adequately capturing the range of the latent variable (θ) intended to be measured. Although evaluation for scale attenuation effects is standard practice when psychometrically evaluating an instrument, there is no standard threshold that is agreed upon in the field for defining when they become a problem. The purpose of the current study is to generate empirical evidence to inform the evaluation of scale attenuation effects.
METHODS: A simulation study was designed to determine the consequences of varying degrees of scale attenuation effects. A continuous-normal θ distribution (population mean=0, standard deviation=1) was simulated for N=500, representing severity of an underlying condition. Ordinal instruments were constructed with average item response category thresholds that either did or did not align with the population distribution of θ. To simulate attenuation, mis-aligned items were created by subtracting a constant from the aligned item response thresholds. Experimental conditions varied the degree of scale attenuation, based on the subtracted constant’s strength or number of mis-aligned items, and the number of response categories in each item.
RESULTS: The bias of the resulting instrument total scores was examined for each condition. The observed sample percentage endorsing the highest response category was compared to when bias exceeded a threshold set at half of the distribution’s standard deviation. Results indicate that the number of response categories and number of items observed to have skewed response distributions were important in addition to the overall proportion of the sample endorsing the highest response category.
CONCLUSIONS: Definitions for problematic floor and ceiling effects must account for key characteristics of the instrument.
METHODS: A simulation study was designed to determine the consequences of varying degrees of scale attenuation effects. A continuous-normal θ distribution (population mean=0, standard deviation=1) was simulated for N=500, representing severity of an underlying condition. Ordinal instruments were constructed with average item response category thresholds that either did or did not align with the population distribution of θ. To simulate attenuation, mis-aligned items were created by subtracting a constant from the aligned item response thresholds. Experimental conditions varied the degree of scale attenuation, based on the subtracted constant’s strength or number of mis-aligned items, and the number of response categories in each item.
RESULTS: The bias of the resulting instrument total scores was examined for each condition. The observed sample percentage endorsing the highest response category was compared to when bias exceeded a threshold set at half of the distribution’s standard deviation. Results indicate that the number of response categories and number of items observed to have skewed response distributions were important in addition to the overall proportion of the sample endorsing the highest response category.
CONCLUSIONS: Definitions for problematic floor and ceiling effects must account for key characteristics of the instrument.
Conference/Value in Health Info
2025-05, ISPOR 2025, Montréal, Quebec, CA
Value in Health, Volume 28, Issue S1
Code
MSR23
Topic
Methodological & Statistical Research
Topic Subcategory
PRO & Related Methods
Disease
No Additional Disease & Conditions/Specialized Treatment Areas