Objective: Psychometric analysis is often used in health outcomes research to evaluate the validity of survey instruments. The kernel regression approach is a nonparametric alternative to parametric item response theory (IRT) models, which assume that item responses follow a more restrictive, parametric distribution. The purpose of this study was to demonstrate the effectiveness of kernel regression for psychometric analysis of data arising from a small sample pilot study. Methods: The kernel approach to IRT was applied to data obtained from 24 pharmacists who completed a 99-item questionnaire on herbal and dietary supplements. The questionnaire was designed to measure five different traits, two of which included knowledge and performance of patient counseling. Using visual plots, each item was analyzed by its estimated category response function (CRF), the probability that an item response category is endorsed as a function of ability (trait) level, i.e., total score. For categories that indicate an item score (a "correct" option in a multiple-choice [MC] item or an ordinal category in a rating scale item), CRF should increase with the level of ability. For illustrative purposes, we report the performances of two of 56-MC items that assessed pharmacists' knowledge (knowledge1, knowledge2), and two of 11, 5-point rating scale items that examined pharmacists' provision of patient counseling (counseling1, counseling2). Results: The knowledge1 and counseling2 items fulfilled the psychometric criteria. Knowledge2 was removed from the survey because the CRF of the correct response option did not increase with level of ability, i.e., pharmacist knowledge. The counseling1 item was removed because CRFs revealed that the ordinal categories did not increase with pharmacists' patient counseling ability. Conclusion: The kernel regression approach to IRT provides a flexible and informative approach to refine a survey instrument and evaluate its properties using pilot data.