OBJECTIVES: To assess the criterion validity of HUI Mark 2 (HUI2) and Mark 3 (HUI3) utility scores from a new, centralized on-line questionnaire administration system. METHODS: The system presents HUI questionnaires to patients and provides results to clinicians by email. Questionnaire results include responses and 32 derived variables (14 attribute levels; two overall health state vectors; 14 single-attribute utility scores; and two multi-attribute utility scores of health-related quality of life (HRQL)). SPSS code, validated to Health Utilities Inc. decision tables for determining attribute levels and published utility functions, is the criterion method for determining HUI derived variables. Testing used a data set that included questionnaire response combinations for all HUI2 and HUI3 attribute levels. Criterion validity was evaluated using percent exact agreement, and single-measure intra-class correlation coefficients (ICC), between scores from the new system and scores from the criterion method. RESULTS: The test data set generated 240 utility scores. There was exact agreement for 99.2% (n=238) of the scores. Disagreement was limited to HUI2 sensation (ICC=0.805, p<0.01), and HUI2 overall HRQL (ICC=0.966, p<0.01), scores in one test case. Results were received by email from the on-line system within approximately one minute of completing each questionnaire. There were no missing or incomplete questionnaire data from the on-line system. CONCLUSION: The results indicate that most of the utility scores from the new on-line questionnaire system have criterion validity and there is a problem with the coding algorithm for at least one set of questionnaire response combinations associated with HUI2 sensation. The coding problem should be corrected, and more rigorous testing should be completed, before public release of the system. The system should be considered an alternative to traditional methods for future HUI data collection, especially for applications requiring immediate results such as clinical settings.