OBJECTIVES:  To examine impact of rater severity/leniency on measures of neurobehavioral functioning. Observed performance, where a clinician observes and then rates patient’s performance, is required when assessing patients with severe traumatic brain injury (TBI). When some raters are more severe or lenient in how they assign scores measurement treatment effectiveness will be under- or over- estimated. METHODS: Prospective, observational, cohort study. 57 clinicians administered Disorders of Consciouness Scale (DOCS) to 174 patients with severe TBI who were vegetative or minimally conscious at time of study enrollment and within 180 days of injury. To complete the DOCS, clinicians present 25 sensory stimuli to patients and rate the elicited response on a 3-point rating scale. Data were analyzed using the multi-faceted Rasch model (MFRM). A facet is any factor that contributes to making a patient appear to have more or less of a trait than they actually do. MFRM is an extension of the Rasch model that enables patient measures to be adjusted for rater severity/leniency. RESULTS: Overall, mean DOCS measure was 50.8 units unadjusted and was 51.7 units after adjusting for rater severity; t=-2.25, P=.03. However, 35% of individual DOCS measures differed more than the established minimally detectable change of 5 units; some patient measures differed by as much as 37 units; a minimally clinically important difference (MCID) for the DOCS is 7 units. 28 patients were rated too severely (IQR 5.6 - 9.9 units too severely) and 13 were patients rated too leniently (IQR 7.0 - 14.0 units too leniently). CONCLUSIONS: Accounting for rater severity/leniency is important for all endpoints in which a rater observes, and then judges, patient behavior/response. Raters can introduce unwanted variation that threatens the interpretation of clinical trial endpoints. By adjusting for rater severity/leniency accurate change in endpoints across time can be established, despite data being collected by different raters.