OBJECTIVES: Despite their complexity appropriate use of asthma inhaled medicines is crucial to ensure optimal drug delivery to the airways. We describe the validation of an instrument to assess inhalation skills in children. METHODS: The instrument includes a breakdown of the steps necessary for appropriate inhalation. We videotaped 25 children taking a placebo inhaler (metered dose inhaler (MDI), MDI with AeroChamber® (MDI-AE®), and Diskus®). A gold standard (GS) was developed by agreement of two asthma experts watching the videotaped demonstrations. Twenty-one raters scored the randomly ordered demonstrations twice within a 2-week interval (sessions 1 and 2). Intra-class correlation coefficients (ICCs) were calculated to assess validity (comparing GS to raters' scores), interrater reliability, and test-retest reliability for each step of the inhalation. RESULTS: ICCs varied considerably by both, the device and the step. In session 1, a small proportion of raters agreed with the GS on whether patients actuated the MDI and inhaled simultaneously (9.5%, ICCs 0.62 to 0.74) and whether patients hold their breath (19%, ICCs 0.62 to 1.00). A better agreement was observed for the MDI-AE® where actuation (43%, ICCs 0.43 to 0.56) and inhalation (57%, ICCs 0.43) are two separate steps. The best interrater agreement was on the shaking of the MDI (ICC=0.83) and the MDI-AE® (ICC=0.74). Agreement for the Diskus® was poor for all steps. Results for session 2 were similar. The best intra-rater agreement was for the Diskus® (ICCs=1 for 5 steps), though only a small proportion of raters agreed on these steps (5% to 21%). CONCLUSIONS: There was large variability within and between raters' scores. Some steps were better assessed than others. These results suggest that in addition to a detailed instrument, training of raters is crucial to obtain a valid assessment of the children's inhalation technique.