BACKGROUND: Factor analysis is a widely accepted approach to assess the suitability of an instrument structure and can be used in the content and psychometric validation of clinical outcome assessments (COAs). Qualitative insights into the importance of items and concepts should also be considered when developing a scoring algorithm, outlining how to combine individual items into a score in a meaningful way. OBJECTIVES: This study outlines quantitative methods to define a suitable item-scale factor structure for COAs and qualitative approaches to develop a scoring algorithm to weight items by importance. RESULTS: Factor analysis is used to assess the suitability of a hypothesized conceptual framework for a COA (confirmatory factor analysis) or to identify a suitable item-scale structure in the absence of a pre-defined conceptual framework (exploratory factor analysis). First or second-order confirmatory factor analysis can be used depending on the hierarchy and structure of the concept of interest. Once the item-scale factor structure is finalized a scoring algorithm can be developed. Many COAs are scored by assigning an equal weight to all items and summing or averaging items to form domain and total scores. Qualitative ranking exercises with patients and clinical experts can determine the relative importance of items. Weighting can then provide the percentage that each item should contribute towards an overall score. This can provide insights into which items are of greater relevance to a condition, and highlight concepts that are crucial or less critical to a health condition. Factor analysis and qualitative approaches to develop scoring algorithms should be used in combination to ensure that item-scale structures of COAs are both quantitatively and qualitatively valid. CONCLUSIONS: When developing a COA factor analysis and qualitative weighting or ranking exercises can be used in combination to determine item-scale structures and scoring for COAs.