OPRR s suspension of Duke s IRB and the HHS report, Institutional Review Boards: A Time for Reform makes clear that effectiveness of IRBs is at issue. Overburdened, without formal training or established monitoring mechanisms, IRBs are vulnerable to compromise as protocol submissions skyrocket. Development methodologies for measures of such a complex institution are needed. OBJECTIVE: To establish a methodology for constructing and evaluating measures assessing IRBs. METHODS: Evaluation of institutional entities such as work by NCQA is reviewed. IRBs are complex entities comprised of levels of individuals acting on the IRB s behalf. Uses of Hierarchical Linear Models to understand interrelationships among levels assists measure development. Generalizability theory, a complex cousin of reliability, provides estimation of the required number of IRB members, protocol-submitting doctors and other participants within each institution as well as the number of items necessary to obtain reliable measures. Multi-trait, multi-method analyses are used to assess the convergent and divergent properties of measures particularly of value in the absence of a gold standard . RESULTS: Simulated data show generalizability of measures are increased with an increased number of participants assessed and/or number of items used and varies based on the homogeneity of subjects within institutions. Multi-trait multi-method analyses indicate that measures with the highest reliability that are more highly correlated with alternative measures of the same domain and less correlated with measures of different domains are the best candidates for measures in the absence of a gold standard CONCLUSIONS: During the politicized process of determining the substance of IRB measures that are trying to meet the needs of diverse stakeholders including patients, medical institutions and pharmaceutical companies, methodological research can assess measures focusing on statistical characteristics. This approach companions the substantive process, providing a basis for divergent interests to make progress in establishing measures of IRBs.