OBJECTIVES: Reimbursement decisions require evidence of effectiveness and a randomised controlled trial (RCT) is seen as the best study design to demonstrate effectiveness. However, there may be situations where a (double-blind) RCT will not be considered necessary, appropriate, methodologically feasible, or ethical. The aim of this study was to develop a decision algorithm to determine the appropriate level of evidence when assessing the effectiveness of a medical intervention. METHODS: The initial algorithm was based on the literature and interviews with personnel at the Health Care Insurance Board (CVZ), the central reimbursement authority in the Netherlands. In addition to the results of a previous study of 72 reimbursement dossiers concerning medical specialist care, we also retrospectively studied 20 reimbursement dossiers made by CVZ to identify any arguments why lower level evidence could be accepted. We then interviewed several Dutch and foreign experts. Our algorithm was continuously refined during the study and prospectively validated using new reimbursement dossiers. RESULTS: RCT evidence was lacking in most positive reimbursement decisions (8/9), but also in most negative reimbursement decisions. Methodological issues can play a role in accepting lower levels of evidence, e.g. when blinding is impossible. Moreover, an RCT may be unsuitable (e.g. due to time constraints) or viewed as unnecessary (e.g. in testing parachutes). Finally, ethical reasons can play a role in accepting lower level evidence. Our decision algorithm contains a stepwise approach to determine the appropriate evidence level, which includes (double-blind) RCTs, observational comparative effectiveness research or non-comparative effectiveness research. CONCLUSIONS: Policy regarding acceptance of lower level evidence in reimbursement decisions needs to be transparent. Our decision algorithm can guide decision makers in reaching a structured and well-founded decision as to whether lower level evidence of effectiveness is appropriate.