OBJECTIVES: Real-world evidence (RWE) from non-randomized studies is increasingly valuable for comparative effectiveness research for regulatory and reimbursement decision-making. This study aimed to systematically review published RWE guidance in assessing health technologies and to identify potential gaps.

METHODS: Systematic searches were conducted in EMBASE, websites of regulatory and reimbursement decision-making bodies in Europe and the United States, and key organizations or initiatives to identify guidance: published in English (between January 2015 and June 2020), reporting on the use of RWE, supporting regulatory and/or reimbursement submissions for health technologies. Screening and data extraction followed pre-specified protocol. Condition-specific guidance, post-market surveillance, commentaries, and conference abstracts were excluded.

RESULTS: Of the 245 hits, twelve references (reporting on nine guidance documents) were deemed relevant. Fourteen additional guidance documents were identified through website searches.

Several general frameworks developed by the Food and Drug Administration’s (FDA) RWE Program, the Institute for Clinical and Economic Review (ICER), and academic departments for health policy were identified. There is a paucity of guidance on integration of RWE studies in cross-study synthesis with randomized studies, and methodological guidance on selection and analysis of such evidence is limited. Most guidance focused on RWE reliability and relevance of the data (including the ISPOR/ISPE Taskforce), whereas only two documents (ICER, RWE Decoder) presented a structured, multi-step process of determining RWE use based on the evidentiary needs of a decision-making problem and tailoring the most appropriate analytical methodologies to ensure trust and transparency in results. Other guidance focused on study design and registration, data quality including selection bias, data linkage, and data replicability.

CONCLUSIONS: Guidance on RWE use is currently focused on specific issues. Future guidance should target areas that is crucial for healthcare decision-makers, such as combining RWE with randomized studies, statistical methods for estimating comparative effects across both designs, and exploring uncertainty.