OBJECTIVES:

Real-world evidence (RWE) has become a popular data source for health technology assessment (HTA), and has been frequently incorporated into HTA decision-making processes. Given the concerns about RWE quality, HTA decision-makers have called for comprehensive RWE guidance. Applying RWE appraisal tools which are commonly used in epidemiological studies may provide this guidance. However, information is lacking on how these appraisal tools can satisfy the needs of HTA decision-makers.

METHODS:

A targeted search of RWE appraisal tools published since 2002 was conducted in two approaches. First, we performed a snowball search, defined by Wohlin2016, to identify the reviews from which we could extract RWE appraisal tools. Second, we searched Medline systematically for RWE appraisal tools not covered by the previous reviews (i.e. the tools published since 2019). After identifying the eligible tools, we conducted a content analysis to categorize the tools based on needs of HTA decision-makers on RWE guidance, which were summarized by Jaksa2021.

RESULTS:

From the 1212 obtained hits in this review, 44 critical appraisal tools were identified. Among these tools, 10 (23%) were applicable to all types of RWE, while the others were applicable only to certain types (e.g. cohort studies). Many of these tools addressed issues that HTA decision-makers are concerned about, such as confounding (n=29), quality of primary data sources (n=9), and transparency (n=22). However, the criteria defined by each tool to address these concerns differed significantly in numbers, content, and ways of interpretation.

CONCLUSIONS:

RWE appraisal tools have great potential to be applied as guidance for HTA decision-making, but their applicability is limited by the large heterogeneity in how they address the concerns of HTA decision-makers. To improve the applicability for HTA decision-making, HTA decision-makers may need to be involved in the development and implementation process of appraisal tools.