Real-World Evidence in Canadian Health Technology Assessment: Is it Worth It?

Cal Shephard, MSc; Eon Ting, MSc, AstraZeneca Canada, Mississauga, ON, Canada

RWE in HTA: What’s the Issue?

In Canada, to have new medicines reimbursed by the public healthcare system, drug manufacturers are required to submit evidence packages to health technology assessment (HTA) agencies, such as Canada’s Drug Agency (CDA-AMC) and Institut national d’excellence en santé et en services sociaux (INESSS).

In addition to randomized controlled trials (RCTs), manufacturers may include real-world evidence (RWE) as part of their submission package to address evidence gaps. RCTs, while the gold standard of clinical evidence, do not always have all the answers needed for HTA agencies to make funding recommendations. For instance, multinational RCTs might not reflect Canadian practice. Furthermore, RCTs often follow strict protocols designed for regulatory purposes and not fully reflect real-world situations such as discrepancies in access to care, demographics, and socioeconomics. Well-conducted RWE can illuminate evidence gaps that RCTs miss.

There are many types of RWE, ranging from provincial and hospital health record data to privately held patient support program data. It is challenging to know specifically how to balance what is considered “best,” practically feasible, and most useful for HTA given the circumstances.

Why Include RWE in HTA Submissions?

In many cases, drug manufacturers conduct RWE to answer questions RCTs would not be able to answer such as:

• How many patients are currently using Drug X?

• How long are patients adhering to Drug Y?

• How do patients on Drug Z perform in the long-term?

• How many times do patients end up at the emergency room?

• What costs does the healthcare system incur?

In some cases, RWE can also be used to construct a “synthetic” control arm to help demonstrate comparative effectiveness when the clinical trial only has a single intervention arm.

So why not always perform RWE studies? A single RWE study potentially takes several months and hundreds of thousands of dollars to complete, a significant investment of time, money, and human resources. Before initiating a study, we must ask: Is it worth it?

There are many types of real-world evidence, ranging from provincial and hospital health record data to privately held patient support program data. It is challenging to know which type is “best,” most feasible, and most useful for health technology assessment.

How Did We Investigate This Issue?

With the help of researchers from the University of Toronto and University of Calgary, we commissioned a comprehensive environmental scan to better understand the RWE included in HTA submissions to CDA-AMC. Reviewers assessed HTA review reports published between January 2020 and June 2024 and thematically summarized the findings; the results were presented at the ISPOR 2025 conference in Montreal.

We found that between January 2020 and June 2024, drug manufacturers submitted a total of 274 HTA submissions to CDA-AMC, with 70 (25.5%) including RWE and 27 (9.9%) including more than one source of RWE. Table 1 summarizes the most common purposes of RWE beyond describing treatment patterns and patient characteristics:

1. Inform comparative clinical effectiveness (54%),
2. Address evidence gaps from pivotal trials or systematic reviews (52.2%), and
3. Support pharmacoeconomic models (54.9%).

Retrospective cohort design (57.5%) represented most studies, especially for oncology indications. Prospective cohort studies (20.4%) were more common for nononcology indications (30.4% vs. 4.5% for oncology indications). RWE were predominantly from the United States (39%) or Europe (38%); studies from Canada (11%) were in the minority (Table 1). The majority of submissions (82.9%) received a “Reimburse with Conditions” recommendation.

Reviewer criticism of RWE is presented narratively in their review reports in an unstructured format. A summary of the criticisms of RWE is presented in Table 2; 70% of studies had data quality issues, such as collection protocols, data provenance, and analytical validity.

Methodological concerns about bias and residual confounding were noted in 50% of studies, stemming from the reliance on retrospective cohort designs (57.5%) which lack a priori protocol-driven data collection. Short follow-up duration (52.9%) and sample size limitations (48.6%) further constrained the ability of RWE to address long-term effectiveness questions. Generalizability concerns (61%) encompassed patient populations, clinical practice patterns, and treatment sequencing.

In summary, these findings demonstrate that RWE might support HTA submissions by addressing certain clinical, comparative, and economic evidence gaps. However, there were many criticisms, and it is not clear whether those RWE studies even played a role in the expert committee’s deliberations.

Because RWE are considered supplementary in a HTA submission, it is difficult to assess the counterfactual: What would the outcome of an HTA have been if RWE had not been submitted?

Table 1. Summary of Real-World Evidence Studies Included in Health Technology Assessment Submissions

Table 2. Summary of Health Technology Assessment Agency Criticisms of Real-World Evidence Studies

RWE as Payer Evidence: the 3R’s

At AstraZeneca Canada, our approach is to support HTA agencies in their assessments so that medicines are made available to Canadian patients. In many cases we have generated and submitted RWE; however, it was not clear whether these RWE studies made a difference in the HTA. In fact, our findings suggest that RWE studies across the board had varying degrees of quality, and as such, were often heavily criticized and even diminished in the review.

We propose a set of complementary principles to help build real-world payer evidence that is real, robust, and relevant to the decision problem faced by health technology assessment agencies.

Despite this, we are not discouraged by these results but suggest a shift in how RWE should be generated and used. We encourage the development of meaningful real-world payer evidence and believe that it is a shared responsibility of the evidence generator and reviewing agencies to distinguish what “good quality” payer evidence is so it can be used in a review.

In May 2023, CDA-AMC issued guidance on the reporting of RWE, but there is still an opportunity to better define what “good quality” RWE is and how it could be used in HTA. Therefore, we propose a set of complementary foundational principles—the 3R’s quality framework—to help build real-world payer evidence that is real, robust, and relevant to the decision problem faced by HTA agencies:

• Real: The findings from the RWE are reflective of and generalizable to the payer’s decision setting.

• Robust: The methods for generating the RWE minimize biases and are transparently reported. Sample sizes and duration of study are reasonable and sufficient.

• Relevant: The study parameters of the RWE are applicable and purposefully designed to support payer decision making.

It is important that evidence generation leads to “good quality” RWE so that it then can be appropriately used by HTA agencies. But without explicit processes for how that RWE will be used in HTA, manufacturers (as we uncovered in our research) may submit RWE that is not considered real enough, robust enough, or relevant enough, resulting in these studies being perceived as unhelpful for the review. This further undermines HTA reviewers’ trust in RWE in a submission, increasing the risk that it could be categorically disregarded without the opportunity to show its true potential. While there is hesitancy to overprescribe how evidence should be generated, foundational principles, such as the 3R’s above, may help facilitate the inclusion of better quality, real-world payer evidence.

Refocusing for the Future

With the world of clinical trials becoming more complex, RWE will undoubtedly play a role in answering pertinent clinical questions. When considering what is needed for HTA, RWE needs to be reclassified as real-world “payer evidence” so that evidence generators, such as drug manufacturers, can better design the “what”, “how”, and “why” of studies. Evidence generated for the purpose of supporting reviews can be very helpful; however, our research findings suggest that RWE currently does not have a clear role in HTA and may be less readily accepted for review in the future.

Real-world evidence that is intentionally designed as real-world payer evidence could be very useful for health technology assessment purposes.

Aside from our research findings, it is also important to consider the logistics of developing RWE in Canada. Canada’s health databases are fragmented across different provinces and disparate agencies. An integrated pan-Canadian database could support future studies with larger, more reflective study sizes. The time required for large-scale provincial database studies hampers the creation of RWE locally, so in some cases, nonlocal RWE may be adapted through transportability analysis for use in local HTA submissions.

It is up to drug manufacturers to decide whether the development of RWE is worth the time and cost. But without clear agency processes for differentiating what is good (and useful) from what is not (and of poor quality), generating future evidence for HTA may prove to be a fruitless endeavor.

RWE that is intentionally designed as real-world payer evidence could be very useful for HTA purposes. While our research findings did not find any definitive association between the inclusion of RWE and favorable funding recommendations, we still believe that good quality payer evidence plays an important role in HTA decision making and Canadian health research. With improvements such as clearer use-case guidance, definitions of quality, and a coordinated data infrastructure, Canadian real-world payer evidence may be worth it in the future.