OBJECTIVES: This conceptual paper introduces enhanced metrics to evaluate how well RWE derived from Target Trial Emulations (TTEs) agrees with RCTs for benchmarking purposes, aiming to improve the assessment of RWE reliability for HTA. While existing metrics focus on assessing the agreement of relative effects (e.g., risk ratio), the validity of absolute effects (e.g., survival) is equally important in HTA. We propose an enhanced metrics to address this.

METHODS: Within the Sheffield RECReATE project (Researching the use of England's Cancer Registry data for Assessing Treatment Effectiveness), we designed a series of TTE benchmarking case studies to evaluate the feasibility of deriving reliable clinical evidence using real-world data. Extended RCT-RWE agreement metrics were developed to assess TTEs that replicate benchmark RCTs. These include three criteria from existing metrics examining the alignment of relative effect estimates: (1) Regulatory Agreement, evaluating if RWE replicates RCT estimates in direction and statistical significance; (2) Estimate Agreement, checking if RWE point estimates fall within the 95% confidence intervals (CIs) of RCTs; and (3) Standardised Differences, with an absolute Z-value < 1.96 indicating no significant difference between RCT and RWE estimates. Our extended metrics enhance these by applying criterion (2) for non-relative effect estimates for each treatment arm separately. Additionally, we propose an exploratory criterion: (4) Survival Curve Comparison, checking if RWE survival curves fall within RCTs’ 95% CIs.

RESULTS: Our findings revealed that omitting the assessment of absolute outcomes can be problematic in HTA. When systematic bias exists across treatment arms (e.g., immortal time bias), solely examining the RCT-RWE agreement of relative effects can mask incorrect estimates of absolute effect differences in RWE, leading to biased decisions if used in cost-effectiveness models.

CONCLUSIONS: We propose incorporating extended metrics to assess RCT-RWE agreement in TTE benchmarking studies. This approach provides a more nuanced examination of RWE’s reliability and applicability in HTA.