Much Ado About Little: Dealing With Limited RCT Evidence for Early HTA and Reimbursement Decisions
Christian Lamping, MSc and Felix Wolfrum, MSc, TH Köln - University of Applied Sciences, Cologne, Germany
The Challenge of Accelerated Approvals
The third plenary session was moderated by Mark Sculpher, PhD, University of York, York, United Kingdom. The panelists discussed how to tackle the challenges due to an increasing number of accelerated approvals (Figure 1). As the increase in speed often comes with a decrease in rigor (eg, little or no randomized controlled trials, more surrogates, etc), Sculpher framed the discussion by posing this question to the panelists, “How much additional health do these products provide?”
Causal Methods
Uwe Siebert, MD, MPH, MSc, ScD, University for Health Sciences, Medical Informatics and Technology, and Harvard Chan School of Public Health, Tirol, Austria presented one approach to address this question by explaining when to use causal methods (no randomization or randomized treatment assignment is violated). Siebert further presented methods (target-trial and g-methods) to be used in this case. His clear message regarding the topic was, “Never ever compare ‘ever’ versus ‘never’!”
"Never ever compare 'ever' versus 'never'!”—Uwe Siebert, MD, MPH, MSc, ScD
Surrogate Endpoints: Uncertainty and Validation
Sylwia Bujkiewicz, PhD, MSc, University of Leicester, Leicester, United Kingdom took up the subject of rigor and presented methods to increase it. She focused on surrogate endpoints that are important in health technology assessment (HTA) in situations of limited evidence on final clinical outcome. She emphasized that the use of surrogate endpoints may bring further uncertainty requiring appropriate validation of surrogate endpoints. Bujkiewicz proposed bivariate meta-analysis for validation and the use of surrogate relationships (hierarchical meta-analytic method) to reduce uncertainty and increase precision.
Opportunities and Challenges
Stephen Palmer, PhD, University of York, York, United Kingdom discussed modeling approaches for the evaluation of histology independent treatments. He sees opportunities as well as obstacles in the innovative study designs such as basket trials (efficiency advantages vs challenges for HTA). He emphasized the necessity of the extension of Bayesian hierarchical modeling to other endpoints and proposed to move beyond the ICER, focusing instead on the impact on the population’s health, for example.
“When we look at response as a predictor of progression-free survival, the fact that progression-free survival can also be a surrogate does not enter this type of analysis in any way."—Sylwia Bujkiewicz, PhD, MS
Preventing Deception
Eva Susanne Dietrich, University of Bonn Pharmaceutical Institute, Bonn, Germany raised skepticism about some of the new methods. In her experience, Dietrich feels that data need to be gathered tediously during approval while only a small amount of follow-up data is provided after approval. Her aim is that decision makers will be able to detect deception potentially caused by the methods currently used. Finally, she called for transparent, understandable, and predictable reimbursement procedures.
Addressing Questions From Attendees
The Q&A discussion addressed how uncertainty in postlaunch research should be used in decision-making processes. Palmer responded that there is currently an overemphasis on efficiency and errors but not thinking about the meaning itself. The uncertainty could potentially be smaller and one should scale in on potential benefits. The use of a pricing scheme to address these challenges of uncertainty was also proposed. Siebert highlighted again the fundamental risk aversion and pointed out that P values are not always optimum.
“Data need to be gathered tediously during approval while only a small amount of follow-up data is provided after approval.”—Eva Susanne Dietrich
Another question raised by the audience referred to unknown confounders in causal inference and whether they are a big danger, especially in new treatments. Siebert responded, “Yes and no. As in all observational studies, the validity of the results depends on capturing all relevant confounders. Therefore, randomization remains the best thing to do (if you can). But if you have a randomized controlled trial with postrandomization confounding, there are specific g-methods for the adjustment that use the randomization. Therefore, the assumption of no unmeasured confounders is less relevant (eg, the g-method "structural nested models with g-estimation)." Many of these have successfully been used, he continued, leading to lower ICERs and positive reimbursement in trials with treatment switching.
The question of whether the CRC analysis of response as a surrogate endpoint for progression-free survival would cause additional challenges related to the fact that progression-free survival is itself a surrogate endpoint was answered by Bujkiewicz.
“Not really, she said. “When we look at response as a predictor of progression-free survival, the fact that progression-free survival can also be a surrogate does not enter this type of analysis in any way. We just focus on these two outcomes.”
At the end of the fruitful discussion in which many different proposals were presented, it seemed clear that these methodologies need further assessment and validation, including unbiased comparisons with already existing and established methods.
