Real-world evidence (RWE) plays an increasingly important role in health technology assessment (HTA), as well as regulatory and clinical decision making. RWE studies, however, are subject to multiple sources of bias, which are often not easy to identify, impeding credibility and inclusion of RWE in decision making. A comprehensive, fit-for-purpose, and easy-to-use bias assessment tool would help streamline RWE evaluation, enabling efficient utilization of RWE for decision making globally.
A working group of the International Society for Pharmacoepidemiology collaborated with HTA experts to develop a tool that could guide bias assessment in observational studies on the comparative safety and effectiveness of medications, building upon existing methodological tools, best practice guidelines, and checklists for the analysis of real-world data. The tool was further tested and refined in collaboration with HTA agencies.
APPRAISE ( praisal of otential for Bias in e l-World Ev dence tudi s) covers key domains through which bias might be introduced into an RWE study: inappropriate study design and analysis, exposure and outcome misclassification, and confounding. Each domain contains a series of questions. Responses to questions autopopulate a summary of the potential for bias within each domain and of the actions to take to avoid, mitigate, or explore the impact of bias.
APPRAISE is a tool to guide bias assessment in observational studies on medication comparative effectiveness or safety. Although the tool was designed for HTA, it will be useful for many other users of RWE and will help guide optimized RWE generation.
What is it about? Real-world data (RWD) collected in the process of healthcare is increasingly important for health technology assessments (HTA) and decision making in healthcare, regulatory, and policy contexts. However, nonrandomized real-world evidence (RWE) studies often face multiple biases that can affect their credibility and usefulness. This study addresses the challenge of identifying bias in RWE studies via APPRAISE, a tool designed to streamline the evaluation of bias in RWE comparative studies of treatment effects. This study contributes significantly to understanding how biases can be systematically identified and addressed, enhancing the reliability and interpretation of RWE studies.
How was the research conducted? The study's goal was to develop a tool to assess bias in observational studies that use RWD to evaluate medication effectiveness or safety. This was done by experts from the International Society for Pharmacoepidemiology who collaborated with HTA agencies to ensure the usefulness and usability of the tool. The tool built upon existing tools, methodological guidelines, and checklists for analyzing RWD. Researchers conducted a literature review, consulted with experts, and tested the tool with HTA agencies to refine it. This approach was chosen to ensure the tool's relevance and applicability in assessing bias in diverse study designs.
What were the results? The APPRAISE tool effectively identifies potential biases across key domains, including study design, exposure and outcome misclassification, and confounding. Responses to questions populate a summary of potential biases and suggest actions to mitigate them. The tool also provides examples and references for further information on sources of bias and ways to avoid them. The tool's comprehensive nature and user-friendly design make it valuable for various users of RWE, beyond HTA agencies, including learners in academic clinical medicine, public health, and industry, and regulatory professionals new to the design and conduct of RWD-based studies of treatment effects.
Why are the results important? This is a new and valuable tool for HTA agencies and other decision makers, as it provides a structured way to evaluate potential for bias in RWE studies. The findings can lead to concrete changes in research practices by improving the reliability of evidence used in decision making. Patients, healthcare providers, and policy makers benefit from more accurate assessments of treatment effects, leading to better healthcare outcomes. In the long-term, this tool could also improve the quality and credibility of evidence produced by RWE studies, influencing future research, clinical practice, and policy development.
What are the strengths and weaknesses of this study? The main strength of this study is the development of a comprehensive, user-friendly tool that systematically assesses the potential for bias in RWE studies. The tool was developed in consultation with HTA experts and is endorsed by the International Society for Pharmacoepidemiology. A limitation is that the tool does not quantify bias, which can vary across studies. Future research could explore quantifying bias levels and expanding the tool's applicability to other studies of other interventions, such as vaccines or nonpharmacological treatments.
Note: This content was created with assistance from artificial intelligence (AI) and has been reviewed and edited by ISPOR staff. For more information or for inquiries on ISPOR’s AI policy, click here or contact us at info@ispor.org.