Plain Language Summary

What is it about? Real-world data (RWD) collected in the process of healthcare is increasingly important for health technology assessments (HTA) and decision making in healthcare, regulatory, and policy contexts. However, nonrandomized real-world evidence (RWE) studies often face multiple biases that can affect their credibility and usefulness. This study addresses the challenge of identifying bias in RWE studies via APPRAISE, a tool designed to streamline the evaluation of bias in RWE comparative studies of treatment effects. This study contributes significantly to understanding how biases can be systematically identified and addressed, enhancing the reliability and interpretation of RWE studies.

How was the research conducted? The study's goal was to develop a tool to assess bias in observational studies that use RWD to evaluate medication effectiveness or safety. This was done by experts from the International Society for Pharmacoepidemiology who collaborated with HTA agencies to ensure the usefulness and usability of the tool. The tool built upon existing tools, methodological guidelines, and checklists for analyzing RWD. Researchers conducted a literature review, consulted with experts, and tested the tool with HTA agencies to refine it. This approach was chosen to ensure the tool's relevance and applicability in assessing bias in diverse study designs.

What were the results? The APPRAISE tool effectively identifies potential biases across key domains, including study design, exposure and outcome misclassification, and confounding. Responses to questions populate a summary of potential biases and suggest actions to mitigate them. The tool also provides examples and references for further information on sources of bias and ways to avoid them. The tool's comprehensive nature and user-friendly design make it valuable for various users of RWE, beyond HTA agencies, including learners in academic clinical medicine, public health, and industry, and regulatory professionals new to the design and conduct of RWD-based studies of treatment effects.

Why are the results important? This is a new and valuable tool for HTA agencies and other decision makers, as it provides a structured way to evaluate potential for bias in RWE studies. The findings can lead to concrete changes in research practices by improving the reliability of evidence used in decision making. Patients, healthcare providers, and policy makers benefit from more accurate assessments of treatment effects, leading to better healthcare outcomes. In the long-term, this tool could also improve the quality and credibility of evidence produced by RWE studies, influencing future research, clinical practice, and policy development.

What are the strengths and weaknesses of this study? The main strength of this study is the development of a comprehensive, user-friendly tool that systematically assesses the potential for bias in RWE studies. The tool was developed in consultation with HTA experts and is endorsed by the International Society for Pharmacoepidemiology. A limitation is that the tool does not quantify bias, which can vary across studies. Future research could explore quantifying bias levels and expanding the tool's applicability to other studies of other interventions, such as vaccines or nonpharmacological treatments.

^{Note: This content was created with assistance from artificial intelligence (AI) and has been reviewed and edited by ISPOR staff. For more information or for inquiries on ISPOR’s AI policy, click here or contact us at info@ispor.org.}