OBJECTIVES

RWE studies addressing the same research question using the same data and analysis methods may arrive at different results and conclusions. This variability in practice suggests bias, unmeasured confounders, and undesirable heterogeneity that may cause an inability to validate and replicate results. There is a growing body of work about the need for the quality and relevance of real world data as well as the transparency and replicability of RWE. We explore why variability may occur in a fundamental building block of RWE, the “algorithm.” An algorithm is a set of rules to be followed in calculations or problem-solving operations. RWE algorithms are comprised of real-world data, coding schemes, metadata, and logic. We address four types of algorithms: cohort and subgroup identification, exposures, other covariates, and outcomes measures. While algorithms may be transparent, the criteria used to evaluate whether they are “fit-for-purpose,” including their “operating characteristics,” are often not.

METHODS

We looked for standards and recommendations from regulatory bodies (i.e., FDA, EMA, ICH), other governmental agencies (i.e., AHRQ, PCORI), external agencies (i.e., Duke Margolis), payer organizations (i.e., NCQA, NQF), and professional societies (i.e., ISPOR, ISPE). We abstracted the criteria proposed by which algorithms should be evaluated as well as statistics and operating characteristics to be reported.

RESULTS

Our research yielded the following recommended criteria: Clinical importance, Data Quality, Feasibility, Interpretation of scores (Outcomes), Relevance, Reliability or reproducibility, Responsiveness, Safety (i.e., no harm to subjects), Transparency & Replicability, and Validity. Specific statistic recommendations are: Sensitivity, Specificity and Positive & Negative Value, Sample size available as well as other appropriate measures of and validity and reliability. The presentation will define and map these criteria to their source with the evaluation metrics.

CONCLUSIONS

Applying these criteria for algorithms allow RWE to be judged both on its transparency and quality.