Non-Randomized Studies in Comparative Effectiveness Research; How Systematically Are They Identified By Existing Search Filters? Results from a Systematic Literature Review

OBJECTIVES: Comparative effectiveness research (CER) aims to generate real-world data (RWD) from non-randomized studies (NRS) to inform healthcare decision-making. The unbiased selection of NRS in systematic literature reviews (SLR) is critical to ensure confidence in RWD. However, limited research exists on developing and validating NRS search filters in CER for bibliographic databases. This study aimed to systematically identify existing NRS search filters and discuss their performance in CER.

METHODS: An SLR was conducted to identify NRS filters for CER published in English between 2012 and 2022 from electronic databases (e.g., Embase, MEDLINE, Cochrane Database of Systematic Reviews) and key research and health technology assessment (HTA) organizations. Screening and data extraction (NRS filter description, study types targeted, performance metrics) were performed by two reviewers.

RESULTS: Database searches retrieved 2,524 publications. Two publications summarizing 14 unique NRS search filters were identified. In addition, one publication from the Institute for Quality and Efficiency in Health Care was included that presented a de novo NRS search filter. All NRS search filters were published after 2001 and were developed for use in PubMed, MEDLINE, or Embase. Six NRS search filters were proposed by key HTA organizations. Most NRS filters covered several observational study designs, but only one filter limited to controlled NRS achieved adequate sensitivity (92%), a precondition for comprehensive information retrieval. Embase search strategies showed higher sensitivity and precision than those of MEDLINE. None of the identified filters targeted CER terms.

CONCLUSIONS: Limited validated NRS search filters for CER may be impacted by a lack of standardized definitions and labeling of NRS study types. Future research should focus on developing and validating NRS search filters with reference sets outside the context of filter development to improve consistency in database indexing of publications and allow unbiased selection of these studies in decision-making.