The Official News & Technical Journal Of The International Society For Pharmacoeconomics And Outcomes Research

Kim Saverno RPhConducting Literature Reviews to Support the Use of Patient-Reported Outcomes (PRO) Measures in Clinical Trials The Benefits of a Systematic Search Strategy

Debbie Smith MSc, PhD, CPsychol, Matt Reaney MSc, and Jane Speight MSc, PhD, CPsychol, AHP Research, Uxbridge, UK

Systematic reviews are regarded as one of the strongest forms of medical evidence (second only to meta-analyses). However, a recent review of 300 systematic reviews found that not all systematic reviews are equally reliable [1]. If designed and conducted appropriately, systematic reviews can prove invaluable to clinicians as an efficient way of keeping abreast of new research, guideline developers and regulatory agencies for consideration in practice, decisions and recommendations, and granting awarding bodies and research ethics boards as justification for new research bids.

Our aim in this brief paper is to explore the benefits of the systematic review for outcomes research and, in particular, to encourage use of a standardized systematic search strategy, which maximizes rigor, flexibility, ease of interpretation and replicability. In addition, consistent use of a standardized strategy will ensure transparency, minimize the risk of error and bias, and allow for direct comparison between reviews.

The Role of Literature Reviews in Patient-Reported Outcomes (PRO) Research
Few pharmaceutical-led clinical programmes begin without a review of the literature but, in recent years, systematic reviews have become increasingly important in outcomes research. The U.S. Food and Drug Administration (FDA) draft guidance on the use of PROs in medical product development [2] offers the opportunity for pharmaceutical companies to make labeling claims based on PRO assessment. However, high-level evidence is required to ensure that PRO data are meaningful and that the best available PRO instruments are used to measure a preferred endpoint. Systematic identification of condition-specific issues and the appraisal of PRO endpoints and existing instruments are therefore essential to:

• determine burden of illness
• identify potentially relevant PRO instruments
• critically review relevant PRO instruments to determine the most appropriate for the purpose
• inform the design or adaptation of PRO instruments

The FDA draft guidance has thus given renewed weight to the role of the systematic literature review as the cornerstone of the clinical programme. The identification (and critical appraisal) of previously developed PRO questionnaires purporting to measure selected endpoints is one of the fundamental questions on which all PRO measurement strategies need to be built to maximize the chance of having data suitable for consideration in a product label. However, the principles of this regulatory guidance (and that provided in 2005 by the CHMP of the EMEA [3]) do not end at the achievement of a label claim, but apply equally to post-marketing studies.

While many aspects of the FDA’s draft guidance are detailed and in some instances somewhat prescriptive, the messages regarding the rigor required of this most fundamental aspect of the PRO measurement strategy are less clear. Whilst the validity of the selected PRO instrument is rightly questioned by the FDA’s draft guidance, the validity of the literature review that informs its selection is not. Systematic reviews are scientific exercises and this means that they require the same rigor as other aspects of research, yet current methods used to conduct so-called “systematic reviews” remain variable, meaning that the quality and comparability of systematic reviews is not assured. Thus, we consider the benefits of adopting and standardizing a systematic approach to reviewing the PRO literature to inform clinical research.

The Purpose and Value of a Systematic Review
A systematic review addresses a clearly defined research question through use of a clear and comprehensive methodology to identify, select and critically appraise relevant research studies, and to synthesize data from these research studies. In using a rigorous methodology, systematic reviews aim to minimize bias associated with study selection, identify all relevant research studies from the sources searched and be transparent in their methods and conclusions.

Developing a Systematic Search Strategy
There are seven stages to a systematic review (see Figure 1).

The focus of this paper is on formulating the systematic search strategy. A good systematic review starts with a well defined and clear research question. The conduct and reporting of systematic reviews may be improved by implementation of a set of standards to guide the search. The ‘syntax search’ strategy has been traditionally employed to conduct systematic reviews of the medical and psychosocial literature to identify, review and select PRO measures for clinical research. The syntax search involves the identification of search terms, and their combination using Boolean operators in a string of syntax. The Cochrane Collaboration is a set of specific groups (e.g. Cochrane Oral Health Group) that have the primary aim of helping to prepare and disseminate systematic reviews of the effects of health care. All Cochrane systematic reviews follow a standardized format according to the Cochrane Reviewers handbook and are maintained on the Cochrane Database of Systematic Reviews. The Cochrane Collaboration promotes use of the “PICO” strategy as a framework for designing the research question to ensure clarity and focus. This involves splitting the search terms into four categories, and combining them in ways relevant to answering the research question. PICO stands for:

P Population / problem of interest
I Intervention
C Comparison
O Outcome

However, this is rarely employed in systematic reviews conducted within the pharmaceutical arena. We explored the medical literature using both the traditional ‘syntax search’ and the Cochrane Collaboration preferred PICO strategy to answer the question “which PROs have been used in islet transplants and other diabetes-related transplants that may be relevant for evaluating future islet programmes?” Searches were conducted in the Scopus database; a search interface which includes journals indexed via Medline (100%), Embase (100%), PubMed (100%), PsycINFO (65%) and CINAHL (65%). Results are displayed in Figures 2 and 3.

To ensure reliability, the output resulting from each strategy was independently evaluated by two researchers (AH and LK) and the methods critiqued. Both the ‘syntax search’ and the Cochrane Collaboration preferred method without outcome terms (a standard methodology to identify the general scope of the literature) returned 10,339 ‘hits’ (abstracts) for review. The research team identified ways in which to combine search terms to present a more manageable number of ‘hits’ for abstract screening.

The Research Teams Views of the Two Search Strategies
Both researchers agreed that the PICO strategy allowed for greater adaptability and targeted reviewing without compromising the quality of the search or results. Combining a priori search terms systematically according to [P and (I or C) and O], resulted in a much more manageable 659 abstracts (see Table 1). Producing similar combinations using the ‘syntax search’ proved more complex for the research team and was prone to error. The research team concluded that the PICO framework offers greater clarity, achieves a clearer search strategy, allows for greater adaptability, and facilitates targeted reviewing without compromising quality. The researchers felt more confident in the search conducted using PICO and stated:

“PICO allows me to save and track previous searches and re-use them. This really saves time. It also ensures that all relevant search terms are included. It makes it easier to present the search strategy to clients so they can assess which search terms can be added or omitted” “The adaptability of PICO is particularly beneficial as once a search has been started, it is very easy to see what the effect is of manipulating one area (such as population) without it confusing other contributing search terms belonging to other categories”

The quality of a systematic review depends on the extent to which scientific methods are used to minimize the risk of error and bias, but also the extent to which the search strategy is flexible (necessary to modify the research questions if too many abstracts are returned) and replicable. The replicability of literature reviews is vital to ensure that they can be updated with minimal efforts (and do not become out-of-date) as new literature is published. We found that the PICO method is comparable to the standard syntax search in terms of returned abstracts, but offers the added benefits of being:

1. Easy to implement
2. Clear to present
3. Sufficiently versatile to allow further targeting according to subtle adaptations to the research question
4. Grounded in good science, as it in line with the Cochrane Collaboration recommendations.

As a result is much easier to examine the validity of the research question and its related search when applying the PICO framework, thus ensuring that the search strategy is appropriate for answering the research question(s). Importantly, despite the lack of specific guidance on the methods required for literature reviews, adopting this systematic search strategy is much more likely to satisfy the stringent criteria for documentation of the systematic identification and selection of PRO instruments for clinical trials (i.e. the PRO evidence dossier) as suggested by the FDA.

The authors wish to thank Leena Khagram and Ana Howarth for their assistance in interpreting and evaluating the output from the syntax and PICO searches.

1. Moher D, Tetzlaff J, Tricco AC, et al. Epidemiology and reporting characteristics of systematic reviews. PLoS Med 2007;4:e78.
2. Food and Drug Administration (FDA). Guidance for industry: patient-reported outcome measures: use in medical product development to support labelling claims: draft guidance. Health Qua Life Outcomes doi:10.1186/1477-7525-4-79.
3. Committee for Medicinal Products for Human Use (CHMP): Reflection Paper on the Regulatory Guidance for the Use of Health-Related Quality of life (HRQL) Measures in the Evaluation of Medicinal Products. Retrieved from:

  Issues Index | 2009 Issues Index