PRO Mixed Methods & Modes Manuscript Outline (October 26, 2010)

USING MIXED MODES AND/OR METHODS TO COLLECT PATIENT-REPORTED OUTCOMES DATA IN CLINICAL TRIALS

1. Introduction
   

The incorporation of the patient perspective in the evaluation of medical products (i.e., drugs, biologicals, devices) has been increasingly viewed as important, if not essential.  Medical products aimed at relieving patients’ symptoms and/or improving levels of self-reported functioning will require patient-reported outcomes (PROs) as endpoints in clinical trials.  As stated in the US Food and Drug Administration (FDA) Guidance for Industry titled Patient-Reported Outcome Measures: Use in Medical Product Development to Support Labeling Claims, “Use of a PRO instrument is advised when measuring a concept best known by the patient or best measured from the patient perspective” (FDA, 2009). 

There is no doubt that the release of this “PRO Guidance” by the FDA has focused increased attention on the scientifically sound measurement of PRO endpoints in clinical trials.  Contemporaneous with the increasing attention on the appropriate use of PRO measures as efficacy endpoints, the use of technology in clinical trials expanded substantially.  There is growing recognition of the many advantages of electronic technologies for PRO measures (i.e., ePROs), including less subject burden, avoidance of secondary data entry errors, easier implementation of skip patterns, date and time stamping, and more accurate and complete data (Stone et al., 2002).  Migrating from paper to electronic data collection is one of the most significant movements in the PRO measurement field.

Alongside the emergence of many technology-based ways of capturing PRO data is the need to assure measurement equivalence across and among these methods and modes of administration (Coons et al., 2009). This becomes especially important if multiple methods or modes are utilized within a single trial.  For the purpose of this paper, we have adopted the distinction made in the PRO Guidance between PRO instrument administration modes and data collection methods.  Administration mode refers to self- vs. interviewer-administered, while method refers to the tool used for capturing the data such as paper-based questionnaires, web-based data entry, interactive voice response systems (IVRS), or any of the other ePRO devices.  However, regardless of whether different modes or methods are being considered, the comparability of the data obtained via the original and alternative data collection approaches must be assured.  If a new or different data collection approach introduces response bias, the appropriateness of pooling data within a trial or comparing data across trials is in question.

The methodological purists among us recommend that PRO data capture modes and/or methods not be varied within a single clinical trial or between trials that seek to provide comparable data.  In general, anything that has the potential to introduce measurement error into a trial should be avoided (Streiner and Norman, 2006).  Measurement error is, in essence, noise (error variance) that reduces statistical power and attenuates the ability of the trial to detect real change (i.e., treatment effect) in the trial endpoint.  Clinical trial designs should avoid as many sources of error variance in the data as possible.  In the context of this paper, potential error variance can be introduced into the trial design by different data collection modes or methods used within the trial that do not provide comparable data (i.e., the methods and/or modes lack sufficient measurement equivalence.)

However, although it may not be optimal, mixing of PRO data collection methods and modes within trials does occur and has to be addressed pragmatically.  It is clear from the FDA’s PRO Guidance that this situation is anticipated to occur within clinical trials.  Specifically, the Guidance states that “We intend to review the comparability of data obtained when using multiple data collection methods or administration modes within a single clinical trial to determine whether the treatment effect varies by methods or modes” (FDA, 2009).  The Guidance does not, however, discuss ways for clinical trial designs to ensure the comparability of the data when mixed methods or modes are used.

A prior ISPOR task force report addressed the evidence necessary to demonstrate measurement equivalence between electronic and paper-based PRO measures (Coons et al, 2009).  Although the task force recommendations focused on the migration of paper-based PRO measures to electronic platforms, the same principles apply to the assessment of measurement equivalence across and among all PRO data collection methods and modes.  However, here again, the use of mixed modes and/methods of data capture within clinical trials was not substantively addressed.

The purpose of this paper is to provide recommendations regarding good research practices for studies in which PRO data collection methods and/or administration modes are mixed within a single trial, or across trials that are intended for direct comparison.  The objective is to address issues that must be considered to avoid sources of measurement error that materially impact the measurement properties of the instrument being used to capture PRO endpoints in clinical trials.   Our intent is to provide practical means of optimizing data integrity when mixed modes and/or methods are unavoidable or when the benefit of mixed modes or methods is perceived to justify the risk. 

Research settings: While mixed methods and modes may occur in all research settings, this article will focus on their use in clinical trials, in particular, as the stakes and risks are highest for sponsors and investigators in this setting.  In clinical trials, mixed methods and modes may occur within and between the following levels:

1. Drug development programs
2. Clinical trials within a program
3. Countries within a clinical trial
4. Sites within trial or a country if multinational trial
5. Patients within a site

Process:  to address this complex issue, ISPOR has initiated a working group to develop good research practice recommendations for the implementation of mixed methods and modes in clinical trials and the analysis of the resulting data.

2. What are the issues with using mixed methods and modes in clinical trials? (Suggestion: to list the types of modes and methods and potential issues with each)

1. Mixed modes of administration (i.e., self vs. interviewer)
1. Brief review of the psychology of self-report to help the reader understand the source of mode effects, e.g., question comprehension, recall, sensitivity of questions, comfort, etc.;
2. Mixed methods of data collection
1. Paper – Handheld device
2. Paper – Tablet
3. Paper – Web
4. Paper – IVR
5. IVR – Web
6. Handheld device – Tablet – Web
3. Impact of mixed method/mode effects
1. How method/mode effects manifest themselves, differences in mean, variance, reliability; or alternatively, differential item functioning
2. Reported magnitude of the various method/mode effects to put into context the appropriate analytic approaches;
3. How likely is it that method/mode effects have caused incorrect conclusions to be drawn in clinical trials?  For example, if both treatment arm efficacy estimates have an upward bias, then mathematically that bias is accounted for automatically in any comparative analysis.
4. Given the risks, why use mixed methods or modes in clinical trials?
1. Minimize missing data due to device loss or failure or patient not using the preferred method or mode
2. Accommodate technology-challenged countries, sites or patients
3. Improve effect estimates by using triangulation of results across sources.
5. Literature review (accounting for study populations (pediatric, adults, geriatric), disease areas (chronic, acute, oncology, etc…), type of PRO endpoints, etc.  The mixed methods/modes issue should be addressed in well-defined contexts to answer the question: Does the context make a difference?)

1. Mixed modes of administration studies
2. Mixed methods of data collection studies (within self-report mode)
3. Review of FDA Guidance and other regulatory documents
   

3. Strategies for appropriate use of mixed modalities in clinical trials
1. Non-statistical approaches to minimizing method or mode effects
1. Staff training
2. Respondent training and instructions
3. Efforts to minimize differences in questionnaire completion rates
4. Sound survey design practices (Dilman and Salant)
2. Considerations by level starting with drug development programs
1. Older studies most likely used paper, newer studies more likely to use ePRO
2. Fewer issues between programs for same compound
3. Clinical trials within a program
1. Between Phases
1. Changing methods between Phase II to III within a program
2. Changing methods between Phase III and IV
2. Within Phase
1. Issues specific to Phase II or III trials
2. Using different methods between same phase trials especially Phase III
3. Within a trial:
1. Baseline in one method (at clinic) and follow up in another (at home)
2. Countries within a clinical trial
1. Between countries
1. Mixing methods between countries may be unavoidable in ePRO studies where technology may not be suitable for specific countries
2. Recommendation: mixing between countries to control for nesting effect of language and method acceptable
2. Within a country (from here down not recommended in clinical trial except for emergency needs)
1. Between sites
2. Within a site
1. Between patients
2. Within patient  (Backup data collection methods specified in protocol if device is lost or fails or patient can’t get to IVR)
3. Within instrument and administration (should patient be able to start in one method and finish in another, in the same administration?)
3. Considerations when mixing modes as well as methods in a trial
   
   
4. Analytical approaches for evaluating mixed methods/modes data using specific examples
1. Background
1. Brief lit review of statistical approaches to analysis of data from mixed methods/modes studies. This can draw largely from methods to examine order effect and alternative formatting (e.g., Grunberg, et al. QOLR). There are methods to avoid/assess the amount of cascading errors in general.  These methods are (or are not) different from what is typically done to analyze PRO data.
2. Instrument development time point: prior to use in a trial, look at different methods/modes and then how to address them in a subsequent trial (this should be mentioned earlier as well, indicate that both item and scale development is involved)
3. A comprehensive statistical analysis plan outline for trial data
4. Discussion of the type of mixing observed and the implications. Pick up the point in main paper about different methods/modes within a patient as opposed to different methods/modes across sites/countries that are nevertheless consistent within patients.
5. Discuss effect of administration interval and how it would impact effect size. Solutions here would be case-specific, there is no blanket recommendation that we can make.
2. Diagnostics
1. Variance stabilizing and interpretability transformations. (small section on visual analogue scales)
   - advantages and disadvantages of both approaches.  brief section.
1. t-scores (use example of the SF tools; PROMIS)
2. 0-100 comparable scaling (use example from cancer control studies with multiple measures of fatigue, qol).
2. Comparing scales (indicate how to carry out the diagnostic analysis and discuss the implications)
1. Equivalence (mean, variance, Bland-Altman, regression model)
1. What level of evidence should a sponsor/investigator consider providing about equivalence between methods or modes in a pivotal trial vis a vis the FDA PRO Guidance and recommendations of the first task force
2. Recommendations for standards of equivalence for levels described above (i.e., highest standard for within patient mixing, lower for site, country, etc.)
3. Reiterate recommendations from previous document. Indicate that mixing modes/methods might add to the error variance and hence reduce reliability. Demonstrate using power calculation examples that you will need to adjust sample size up if you are assuming increased variability due to mixed methods/modes.
2. Baseline in one mode/method, follow up in another
1. implications for responder analyses: indicate importance of defining cutpoints and defending
2. implications if eligibility criteria defined based on PROs –there are examples of mixed modes/methods in this respect in the literature
3. Sensitivity Analysis with and without the alternative mode or method
1. to determine effect size impact
2. effect of methods or modes on responsiveness, reliability, validity
4. Latent variable modeling/factor analysis
1. use as a diagnostic for impact of mixed modes/methods
5. Differential functioning item analysis
1. describe how it can be used to assess impact of mixed modes/methods
3. Combining data
1. simply combine the data as if it came from the same source if diagnostics so indicate
2. if diagnostics indicate non-equivalence
• add in a factor for the source of the data;
• deal with it as we would with any other source of variation.
• Analogous to treatment by center interaction for example.
4. Missing data considerations (small section giving refs)
1. Indicate imputation approaches, a particular advantage of mixed methods or modes to provide different sources for the same info perhaps or just leave as the methods are the same for one method or mode (which for most applications they would be).
2. Discussion of reliability and increased number of observations. Increase in precision maybe with illustration table if values were MAR. Indicate that the most common situation may be equal, or higher, reliability and increased observations, but the observations that are missing (which were suggested to be MNAR) might be more problematic (than those MAR) if a systematic bias is introduced because of mixing modes/methods.
5. A template for presenting results: summarize i-iii from (b) in an example report (supplemental appendix)
   - use the example of a clinical trial to compare effects of a dummy agent to a placebo to drive the points. Nextamethosone was an example used for ASCO in this manner (Stockler et al, ASCO 2006). In this way we can go between the general procedure and the specific example (pain, anxiety, mood, tx schedule, provides contextual help for readers)
   
6. Discussion of analytics section

- mixed methods and modes will occur, so methods for best practices are needed
- discuss the balance between statistical, theoretical purity and empirical practicality
- put into context the order of magnitude of the likely effect and size of the “problem”. Does the evidence suggest that the problems induced by mixed methods/modes are trivial, prohibitive, or somewhere in between?  (it depends)
We discussed the tonality of the end result within a context of comparable analogies such as:
a. mixing modes of blood pressure cuff (person versus machine)?
b. Another example is glucose counts by machine versus quickie tests

7. Conclusions of analytics section
   - the size of the error is (or is not) a major concern
   - methods are (or are not) readily available to correct for such bias
   - investigators and sponsors are encouraged to use (or warned off using) mixed methods/modes

V. Conclusion

VI. References