The Application of Post-Market Registries and other Evidence for Medical Devices - Part III: Post-Market Studies Under Medicare Coverage with Evidence Development: When Are They the Wrong Answer to the Right Question?

This is a summary of the presentation given at the ISPOR Medical Device and Diagnostics Council Forum at the ISPOR 10th Annual International Meeting, May 17, 200, Washington, DC, USA.

Introduction
On April 7, 2005, the Centers for Medicare and Medicaid Services (CMS) issued a draft guidance on data collection that may be required in the context of Medicare national coverage determinations (NCDs) [1]. The Draft Guidance stated that its purpose is to describe factors that CMS may consider in deciding to extend national coverage for certain items and services with coverage that is linked to a requirement for prospective data collection, known as “coverage with evidence development” (CED). The primary purpose of obtaining such additional evidence is for CMS’s use in making payment determinations, i.e., that a treatment or other intervention is “reasonable and necessary” under Medicare.

In many instances, the prospective data collection described in the Draft Guidance will comprise post-market studies; that is, it will be for drugs, devices, and biotechnologies that will have been approved by the Food and Drug Administration (FDA) for marketing. This presentation is not a review or critique of the full April 2005 Draft Guidance. However, it does address key aspects of the types of post-market studies that are described in the Draft Guidance, including where there may be mismatches between the proposed study designs and the intent and circumstances of CED.

Evidence as Basis for “Reasonable and Necessary”
The concept of “reasonable and necessary” under Medicare is pivotal to the rationale for CED and its implementation. The CMS authority to make coverage decisions derives from Section 1862(a)(1)(A) of the Social Security Act, which restricts all coverage and payment to that which is found “reasonable and necessary” for the diagnosis or treatment of illness or injury. It state that “ . . . no payment may be made . . . for expenses incurred for items or services . . . [which] are not reasonable and necessary for the diagnosis or treatment of illness or injury or to improve the functioning of a malformed body member.” This provision gives the HHS Secretary, acting through CMS, the authority to determine the coverage of services under Medicare.

Across a series of assertions, the Draft Guidance interprets reasonable and necessary as being explicitly evidence based: “The primary purpose of obtaining additional evidence through CED is for the agency’s use in making payment determinations, i.e., that a treatment is reasonable and necessary” (Draft Guidance p.2) … “In general, the core consideration in determining when an item or service is ‘reasonable and necessary’ is the quality of the evidence available to assess whether it improves net health outcomes” (Draft Guidance, p.3) . . . “In some cases, CMS will determine that an item or service is only reasonable and necessary when specific data collections accompany the provision of a service” (Draft Guidance, p.6) [1].

As such, CMS may require certain additional evidence in order to establish the payment threshold of reasonable and necessary. The main consideration will be not just what this evidence suggests but the quality of the evidence. Further, the evidence that counts is that which demonstrates whether the intervention results in improved health outcomes. Finally, the very act of collecting data at the time a service is rendered could convert that service from not being reasonable and necessary to one that is.

Circumstances and Questions for CED

The Draft Guidance describes two general circumstances for applying CED. The first is for interventions that may have been demonstrated to improve health outcomes in a broad patient population, but for which important questions remain about which patients are likely to derive benefit from the intervention. The second circumstance concerns interventions for which existing evidence of improved health outcomes is not conclusive, yet suggests that the intervention may provide important benefit. This means that CED may proceed under conditions ranging from scarce to plentiful evidence.

According to the Draft Guidance, CED may be appropriate when there are questions pertaining to:
• gaps on safety, side effects;
• risks and benefits not described in literature;
• risks and benefits in specific patient subgroups;
• long-term risks and benefits, quality of life, utilization, costs, other real-world outcomes;
• risks and benefits of procedures not subject to FDA approval;
• effectiveness of interventions for rare diseases;
• available evidence not generalizable to providers/facilities or Medicare population;
• comparative effectiveness of new vs. standard interventions; and
• clinical significance, given statistical significance (Draft Guidance, p.9-10) [1].

The diversity in types of outcomes that may be in question suggests that a variety of study designs and data sources may be appropriate.

Types of Study Design
The Draft Guidance provides “a list of study designs that may be used to develop an evidence base,” including “databases,” which “require entry of baseline data … used to monitor patient safety and benefit,” “longitudinal or cohort studies” in which “patients are followed over time after baseline clinical information is collected,” “prospective comparative studies (also called practical clinical trials [PCTs])” that “require a formal comparison group [and] can include randomization,” and randomized clinical trials (RCTs).

A challenge of CED will be fitting a data collection study design to an evidence question and to any budgetary or other practical constraints. While RCTs or PCTs offer the best study designs (with tradeoffs of internal and external validity) for determining the causeand- effect relationship between an intervention and health outcomes, they are not necessarily the best designs for answering all types of evidence questions that may arise in the context of CED or, for that matter, of any effort to collect data under real-world conditions. For example, a non-randomized trial or case series may be appropriate for determining the potential effectiveness of an intervention for an otherwise fatal condition. For assessing the accuracy of a diagnostic test, a cross-sectional study can compare a new test to a “gold standard” in patients suspected of having disease or disorder. Surveillance or registries may be used to determine the incidence of rare, serious adverse events potentially due to an intervention. Registries with follow-up may be used to assess the safety or effectiveness of incrementally modified technologies posing no known additional risk.

Registries in Particular
The draft guidance refers to “registry” twice in the Draft Guidance. One instance arises in the example of implantable cardioverter defibrillators (ICDs), given to demonstrate the first general type of circumstance for applying CED, where: “The collection and review of baseline data by the implanting physician will help ensure that individual patients are being provided with care that is appropriate . . . The data gathered in this way should also help provide additional information on risks and benefits of the procedure . . . CMS implemented this initial registry using an existing electronic data submission system present in all hospitals . . . ” (Draft Guidance, p.7) [1]. The second instance refers to complementing registries with other data sources: “In many cases, it will be possible to link administrative data to data gathered for registries and practical trials . . .” This latter usage resembles the Draft Guidance description of “database.”

The term “registry” can refer to a broad set of data collection methods. This could be a focused design that is initiated by a particular intervention (such as a device implantation) and tracks a narrow set of measures, e.g., adverse events. It also could be a large, comprehensive architecture that captures a wide range of data elements, including patient demographic information, diagnostic and treatment information, medical claims, medication use, etc.

The potential utility of registries depends on a variety of factors, including the selection and frequency of follow-up data collection. A registry that comprises baseline data only cannot inform a determination of the effectiveness or safety of an intervention. It can help to ascertain and track trends of the patient population, site of service, providers, and other aspects of the initial procedure, and it can be used to identify and contact patients in response to other adverse event reports. With periodic follow-up data, a registry can monitor patient safety as well as individual and population changes in effectiveness, though not causation in absence of control group. If linked to claims data, registries can be used to determine which patients later received other follow-up care (hospitalization, other procedures, etc.) or who experienced adverse events or death, which may be associated with the original intervention captured in the registry. In concert with other data sources, registries can enable capturing data on diverse populations and practice settings for assessing the generalizability of findings regarding a technology. Registries can generate hypotheses that might be tested with other methods, preferably prospective ones with controls.

Evidence Requirements to Fit the Technology
In its coverage determinations and related documentation, CMS iterates the fundamental attributes of strong evidence that form the basis of evidence hierarchies used around the world. These include striving where possible for prospective studies, contemporaneous controls, placebo controls, randomization, sufficiently large sample sizes, blinding of patient assignment, etc. CMS refers to evidence hierarchies similar to those used elsewhere that typically show RCTs at the top (sometimes just below meta-analyses of RCTs), followed by non-randomized controlled trials and other less rigorous observational studies, with consecutive case series and single case reports at the bottom [2].

Of course, traditional evidence principles can be inappropriate, e.g., for many medical devices and surgical procedures. These evidence principles have been recognized largely in the course of clinical research designed for pharmaceuticals. For devices and many procedures, placebo controls and blinding are difficult if not impossible. The affected populations may be too small to achieve sample sizes used in typical drug trials. Also, as opposed to most drug therapies, it is often difficult to isolate the health impact of a diagnostic technology or other device apart from accompanying technologies and techniques [3].

Direct Evidence Not Always Necessary
For CED or any other type of assessment, evidence expectations should depend on the type of intervention and the context. Developing direct evidence for impact on health outcomes of a preventive, screening, or diagnostic technology may be impractical. The causal relationship between these types of technologies can be confounded by multiple factors intervening between an accurate test and health outcomes, including whether and how the test findings are used, patient compliance with any treatment or regimen, whether the treatment and compliance are being monitored, and the impact of the treatment on outcomes. If a chain of evidence already exists that links test results to treatment decisions, patient compliance, and outcomes is well established, then the more proximal evidence of test accuracy and potential side effects of the test may suffice.

For informing a coverage decision, study design should address direct and indirect evidence requirements reflecting patient indication, technology type, and application. The Draft Guidance recognizes the utility of indirect evidence in its examples of recent NCDs that exemplify how CED might be implemented. In describing coverage for FDGPET, the Draft Guidance states: “Based on studies of FDG PET’s usefulness as a cancer biomarker and for cancer staging and diagnosis, CMS now provides coverage if certain patient safeguards for patients are provided, including mandatory collection of clinical data. Under these circumstances, FDG-PET has the potential to improve health outcomes by influencing patient management; and by helping physicians appropriately evaluate the PET scan results … ” (Draft Guidance, p.8) [1].

Study Protocols for Answering Evidence Questions
CED study protocols should not be developed in the absence of explicit questions that are answerable by generating evidence. Such study protocols must be practical within the context of making coverage determinations for Medicare. An evidence question should specify at least the: health problem, patient population, technology, comparator (standard of care, whether drugs, procedures, other interventions, as appropriate), health outcomes/ endpoints of interest, setting of care, and providers.

At the very least, a study protocol designed to generate data to answer the evidence questions should include information about the proposed study design, duration, statistical requirements, and patient eligibility and enrollment. CMS and other stakeholders should consider the viability of the proposed data collection, including whether the data collection protocol is practical and ethical for enrolling and observing patients in a timely manner, whether its results will be worth the cost of the data collection, and whether the results will be actionable.

The design of a protocol for data collection under CED must consider that this will be conducted in the context of coverage, rather than in a traditional pre-market research context as might be the case for, say, other biomedical research or data collection for pursuing FDA market clearance. Any drugs, devices, or biotechnology that are the subject of an NCD are already likely to have been approved by the FDA and available for use in any patient as prescribed by a physician, which may confound patient enrollment in controlled studies. Research costs are at issue; Medicare is not normally in the business of paying for clinical research costs, and technology sponsors may not have the incentive or, at least for some smaller companies, the resources to pay for further data collection. Further, a new data collection burden would be imposed in practice settings, not investigational ones that typically receive funding for their research.

Before Requiring CED . . .
The Draft Guidance acknowledges certain qualifications about evidence collection that have been raised in open door forums hosted by CMS and by various stakeholders, including providers, industry groups, patient advocacy groups, and others [4]. To paraphrase the Draft Guidance, these include that CED should: • only be used to address specific evidence questions; • not duplicate existing data collection efforts of FDA or other public or private sector entities; • be worth its cost; and • should minimize financial and other resource burdens. As CED evolves, CMS and other stakeholders in CED should seek to ensure that these important qualifications of CED are sustained.

Methodological Framework for CED?
If CED is to be successful, it should be systematic and transparent. It is in the interests of CMS and the diverse stakeholders in Medicare coverage policies that the potential pathways to coverage-if not the determinations themselves-be clearly delineated at the outset. As CMS and stakeholders consider how to implement CED, they might consider forming an independent, properly qualified body to establish, transparently, a framework for study designs for CED. This would address, e.g., relative strengths and weaknesses of designs; capability to address particular types of evidence questions; and tradeoffs involving internal and external validity, costs, duration. Such a body could have representation of methodologists, clinicians, innovators, other stakeholders.

An Update
The April 7, 2005, Draft Guidance called for comments to be submitted to CMS by June 6, 2005. On July 12, 2005, CMS issued a fact sheet responding to comments from 65 stakeholder organizations [5]. The fact sheet provided answers to 12 questions that represented commenters’ most frequently stated concerns. These address such matters as when and how often CED will be applied, the authority of CMS to collect data in return for payment, patient privacy, which technologies would be affected, the role of FDA, and avoiding duplication of effort in data collection. One answer addressed whether CMS would develop a formal process for implementing CED, including designing methods by which the additional CED data must be collected. Without providing further detail, the July fact sheet indicated that CMS would issue more specific criteria for applying CED in a subsequent draft guidance document. This next version is to be followed by a 30- day comment period. According to the July 2005 fact sheet, CMS anticipates releasing the final CED draft guidance by the end of the year.

REFERENCES

1. CMS. Draft Guidance for the Public, Industry, and CMS Staff. Factors CMS Considers in Making a Determination of Coverage with Evidence Development. April 7, 2005. (Henceforth “Draft Guidance.”) http://www.cms.hhs.gov/coverage/download/guidanceced.pdf. Last accessed October 10, 2005.
2. CMS. Decision Memo: VADs as Destination Therapy (CAG-00119N), October 1, 2003. http://www.cms.hhs.gov/mcd/viewdecisionmemo.asp?id=79. Last accessed October 10, 2005.
3. Goodman C. HTA 101. Introduction to Health Technology Assessment. National Information Center for Health Services Research and Technology Assessment, National Library of Medicine. 2004. http://www.nlm.nih.gov/nichsr/hta101/hta.101.pdf . Last accessed October 10, 2005.
4. For example: AdvaMed. AdvaMed Submits Comments to CMS on Coverage with Evidence Development Draft Guidance. June 6, 2005. http://www.advamed.org/publicdocs/6-6-05-cms.shtml. Last accessed October 10, 2005.
5. CMS. Fact Sheet: CMS Responds to Stakeholder Feedback Regarding Coverage with Evidence Development. July 12, 2005. http://63.241.27.79/coverage/download/guidfactsheet.pdf. Last accessed October 10, 2005.

  Issues Index | 2005 Issues Index