|
Issue VI
ADDRESSING QUESTIONS OF BIAS, CREDIBILITY AND QUALITY IN HEALTH ECONOMIC EVALUATIONS
Goal
Identify key issues in addressing bias, credibility, and quality of pharmacoeconomic evaluations.
Specific Objectives
- Identify and prioritize the key issues associated with reducing bias and increasing credibility and quality
of pharmacoeconomic evaluations
- Identify a plan of action to resolve these issues
- Recommend next steps
Co-chairs
- Rod Barnes MBA, Director, Health Outcomes, Alcon Laboratories, Inc.
- Alan Heaton PharmD, Pharmacy Gold Inc.
Panelists
- Carole Magoffin MS, Senior Advisor, National Pharmaceutical Council
- Jack McMillan PhD, Director & Team Leader, Outcomes Research, Pfizer Inc.
- Tom Taylor PhD, Associate Professor & Co-director of the University of Iowa College of Pharmacy
- Albert Wertheimer PhD MBA, Director, Outcomes Research & Management, Merck & Co Inc
Background and Context
Bias, quality, and credibility are long-term research issues addressed by experts in many fields. Like other disciplines,
the field of health economics continues to evolve and challenge itself in this regard. Improvements in this field of
research necessitate support and adhesion to the highest quality work and integrity, promotion of continuous quality
enhancement and open dialogue.
Definition of bias in the field of health economic research refers to a meaning beyond the Stanley and
Campbell (Campbell 63 667; Cook 81 668) statistical concept, i.e., as a threat to validity. Bias is also an ethical issue dealing
with disclosure and conflict of interest. The following definition of bias was proposed by The Task on Principles for
Economic Analysis of Health Care Technology in 1995 (Task Force 479): as, "A range of factors that systematically influence
the measures undertaken independent of the studied intervention; a tendency, intentional or unintentional, to
inappropriately or unfairly favor one or more of the interventions being evaluated."
Resolution of bias, quality and credibility issues in health economics is complicated by a host of factors, especially
the variety of stakeholders and their unique perspectives and information needs. These stakeholders include users of
research such as managed care organizations, governments, payers and providers, and the producers of research such
as government agencies, pharmaceutical firms, payers and providers, academics, consultants, and foundations.
The objectives of this panel were to address the issues of bias, quality, and credibility of health economic research
from the perspective of both producers and users of these data with patient care as the underlying concern. The scope of
the panel's deliberation included public dissemination of research data, methods to improve quality, minimize bias, and
thus enhance credibility of health economic studies. Methodology issues are elsewhere. The panel anticipates that as
advancements are made over time, concerns over bias, quality, and credibility of health economic research will diminish.
Problem Statement
Multiple published studies have criticized the rigor, relevance, objectivity, methods, and reports produced within the
health economic research domain (Drummond 378; Udvarhelyi 417; Coyle 114; Bradley 30; Balas 363; Iskedjian 394;
Hillman 669; Blades 670; Lee 671). Consequently, health economic research findings are not used as extensively as they
could be and rational decision processes about the efficient use of health care resources may not be fully informed.
Ultimately, care for patients and populations may be adversely affected. In this context, there is a need for continued improvement
in the quality of economic research conducted.
Issues
The panel identified the following key issues surrounding these matters:
- Quality: are best methods being used?
- Bias: whether real or perceived, how do we deal with it?
- Credibility: do we have a problem with believability or with relevance?
1. Quality
Health economics is a relatively new science without strong consensus on all methods. There is a need to
develop consensus on methodology for quality issues to be resolved. The multidisciplinary nature of this field of research
makes peer identification for manuscript review difficult. In addition, there is a lack of consensus on evaluation criteria,
rendering the peer review process difficult.
2. Bias
Bias can be divided into "intentional bias" and "unintentional or subtle bias" which includes design flaws and
inappropriate conclusions irrespective of the medical interventions under study.
Bias can be related to the underpowered nature of many clinical and database studies which would require much
larger samples. There is a perception that financial sponsorship will bias study results. However, despite objectivity
problems, industry is still the major funding body for healthcare economic information. Withholding negative findings and failure
to submit data to public scrutiny can bias the literature and applications of research data which are disseminated. There
are few occasions when circumstances would make the withholding of results acceptable (Yee 97 419).
3.Credibility
The relevance of health economic research data is questioned by decision-makers with respect to populations
studied and the disconnect between decision makers' criteria (business decisions under risk or uncertainty where majority
rules) and methodologists' concerns (scientific standards seeking statistical significance) Stakeholders have different
objectives. Methods are geared to societal decision-making but practical applications are at a different level. In addition,
decisions have to be made right away and cannot wait for the development of better methods. Some groups and journals such
as the New England Journal of Medicine (Kassirer 91) have questioned the credibility of health economic analyses
and restricted the dissemination of data. The ownership of health economic data must be defined, as well as access to
this information. Health economic tools are expected to be employed to optimize use of the resources of society, but
few decision makers use them that way.
Recommendations and Next Steps
The following recommendations will improve the quality and credibility of health economic research. These
recommendations pertain to four major domains:
- Design and research practices
- Sponsorship
- Publication and dissemination
- Role of professional societies and organizations
- Follow-up conference
1. Design and research practices
Researchers should design and conduct studies using the best available practices consistent with the study
objectives. Methods should be specified in advance as well as explicitly and transparently reported. Health economic studies
should generally follow ethical Good Clinical Practices provisions such as those described by the American Federation of
Clinical Research (AFCR Good Clinical Practices) or other authoritative bodies. Prospectively designed pilot studies are
essential for the evaluation of feasibility and the planning of future research. In addition, studies should not be terminated early
in an attempt to hide unwanted health economic results of potential interest. Base case assumptions should be clear,
and the sensitivity analyses should include conservative assumptions for the new technology being assessed.
2. Sponsorship
Full disclosure of any financial relationships which authors and speakers have linking them directly or indirectly to
the interventions under study should be listed. Sponsored research should have a written protocol agreed to by
researchers and sponsors in advance. Unanticipated issues should be handled by mutually agreed protocol amendments. Access
to relevant documents and data, project control, presentation, and publication rights should be defined in advance in
the contract (Schulman 415). The contract would reflect the code of ethics recommended by this panel.
3. Publication and dissemination
Authorship should conform to generally recognized practices among the peer research community. Research data,
given full disclosure, transparency, and sufficient information to replicate the study, should be judged on the merits of its
content. One or more authors should be receptive to and available for reader inquiries in those cases where it is clearly not
practical or possible to provide all information necessary for replication of the study within a manuscript. Publication
decisions by journals should use the same criteria for evaluating merit of health economic studies that are used for other types
of manuscripts, and should not preclude publication of health economic work due solely to funding arrangements or
author affiliations. Published reports should address the criteria established by experts and be completely transparent to
facilitate evaluation, comparison or reproduction.
Mandating full publication rights in a code of ethics for health economic research remains an area of debate among
panel members. Certain members believe that this issue is at the heart of bias and credibility problems and they advocate
full publication rights for contractors performing health economic research. Other panel members anticipate that
mandatory rights in certain health economic studies could be problematic. Organizations, industries or government bodies
using contractors to perform studies are not likely to allow full publication rights for their services.
4. Role of professional societies and organizations
A professional society such as the International society for Health economics and Outcomes Research (ISPOR),
the Association for Health Services Research (AHSR) or the Society for Medical Decision-making (SMDM) should
consider developing a code of ethics for health economic researchers. Those professional societies need to collaborate on
education and best practices for the benefit of association members, the journal community and research users, and to
improve quality, reduce bias, and enhance credibility of health economic research. Professional societies should also help
journal editors identify peer reviewers and advise on their review procedures. A foundation to address these ideas may be
in order. ISPOR and similar bodies should also publicize awards for high quality research, including student or
fellowship awards.
The concept of an independent body that would provide confidential advice and would certify voluntarily submitted
health economic study designs and reports deserves further discussion. This body could be a quasi-public organization
that discloses all its financial relationships. Importantly, a similar body might also investigate means of evaluating
health economic research for continuous quality improvement and development of the field.
5. Development of Study Methods and Ethics Standards
In the near future, clear best methodological and ethical practice statements should be developed and
disseminated. Meetings with journal editors and users should be organized to educate and offer assistance to stakeholders.
Specific recommendations are:
- Researchers should design and conduct studies using the best practices consistent with the study
objectives. Methods should be specified in advance, explicitly and transparently reported.
- Health economic studies should generally follow the ethical Good Clinical Practices provisions described
by authoritative organizations such as the American Federation of Clinical Research.
- Authorship should conform to generally recognized practices among the peer research community.
- Full disclosure of financial relationships of authors and speakers should be listed.
6. Follow-up conference
A conference similar to this one should be convened in two years to review progress made as a result of these
deliberations. ISPOR should work with other societies and organizations to further explore all the other recommendations
made by the panel which require additional thought.
In addition to the above recommendations, ISPOR should develop the following activities:
- Establishment of a code of ethics addressing among other issues the use of appropriate methodology,
reproducibility, publication rights and disclosure of potential conflicts of interest. Internal distribution and acceptance should
be sought in the coming year, with subsequent external promotion.
- Creation of an office to continue work on the issues raised by the panel, to coordinate collaborative efforts
with other organizations, to provide regular commentary and input and to facilitate the exchange of ideas in the
ISPOR journal.
- Dissemination and high exposure of information related to awards obtained for high quality research
including annual meeting presentations, journal articles and student fellowships.
- Establishment of a working relationship and coordination of regular meetings with the Food and Drug
Administration (FDA) to develop health economic guidelines to address concerns related to bias, quality, credibility and ethics.
Summary
Criticism of health economic research has resulted in limited utilization of this data by decision-makers and
end-users, potentially affecting care for populations. A number of recommendations were proposed by the panel to address
the issues of bias, quality, and credibility of health economic research from the perspective of both producers and users
of these data with patient care as the underlying concern. Suggestions pertain to four major domains, including design
and research practice, sponsorship, publication and dissemination of research data and the role of professional
associations and organizations.
Advisory Reports Main Page
|
