Document text
ACIP Guidance for Health Economics Studies
Guidance for Health Economics Studies P resented to the Advisory Committee on
Immunization Practices (ACIP)
Prepared by
The ACIP Ad Hoc WG on Economic Analyses
(Members listed alphabetically)
Tracy Lieu, M.D., M.P.H.
Professor and Director, Center for Child Health Care Studies Departme nt of Ambulatory Care
and Prevention
Harvard Pilgrim Health Care and Harvard Medical School
ACIP Member July 2004-June 2008
Martin I. Meltzer, M.S., Ph.D.
Senior Health Economist & Distinguished Consultant
CDC/NCPDCID/DEISS
Mark L. Messonnier, M.S., Ph.D.
Lead Economist
CDC/NCIRD/ISD
11/13/2007 Page 1 of 9
ACIP Guidance for Health Economics Studies
Guidance for Health Economics Studies P resented to the Advisory Committee on
Immunization Practices (ACIP)
Rationale for this guidance
The Charter of the Advisory Committee on Immuni zation Practices (ACIP) states that, when
considering recommendations for use of a vaccine , ACIP members’ deliberations should include
consideration of vaccine efficacy as well as cost:b eneft and risk:benefit anal yses. In recent years
there has been a trend toward an increasing numbe r of economic studies of vaccines presented as
evidence for the Committee to consider. Precedent has been established by journals such as the
British Medical Journal and Vaccine , in that they have adopted standards by which editors and
reviewers should consider the quality of subm itted economic studies. The Academy of Managed
Care Pharmacy also has adopted guidelines for economic data submitted in support of formulary
consideration. This Guidance has been developed to help ensure the quality of economic data
presented to the ACIP and its work groups (WG).
To ensure the value and consistency of economi c data presented to the ACIP, WG chairs are
advised to involve CDC economists from the inception of the WG. Participation of CDC
economists can be facilitated by the National Cent er for Immunization and Respiratory Diseases
(NCIRD) lead economist, who can coordinate such involvement.
Objective of this guidance
It would be impossible to provide a set of standards for methods, inputs and values for inputs that
would cover all the diverse vaccin e-related issues that the ACIP considers. The objective of this
guidance is to provide a framework for the desc ription and presentation of methods used to
examine the economics of a vaccine-related issue, such that the ACIP readily can understand the
science of the analyses. To ensure that su ch standards have been achieved, this guidance
specifically incorporates technical review of any economic material s that are to be presented to
the ACIP.
GUIDANCE
Materials to be presented
All researchers (internal or exte rnal to CDC) wishing to presen t a health economics study (either
new or revised/updated) to the ACIP (or one of its WGs) shall present the following material:
1) Document or report providing methods and results The document should provide a
detailed description of the methods and results. The level of detail s hould be similar to that
provided in a manuscript that would be submitted to a peer-reviewed journal. However, the
document should not be bound by journal restricti ons on length of the methods and results
sections. There is no need for a detailed intr oduction section. The discus sion section typically
would be shorter than that found in a published manuscript. Additional details regarding the
content of this document are provided below.
11/13/2007 Page 2 of 9
ACIP Guidance for Health Economics Studies
2) Slide sets and other presenta tion materials (e.g., handouts ) The principles and template
for slides intended for presentation to the AC IP which outline the methods and results, are
provided in the “Guidance: format for presentation of slides” section. Additional details regarding slide content also are provided ( See attached set of “template slides.”)
Timeline for materials to be made available and review process
The report, slides, and ot her presentation material must be submitted to the relevant ACIP WG
chair and CDC lead staff person no later than 8 weeks before the ACIP general meeting or WG
meeting at which the analysis is proposed to be presented. Under extraordinary circumstance, an
appeal can be made to the WG chair, the CDC lead staff person, and the NCIRD lead economist
to submit the report fewer than 8 weeks before th e ACIP meeting. In case of such an appeal, the
lead economist will document the justification an d granting of exceptions in collaboration with
the WG chair and the CDC lead staff person.
The NCIRD lead economist (or designee) will work with the WG chair and CDC lead staff
person to identify reviewers for an anonymous peer review. Non-CDC revi ewers may be used if
a particular area of e xpertise is not available among CDC ec onomists. Reviewers will consult
with relevant CDC subject-matter experts and return comments a nd questions in writing to the
NCIRD lead economist (or designee) at least 4 weeks in advance of the formal presentation.
These comments will be forwarded to the WG ch air and CDC lead staff person who will forward
them to the appropriate WG members and to th e researchers submitting the economic analysis.
This process will allow time for at least one round of comments and revisions or responses prior
to the formal presentation. Th e relevant WG chair, the CDC lead staff person, and the NCIRD
lead economist (or designee) will determine if revisions and responses are sufficient to allow
presentation. If differences persist, the WG chair and the CDC lead staff person, in conjunction
with all members of the WG, w ill decide if the information w ill be presented to the ACIP.
Reviewer comments also will be provided to th e WG and the ACIP to explain the differences
that may exist.
11/13/2007 Page 3 of 9
ACIP Guidance for Health Economics Studies
GUIDANCE: Content of document
A document describing the methods and results of a health economics study for consideration by
the ACIP will contain the follo wing sections and elements:
1) Affiliations
All authors shall incl ude their affiliations.
2) Statements of conflict or po tential conflicts of interest
A separate section listing any potenti al conflicts of interest shall be included for each author. If
there are no potential conflicts of interest, a statement to that effect must be included (e.g.,
Author A: No conflicts of interest).
METHODS
3) Methods: the study question
The study question must be explicitly stated in detail (e.g., In this study, we present results of a
cost-effectiveness analysis of routinely vacci nating age group XX against Disease Y, using
vaccine Z, using three doses given once per year over three consecutive years.)
4) Methods: the perspective
The study must be conducted from the societal perspective unless strong justification is provided
for doing otherwise.
Other perspectives may be in cluded when relevant justif ication is provided for their
consideration. The study perspective(s) must be explicitly stated (e.g., “The perspective used in this study is
societal.”)
5) Methods: intervention strategies
Intervention strategies must be clearly identifi ed and completely described. A baseline strategy
must be included.
Authors must describe all releva nt interventions. If the analys is does not include all possible
interventions, the authors sha ll provide the rationale for including or excluding them. For
example, the authors of an analysis of vaccinati on of adults against pert ussis would discuss the
rationale for inclusion or exclusion of a select ive strategy of vaccinati ng care providers of young
infants.
6) Methods: time frame and analytic horizon
The time frame and analytic horizon for each stra tegy must be defined clearly, complete with
appropriate justification (e.g., “W e analyzed benefits and cost s over a ten year period. The
analytic horizon was selected because availabl e data suggest that af ter XX years, vaccine-
11/13/2007 Page 4 of 9
ACIP Guidance for Health Economics Studies
induced immunity may wane to the point where a large proportion of vaccinees would require a
booster dose to maintain adequa te levels of immunity.”).
7) Methods: the economic model
a) The analytic method used must be specified (e.g., cost-benefit, cost-e ffectiveness, or cost-
utility analysis). The summary me asure must be defined/ identified.
b) The basic model used in the analysis must be explained in the text without use of
mathematical notation. Authors may consider en hancing the written desc ription with a word
“equation”. If deemed necessary by the author s, an equation, or se t of equations, using
mathematical notation also can be provided in a technical appendix.
8) Methods: health outcomes of interest Health outcomes must be clearly identified (e.g., deaths, hospitalizati ons, outpatient visits,
quality adjusted life years).
Authors must ensure that health outcomes are re levant to the perspective. Although almost any
health outcome may be relevant for a societal perspective, some health outcomes may be
irrelevant for any additional perspectives.
9) Methods: inclusion of epidemiologic models
When an epidemiologic model is an integral part of a health economics study to be presented to
the ACIP, the authors shall include a descripti on of the model. The authors must include a
schematic diagram illustrating the model. Annotat ion in such a diagram must be done without
use of mathematical notation. Authors are directed to ensure that such schematic diagrams can be
readily understood without extensive reading of the main text (i.e., can “stand alone”).
The authors must state the time step used in the epidemiologic model. “Time step” refers to the
time associated with the probabi lities used in the model. For example, if the authors use
probabilities of death per unit population per year, then the de facto time step of such a model
would be one year. Probabilities, and time step, can be in almost any unit of time. The total
number of time steps must match the time fr ame of the economic model unless a specific
explanation is given.
Authors should note that it is insufficient mere ly to reference another source (e.g., reference a
previously published paper).
10) Methods: Inputs: values and sources
a) Probabilities: Values and sources must be presented. Authors mu st make sure that
probabilities used in the analysis are “reasonable.”
b) Costs: Values and sources must be presen ted. The study shall differe ntiate between direct
medical, direct non-medical, indire ct (i.e., productivity), and in tangible costs (included when
relevant). The year of cost data also must be st ated. Authors must make sure that included costs
are relevant to the stated perspective.
11/13/2007 Page 5 of 9
ACIP Guidance for Health Economics Studies
c) Other inputs: Values and sources of any othe r inputs (e.g., Quality Adjusted Life Years) must
be presented.
Each value used in the model must have at least one clearly identi fied source. Values that are
assumed, or calculated (e.g., a resi dual probability), or the results of expert opinion shall be
identified as such. Authors are advised that the grea test clarity is often achieved by presenting all input values and
sources in tabular format. Such tables must in clude sufficient footnotes to enable a reader to
readily understand the table without extensive reading of the main text (i.e., such tables must
“stand alone”).
11) Methods: discounting
All future costs and benefits shal l be discounted to present value. This includes future health
outcomes (e.g., future lives saved must be discounted to present).
The discount rate used must be specified, and au thors must ensure that the discount rate is
relevant to the stated perspective.
12) Methods: sensitivity analyses
The general goal of sensitivity an alyses is to demonstrate how conclusions might change with
changes in input values.
Sensitivity analyses, including appropriate thresh old analyses, shall be conducted and reported
on cost and incidence variables. The type of sensitivity analysis conduc ted must be described.
Authors are strongly encouraged to conduct sensitivity analyses that allow them to identify
which variables within the model are most influential in determining the overall results. The ranges and sources of the values used in the sensitivity analyses must be clearly reported.
Authors shall report values used in sensitivity an alyses in the same table in which they report
values of all inputs (see Point 10). Univariate sensitivity analyses (altering only one value at a time, and keeping all other values
fixed at original values ) are unlikely to be considered adequa te. Authors shall present some form
of multivariate sensitivity analysis or an explan ation as to why such analysis is not presented.
Authors must use sensitivity analysis ranges that are based on clinical ly relevant or policy
relevant cutoffs rather than those based on arbitr ary changes in input values (e.g., plus or minus
10% of initially used values). As a form of sensitivity analysis, in addition to presenting results that have been discounted to
present values (see Point 11), undiscounted costs and benefits may be presented.
13) Methods: independent replication
11/13/2007 Page 6 of 9
ACIP Guidance for Health Economics Studies
In general, information must be presented that would allow a researcher, with sufficient interest
and relevant skills, to inde pendently replicate the study..
In order to meet such a standard, where necessa ry, authors may provide additional details in a
technical appendix.
RESULTS
14) Results: summary measures
Results that answer the study question must be identified. Th e summary economic measure(s)
that answers the study question must be presented.
The summary measure must be appropriat e for the perspective used in the study.
If deemed necessary, authors also may wish to present other results calculated during the
analysis, such as total number of cases averted.
15) Results: tables and graphs
All tables and graphs used to present result s must be readily understood without extensive
reading of the main text (i.e., such graphs and tables can “stand alone”). Authors must include
footnotes that will help a read er understand each table and graph.
Tables and graphs must add value to the report. Graphs or tables that c ontain results not central
to the summary measure (see Point 16) ma y be included in a technical appendix.
Graphs shall be drawn using the standard guidelines for graphical representation of data.
Detailed guidelines can be found in texts such as Tufte (Tufte ER. The Visual Display of
Quantitative Information . Graphics Press, Cheshire Connect icut, 1983: pp. 197). Examples of
such guidelines include:
1) Pie charts are almost always unacceptable.
2) Horizontal and vertical grid lines usually are not needed.
3) For line charts, typically only 4 lin es (variables) drawn per chart.
4) For column and bar charts, typically only 4 bars (variables) included per chart. Avoid
using “stacked” bars or columns (e.g., a column that has several elements that add to
100%).
5) To allow easy printing and copying, graphs and figures should be drawn in gray-
scale. Color in graphs and figures should be used sparingly, if at all.
16) Results: sensitivity anal yses and influential variables
Authors shall present their sensit ivity analyses in a clearly id entified section, complete with
relevant tables and graphs.
Authors shall, whenever possible, present a list of “most influential” variables as identified
through sensitivity analysis (see Point 12).
DISCUSSION
11/13/2007 Page 7 of 9
ACIP Guidance for Health Economics Studies
17) Discussion: overall
The discussion section should be more limited than that typically found in a peer-reviewed
manuscript.
18) Discussion: limitations
Study limitations must be discussed. Limitations shall include accuracy of any epidemiologic
model and input data (see Points 9 and 10).
Any implicit assumptions, such as an adequa te supply of vaccine, should be mentioned.
19) Discussion: relation to other relevant studies
Results must be discussed in relation to other similar studies if such studies are available.
20) Discussion: how results may change
There must be an explicit discussion, typically drawing from the results of the sensitivity
analyses, of how results would change if ke y assumptions or valu es were to change.
21) Discussion: no policy implications Unless otherwise requested by the relevant WG ch air, the report must NOT include a discussion
of the policy implications of the results and limitations. Because it is CDC’s and ACIP's
responsibility to make policy interpretations, such discussion will be deleted if included in the
document distributed for ACIP discussion, unless the WG chair specifically requests it be
included.
22) Details not addressed in this guidance
For further guidance regarding methods a nd results, researchers may follow the
recommendations described in the follo wing standard texts (or equivalent):
1) Prevention Effectiveness: A Guide to D ecision Analysis and Economic Evaluation, 2d . Ed.
Anne Haddix, Steven Teutsch, and Phaedra Cors o, editors. (New York: Oxford University
Press, 2003)
2) Cost-Effectiveness in Health and Medicine. Joanna Siegel, Louise Russell, Milton Weinstein,
and Marthe Gold, editors. (New York: Oxford University Press, 1996).
23) Additional information
1) Link to the Community Guide Economic Eval uation abstraction form where all of the
important components of an EE analysis are pres ented. The form has a quality rating of EEs at
the end. www.thecommunityguide.org/methods/econ-abs-form.pdf
2) Many standards for conducting economic eval uations are also on the CDC on-line Economic
Evaluation course. www.cdc.gov/owcd/EET/Preface/Preface.html
3) The Guide to Community Preventive Servi ces book, chapter 11 is titl ed “Interpreting and
Using Economic Evidence”.
11/13/2007 Page 8 of 9
ACIP Guidance for Health Economics Studies
4) Detsky AS, Laupacis A. Relevance of cost-eff ectiveness analysis to clinicians and policy
makers. JAMA. 2007;298:221-4.
GUIDANCE: format for presentation of slides
Principles for presenting a health economics study to ACIP:
After the document has been peer-reviewed internally, and the WG chair and CDC lead staff
person have reviewed and discusse d the contents of the document a nd the relevant analysis, they
will decide whether to proceed with invi ting a presentation to the ACIP or WG.
It is realistic to assume that many ACIP members will not have had the opportunity to read the document. Further, the ACIP general meetings (but not WG meetings) are open to the public. Thus, the presenter of the health economics study should realize that most of the audience at the
presentation will not be aware of the study, and that the audience members have diverse
backgrounds. Further, especially fo r presentations at an ACIP gene ral meeting, it is likely that a
relatively short amount of time will be allotte d for the presentation, typically 20-30 minutes
including time for questions. A presentation made in the closed setting of a WG teleconference
or meeting may be longer and more detailed.
Template slides:
The attached set of template slides provides an outline of the type of slides and suggested layout
of each type of slide. The subj ect matter of the slides in the template is designed so that a
presenter can be assured of presenting the main f acts as they relate to the methods, results and
limitations of the study.
11/13/2007 Page 9 of 9