Document text
1
A
CIP Evidence to Recommendation User’s Guide
October 1 , 2020
2
Contents
Introduction .................................................................................................................................................. 3
How to Use This Guide .................................................................................................................................. 3
The EtR Framework ................................................................................................................................... 4
Organization of this Guide ........................................................................................................................ 5
Overarching Framework Completion Guidance ....................................................................................... 5
Specific Criteria ............................................................................................................................................. 6
Domain 1: The Problem ............................................................................................................................ 6
Criterion: Public Health Priority ............................................................................................................ 6
Domain 2: Benefits & Harms of the Options ............................................................................................ 6
Criterion 1: Magnitude of desirable anticipated effects ....................................................................... 6
Criterion 2: Magnitude of undesirable anticipated effects .................................................................. 7
Criterion 3: Balance of desirable versus undesirable anticipated effects ............................................ 8
Criterion 4: Certainty of evidence for outcomes .................................................................................. 8
Domain 3: Values and Preferences ........................................................................................................... 9
Criterion 1: Target population perception of value .............................................................................. 9
Criterion 2: Uncertainty around target population perception of value .............................................. 9
Domain 4: Acceptability .......................................................................................................................... 10
Criterion: Acceptability to key stakeholders ....................................................................................... 10
Domain 5: Resource Use ......................................................................................................................... 10
Criterion: Resource allocation ............................................................................................................ 10
Domain 6: Equity ..................................................................................................................................... 11
Criterion: Healt h equity ...................................................................................................................... 11
Domain 7: Feasibility ................................ ............................................................................................... 12
Criterion: Implementation feasibility .................................................................................................. 12
Conclusions and Additional Considerations ............................................................................................ 12
Summary of the ACIP’s deliberations and final recommendations ........................................................ 13
Additional Resources .................................................................................................................................. 13
Appendix 1: ACIP Evidence to Recommendations Framework .................................................................. 14
Appendix 2 When to Prepare GRADE Evidence Profile Algorithm ............................................................. 21
Appendix 3 Implementation Considerations Checklist ............................................................................... 22
3
Introduction
In 2010, the Advisory Committee on Immunization Practice ( ACIP ) formally adopted the Grading of
Recommendations Assessment, Development and Evaluation (GRADE) approach for assessing the
evidence and developing evidence -based recommendations. Since then, an assessment of the certainty
of available evidence using GRADE ha s been an integral part of ACIP recommendation development . In
addition to assessing the certainty of evidence, guideline panel s consider other domains when making a
recommendation (e.g., the balance of benefits and harms, patient’s values and preferences, etc.). This
document presents the information that must be considered when making a recommendation. Previous
documents/publica tions provide information on assessing the certainty of the evidence (Ahmed 2011 ).
D
uring the process of recommendation formulation, panels consider a range of factors in addition to the
certainty in the evidence. Elucidation of these factors and the j udgments behind them facilitates
transparency, consistency, and communication of recommendations to healthcare providers, partner
organizations, and the public (Alonso -Cuello 2017a & b) . To structure the discussion of these additional
factors and to make the judgments transparent, an evidence to decision -making (EtD) framework is
used.
The EtD structures decision -making by:
• Inform ing panel members’ judgments about the pros and cons of each intervention under
consideration;
• Ensuring the important factors that determine a decision (criteria) are considered;
• Provid ing a concise summary of the best available evidence to inform judgements about each
criterion;
• Helping structure discussion and identify reasons for disagreements when they occur; and
• Making the b asis for decisions transparent to guideline users or those affected by a policy
decision.
The ACIP has continued to follow and build upon the methodological advances in the GRADE approach
and, as a result, has developed a modified Evidence to Recommendation (EtR) framework tailored to the needs of ACIP (Appendix 1). The purpose of EtR framework is to help panels making recommendations
move from evidence to decisions, and to provide transparency around the impact of additional factors on deliberations when considering a recommendation .
How to Use This Guide
The following supplementary information is intended to guide ACIP work groups while they complete
the EtR framework , as they address specific policy questions and develop policy options for ACIP ’s
consideration .
ACIP’s EtR frameworks are prepared by ACIP work groups for presentation to ACIP. As such, the EtR
frameworks represent the information reviewed and considered by the ACIP w ork group in developing
policy options for ACIP’s consideration. The ACIP’s final recommendation and any additional considerations of ACIP during its deliberations are captured in a separate table titled ‘Final deliberation
and decision by ACIP’ at the bottom of the EtR framework (Appendix 1) .
4
The ACIP’s EtR frameworks and accompanying GRADE evidence profiles are posted on the ACIP website
(https://www.cdc.gov/vaccines/acip/recs/index.html ), and are referenced with links in the MMWR
Policy Note or Recommendations and Reports document s, upon publication in MMWR .
The EtR Framework
The EtR frameworks are designed to include key background information, criteria for making a decision,
and conclusions .
The background section of the EtR is intended to provide details of the policy question that is the topic
of the framework. In the EtR framew ork table, this section is opened by a statement of the policy
question and the PICO (Population, Intervention, Comparator, Outcome), followed by a brief summary of information needed to understand the question and why a recommendation is needed. Specific
items that should be addressed include whether the recommendation would be for “off -label” use (i.e.,
an indication not specified in the FDA approved label) or a preferential recommendation.
The second part of the framework is composed of rows delineating the factors or criteria considered
during recommendation development, which are grouped into “domains.” These questions are
intended to frame the discussions and guide literature searches and collection of relevant data.
T
he following columns are included for each criterion:
• Judgments that the work group members must make in relation to each criterion are summarized in
the first column of the framework as checkboxes. In most cases, it is anticipated that the framework
will be developed with judgments for each criterion suggested by the work group and subject
matter experts (SMEs) who have prepared the framework. The judgments should be a result of the work group reaching a consensus; however, minority opinions expressed during the discussion
should be captured in the “additional considerations” section (see below).
• Evidence to inform each of those judgments is presented in the second column of the table.
Evidence refers to facts used to inform the work group’s judgments that are derived from studies
that used systematic and explicit methods.1 If published evidence is available, a paragraph or
bulleted list summarizing the important considerations is sufficient, with mention of the most
critical references or links to more detailed summaries of the evidence such as the GRADE evidence
profile1. If no peer -reviewed body of evidence is available, this should be simply stated and any
additional information used to inform the judgment indicated. The intent is to be transparent about
the information that was used to make the judgment, not to imply the need for the development of
evidence when it is not available.
• Additional information that informs or justifies each judgment may be provided in the final, right -
hand c olumn of the table. Additional information can include other data, assumptions, and logic
used to make a judgment. Work groups may make different judgments for one or more subgroups
(such as patients who are older or who have more severe disease) in relation to some or all of the criteria. When relevant, they may also report additional details, such as dissenting views of work
group members or the results of voting on judgments where there was disagreement.
Minority
opinions voiced during discussions should be presented to increase transparency around the
deliberation process. In addition, interpretations of the evidence may be presented here.
1 If using GRADEpro Guideline Development Tool software ( https://gradepro.org/ ), a GRADE evidence profile can
be automatically inserted into the EtR framework.
5
Organization of this Guide
This document provides additional information for each domain and criterion, including the criterion
question as presented in the EtR framework, related questions that expand on the issues to consider for each criterion (adapted from the GRADE Handbook ), and completion guidance. A Discussion section is
also included to provide further information and clarification on issues specific to each criterion/question. This additional information is intended to assist work groups and the ACIP in providing the relevant information to populate the framework.
Overarching Framework Completion Guid ance
Despite availability of a framework, individuals and groups will need to continue to make judgments at several points, especially when data are minimal or absent. One purpose of the framework is to make
those judgments explicit. Therefore, when evid ence is not available, this should be stated and a
summary of the discussion included. It is also anticipated that there will be differences of opinion within
the work group, and the framework should be used to reflect these differences, particularly with regard
to the final recommendation. Since one goal of the framework is to enhance transparency and communication of deliberations, significant differences of opinion or discussion points should be captured within each domain when relevant.
6
Specific Criteria
Domain 1: The Problem
Criterion: Public Health Priority
Criterion question: Is the problem of public health importance?
Related questions:
• Are the consequences of the problem serious (i.e. severe or important in terms of the potential
benefits or savings)?
• Is the problem urgent?
• Are a large number of people affected by the problem?
• Is the problem related to emerging diseases, antimicrobial resistance, or epidemic potential?
• Are there disadvantaged groups or populations disproportionately/differentially affected by this problem?
C
ompletion Guidance: Provide a short summary of the background and significant elements of the
problem, including a description of epidemiology, clinical features , and sequelae of the
disease/condition in terms of public health consequences. This may be excerpted or summary information from presentations to the full ACIP. Specific items to consider may include:
• F
requency of the disease/condition (e.g., incidence, prevalence, secular trends)
• Severity of the disease/condition (e.g., mortality, morbidity)
• Social impact of the disease/condition (e.g., hospitalization rate, sickness absenteeism, effects on high -risk groups and vulnerable populations, clinical features, perception of importance, and
the existence of other preventive measures)
A
ny additional considerations regarding the public health priority may also be included.
Domain 2: Benefits & Harms of the Options
Criterion 1: Magnitude of desirable anticipated effects
Criterion question: How substantial are the desirable anticipated effects?
Related question:
• How substantial is the anticipated effect for each main outcome for which there is a desirable effect?
C
ompletion Guidance: Across the critical outcomes (as specified in the PICO question and elaborated in
the evidence profiles), evaluate the magnitude of the anticipated effect. The anticipated direct and indirect ( e.g., herd immunity) benefits of vaccination or other interventions, should then be listed in the
framework in terms of specified outcomes with a description of how substantial the effects are. For effectiveness or efficacy, specific items that may be considered are immunogenicity, strain coverage, capacity to reduce the disease incidence, capacity to disrupt carriage, duration of protection, and/or serotype replacements.
T
he range of effectiveness/efficacy estimates should be included here, but specific information
regarding the confidence in point estimates should be pr esented in the certainty of evidence section
(Criterion 4) below as part of the GRADE evidence profile. Describe any differences in
7
effectiveness/efficacy estimates for special populations (e.g., immunocompromised, specific age groups,
etc.) and disadvantaged groups or settings (i.e. baseline risk) . Work groups should also consider the
uncertainties (if any) related to benefit with regard to durability of effectiveness, and effect on serious health outcomes (e.g., death and hospitalization), which are not often measured pre -licensure because
such studies would require a sample size too large to be feasible. The numbers of illnesses, hospitalizations and deaths that can be averted by use of the vaccine may be presented here but is not required.
In
addition, information may be included in the “additional considerations” section to highlight where
evidence may be lacking (e.g., subgroups where additional data are needed to assess the benefits).
Discussion
A
dditional considerations here can include quality of life, acute and chronic pain, and functional status,
among those who receive the vaccine and those who do not, including those who suffer the condition that could be prevented or ameliorated by vaccination. Work groups are encouraged to perform a
number needed to vaccinate (NNV) analysis. If a number needed to vaccinate (NNV) analysis is performed, any important uncertainty or assumptions should be included. There are no defined thresholds for an “acceptable” NNV .
G
uidance surrounding the use of immunogenicity data as a surrogate for efficacy or effectiveness can be
found in the GRADE Handbook (Schünemann 2013 ). It is important to use the same language around
judgments (e.g., “ minimal,” “small,” “moderate,” “large”) as included in the framewo rk. The framework
requires that the work group check one box and rationale should be provided for the judgment selected
by the work group. However , any differences in opinions on the judgement selected should be reflected
in the additional Information se ction to ensure transparency.
Criterion 2: Magnitude of undesirable anticipated effects
Criterion question: How substantial are the undesirable anticipated effects?
Related question:
• How substantial is the anticipated effect for each main outcome for which there is an undesirable effect (taking into account the severity or importance of the adverse effects and the number of people affected)?
C
ompletion Guidance: For each of the critical outcomes (as specified in the PICO question and
elaborated in the evidence profiles), evaluate the magnitude of the undesirable anticipated effect. List
the anticipated harms of vaccination, in terms of chosen outcomes and describe how substantial those
effects are. Specific items to consider may include reactogenicity, adverse events, and interactions with other vaccines. Describe any differences in harms for disadvantaged groups or settings. As above, information may be included in the Additional Information section to highlight where evidence may be lacking.
D
iscussi on
W
hen answering this question, work groups should consider the size and strength of the pre -licensure
safety database to sufficiently examine safety outcomes of interest. Groups can also discuss safety
experience with vaccines from the same product cla ss, and any uncertainties related to the safety of the
8
product based on limitations of study design, limitations of size of safety database for rare outcomes,
and duration of follow up. Work groups should consider “risk of harm” (e.g., unknown long -term effects
of new adjuvants, having only clinical trial data vs . long -term real- world experience).
Just
as for the benefits section, it is important to use the same language around judgments (e.g.,
“minimal,” “small,” “moderate,” “large”) as included in the framework. Only one box should be
checked, but differences in opinions should be reflected in the Additional Information section to ensure
transparency.
Similar t
o the previous criterion, work groups should consider the existing safety information for both
the product under consideration, as well as similar products (e.g., vaccines in the same class). Any
uncertainties related to safety should be clearly articulate d and discussed.
Criterion 3: Balance of desirable versus undesirable anticipated effects
Criterion question: Do the desirable effects outweigh the undesirable effects?
Related question:
• What is the balance between the desirable effects relative to the undesirable effects?
Co
mpletion Guidance: The work group should describe the deliberations on this topic, ensuring that any
minority opinions are included. If specific subgroups have a different benefit to harm balance than the general population, describe those considerations here (especially if those groups are not included in
the recommendation on the basis of this altered balance).
Di
scussion
Wh
en the comparator is “no vaccination,” this section is focused on comparing the balance of risks and
benefit s of the same vaccine. Comparison of the adverse events profile of this vaccine to other
vaccine(s) available to prevent the same disease can be addressed in the implementation section (e.g.,
communication with providers and patients to ensure recipients are counseled about severity of adverse events) or in the ‘Additional Considerations ’ section. Since it may be difficult to compare benefits and
risks, as measures are inherently different, judgment must be exercised.
Criterion 4: Certainty of evidence for outcomes
Criterion question: What is the overall certainty of this evidence for the critical outcomes?
Related question:
• What is the overall certainty of this evidence of effects, across all of the outcomes that are critical to making a decision?
Co
mpletion Guidance: For most situations, the overall certainty of the evidence is the lowest certainty of
the critical outcomes. See the ACIP GRADE Handbook regarding the process of making detailed judgments about the quality of evidence or certainty in e stimates of effects and preparation of evidence
profiles. The GRADE evidence profiles and associated documents should be referenced here. If there is uncertainty about whether to prepare evidence profiles, please follow the “When to GRADE” algorithm (Appe ndix 2) and consult the ACIP Secretariat. If evidence profiles are not prepared, a rationale and
summary of any alternative methods should be provided.
9
Discussion
I
t is expected that the overall certainty of the evidence will be informed by the GRADE e vidence profiles
used to inform Criteria 2 and 3 (benefits and harms) . If evidence profiles are not prepared, a rationale
and summary of any alternative methods should be provided.
Domain 3: Values and Preferences
Criterion 1: Target population perception of value
Criterion question: Does the target population feel that the desirable effects are large relative to
undesirable effects?
Related questions:
• How does the target population (i.e., those affected or potentially affected by the disease) v iew
the balance of desirable effects versus undesirable effects?
• Would patients and caregivers feel that the benefits outweigh the harms and burden?
• Does the population appreciate and value the vaccination?
C
ompletion Guidance: Provide information regarding the perspectives and perceptions of potential
recipients about the disease and the vaccine. Describe the source of target population values and
preferences for critical outcomes (e.g., targeted research, questionnaires). Discussion and evidence regarding the possible value to the population due to herd immunity may also be included, where applicable.
D
iscussion
W
ork groups may want to consider the following topics:
• How much less people value outcomes that occur in the future versus outcomes that occur now
• Potential recipient attitudes to undesirable effects (i.e. how risk averse they are)
• Risk perception – whether some segment of the population is inclined to view itself as not at risk
for the disease
I
f the evidence is limited, wo rk group deliberations can be used. If evaluation of values is desirable and
there is sufficient time to conduct research, there are existing survey mechanisms that may be used when funding and space are available. Work groups interested in this should c onsult with the ACIP
Secretariat.
Criterion 2: Uncertainty around target population perception of value
Criterion question: Is there important uncertainty about , or variability in , how much people value the
main outcomes?
Related questions:
• How much do individuals value each of the outcomes in relation to the other outcomes (i.e. ,
what is the relative importance of the outcomes)?
• Is there evidence to support those value judgements, or is there evidence that the variability in
those values is large enough to lead to different decisions?
C
ompletion Guidance: It is not anticipated that there will be a need to routinely gather evidence if none
is currently available; rather, any paucity of information should be indicated here. However, if the
10
variability is estimated to be so significant that this factor alone could lead to a different decision, this
should be indicated and consideration given to obtaining additional evidence. The more likely it is that differences in values would lead to different decisions, the less likely it is that there will be a consensus
that an option is a priority from both an individual and public health perspective and the more
important it is to obtain evidence of the values of those potentially affected by the disease.
Domain 4: Acceptability
Criterion: Acceptability to key stakeholders
Criterion question: Is the option acceptable to key stakeholders?
Related question:
• Are there key stakeholders that would not accept the distribution of benefits, harms , and costs?
• Are there key stakeholders that would not accept the costs or undesirable effects in the short
term for the desirable effects (benefits) in the future?
C
ompletion Guidance: This will often represent the opinion of the work group . Input from ACIP
members may also be obtained through commentary and discussion at meetings and incorporated into
the final framework. Stakeholders may include professional societies, liaison organizations, providers, pharmaceutical companies, advocacy groups, and the general public. Acceptability may vary across
stakeh older groups and any such variability, and the rationale for it , should be captured here. Similar to
other domains, the work group consensus opinion on whether it would be acceptable to the majority of stakeholders will be the basis for the final judgment ; there is not a specified “threshold” to determine
the answer.
D
iscussion
W
ork groups should carefully consider and define which specific stakeholders ( e.g., providers in
healthcare settings, providers in community settings, providers in public health s ettings, healthcare
delivery systems, and the public) are considered in their discussions. Critical stakeholders for each
circumstance may differ and therefore there is no pre -specified list of stakeholders that should always
be considered, rather work gr oups should determine those groups that are relevant for their given
situation. Liaison members on work groups can often provide perspective for their organizations that may be useful in deliberations.
D
omain 5: Resource Use
Criterion: Resource allocation
Criterion question: Is the option a reasonable and efficient allocation of resources?
Related questions:
• What is the cost -effectiveness of the vaccination?
• How does the cost -effectiveness of the vaccination vary in any sensitivity analyses?
• How does the cost -effectiveness change in response to changes in context, assumptions, model
structure, across different studies, etc.?
C
ompletion Guidance: The objective of this section is to describe the available evidence from cost-
effectiveness analyses (CEAs) as well as any major factors that could affect the cost -effectiveness profile
11
of the vaccine. Identify the cost- effectiveness results, including the base- case and a sensitivity range,
from any available cost -effectiveness studies. When possible, indicate which assumptions cause the
greatest change in results by examining the sensitivity analyses of a study. If there are two or more
studies, identify any major differences between the studies. Identify any other important factors that
may affect the cost -effectiveness profile of the vaccination. For example, issues identified in other
domains of the EtR framework regarding the overall certainty of the evidence for critical outcomes may
also be important factors that affect vaccination cost -effectiveness.
D
iscussion
W
ork groups should assemble economic evidence on the study question and the intervention strategies
being evaluated. It may be helpful for the health economic evidence to be assembled in consultation
with the ACIP Economics Lead and/or the NC IRD Lead Economist. The evidence collected can include
analyses conducted by independent researchers, the vaccine industry and CDC economists. The work group lead and the ACIP Secretariat will determine if an internal CDC CEA should be conducted. This w ill
generally be needed for new vaccines and new recommendations with major programmatic economic impact.
T
he findings from the health economic analyses should be reviewed and presented in accordance with
ACIP guidance on economic studies (https://www.cdc.gov/vaccines/acip/committee/guidance/economic -studies.html
). The work group
should consider the quality of the CEAs as described by the Second Panel on Cost -Effect iveness in Health
and Medicine. 4 This may include review of the model design and inputs to determine if the estimates
are reasonable in terms of disease transmission, intervention effects, vaccine costs, implementation costs, disease outcome costs, and c ommunity vs. patient preferences. It may be important to
understand and articulate whether there is variability in local epidemiology, infrastructure, or costs that influence impact of the intervention.
Domain 6: Equity
Criterion: Health equity
Criterion question: What would be the impact on health equity?
Related questions:
• Are there any groups or settings that might be disadvantaged in relation to the problem or options that are considered?
• Are there plausible reasons for anticipating differences in the relative effectiveness of the option for disadvantaged groups or settings?
• Are there different baseline conditions across groups or settings that affect the absolute effectiveness of the option or the importance of the problem for disadvantaged groups or settings?
• Are there important considerations that should be made when implementing the intervention (option) in order to ensure that inequities are reduced, if possible, and that they are not increased?
C
ompletion Guidance: Clinical and p ublic health guidelines need to explicitly consider health equity
(Welch et al, 2017). Health inequities are differences in health considered unfair or unjust and could
have been avoided . This domain facilitates transparent and explicit consideration of t he impact of the
intervention when compared with the alternative option on the target population, specifically to identify if any persons would be disadvantaged as a result of the intervention. The identification of factors that
12
may lead to health inequiti es may lead to modified recommendations that apply to the target
population or separate recommendations tailored to disadvantaged groups. Work groups may conduct a
high -level search to identify any publications reporting on issues of health equity or inequity on the topic
under consideration. Relevant publications may be qualitative or quantitative. If no evidence is
identified, then the work group should provide additional considerations addressing the related questions to the best of their ability.
Doma in 7: Feasibility
Criterion: Implementation feasibility
Criterion question: Is the option feasible to implement?
Related questions:
• Is the intervention sustainable?
• Are there important barriers that are likely to limit the feasibility of implementing the intervention or require consideration when implementing it?
• Is access to the vaccine an important concern?
• Would the vaccine recommendation have any impact on health equity?
• Are there important considerations when implementing the intervention in order to ensure that inequities are reduced, if possible, and that they are not increased?
Co
mpletion Guidance: The Implementation Checklist tool (Appendix 3) summarizes potential barriers to
implementation and can be used to guide these discussions. Impact on health equity is a component that factors into several areas of the tool, but any outstanding concerns can be explicitly described. Information regarding barriers that would be difficult to overcome should be provided. Implementation
issue s are not expected to drive the recommendation, but it is possible that implementation
considerations may change the type of recommendation, influence the wording of the recommendation
or only inform the guidance that accompanies the recommendation. If th ere are specific factors that
influence the ultimate decision regarding the recommendation or its wording, the rationale would be important to include here. As with other areas of the framework, if data are lacking this should be stated. Factors that may impact guidance accompanying the recommendation may be briefly listed
here, with additional information included in the “Additional Considerations” section of the framework.
Conclusions and Additional Considerations
Finally, the conclusions that the work group reaches and any additional considerations that the work
group would like to present regarding the policy question or recommendation are presented in the last
four rows of the EtR framework (Appendix 1).
The t
ype of recommendation and actual text o f the recommendation are based on the judgments made
for all of the criteria and relate directly to the summary “Balance of Consequences” judgement presented in the first row of the conclusions section. The specific wording of each of the three recommenda tion types will stand alone. The three recommendation options include: ACIP recommend s
the intervention; ACIP recommend s the intervention for individuals based on shared clinical decision -
making; and, ACIP does not recommend the intervention ( Even though the Intervention may be used
within FDA licensed indications) .
13
The final row of the EtR framework table provides space to emphasize any additional considerations that
are pertinent to the recommendation, including suggestions for overcoming implementation barriers,
proposed monitoring and evaluation needs, and/or areas requiring additional research to inform future decisions.
Summary of the ACIP ’s deliberation s and final recommendations
A summary of the ACIP discussions regarding the work group recommendations and the final wording
approved should be included in the table ‘Final deliberation and decision by the ACIP .’ This should
include a brief description of the rationale supporting any ACIP modification of, or disagreement with,
the work group recommendation. This section will be completed by the ACIP work group lead and the ACIP work group chair collaboratively.
Additional Resources
1. Alonso -Coello P, Schünemann HJ, Moberg J, Brignardello -Petersen R, Akl EA, Davoli M, Treweek
S, Mustafa RA, Rada G, Rosenbaum S, Morelli A. GRADE Evidence to Decision (EtD) frameworks:
a systematic and transparent approach to making well informed healthcare choices. 1: Introduction. bmj. 2016 Jun 28;353:i2016.
2. Alonso -Coello P, Oxman AD, Moberg J, Brignardello -Petersen R, Akl EA, Davoli M, Treweek S,
Mustafa RA, Vandvik PO, Meerpohl J, Guyatt GH. GRADE Evidence to Decision (EtD) frameworks: a systematic and transparent approach to making well informed healthcare choices. 2: Clinical practice guidelines. bmj. 2016 Jun 30;353:i2089.
3. Ahmed, F., Temte, J.L., Campos -Outcalt, D., Schünemann, H.J. and ACIP Evidence Based
Recommendations Work Group (EBRWG, 2011. Methods for developing evidence-based recommendations by the Advisory Committee on Immunization Practices (ACIP) of the US Centers for Disease Control and Prevention (CDC). Vaccine , 29(49), pp.9171 -9176.
4. Schünemann H, Brożek J, Guyatt G, Oxman A. GRADE handbook for grading quality of evidence and strength of recommendations. Updated October 2013. The GRADE Working Group, 2013.
Available from guidelinedevelopment.org/handbook. 2013.
5. Welch, V. A., Akl, E. A., Guyatt, G., Pottie, K., Eslava -Schmalbach, J., Ansari, M. T., ... &
Hultcrantz, M. (2017). GRADE equity guidelines 1: considering health equity in GRADE guideline
development: introduction and rationale. Journal of clinical epidemiology , 90, 59-67.
14
Appendix 1: ACIP Evidence to Recommendations Framework
Question: Overarching policy question to be answered by the guideline panel (ACIP) using the Evidence to Recommendations (EtR) frame work.
The quest ion shou ld be precise and ident ify the specific interv ention, compar ison, and outcome, as w ell as the tar get popul ation and t he setting
(specific s ubpopul ations) in PICO format.
Popula tion: Target popul ation for vaccine (e.g., a ge range, se x, immune sta tus, pregnancy )
Intervention: Vaccination (if appl icable, dosag e and sc hedul e)
Compa rison(s): No Vaccination/S tandard of care/An existing vaccine/O ther prevention option
Outcome : Outcome(s) associated with vaccination (e.g., prevention outcomes or adverse effects)
Background: The addressed PICO question should be described in detail, and important background information for understanding the questio n
and why a recommendation or decision is needed should be briefly provided. If a recommendation is preferential or represents off-label use,
this should be indicated.
Include sample language: Additional background information supporting the ACIP recommendations on the use of xxx vaccine can be found in
the relevant publication of the recommendation referenced on the ACIP website .
WORK GROUP JUDGMENTS EVIDENCE ADDITIONAL INFORMATION PROBLEMIs the problem of public health importance?
○
○ ○
○
○
○ No
P
robably no
Probably yes
Yes
Varies
Don't know Provide available scientific evidence on
burden of disease, preferably within the
target population for the
recommendation.
I
f no published evidence is available,
provide expert judgment on the public
health priority considerations. Identify any additional public health
priority considerations, including
consideration of disparities.
15 WORK GROUP JUDGMENTS EVIDENCE ADDITIONAL INFORMATION BENEFITS & HARMS How substantial are the desirable
anticipated effects?
○
○
○
○
○
○
○
○○
○
○
○
○
○ ○
○
○ ○ Minimal
Small
Moderate
Large
Varies
Don't know Describe the magnitude of the beneficial
effects of vaccination on individual
(vaccine effectiveness, duration of
protection) and population (herd
immunity) levels. Take into consideration:
Is the baseline benefit similar across
subgroups (by age, sex, pregnancy or
lactation s tatus, occupation [ i.e., healthcare
workers], immune s tatus, r ace, S ES, and
other gr oups)?
Are there indirect ef fects that should b e
considered ( e.g., herd i mmunity)?
How substantial are the undesirable
anticipated effects?
Minimal
S
mall
Moderate
Large
Varies
Don't know Are there undesirable effects of the
vaccine, either on the individual (e.g.,
adverse events following immunization) or population (e .g., age -shift of disease,
serotype replacement) levels? Take into consideration:
Is the baseline risk for harm similar across subgroups (see above)?
S
hould there be separate recommendations
for subgroups based on harms?
Do the desirable effects outweigh the
undesirable effects?
Favors interventionl
F
avors comparison
Favors both
Favors neither
Varies
Don't know Describe the balance of benefits of the
vaccine with possible harms (individual and population level).
16
○
○
○
○
○○○
○
○
○
○
○
○
○
○
○
WORK GROUP JUDGMENTS EVIDENCE ADDITIONAL INFORMATION
What is the overall certainty of this
evidence for the critical outcomes ?
Effectiveness of the intervention
No studies found
4 (very low)
3 (low)
2 (moderate)
1 (high)
Safety of the intervention
No studies found
4 (very low)
3 (low)
2 (moderate)
1 (high) Please refer to GRADE evidence profiles for
detailed assessment of the certainty of the
evidence. For more information, please see the
ACIP Handbook for Developing Evidence -
Based Recommendations. If GRADE was n ot used to evaluate the
certainty of evidence, please provide
justification and the method and outcome of any other tools used to evaluate the body of evidence relevant to the critical outcomes. VALUES Does the target population feel that the
desirable effects are large relative to undesirable effects?
No
P
robably no
Probably yes
Yes
Varies
Don't know Provide any available evidence on target
population values & preferences related to vaccination and comparative health benefits
and risks. Describe the source of these
estimates. Are values and preferences for relevant
outcomes measured? Are the benefits, harms and costs of vaccination valued
differently by different subgroups?
If the target group doesn’t value the intervention, or attributes little value to the
harms and benefits, consider whether potential education measures are needed.
17
WORK GROUP JUDGMENTS EVIDENCE ADDITIONAL INFORMATION
Is there important uncertainty about or
variability in how much people value the
main outcomes?
○
○
○
○
○
○
○
○ ○
○
○
○
○
○
○
○
○Important uncertainty or variability
Probably important uncertainty or variability
Probabl not important uncertainty or variability
No important uncertainty or variability
No known undesireable outcomes Please provide available data used to
determine the relative importance that the target population attributes to the desirable and the undesirable outcomes related to the intervention as well as the comparison. Describe the source of variability, if any.
Are there methods for determining values satisfactory for this recommendation?
If not, systematic assessment of values and preferences of target group may be
considered. ACCEPTABILITY Is the intervention acceptable to key
stakeholders?
No
P
robably no
Probably yes
Yes
Varies
Don't know Provide assessment of whether intervention
would be acceptable to stakeholders
(ethically, programmatically, financially, etc.) RESOURCE USE Is the intervention a reasonable and
efficient allocation of resources?
No
P
robably no
Probably yes
Yes
Varies
Don't know
Provide summary of cost -effectiveness
analyses (CEAs) of the vaccine in the target population. Include base case results and a sensitivity range. Include any other notable findings, for example, specific policy -
relevant scenario s. Overall findings: Summarize the findings
from available CEAs, including major differences in baseline assumptions.
Uncertainty: Does the analysis capture the full range of uncertainty? For example, are
the findings from the uncertainty of
evidence ana lysis, identified earlier in this
document (the EtR Framework), appropriately represented in the methods of the CEAs?
Multiple assessments: Are there multiple CEAs? If so, what are the major differences
in methods and results?
18
WORK GROUP JUDGMENTS EVIDENCE ADDITIONAL INFORMATION EQUITY What would be the impact on health
equity?
○
○
○
○
○
○
○
○
○
○
○
○
○
Reduced
Pr
obably reduced
Probably no impact
Probably increased
Increased
Varies
Don't know Summarize the findings from a review of the
literature addressing issues of health
inequities or groups who may be disadvantaged. Consider from the evidence or guideline
panel:
• Are there any groups or settings that might be disadvantaged in relation to the problem or o ptions that are
considered?
• Are there plausible reasons for anticipating differences in the relative effectiveness of the option for disadvantaged groups or settings?
• Are there different baseline conditions across groups or settings that affect the absolut e effectiveness of the option or
the importance of the problem for disadvantaged groups or settings?
• Are there important considerations
that should be made when implementing the intervention
(option) in order to ensure that
inequities are reduced, if possible,
and that they are not increased? FEASIBILITY Is the intervention feasible to implement?
No
Pro
bably no
Probably yes
Yes
Varies
Don't know Are there any barriers to implementation?
Please refer to the Implementation
Considerations checklist.
19
Balance of
consequences Undesirable
consequences
clearly outweigh
desirable
consequences
in most settings Undesirable
consequences
probably outweigh
desirable
consequences
in most settings The balance
between
desirable and
undesirable
consequences
is closely balanced
or uncertain
Desirable
consequences
probably outweigh
undesirable
consequences
in most settings Desirable
consequences
clearly outweigh
undesirable
consequences
in most s ettings There is
insufficient
evidence to
determine the
balance of
consequences
○ ○ ○ ○ ○ ○
Is there sufficient information to move forward with a recommendation?
Yes ○ No ○
○ ○ ○ Policy options for
ACIP consideration ACIP does not recommend the
i
ntervention*
*Intervention may be used within
FDA licensed indications
ACIP recommends the intervention
for individuals based on shared
clinical decision -making
ACIP recommends the intervention
Draft
recommendation
(text) Please provide the draft recommendations proposed to ACIP.
Additional
considerations
(optional) Please outline any significant additional considerations (e.g., aspects related to implementation, monitoring and
evaluation, research priorities, etc.).
Fina
l deliberation and decision by the ACIP
20
Final ACIP
recommendation
○ ○ ACIP does not recommend the
intervention *
*Intervention m ay be used within FDA
licensed indications ACIP recommends the intervention for
in
dividuals based on shared clinical
decision -making
○ ACIP recommends the intervention
Additional ACIP
considerations
Wording as accepted in the guide
21
Appendix 2 When to Prepare GRADE Evidence Profile Algorithm
22
Appendix 3 Implementation Considerations Checklist