Program Evaluation
Strategies for Assessing How Information Dissemination Contributes to Agency Goals Gao ID: GAO-02-923 September 30, 2002Federal agencies are increasingly expected to focus on achieving results and to demonstrate, in annual performance reports and budget requests, how their activities will help achieve agency or governmentwide goals. Assessing a program's impact or benefit is often difficult, but the dissemination programs GAO reviewed faced a number of evaluation challenges--either individually or in common. The breadth and flexibility of some of the programs made it difficult to measure national progress toward common goals. The programs had limited opportunity to see whether desired behavior changes occurred because change was expected after people made contact with the program, when they returned home or to work. The five programs GAO reviewed addressed these challenges with a variety of strategies, assessing program effects primarily on short-term and intermediate outcomes. Two flexible programs developed common measures to conduct nationwide evaluations; two others encouraged communities to tailor local evaluations to their own goals. Congressional interest was key to initiating most of these evaluations; collaboration with program partners, previous research, and evaluation expertise helped carry them out. Congressional concern about program effectiveness spurred two formal evaluation mandates and other program activities. Collaborations helped ensure that an evaluation would meet the needs of diverse stakeholders.
GAO-02-923, Program Evaluation: Strategies for Assessing How Information Dissemination Contributes to Agency Goals
This is the accessible text file for GAO report number GAO-02-923
entitled 'Program Evaluation: Strategies for Assessing How Information
Dissemination Contributes to Agency Goals' which was released on
September 30, 2002.
This text file was formatted by the U.S. General Accounting Office
(GAO) to be accessible to users with visual impairments, as part of a
longer term project to improve GAO products' accessibility. Every
attempt has been made to maintain the structural and data integrity of
the original printed product. Accessibility features, such as text
descriptions of tables, consecutively numbered footnotes placed at the
end of the file, and the text of agency comment letters, are provided
but may not exactly duplicate the presentation or format of the printed
version. The portable document format (PDF) file is an exact electronic
replica of the printed version. We welcome your feedback. Please E-mail
your comments regarding the contents or accessibility features of this
document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
United States General Accounting Office:
GAO:
Report to Congressional Committees:
September 2002:
Program Evaluation:
Strategies for Assessing How Information Dissemination Contributes to
Agency Goals:
GAO-02-923:
Contents:
Letter:
Results in Brief:
Background:
Scope and Methodology:
Case Descriptions:
Program Flexibility, Delayed Effects, and External Influences Posed
Major Evaluation Challenges:
Surveys and Logic Models Helped Address Most Challenges, but External
Factors Were Rarely Addressed:
Congressional Interest, Collaboration, Available Information and
Expertise Supported These Evaluations:
Observations:
Agency Comments:
Bibliography:
Case Evaluations and Guidance:
Other Evaluation Guidance and Tools:
Related GAO Products:
Table:
Table 1: The Programs‘ Challenges and Their Strategies:
Figures:
Figure 1: Information Dissemination Program Logic Model:
Figure 2: University of Wisconsin Cooperative Extension Logic Model:
Figure 3: Logic Model for the National Youth Anti-Drug Media Campaign
Evaluation:
Figure 4: CDC Tobacco Use Prevention and Control Logic Model:
Abbreviations:
CDC: Centers for Disease Control and Prevention:
CSREES: Cooperative State Research, Education, and Extension Service:
EFNEP: Expanded Food and Nutrition Education Program:
EPA: Environmental Protection Agency:
ETS: environmental tobacco smoke:
HHS: Department of Health and Human Services:
NIDA: National Institute on Drug Abuse:
NSPY: National Survey of Parents and Youth:
OECA: Office of Enforcement and Compliance Assurance:
OMB: Office of Management and Budget:
ONDCP: Office of National Drug Control Policy:
USDA: U.S. Department of Agriculture:
[End of section]
United States General Accounting Office:
Washington, DC 20548:
September 30, 2002:
The Honorable Fred Thompson:
Ranking Minority Member:
Committee on Governmental Affairs:
United States Senate:
The Honorable Stephen Horn:
Chairman:
The Honorable Janice D. Schakowsky:
Ranking Minority Member:
Subcommittee on Government Efficiency, Financial Management, and
Intergovernmental Relations:
Committee on Government Reform:
House of Representatives:
Federal agencies are increasingly expected to focus on achieving results
and to demonstrate, in annual performance reports and budget requests,
how their activities will help achieve agency or governmentwide goals.
We have noted that agencies have had difficulty explaining in their
performance reports how their programs and activities represent
strategies for achieving their annual performance goals. Agencies use
information dissemination programs as one of several tools to achieve
various social or environmental goals. In programs in which agencies do
not act directly to achieve their goals, but inform and persuade others
to act to achieve a desired outcome, it would seem all the more
important to assure decision makers that this strategy is credible and
likely to succeed. Various agencies, however, fail to show how
disseminating information has contributed, or will contribute, to
achieving their outcome-oriented goals.
To assist agency efforts to evaluate and improve the effectiveness of
such programs, we examined evaluations of five federal information
dissemination program cases: Environmental Protection Agency (EPA)
Compliance Assistance, the Eisenhower Professional Development
Program, the Expanded Food and Nutrition Education Program (EFNEP),
the National Tobacco Control Program, and the National Youth Anti-Drug
Media Campaign. We identified useful evaluation strategies that other
agencies might adopt. In this report, prepared under our own
initiative, we discuss the strategies by which these five cases
addressed their evaluation challenges. We are addressing this report to
you because of your interest in encouraging results-based management.
To identify the five cases, we reviewed agency and program documents
and evaluation studies. We selected these five cases because of their
diverse methods: two media campaigns were aimed at health outcomes,
and three programs provided assistance or instruction aimed at
environmental, educational, and health outcomes. We reviewed agency
evaluation studies and guidance and interviewed agency officials to
identify (1) the evaluation challenges these programs faced, (2) their
evaluation strategies to address those challenges, and (3) the
resources or circumstances that were important in conducting these
evaluations.
Results in Brief:
Assessing a program‘s impact or benefit is often difficult, but the
dissemination programs we reviewed faced a number of evaluation
challenges”either individually or in common. The breadth and flexibility
of some of the programs made it difficult to measure national progress
toward common goals. The programs had limited opportunity to see
whether desired behavior changes occurred because change was expected
after people made contact with the program, when they returned home or
to work. Asking participants to report on their own attitude or behavior
changes can produce false or misleading information. Most importantly,
long-term environmental, health, or other social outcomes take time to
develop, and it is difficult to isolate a program‘s effect from other
influences.
The five programs we reviewed addressed these challenges with a variety
of strategies, assessing program effects primarily on short-term and
intermediate outcomes. Two flexible programs developed common
measures to conduct nationwide evaluations; two others encouraged
communities to tailor local evaluations to their own goals. Agencies
conducted special surveys to identify audience reaction to the media
campaigns or to assess changes in knowledge, attitudes, and behavior
following instruction. Articulating the logic of their programs helped
them identify expected short-term, intermediate, and long-term outcomes
and how to measure them. However, only EPA developed an approach for
measuring the environmental outcomes of desired behavior changes. Most
of the programs we reviewed assumed that program exposure or
participation was responsible for observed behavioral changes and failed
to address the influence of external factors. The National Youth Anti-
Drug Media Campaign evaluation used statistical controls to limit the
influence of other factors on its desired outcomes.
Congressional interest was key to initiating most of these evaluations;
collaboration with program partners, previous research, and evaluation
expertise helped carry them out. Congressional concern about program
effectiveness spurred two formal evaluation mandates and other program
assessment activities. Collaborations helped ensure that an evaluation
would meet the needs of diverse stakeholders. Officials used existing
research to design program strategies and establish links to agency
goals. Agency evaluation expertise and logic models guided several
evaluations in articulating program strategy and expected outcomes.
Other agencies could benefit from following the evaluation strategies
we describe in this report when they evaluate their information
campaigns.
Background:
Federal agencies are increasingly expected to demonstrate how their
activities contribute to achieving agency or governmentwide goals. The
Government Performance and Results Act of 1993 requires federal
agencies to report annually on their progress in achieving their agency
and program goals. In spring 2002, the Office of Management and Budget
(OMB) launched an effort as part of the President‘s Budget and
Performance Integration Management Initiative to highlight what is known
about program results. Formal effectiveness ratings for 20 percent of
federal programs will initially be conducted under the executive budget
formulation process for fiscal year 2004. However, agencies have had
difficulty assessing outcomes that are not quickly achieved or readily
observed or over which they have little control.
One type of program whose effectiveness is difficult to assess attempts
to achieve social or environmental outcomes by informing or persuading
others to take actions that are believed to lead to those outcomes.
Examples are media campaigns to encourage health-promoting behavior
and instruction in adopting practices to reduce environmental pollution.
Their effectiveness can be difficult to evaluate because their success
depends on the effectiveness of several steps that entail changing
knowledge, awareness, and individual behavior that result in changed
health conditions or environmental conditions. These programs are
expected to achieve their goals in the following ways:
* The program will provide information about a particular problem, why
it is important, and how the audience can act to prevent or mitigate
it.
* The audience hears the message, gains knowledge, and changes its
attitude about the problem and the need to act.
* The audience changes its behavior and adopts more effective or
healthful practices.
* The changed behavior leads to improved social, health, or
environmental outcomes for the audience individually and, in the
aggregate, for the population or system.
How this process can work is viewed from different perspectives. Viewed
as persuasive communication, the characteristics of the person who
presents the message, the message itself, and the way it is conveyed are
expected to influence how the audience responds to and accepts the
message. Another perspective sees the targeting of audience beliefs as
important factors in motivating change. Still another perspective sees
behavior change as a series of steps”increasing awareness,
contemplating change, forming an intention to change, actually changing,
and maintaining changed behavior. Some programs assume the need for
some of but not all these steps and assume that behavior change is not a
linear or sequential process. Thus, programs operate differently,
reflecting different assumptions about what fosters or impedes the
desired outcome or desired behavior change. Some programs, for example,
combine information activities with regulatory enforcement or other
activities to address factors that are deemed critical to enabling
change or reinforcing the program‘s message.
A program logic model is an evaluation tool used to describe a program‘s
components and desired results and explain the strategy”or logic”by
which the program is expected to achieve its goals. By specifying the
program‘s theory of what is expected at each step, a logic model can
help evaluators define measures of the program‘s progress toward its
ultimate goals. Figure 1 is a simplified logic model for two types of
generic information dissemination programs.
Figure 1: Information Dissemination Program Logic Model:
[See PDF for image]
This figure illustrates the Information Dissemination Program Logic
Model in the following manner:
Inputs:
Staff;
Equipment;
Materials;
Partnerships;
Time.
Activities:
Media campaign: Broadcast TV or radio advertisements to the targeted
population;
Instruction: Inform or train interested parties (e.g., hold workshops,
answer calls for assistance, distribute brochures);
External factors: Other environmental influences on program operations
or results.
Outputs:
Number of people reached;
Number of activities completed (e.g., advertisements run, calls
answered, brochures distributed);
External factors: Other environmental influences on program operations
or results.
Outcomes, short-term:
Audience familiarity with advertisements;
Change in audience or participants' knowledge, awareness, attitudes,
skills, or intent to change;
External factors: Other environmental influences on program operations
or results.
Outcomes, Intermediate:
Change in audience or participants' behavior (e.g., reduced smoking
initiation among youth);
Adoption of suggested practices by participants or facilities;
External factors: Other environmental influences on program operations
or results.
Outcomes, Long-term:
Change in targeted social, health, or environmental conditions (e.g.,
reduced smoking-related illness);
External factors: Other environmental influences on program operations
or results.
Source: GAO analysis.
[End of figure]
A program evaluation is a systematic study using objective measures to
analyze how well a program is working. An evaluation that examines how
a program was implemented and whether it achieved its short-term and
intermediate results can provide important information about why a
program did or did not succeed on its long-term results. Scientific
research methods can help establish a causal connection between program
activities and outcomes and can isolate the program‘s contribution to
them. Evaluating the effectiveness of information dissemination programs
entails answering several questions about the different stages of the
logic model:
* Short-term outcomes: Did the audience consider the message credible
and worth considering? Were there changes in audience knowledge,
attitudes, and intentions to change behavior?
* Intermediate outcomes: Did the audience‘s behavior change? [Footnote
1]
* Long-term outcomes: Did the desired social, health, or environmental
conditions come about?
Scope and Methodology:
To identify ways that agencies can evaluate how their information
dissemination programs contribute to their goals, we conducted case
studies of how five agencies evaluate their media campaign or
instructional programs. To select the cases, we reviewed departmental
and agency performance plans and reports and evaluation reports. We
selected cases to represent a variety of evaluation approaches and
methods. Four of the cases consisted of individual programs; one
represented an office assisting several programs. We describe all five
cases in the next section.
To identify the analytic challenges that the agencies faced, we reviewed
agency and program materials. We confirmed our understanding with
agency officials and obtained additional information on the
circumstances that led them to conduct their evaluations. Our findings
are limited to the examples reviewed and thus do not necessarily
reflect the full scope of these programs‘ or agencies‘ evaluation
activities.
We conducted our work between October 2001 and July 2002 in accordance
with generally accepted government auditing standards.
We requested comments on a draft of this report from the heads of the
agencies responsible for the five cases. The U.S. Department of
Agriculture (USDA), the Department of Health and Human Services (HHS),
and EPA provided technical comments that we incorporated where
appropriate throughout the report.
Case Descriptions:
We describe the goals, major activities, and evaluation approaches and
methods for the five cases in this section.
EPA Compliance Assistance:
EPA‘s Compliance Assistance Program disseminates industry-specific and
statute-specific information to entities that request it to help them
gain compliance with EPA‘s regulations and thus improve environmental
performance. Overseen and implemented by the Office of Enforcement
and Compliance Assurance (OECA) and regional offices, compliance
assistance consists of telephone help lines, self-audit checklists,
written guides, expert systems, workshops, and site visits of regulated
industries. OECA provides regional offices with evaluation guidance
that illustrates how postsession surveys and administrative data can be
used to assess changes in knowledge or awareness of relevant
regulations or statutes and adoption of practices. EPA encourages the
evaluation of local projects to measure their contribution to achieving
the agency‘s environmental goals.
Eisenhower Professional Development Program:
In the U.S. Department of Education, the Eisenhower Professional
Development Program supports instructional activities to improve the
quality of elementary and secondary school teaching and, ultimately,
student learning and achievement. Part of school reform efforts, the
program aims to provide primarily mathematics and science teachers with
skills and knowledge to help students meet challenging educational
standards. Program funds are used nationwide for flexible professional
development activities to address local needs related to teaching
practices, curriculum, and student learning styles. The national
evaluation conducted a national survey of program coordinators and
participating teachers to characterize the range of program strategies
and the quality of program-assisted activities. The evaluation also
collected detailed data at three points in time from all mathematics
and science teachers in 10 sites to assess program effects on teachers‘
knowledge and teaching practices.
Expanded Food and Nutrition Education Program and Other Cooperative
Extension Programs:
USDA‘s Cooperative State Research, Education, and Extension Service
(CSREES) conducts EFNEP in partnership with the Cooperative Extension
System, a network of educators in land grant universities and county
offices. EFNEP is an educational program on food safety, food
budgeting, and nutrition to assist low-income families acquire
knowledge, skills, and changed behavior necessary to develop
nutritionally sound diets and improve the total family diet and
nutritional well-being. County extension educators train and supervise
paraprofessionals and volunteers, who teach the curriculum of about 10
sessions. EFNEP programs across the country measure participants‘
nutrition-related behavior at program entry and exit on common
instruments and report the data to USDA through a common reporting
system. In addition, the Cooperative Extension System conducts a
variety of other educational programs to improve agriculture and
communities and strengthen families. State cooperative extension staff
developed and provided evaluation guidance, supported in part by
CSREES, to encourage local cooperative extension projects to assess,
monitor, and report on performance. Evaluation guidance, including
examples of surveys, was provided in seminars and on Web sites to help
extension educators evaluate their workshops and their brochures in the
full range of topics, such as crop management and food safety.
National Tobacco Control Program:
In HHS, the Centers for Disease Control and Prevention (CDC) aims to
reduce youths‘ tobacco use by funding state control programs and
encouraging states to use multiple program interventions, working
together in a comprehensive approach. CDC supports various efforts,
including media campaigns to change youths‘ attitudes and social norms
toward tobacco and to prevent the initiation of smoking. Florida, for
example, developed its own counter-advertising, anti-tobacco mass media
’truth“ campaign. CDC supports the evaluation of local media programs
through funding and technical assistance and with state-based and
national youth tobacco surveys that provide tobacco use data from
representative samples of students. CDC also provides general evaluation
guidance for grantee programs to assess advertisement awareness,
knowledge, attitudes, and behavior.
National Youth Anti-Drug Media Campaign:
The Office of National Drug Control Policy (ONDCP) in the Executive
Office of the President oversees the National Youth Anti-Drug Media
Campaign, which aims to educate and enable youths to reject illegal
drugs. This part of the nation‘s drug control strategy uses a media
campaign to counteract images that are perceived as glamorizing or
condoning drug use and to encourage parents to discuss drug abuse with
their children.
The media campaign, among other activities, consists of broadcasting
paid advertisements and public service announcements that support good
parenting practices and discourage drug abuse. While ONDCP oversees
the campaign in conjunction with media and drug abuse experts,
advertising firms and nonprofit organizations develop the
advertisements, which are broadcast to the target audience several
times a week for several weeks or months across various media (TV,
radio, newspapers, magazines, and billboards) at multiple sites
nationwide. The ongoing national evaluation is being conducted by a
contractor under the direction of the National Institute on Drug Abuse
(NIDA). The evaluation surveys households in the target markets to
assess advertisement awareness, knowledge, attitudes, and behavior,
including drug use, in a representative sample of youths and their
parents or other caretakers.
Program Flexibility, Delayed Effects, and External Influences Posed
Major Evaluation Challenges:
The programs we reviewed faced challenges to evaluating effects at each
step, from conveying information to achieving social and environmental
goals. Specifically:
* Flexible programs were hard to summarize nationally as they varied
their activities, message, and goals to meet local needs.
* Mass media campaigns do not readily know whether their targeted
audience heard the program‘s message.
* Intended changes in knowledge, attitude, and behavior did not
necessarily take place until after audience contact with the program
and were, therefore, difficult to observe.
* Self-reports of knowledge, attitudes, and behavior can be prone to
bias.
* Long-term behavioral changes and environmental, health, or other
social outcomes can take a long time to develop.
* Many factors aside from the program are expected to contribute to the
desired behavioral changes and long-term outcomes.
Local Program Variability Makes Nationwide Evaluation Difficult:
Several programs we reviewed have broad, general goals and delegated to
state or local agencies the authority to determine how to carry out the
programs to meet specific local needs. For two reasons, the resulting
variability in activities and goals across communities constrained the
federal agencies‘ ability to construct national evaluations of the
programs.
First, when states and localities set their own short-term and
intermediate goals, common measures to aggregate across projects are
often lacking, so it is difficult to assess national progress toward a
common goal. Second, these programs also tended to have limited federal
reporting requirements. Thus, little information was available on how
well a national program was progressing toward national goals.
The Eisenhower Professional Development Program, National Tobacco
Control Program, EPA‘s Compliance Assistance, and CSREES provide
financial assistance to states or regional offices with limited federal
direction on activities or goals. Many decisions about who receives
services and what services they receive are made largely at the
regional, county, or school district levels. For example, in the
Eisenhower Professional Development Program, districts select
professional development activities to support their school reform
efforts, including alignment with state and local academic goals and
standards. These standards vary, some districts having more challenging
standards than others. In addition, training may take various forms;
participation in a 2-hour workshop is not comparable to involvement in
an intensive study group or year-long course. Such differences in short-
term goals, duration, and intensity make counting participating
teachers an inadequate way to portray the national program. Such
flexibility enables responsiveness to local conditions but reduces the
availability of common measures to depict a program in its entirety.
These programs also had limited federal reporting requirements.
Cooperative extension and regional EPA offices are asked to report
monitoring data on the number of workshops held and clients served, for
example, but only selected information on results. The local extension
offices are asked to periodically report to state offices monitoring
data and accomplishments that support state-defined goals. The state
offices, in turn, report to the federal office summary data on their
progress in addressing state goals and how they fit into USDA‘s
national goals. The federal program may hold the state and local
offices accountable for meeting their state‘s needs but may have little
summary information on progress toward achieving USDA‘s national goals.
Media Campaigns Lack Interaction with Their Audience:
Media campaigns base the selection of message, format, and frequency of
broadcast advertisements on audience analysis to obtain access to a
desired population. However, a campaign has no direct way of learning
whether it has actually reached its intended audience. The mass media
campaigns ONDCP and CDC supported had no personal contact with their
youth audiences while they received messages from local radio, TV, and
billboard advertisers. ONDCP campaign funds were used to purchase media
time and space for advertisements that were expected to deliver two to
three anti-drug messages a week using various types of media to the
average youth or parent. However, the campaign did not automatically
know what portions of the audience heard or paid any attention to the
advertisements or, especially, changed their attitudes as a result of
the advertisements.
Changes in Behavior Take Place at Home or Work:
The instructional programs had the opportunity to interact with their
audience and assess their knowledge, skills, and attitudes through
questionnaires or observation. However, while knowledge and attitudes
may change during a seminar, most desired behavior change is expected
to take place when the people attending the seminar return home or to
their jobs. Few of these programs had extended contact with their
participants to observe such effects directly. In the Eisenhower
program, a teacher can learn and report an intention to adopt a new
teaching practice, but this does not ensure that the teacher will
actually use it in class.
Participants‘ Self-Reports May Produce Poor-Quality Data:
End-of-session surveys asking for self-reports of participants‘
knowledge, attitudes, and intended behavior are fast and convenient
ways to gain information but can produce data of poor quality. This can
lead to a false assessment of a workshop‘s impact. Respondents may not
be willing to admit to others that they engage in socially sensitive or
stigmatizing activities like smoking or drug use. They may not trust
that their responses will be kept confidential. In addition, they may
choose to give what they believe to be socially desirable or acceptable
answers in order to appear to be doing the ’right thing.“ When surveys
ask how participants will use their learning, participants may feel
pressured to give a positive but not necessarily truthful report.
Participants may also report that they ’understand“ the workshop
information and its message but may not be qualified to judge their own
level of knowledge.
Outcomes Take Time to Develop:
Assessing a program‘s intermediate behavioral outcomes, such as
smoking, or long-term outcomes, such as improved health status, is
hindered by the time they take to develop. To evaluate efforts to
prevent youths from starting to smoke, evaluators need to wait several
years to observe evidence of the expected outcome. ONDCP expects its
media campaign to take about 2 to 3 years to affect drug use. Many
population-based health effects take years to become apparent, far
beyond the reach of these programs to study.
Tracking participants over several years can be difficult and costly.
Even after making special efforts to locate people who have moved, each
year a few more people from the original sample may not be reached or
may refuse to cooperate. In the Eisenhower evaluation, 50 percent of
the initial sample (60 percent of teachers remaining in the schools)
responded to all three surveys. When a sample is tracked for several
years, the cumulative loss of respondents may eventually yield such a
small proportion of the original sample as not to accurately represent
that original sample. Moreover, the proportion affected tends to
diminish at each step of the program logic model, which can reduce the
size of the expected effect on long-term outcomes so small as to be
undetectable. That is, if the program reached half the targeted
audience, changed attitudes among half of those it reached, half of
those people changed their behavior, and half of those experienced
improved health outcomes, then only one-sixteenth of the initial target
audience would be expected to experience the desired health outcome.
Thus, programs may be unlikely to invest in tracking the very large
samples required to detect an effect on their ultimate outcome.
Other Factors Influence Desired Outcomes:
Attributing observed changes in participants to the effect of a program
requires ruling out other plausible explanations. Those who volunteer to
attend a workshop are likely to be more interested, knowledgeable, or
willing to change their behavior than others who do not volunteer.
Environmental factors such as trends in community attitudes toward
smoking could explain changes in youths‘ smoking rates. ONDCP planners
have recognized that sensation seeking among youths is associated with
willingness to take social or physical risks; high-sensation seekers are
more likely to be early users of illegal drugs. Program participants‘
maturing could also explain reductions in risky behavior over time.
Other programs funded with private or other federal money may also
strive for similar goals, making it difficult to separate out the
information program‘s unique contribution. The American Legacy
Foundation, established by the 1998 tobacco settlement, conducted a
national media campaign to discourage youths from smoking while Florida
was carrying out its ’truth“ campaign. Similarly, the Eisenhower
program is just one of many funding sources for teacher development,
but it is the federal government‘s largest investment solely in
developing the knowledge and skills of classroom teachers. The National
Science Foundation also funds professional development initiatives in
mathematics and science. The evaluation found that local grantees
combine Eisenhower grants with other funds to pay for conferences and
workshops.
Surveys and Logic Models Helped Address Most Challenges, but External
Factors Were Rarely Addressed:
The agencies we reviewed used a variety of strategies to address their
evaluation challenges. Two flexible programs developed common, national
measures, while two others promoted locally tailored evaluations. Most
programs used exit or follow-up surveys to gather data on short-term and
intermediate outcomes. Articulating a logic model for their programs
helped some identify appropriate measures and strategies to address
their challenges. Only EPA developed an approach for measuring its
program‘s long-term health and environmental outcomes or benefits. Most
of the programs we reviewed assumed that program exposure or
participation was responsible for observed changes and failed to
address the role of external factors. However, the NIDA evaluation did
use evaluation techniques to limit the influence of nonprogram factors.
Table 1 displays the strategies the five cases used or recommended in
guidance to address the challenges.
Table 1: The Programs‘ Challenges and Their Strategies:
Challenge: Flexible programs were hard to summarize nationally as they
varied their activities, messages, and goals to meet local needs;
Strategy:
* Develop common measures for national program evaluation;
* Encourage local projects to evaluate progress toward their
own goals.
Challenge: Mass media campaigns do not readily know whether their target
audience heard the program‘s message;
Strategy:
* Survey intended audience to ask about program exposure,
knowledge and attitude change.
Challenge: Intended changes in knowledge, attitude, and behavior might
not take place until after contact with the program and were thus
difficult to observe;
Strategy:
* Conduct postworkshop survey or follow-up surveys;
* Conduct observations;
* Use existing administrative or site visit data.
Challenge: Self-report surveys of knowledge, attitudes, or behavior can
be prone to bias;
Strategy:
* Adjust wording of survey questions;
* Ensure confidentiality of survey and its results;
* Compare before-and-after reports to assess change.
Challenge: Long-term behavioral changes and environmental, health, or
other social outcomes can take a long time to develop.
Strategy:
* Assess intermediate outcomes;
* Use logic model to demonstrate links to agency goals;
* Conduct follow-up survey.
Challenge: Many factors aside from the program are expected to
contribute to the desired behavioral changes and long-term outcomes;
Strategy:
* Select outcomes closely associated with the program;
* Use statistical methods to limit external influences;
* Evaluate the combined effect of related activities rather than
trying to limit their influences.
Source: GAO‘s analysis.
[End of table]
Find Common Measures or Encourage Locally Tailored Evaluations:
Two of the four flexible programs developed ways to assess progress
toward national program goals, while the others encouraged local
programs to conduct their own evaluations, tailored to local program
goals.
EFNEP does not have a standard national curriculum, but local programs
share common activities aimed at the same broad goals. A national
committee of EFNEP educators developed a behavior checklist and food
recall log to provide common measures of client knowledge and adoption
of improved nutrition-related practices, which state and local offices
may choose to adopt. The national program office provided state and
local offices with software to record and analyze client data on these
measures and produce tailored federal and state reports. In contrast,
lacking standard reporting on program activities or client outcomes, the
Eisenhower program had to conduct a special evaluation study to obtain
such data. The evaluation contractor surveyed the state program
coordinators to learn what types of training activities teachers were
enrolled in and surveyed teachers to learn about their training
experiences and practices. The evaluation contractor drew on
characteristics identified with high-quality instruction in the
research literature to define measures of quality for this study.
In contrast, EPA and CDC developed guidance on how to plan and
conduct program evaluations and encouraged state and local offices to
assess their own individual efforts. To measure the effects of EPA‘s
enforcement and compliance assurance activities, the agency developed a
performance profile of 11 sets of performance measures to assess the
activities undertaken (including inspections and enforcement, as well as
compliance assistance), changes in the behavior of regulated entities,
and progress toward achieving environmental and health objectives. One
set of measures targets the environmental or health effects of
compliance assistance that must be further specified to apply to the
type of assistance and relevant industry or sector. However, EPA notes
that since the measured outcomes are very specific to the assistance
tool or initiative, aggregating them nationally will be difficult.
Instead, EPA encourages reporting the outcomes as a set of quantitative
or qualitative accomplishments.
In CDC‘s National Tobacco Control Program, states may choose to
conduct any of a variety of activities, such as health promotions,
clinical management of nicotine addiction, advice and counseling, or
enforcing regulations limiting the access minors have to tobacco. With
such intentional flexibility and diversity, it is often difficult to
characterize or summarize the effectiveness of the national program.
Instead, CDC conducted national and multistate surveillance, providing
both baseline and trend data on youths‘ tobacco use, and encouraged
states to evaluate their own programs, including surveying the target
audience‘s awareness and reactions. CDC‘s ’how to“ guide assists
program managers and staff in planning and implementing evaluation by
providing general evaluation guidance that includes example
outcomes”short term, intermediate, and long term”and data sources for
various program activities or interventions. [Footnote 2]
Survey the Population Targeted by the Media Campaign:
Both mass media campaigns surveyed their intended audience to learn how
many heard or responded to the message and, thus, whether the first
step of the program was successful. Such surveys, a common data source
for media campaigns, involved carefully identifying the intended
audience, selecting the survey sample, and developing the questionnaire
to assess the intended effects.
The National Youth Anti-Drug Media Campaign is designed to discourage
youths from beginning to use drugs by posting advertisements that aim to
change their attitudes about drugs and encourage parents to help prevent
their children from using drugs. Thus, the NIDA evaluation developed a
special survey, the National Survey of Parents and Youth (NSPY), with
parallel forms to address questions about program exposure and effects
on both groups. At the time of our interview, NSPY had fielded three
waves of interviews to assess initial and cumulative responses to the
campaign but planned additional follow-up. Cross-sectional samples of
youths and parents (or caregivers) were drawn to be nationally
representative and produce equal-sized samples within three age
subgroups of particular interest (youths aged 9–11, 12–13, and 14–18).
Separate questionnaires for youths and parents measured their exposure
to both specific advertisements and, more generally, the campaign and
other noncampaign anti-drug messages. In addition, they were asked
about their beliefs, attitudes, and behavior regarding drug use and
factors known to be related to drug use (for youths) or their
interactions with their children (for parents).
Florida‘s tobacco control program integrated an advertisement campaign
to counter the tobacco industry‘s marketing with community involvement,
education, and enforcement activities. The campaign disseminates its
message about tobacco industry advertising through billboards and
broadcasting and by distributing print media and consumer products (such
as hats and T-shirts) at events for teenagers. Florida‘s Anti-tobacco
Media Evaluation surveys have been conducted every 6 months since the
program‘s inception in 1998 to track awareness of the campaign as well
as youths‘ anti-tobacco attitudes, beliefs, and smoking behavior.
Assess Postworkshop Changes with Surveys and Observations:
Most of the instructional programs we reviewed assessed participants‘
short-term changes in knowledge, attitudes, or skills at the end of
their session and relied on follow-up surveys to learn about
intermediate effects that took place later. EFNEP and EPA‘s Compliance
Assistance, which had more extended contact with participants, were
able to collect more direct information on intermediate behavioral
effects.
State cooperative extension and EPA evaluation guidance encouraged
program staff to get immediate feedback on educational workshops,
seminars, and hands-on demonstrations and their results. Reference
materials suggested that postworkshop surveys ask what people think
they gained or intend to do as a result of the program sessions.
[Footnote 3] Questions may ask about benefits in general or perceived
changes in specific knowledge, skills, attitudes, or intended actions.
These surveys can show postprogram changes in knowledge and attitudes
but not whether the participants actually changed their behavior or
adopted the recommended practices. An extension evaluator said that
this is the typical source of evaluation data for some types of
extension programs.
Cooperative extension evaluations have also used other types of on-site
data collection, such as observation during workshops to document how
well participants understood and can use what was taught. [Footnote 4]
The traditional paper-and-pencil survey may be less effective with
children or other audiences with little literacy, so other sources of
data are needed. Program or evaluation staff can observe (directly or
from documents) the use of skills learned in a workshop”for example, a
mother‘s explaining to another nonparticipating mother about the need
to wash hands before food preparation. Staff can ask participants to
role-play a scenario”for example, an 8-year-old‘s saying ’no“ to a
cigarette offered by a friend. These observations could provide
evidence of knowledge, understanding of the skills taught, and ability
to act on the message. [Footnote 5] While these data may be considered
more accurate indicators of knowledge and skill gains than self-report
surveys, they are more resource-intensive to collect and analyze.
Most of the programs we reviewed expected the desired behavior change”
the intermediate outcome”to take place later, after participants
returned home or to their jobs. EFNEP is unusual in using surveys to
measure behavior change at the end of the program. This is possible
because (1) the program collects detailed information on diet,
budgeting, and food handling from participants at the start and end of
the program and (2) its series of 10 to 12 lessons is long enough to
expect to see such changes.
Programs that did not expect behavior to change until later or at work
used follow-up surveys to identify actual change in behavior or the
adoption of suggested practices. Cooperative extension and EPA‘s
Compliance Assistance evaluation guidance encouraged local evaluators
to send a survey several weeks or months later, when participants are
likely to have made behavior changes. Surveys may be conducted by mail,
telephone, or online, depending on what appears to be the best way to
reach potential respondents. An online survey of Web site visitors, for
example, can potentially reach a larger number of respondents than may
be known to the program or evaluator. EPA recommended that the form of
evaluation follow-up match the form and intensity of the intervention,
such as conducting a periodic survey of a sample of those who seek
assistance of a telephone help-desk rather than following up each
contact with an extensive survey. EPA and ONDCP officials noted that
survey planning must accommodate a review by the Office of Management
and Budget to ascertain whether agency proposals for collecting
information comply with the Paperwork Reduction Act. [Footnote 6]
EPA guidance encouraged evaluators to obtain administrative data on
desired behavior changes rather than depending on less-reliable self-
report survey data. Evidence of compliance can come from observations
during follow-up visits to facilities that had received on-site
compliance assistance or from tracking data that the audience may be
required to report for regulatory enforcement purposes. For example,
after a workshop for dry cleaners about the permits needed to meet air
quality regulations, EPA could examine data on how many of the attendees
applied for such permits within 6 months after the workshop. This
administrative data could be combined with survey results to obtain
responses from many respondents yet collect detailed information from
selected participants.
Adjust Self-Report Surveys to Reduce Potential Bias:
Using a survey at the end of a program session to gain information from
a large number of people is fast and convenient, but self-reports may
provide positively biased responses about the session or socially
sensitive or controversial topics. To counteract these tendencies, the
programs we reviewed used various techniques either to avoid
threatening questions that might elicit a socially desirable but
inaccurate response or to reassure interviewees of the confidentiality
of their responses. In addition, the programs recommended caution in
using self-reports of knowledge or behavior changes, encouraging
evaluators”rather than participants”to assess change.
Carefully wording questions can encourage participants to candidly
record unpopular or negative views and can lessen the likelihood of
their giving socially desirable responses. Cooperative extension
evaluation guidance materials suggest that survey questions ask for
both program strengths and weaknesses or for suggestions on how to
improve the program. These materials also encourage avoidance of value-
laden terms. Questions about potentially embarrassing situations might
be preceded by a statement that acknowledges that this happens to
everyone at some time. [Footnote 7]
To reassure respondents, agencies also used the survey setting and
administration to provide greater privacy in answering the questions.
Evaluation guidance encourages collecting unsigned evaluation forms in a
box at the end of the program, unless, of course, individual follow-up
is desired. Because the National Youth Anti-Drug Media Campaign was
dealing with much more sensitive issues than most surveys, its
evaluation took several steps to reassure respondents and improve the
quality of the data it collected. Agency officials noted that decisions
about survey design and collecting quality data involve numerous issues
such as consent, parental presence, feasibility, mode, and data editing
procedures. In this case, they chose a panel study with linked data
from youths and one parent or guardian collected over three
administrations. In addition, they found that obtaining cooperation
from a representative sample of schools with the frequency required by
the evaluation was not feasible. So the evaluation team chose to survey
households in person instead of interviewing youths at school or
conducting a telephone survey.
Hoping to improve the quality of sensitive responses, the surveyors
promised confidentiality and provided respondents with a certificate of
confidentiality from HHS. In addition, the sensitive questions were
self-administered with a touch-screen laptop computer. All sensitive
questions and answer categories appeared on the laptop screen and were
spoken to the respondent by a recorded voice through earphones.
Respondents chose responses by touching the laptop screen. This audio
computer-assisted self-interview instrument was likely to obtain more
honest answers about drug use, because respondents entered their reports
without their answers being observed by the interviewer or their
parents. NIDA reported that a review of the research literature on
surveys indicated that this method resulted in higher reported rates of
substance abuse for youths, compared to paper-and-pencil
administration.
Compare Presession and Postsession Reports to Assess Change:
State cooperative extension and EPA evaluation guidance cautioned that
self-reports may not reflect actual learning or change; they encouraged
local projects to directly test and compare participant knowledge before
and after an activity rather than asking respondents to report their own
changed behavior. Both the EFNEP and Eisenhower evaluators attempted
to reduce social desirability bias in self-reports of change by asking
for concrete, detailed descriptions of what the respondents did before
and after the program. By asking for a detailed log of what
participants ate the day before, EFNEP sought to obtain relatively
objective information to compare with nutrition guidelines. By
repeating this exercise at the beginning and end of the program, EFNEP
obtained more credible evidence than by asking participants whether
they had adopted desired practices, such as eating less fat and more
fruit and vegetables.
The Eisenhower evaluation also relied on asking about very specific
behaviors to minimize subjectivity and potential bias. First, evaluators
analyzed detailed descriptions of their professional development
activities along characteristics identified as important to quality in
prior research”such as length and level of involvement. Thus, they
avoided asking teachers to judge the quality of their professional
development activities. Second, teachers were surveyed at three points
in time to obtain detailed information on their instructional practices
during three successive school years. Teachers were asked to complete
extensive tables on the content and pedagogy used in their course; then
the evaluators analyzed whether these represented high standards and
effective instructional approaches as identified in the research
literature. The evaluators then compared teacher-reported instructional
practices before and after their professional development training to
assess change on key dimensions of quality.
Some cooperative extension guidance noted that pretest-posttest
comparison of self-report results may not always provide accurate
assessment of program effects, because participants may have limited
knowledge at the beginning of the program that prevents them from
accurately assessing baseline behaviors. For example, before instruction
on the sources of certain vitamins, participants may inaccurately assess
the adequacy of their own consumption levels. The ’post-then-pre“ design
can address this problem by asking participants to report at the end of
the program, when they know more about their behavior, both then and as
it was before the program. Evidently, participants may also be more
willing to admit to certain inappropriate behaviors. [Footnote 8]
Use Program Logic Models to Show Links to Unmeasured Long-Term
Outcomes:
Assessing long-term social or health outcomes that were expected to take
more than 2 to 3 years to develop was beyond the scope of most of these
programs. Only EPA developed an approach for measuring long-term
outcomes, such as the environmental effects of desired behavior change
in cases where they can be seen relatively quickly. In most instances,
programs measured only short-term and intermediate outcomes, which
they claimed would contribute to achieving these ultimate benefits.
Several programs used logic models to demonstrate their case; some drew
on associations established in previous research. The Eisenhower and
NIDA evaluations took special effort to track participants long enough
to observe desired intermediate outcomes.
EFNEP routinely measures intermediate behavioral outcomes of improved
nutritional intake but does not regularly assess long-term outcomes of
nutritional or health status, in part because they can take many years
to develop. Instead, the program relies on the associations established
in medical research between diet and heart disease and certain cancers,
for example, to explain how it expects to contribute to achieving
disease reduction goals. Specifically, Virginia Polytechnic Institute
and State University (Virginia Tech) and Virginia cooperative extension
staff developed a model to conduct a cost-benefit analysis of the
health-promoting benefits of its EFNEP program. The study used equations
estimating the health benefits of the program‘s advocated nutritional
changes for each of 10 nutrition-related diseases (such as colorectal
cancer) from medical consensus reports. The study then used program
data on the number of participants who adopted the whole set of targeted
behaviors to calculate the expected level of benefits, assuming they
maintained the behaviors for 5 years.
EPA provided regional staff with guidance that allows them to estimate
environmental benefits from pollution reduction in specific cases of
improved compliance with EPA‘s regulations. To capture and document
the environmental results and benefits of concluded enforcement cases,
EPA developed a form for regional offices to record their actions taken
and pollutant reductions achieved. The guidance provides steps,
formulas, and look-up tables for calculating pollutant reduction or
elimination for specific industries and types of water, air, or solid
waste regulations. [Footnote 9] EPA regional staff are to measure
average concentrations of pollutants before a specific site becomes
compliant and to calculate the estimated total pollutant reduction in
the first year of post-action compliance. Where specific pollution-
reduction measures can be aggregated across sites, EPA can measure
effects nationally and show the contribution to agencywide pollution-
reduction goals. In part because these effects occur in the short
term, EPA was unique among our cases in having developed an approach
for measuring the effects of behavior change.
Logic models helped cooperative extension programs and the evaluation
of ONDCP‘s media campaign identify their potential long-term effects and
the route through which they would be achieved. The University of
Wisconsin Cooperative Extension guidance encourages the use of logic
models to link investments to results. They aim to help projects clarify
linkages among program components; focus on short-term, intermediate,
and long-term outcomes; and plan appropriate data collection and
analysis. The guidance suggests measuring outcomes over which the
program has a fair amount of control and considering, for any important
long-term outcome, whether it will be attained if the other outcomes are
achieved. Figure 2 depicts a generic logic model for an extension
project, showing how it can be linked to long-term social or
environmental goals.
Figure 2: University of Wisconsin Cooperative Extension Logic Model:
[See PDF for image]
This figure is an illustration of the University of Wisconsin
Cooperative Extension Logic Model, as follows:
Situation:
Inputs:
* What we invest:
- staff;
- volunteers;
- time;
- money;
- materials;
- equipment;
- technology;
- partners.
Outputs:
* Activities (What we do):
- workshops;
- meetings;
- counseling;
- facilitation;
- assessment;
- product development;
- media work;
- recruitment;
- training.
* Participants (Who we reach):
- participants;
- customers;
- citizens.
Outcomes-Impact, Short-term (What the short-term results are):
* Learning:
- awareness;
- knowledge;
- attitudes;
- skills;
- opinions;
- aspirations;
- motivations.
Outcomes-Impact, Medium (What the medium-term results are):
* Action:
- behavior;
- practice;
- decisions;
- policies;
- social action.
Outcomes-Impact, Long-term (What the ultimate impact(s) is):
* Conditions:
- social;
- economic;
- civic;
- environmental.
Environment (influential factors):
Affects:
* Outputs;
* Outcomes.
Source: Adapted from Ellen Taylor-Powell, ’The Logic Model: A Program
Performance Framework,“ University of Wisconsin Cooperative Extension,
Madison, Wisconsin, n.d., [hyperlink, http://www.uwex.edu/ces/pdande]
(September 2002).
[End of figure]
The evaluation of the National Youth Anti-Drug Media Campaign followed
closely the logic of how the program was expected to achieve its desired
outcomes, and its logic models show how the campaign contributes to
ONDCP‘s drug-use reduction goals. For example, the campaign had
specific hypotheses about the multiple steps through which exposure to
the media campaign message would influence attitudes and beliefs, which
would then influence behavior. Thus, evaluation surveys tapped various
elements of youths‘ attitudes and beliefs about drug use and social
norms, as well as behaviors that are hypothesized to be influenced
by”or to mediate the influence of”the campaign‘s message. In addition,
NIDA plans to follow for 2 to 3 years those who had been exposed to the
campaign to learn how the campaign affected their later behavior.
Figure 3 shows the multiple steps in the media campaign‘s expected
influence and how personal factors affect the process.
Figure 3: Logic Model for the National Youth Anti-Drug Media Campaign
Evaluation:
[See PDF for image]
This figure depicts the Logic Model for the National Youth Anti-Drug
Media Campaign Evaluation, as follows:
* Campaign activity (including direct media, community organizing,
parent and peer sources);
* Exposure to anti-drug messages from a variety of sources [Influenced
by External factors: Demographics, prior behavior, family and peer
factors, and personal factors may have direct effects or influence
susceptibility to media campaign effects];
* Beliefs, social expectations, skills, and self-efficacy [Influenced
by External factors: Demographics, prior behavior, family and peer
factors, and personal factors may have direct effects or influence
susceptibility to media campaign effects];
* Intentions to use drugs [Influenced by External factors:
Demographics, prior behavior, family and peer factors, and personal
factors may have direct effects or influence susceptibility to media
campaign effects];
* Use of drugs [Influenced by Factors that directly affect drug use
(e.g., price, accessibility, arrest risk)].
Source: Adapted from Robert Hornik and others, Evaluation of the
National Youth Anti-Drug Media Campaign: Historical Trends in Drug Use
and Design of the Phase III Evaluation, prepared for the National
Institute on Drug Abuse (Rockville, Md.: Westat, July 2000).
[End of figure]
Following program participants for years to learn about the effects on
long-term outcomes for specific individuals exceeded the scope of most
of these programs; only the formal evaluation studies of the Eisenhower
and ONDCP programs did this. It can be quite costly to repeatedly
survey a group of people or track individuals‘ locations over time and
may require several attempts in order to obtain an interview or
completed survey. The Eisenhower evaluation employed a couple of
techniques that helped reduce survey costs. First, the evaluation
increased the time period covered by the surveys by surveying teachers
twice in one year: first about their teaching during the previous
school year and then about activities in the current school year. By
surveying teachers in the following spring about that school year, the
evaluators were able to learn about three school years in the space of
1-1/2 actual years. Second, the case study design helped reduce survey
costs by limiting the number of locations the evaluation team had to
revisit. Concentrating their tracking efforts in 10 sites also allowed
the team to increase the sample of teachers and, thus, be more likely
to detect small effects on teaching behavior.
Control for External Influences or Assess Their Combined Effects:
Most of the evaluations we reviewed assumed that program exposure or
participation led to the observed behavioral changes and did not attempt
to control the influence of external factors. However, in order to make
credible claims that these programs were responsible for a change in
behavior, the evaluation design had to go beyond associating program
exposure with outcomes to rule out the influence of other explanations.
NIDA‘s evaluation used statistical controls and other techniques to
limit the influence of other factors on attitudes and behaviors, while
Eisenhower, CDC, and EPA encouraged assessment of the combined effect
of related activities aimed at achieving the same goals.
EFNEP‘s evaluation approach paired program exposure with before-and-
after program measures of outcomes to show a change that was presumed
to stem from the program. Where the recommended behavior is very
specific and exclusive to a program, it can be argued that the program
was probably responsible for its adoption. An EFNEP program official
explained that because program staff work closely with participants to
address factors that could impede progress, they are comfortable using
the data to assess their effectiveness.
Many factors outside ONDCP‘s media campaign were expected to
influence youths‘ drug use, such as other anti-drug programs and youths‘
willingness to take risks, parental attitudes and behavior, peer
attitudes and behavior, and the availability of and access to drugs.
NIDA‘s evaluation used several approaches to limit the effects of other
factors on the behavioral outcomes it was reporting. First, to
distinguish this campaign from other anti-drug messages in the
environment, the campaign used a distinctive message to create a
’brand“ that would provide a recognizable element across advertisements
in the campaign and improve recall of the campaign. The evaluation‘s
survey asked questions about recognition of this brand, attitudes, and
drug use so the analysis could correlate attitudes and behavior changes
with exposure to this particular campaign.
Second, NIDA‘s evaluation used statistical methods to help limit the
influence of other factors on the results. The evaluation lacked a
control group that was not exposed, since the campaign ran nationally,
or baseline data on the audience‘s attitudes before the campaign began,
with which to compare the survey sample‘s reaction. Thus, the
evaluation chose to compare responses to variation in exposure to the
campaign”comparing those with high exposure to those with low
exposure”to assess its effects. This is called a dose-response design
which assesses how risk of disease increases with increasing doses or
exposure. This approach presumes that the advertisements were effective
if you were more likely to adopt the promoted attitudes or behaviors as
you saw more of them.
However, because the audience rather than the evaluator determined how
many advertisements they saw, it is not a random selection process, and
other factors related to drug use may have influenced both audience
viewing habits and drug-related attitudes and behaviors. To limit the
influence of preexisting differences among the exposure groups on the
results, the NIDA evaluation controlled for their influence by using a
special statistical method called propensity scoring. This controls for
any correlation between program exposure and risk factors for drug use,
such as gender, ethnicity, strength of religious feelings, and parental
substance abuse, as well as school attendance and participation in
sensation-seeking activities. This statistical technique requires
detailed data on large numbers of participants and sophisticated
analysis resources.
Some information campaigns are intertwined or closely associated with
another program or activity aimed at the same goals. Both Eisenhower and
the other programs fund teachers‘ professional development activities
that vary in quality, yet they found no significant difference in
quality by funding source in their sample. So the evaluation focused
instead on assessing the effect of high-intensity activities”regardless
of funding source”on teaching practice. EPA‘s Compliance Assistance
program, for example, helps regulated entities comply with regulations
along with its regulatory enforcement responsibilities”a factor not
lost on the entities that are regulated. EPA‘s dual role raises the
question of whether any observed improvements in compliance result from
assistance efforts or the implied threat of inspections and sanctions.
EPA measures the success of its compliance assistance efforts together
with those of incentives that encourage voluntary correction of
violations to promote compliance and reductions in pollution.
An alternative evaluation approach acknowledged the importance of
combining information dissemination with other activities to the total
program design and assessed the outcomes of the combined activities.
This approach, exemplified by CDC and the public health community,
encourages programs to adopt a comprehensive set of reinforcing media
and regulatory and other community-based activities to produce a more
powerful approach to achieving difficult behavior change. The proposed
evaluations seek not to limit the influence of these other factors but
to assess their combined effects on reducing tobacco use. CDC‘s National
Tobacco Control Program uses such a comprehensive approach to obtain
synergistic effects, making moot the issue of the unique contribution of
any one program activity. Figure 4 depicts the model CDC provided to
help articulate the combined, reinforcing effects of media and other
community-based efforts on reducing tobacco use.
Figure 4: CDC Tobacco Use Prevention and Control Logic Model:
[See PDF for image]
This figure illustrates the CDC Tobacco Use Prevention and Control
Logic Model, as follows:
Inputs:
* Federal programs, litigation, and other inputs;
* State tobacco control programs;
* Community and national partners and organizations.
Inputs lead to Activities:
Activities:
* Counter-marketing;
* Community mobilization;
* Policy and regulatory action;
* Efforts targeted to disparate populations.
From Activities come Outputs:
Outputs:
* Exposure to no-smoking/pro-health messages;
* Increased use of services;
* Creation of no-smoking regulations and policies.
Outputs lead to Outcomes:
Short-term Outcomes:
* Changes in knowledge and attitudes;
* Adherence to and enforcement of bans, regulations, and policies.
Intermediate Outcomes:
* Reduced smoking initiation among young people;
* Increased smoking-cessation among young people and adults;
* Increased number of environments with no smoking.
Long-term Outcomes:
* Decreased smoking;
* Reduced exposure to ETS;
* Reduced tobacco-related morbidity and mortality;
* Decreased tobacco-related health disparities.
Note: ETS = environmental tobacco smoke.
Source: Goldie MacDonald and others. Introduction to Program Evaluation
for Comprehensive Tobacco Control Programs (Atlanta, Ga.: CDC, November
2001).
[End of figure]
Congressional Interest, Collaboration, Available Information and
Expertise Supported These Evaluations:
Agencies initiated most of these evaluation efforts in response to
congressional interest and questions about program results. Then,
collaboration with program partners and access to research results and
evaluation expertise helped them carry out and increase the
contributions of these evaluations.
Congressional Interest:
Congressional concern about program effectiveness resulted in two
mandated evaluations and spurred agency performance assessment efforts
in two others. The Congress encouraged school-based education reform to
help students meet challenging academic standards with the Improving
America‘s Schools Act of 1994. [Footnote 10] Concerned about the
quality of professional development to update teaching practices needed
to carry out those reforms, the Congress instituted a number of far-
reaching changes and mandated an evaluation for the Eisenhower
Professional Development Program. The formal 3-year evaluation sought
to determine whether and how Eisenhower-supported activities, which
constitute the largest federal effort dedicated to supporting educator
professional development, contribute to national efforts to improve
schools and help achieve agency goals.
The Congress has also been actively involved in the development and
oversight of the National Youth Anti-Drug Media Campaign. It specified
the program effort in response to nationwide rises in rates of youths‘
drug use and mandated an evaluation of that effort. ONDCP was asked to
develop a detailed implementation plan and a system to measure outcomes
of success and report to the Congress within 2 years on the
effectiveness of the campaign, based on those measurable outcomes.
ONDCP contracted for an evaluation through NIDA to ensure that the
evaluation used the best research design and was seen as independent of
the sponsoring agency. ONDCP requested reports every 6 months on
program effectiveness and impact. However, officials noted that this
reporting schedule created unrealistically high congressional
expectations for seeing results when the program does not expect to see
much change in 6 months.
Congressional interest in sharpening the focus of cooperative extension
activities led to installing national goals that were to focus the work
and encourage the development of performance goals. The Agricultural
Research, Extension, and Education Reform Act of 1998 gave states
authority to set priorities and required them to solicit input from
various stakeholders. [Footnote 11] The act also encouraged USDA to
address high-priority concerns with national or multistate
significance. Under the act, states are required to develop plans of
work that define outcome goals and describe how they will meet them.
Annual performance reports are to describe whether states met their
goals and to report their most significant accomplishments. CSREES
draws on these reports of state outcomes to describe how they help meet
USDA‘s goals. State extension officials noted that the Government
Performance and Results Act of 1993, as well as increased
accountability pressures from their stakeholders, created a
demand for evaluations.
EFNEP‘s performance reporting system was also initiated in response to
congressional interest and is used to satisfy this latter act‘s
requirements. USDA staff noted that the House Committee on Agriculture
asked for data in 1989 to demonstrate the impact of the program to
justify the funding level. On the basis of questions from congressional
staff, program officials and extension partners formed a national
committee that examined the kinds of information that had already been
gathered to respond to stakeholders and developed standard measures of
desired client improvements. State reports are tailored to meet their
information needs, while CSREES uses the core set of common behavioral
items to provide accomplishments for USDA‘s annual performance report.
Collaboration with Program Partners:
In several evaluations we reviewed, collaboration was reported as
important for meeting the information needs of diverse audiences and
expanding the usefulness of the evaluation. ONDCP‘s National Youth Anti-
Drug Media Campaign was implemented in collaboration with the
Partnership for a Drug-Free America and a wide array of nonprofit,
public, and private organizations to reinforce its message across
multiple outlets. The National Institute on Drug Abuse, with input from
ONDCP, designed the evaluation of the campaign and drew on an expert
panel of advisers in drug abuse prevention and media studies. The
evaluation was carried out by a partnership between Westat”bringing
survey and program evaluation expertise”and the University of
Pennsylvania‘s Annenberg School for Communication”bringing expertise in
media studies. Agency officials noted that through frequent
communication with those developing the advertisements and purchasing
media time, evaluators could keep the surveys up to date with the most
recent airings and provide useful feedback on audience reaction.
The Evaluation/Reporting System represented a collaborative effort
among the federal and state programs to demonstrate EFNEP‘s benefits.
USDA staff noted that in the early 1990s, in response to congressional
inquiries about EFNEP‘s effectiveness, a national committee was formed
to develop a national reporting system for data on program results. The
committee held an expert panel with various USDA nutrition policy
experts, arranged for focus groups, and involved state and county EFNEP
representatives and others from across the country. The committee
started by identifying the kinds of information the states had already
gathered to respond to state and local stakeholders‘ needs and then
identified other questions to be answered. The committee developed and
tested the behavior checklist and dietary analysis methodology from
previous nutrition measurement efforts. The partnership among state
programs continues through an annual CSREES Call for Questions that
solicits suggestions from states that other states may choose to adopt.
USDA staff noted that local managers helped design measures that met
their needs, ensuring full cooperation in data collection and the use
of evaluation results.
State extension evaluator staff emphasized that collaborations and
partnerships were an important part of their other extension programs
and evaluations. At one level, extension staff partner with state and
local stakeholders”the state natural resource department, courts, social
service agencies, schools, and agricultural producers”as programs are
developed and implemented. This influences whether and how the
programs are evaluated”what questions are asked and what data are
collected”as those who helped define the program and its goals have a
stake in how to evaluate it. State extension evaluator staff also
counted their relationships with their peers in other states as key
partnerships that provided peer support and technical assistance. In
addition to informal contacts, some staff were involved in formal multi-
state initiatives, and many participate in a formal shared interest
group of the American Evaluation Association. While we were writing our
report, the association‘s Extension Education Evaluation Topical
Interest Group had more than 160 members, a Web site, and a listserv
and held regular meetings (see [hyperlink, http://www.danr.ucop.edu/eee-
aea/]).
Findings form Previous Research:
Using research helped agencies develop measures of program goals and
establish links between program activities and short-term goals and
between short-term and long-term goals. The Eisenhower evaluation team
synthesized existing research on teacher instruction to develop
innovative measures of the quality of teachers‘ professional
development activities, as well as the characteristics of teaching
strategies designed to encourage students‘ high-order thinking. EFNEP
drew on nutrition research to develop standard measures for routine
assessment and performance reporting. Virginia Tech‘s cooperative
extension program also drew on research on health care expenses and
known risk factors for nutrition-related diseases to estimate the
benefits of nutrition education on reducing the incidence and treatment
costs of those diseases.
Both the design of ONDCP‘s National Anti-Drug Media Campaign and its
evaluation drew on lessons learned in earlier research. The message and
structure of the media campaign were based on a review of research
evidence on the factors affecting youths‘ drug use, effective drug-use
prevention practices, and effective public health media campaigns.
Agency officials indicated that the evaluation was strongly influenced
by the ’theory of reasoned action“ perspective to explain behavioral
change. This perspective assumes that intention is an important factor
in determining behavior and that intentions are influenced by attitudes
and beliefs. Exposure to the anti-drug messages is thus expected to
change attitudes, intentions, and ultimately behavior. Similarly, CDC
officials indicated that they learned a great deal about conducting and
evaluating health promotion programs from their experience with HIV-
AIDS prevention demonstration programs conducted in the late 1980s and
early 1990s. In particular, earlier research on health promotions
shaped their belief in the increased effectiveness of programs that
combine media campaigns with other activities having the same goal.
Evaluation Expertise and Logic Models Guided Several Evaluations:
Several programs provided evaluation expertise to guide and encourage
program staff to evaluate their own programs. The guidance encouraged
them to develop program logic models to articulate program strategy and
evaluation questions. Cooperative extension has evaluation specialists
in many of the state land grant universities who offer many useful
evaluation tools and guidance on their Web sites. (See the Bibliography
for a list of resources.) CDC provided the rationale for how the
National Tobacco Control Program addressed the policy problem (youths‘
smoking) and articulated the conceptual framework for how the program
activities were expected to motivate people to change their behavior.
CDC supports local project evaluation with financial and technical
assistance and a framework for program evaluation that provides general
guidance on engaging stakeholders, evaluation design, data collection
and analysis, and ways to ensure that evaluation findings are used. CDC
also encourages grantees to allocate about 10 percent of their program
budget for program monitoring (surveillance) and evaluation. (See
[hyperlink,
http://www.cdc.gov/Tobacco/evaluation_manual/contents.htm]).
CDC, EPA, and cooperative extension evaluation guidance all encouraged
project managers to create program logic models to help articulate their
program strategy and expected outcomes. Logic models characterize how
a program expects to achieve its goals; they link program resources and
activities to program outcomes and identify short-term and long-term
outcome goals. CDC‘s recent evaluation guidance suggests that grantees
use logic models to link inputs and activities to program outcomes and
also to demonstrate how a program connects to the national and state
programs. The University of Wisconsin Cooperative Extension evaluation
guidance noted that local projects would find developing the program
logic model to be useful in program planning, identifying measures, and
explaining the program to others.
Observations:
The agencies whose evaluations we studied employed a variety of
strategies for evaluating their programs‘ effects on short-term and
intermediate goals but still had difficulty assessing their
contributions to long-term agency goals for social and environmental
benefits. As other agencies are pressed to demonstrate the
effectiveness of their information campaigns, the examples in this
report might help them identify how to successfully evaluate their
programs‘ contributions.
Several agencies drew on existing research to identify common measures;
others may find that analysis of the relevant research literature can
aid in designing a program evaluation. Previous research may reveal
useful existing measures or clarify the expected influence of the
program, as well as external factors, on its goals.
Agencies might also benefit from following the evaluation guidance that
has recommended developing logic models that specify the mechanisms
by which programs are expected to achieve results, as well as the
specific short-term, intermediate, and long-term outcomes they are
expected to achieve.
* A logic model can help identify pertinent variables and how, when,
and in whom they should be measured, as well as other factors that
might affect program results. This, in turn, can help set realistic
expectations about the scope of a program‘s likely effects. Specifying
a logical trail from program activities to distant outcomes pushes
program and evaluation planners to articulate the specific behavior
changes and long-term outcomes they expect, thereby indicating the
narrowly defined long-term outcomes that could be attributed to a
program.
* Where program flexibility allows for local variation but risks losing
accountability, developing a logic model can help program stakeholders
talk about how diverse activities contribute to common goals and how
this might be measured. Such discussion can sharpen a program‘s focus
and can lead to the development of commonly accepted standards and
measures for use across sites.
* In comprehensive initiatives that combine various approaches to
achieving a goal, developing a logic model can help articulate how those
approaches are intended to assist and supplement one another and can
help specify how the information dissemination portion of the program is
expected to contribute to their common goal. An evaluation could then
assess the effects of the integrated set of efforts on the desired long-
term outcomes, and it could also describe the short-term and
intermediate contributions of the program‘s components.
Agency Comments:
The agencies provided no written comments, although EPA, HHS, and
USDA provided technical comments that we incorporated where
appropriate throughout the report. EPA noted that the Paperwork
Reduction Act requirements pose an additional challenge in effectively
and efficiently measuring compliance assistance outcomes. We included
this point in the discussion of follow-up surveys.
We are sending copies of this report to other relevant congressional
committees and others who are interested, and we will make copies
available to others on request. In addition, the report will be
available at no charge on GAO‘s Web site at [hyperlink,
http://www.gao.gov].
If you have questions concerning this report, please call me or
Stephanie Shipman at (202) 512-2700. Elaine Vaurio also made key
contributions to this report.
Signed by:
Nancy Kingsbury:
Managing Director, Applied Research and Methods:
[End of section]
Bibliography:
Case Evaluations and Guidance:
Centers for Disease Control and Prevention, Office of Smoking and
Health. Best Practices for Comprehensive Tobacco Control Programs.
Atlanta, Ga.: August 1999. [hyperlink,
http://www.cdc.gov/tobacco/bestprac.htm] (September 2002).
Garet, Michael S., and others. Designing Effective Professional
Development: Lessons from the Eisenhower Program. Document 99-3.
Washington, D.C.: U.S. Department of Education, Planning and Evaluation
Service, December 1999. [hyperlink,
http://www.ed.gov/inits/teachers/eisenhower/] (September 2002).
Hornik, Robert, and others. Evaluation of the National Youth Anti-Drug
Media Campaign: Historical Trends in Drug Use and Design of the
Phase III Evaluation. Prepared for the National Institute on Drug Abuse.
Rockville, Md.: Westat, July 2000.
[hyperlink, http://www.whitehousedrugpolicy.gov/publications]
(September 2002).
Hornik, Robert, and others. Evaluation of the National Youth Anti-Drug
Media Campaign: Third Semi-Annual Report of Findings. Prepared for
the National Institute on Drug Abuse. Rockville, Md.: Westat, October
2001. [hyperlink, http://www.mediacampaign.org/publications/index.html]
(September 2002).
Kiernan, Nancy Ellen. ’Reduce Bias with Retrospective Questions.“ Penn
State University Cooperative Extension Tipsheet 30, University Park,
Pennsylvania, 2001. [hyperlink,
http://www.extension.psu.edu/evaluation/] (September 2002).
Kiernan, Nancy Ellen. ’Using Observation to Evaluate Skills.“ Penn State
University Cooperative Extension Tipsheet 61, University Park,
Pennsylvania, 2001. [hyperlink,
http://www.extension.psu.edu/evaluation/] (September 2002).
MacDonald, Goldie, and others. Introduction to Program Evaluation for
Comprehensive Tobacco Control Programs. Atlanta, Ga.: Centers for
Disease Control and Prevention, November 2001.
[hyperlink, http://www.cdc.gov/tobacco/evaluation_manual/contents.htm]
(September 2002).
Office of National Drug Control Policy. The National Youth Anti-Drug
Media Campaign: Communications Strategy Statement. Washington,
D.C.: Executive Office of the President, n.d.
[hyperlink, http://www.mediacampaign.org/publications/index.html]
(September 2002).
Ohio State University Cooperative Extension. Program Development and
Evaluation. [hyperlink, http://www.ag.ohio-state.edu/~pde/] (September
2002).
Penn State University. College of Agricultural Sciences, Cooperative
Extension and Outreach, Program Evaluation
[hyperlink, http://www.extension.psu.edu/evaluation/] (September 2002).
Porter, Andrew C., and others. Does Professional Development Change
Teaching Practice? Results from a Three-Year Study. Document 2000-04.
Washington, D.C.: U.S. Department of Education, Office of the Under
Secretary, October 2000.
[hyperlink,
http://www.ed.gov/offices/OUS/PES/school_improvement.html#subepdp2]
(September 2002).
Rockwell, S. Kay, and Harriet Kohn. ’Post-Then-Pre Evaluation.“ Journal
of Extension 27:2 (summer 1989).
[hyperlink, http://www.joe.org/joe/1989summer/a5.html] (September
2002).
Taylor-Powell, Ellen. ’The Logic Model: A Program Performance
Framework.“ University of Wisconsin Cooperative Extension, Madison,
Wisconsin, 62 pages, n.d. [hyperlink, http://www.uwex.edu/ces/pdande]
(September 2002).
Taylor-Powell, Ellen, and Marcus Renner. ’Collecting Evaluation Data:
End-of-Session Questionnaires.“ University of Wisconsin Cooperative
Extension document G3658-11, Madison, Wisconsin, September 2000.
[hyperlink, http://www.uwex.edu/ces/pdande] (September 2002).
Taylor-Powell, Ellen, and Sara Steele. ’Collecting Evaluation Data:
Direct Observation.“ University of Wisconsin Cooperative Extension
document G3658-5, Madison, Wisconsin, 1996.
[hyperlink, http://www.uwex.edu/ces/pdande/evaluation/evaldocs.html]
(September 2002).
U.S. Department of Agriculture, Expanded Food and Nutrition Education
Program. EFNEP 2001 Program Impacts Booklet. Washington, D.C.: June
2002. [hyperlink, http://www.reeusda.gov/f4hn/efnep/factsheet.htm]
(September 2002).
U.S. Department of Agriculture, Expanded Food and Nutrition Education
Program. ERS4 (Evaluation/Reporting System). Washington, D.C.: April 9,
2001. [hyperlink, http://www.reeusda.gov/ers4/home.htm] (September
2002).
U.S. Department of Agriculture, Expanded Food and Nutrition Education
Program. Virginia EFNEP Cost Benefit Analysis. Fact Sheet. Washington,
D.C.: n.d. [hyperlink, http://www.reeusda.gov/f4hn/efnep.htm]
(September 2002).
U.S. Environmental Protection Agency, Office of Enforcement and
Compliance Assurance. Guide for Measuring Compliance Assistance
Outcomes. EPA300-B-02-011. Washington, D.C.: June 2002.
[hyperlink, http://www.epa.gov/compliance/planning/results/tools.html]
(September 2002).
U.S. Department of Health and Human Services, Centers for Disease
Control and Prevention. ’Framework for Program Evaluation in Public
Health.“ Morbidity and Mortality Weekly Report 48:RR-11 (1999).
(September 2002).
University of Wisconsin Cooperative Extension. Program Development
and Evaluation, Evaluation.
[hyperlink, http://www.uwex.edu/ces/pdande/evaluation/index.htm]
(September 2002).
Other Evaluation Guidance and Tools:
American Evaluation Association, Extension Education Evaluation
Topical Interest Group
[hyperlink, http://www.danr.ucop.edu/eee-aea/] (September 2002).
CYFERnet. Children, Youth, and Families Education and Research
Network. Evaluation Resources
[hyperlink,
http://twosocks.ces.ncsu.edu/cyfdb/browse_2.php?search=Evaluation]
(September 2002).
Schwarz, Norbert, and Daphna Oyserman. ’Asking Questions about
Behavior: Cognition, Communication, and Questionnaire Construction.“
American Journal of Evaluation 22:2 (summer 2001): 127–60.
Southern Regional Program and Staff Development Committee.
’Evaluation and Accountability Resources: A Collaboration Project of the
Southern Region Program and Staff Development Committee.“ Kentucky
Cooperative Extension Service.
[hyperlink, http://www.ca.uky.edu/agpsd/soregion.htm] (September 2002).
[End of section]
Related GAO Products:
Program Evaluation: Studies Helped Agencies Measure or Explain Program
Performance. GAO/GGD-00-204. Washington, D.C.: September 29, 2000.
Anti-Drug Media Campaign: ONDCP Met Most Mandates, but Evaluations of
Impact Are Inconclusive. GAO/GGD/HEHS-00-153. Washington, D.C.: July
31, 2000.
Managing for Results: Measuring Program Results That Are under Limited
Federal Control. GAO/GGD-99-16. Washington, D.C.: December 11, 1998.
Grant Programs: Design Features Shape Flexibility, Accountability, and
Performance Information. GAO/GGD-98-137. Washington, D.C.: June 22,
1998.
Program Evaluation: Agencies Challenged by New Demand for Information
on Program Results. GAO/GGD-98-53. Washington, D.C.: April 24, 1998.
Managing for Results: Analytic Challenges in Measuring Performance.
GAO/HHS/GGD-97-138. Washington, D.C.: May 30, 1997.
Program Evaluation: Improving the Flow of Information to the Congress.
GAO/PEMD-95-1. Washington, D.C.: January 30, 1995. Designing
Evaluations. GAO/PEMD-10.1.4. Washington, D.C.: May 1991.
[End of section]
Footnotes:
[1] Some intermediate behavioral outcomes may occur in the short term.
[2] Goldie MacDonald and others, Introduction to Program Evaluation for
Comprehensive Tobacco Control Programs (Atlanta, Ga.: Centers for
Disease Control and Prevention, November 2001).
[3] See, for example, Ellen Taylor-Powell and Marcus Renner,
’Collecting Evaluation Data: End-of-Session Questionnaires,“ University
of Wisconsin Cooperative Extension document G3658-11, Madison,
Wisconsin, September 2000. Also see the Bibliography for various
sources of guidance.
[4] See, for example, Ellen Taylor-Powell and Sara Steele, ’Collecting
Evaluation Data: Direct Observation,“ University of Wisconsin
Cooperative Extension document G3658-5, Madison, Wisconsin, 1996.
[5] Nancy Ellen Kiernan, ’Using Observation to Evaluate Skills,“ Penn
State University Cooperative Extension Tipsheet 61, University Park,
Pennsylvania, 2001.
[6] 44 U.S.C. 3501-3520 (2000).
[7] For a review of related research see Norbert Schwarz and Daphna
Oyserman, ’Asking Questions about Behavior: Cognition, Communication,
and Questionnaire Construction,“ American Journal of Evaluation 22:2
(summer 2001): 127–60.
[8] Nancy Ellen Kiernan, ’Reduce Bias with Retrospective Questions,“
Penn State Cooperative Extension Tipsheet 30, University Park,
Pennsylvania, 2001, and S. Kay Rockwell and Harriet Kohn, ’Post-Then-
Pre Evaluation,“ Journal of Extension 27:2 (summer 1989).
[9] EPA, Office of Enforcement and Compliance Assurance, Case
Conclusion Data Sheet, document 2222A (Washington, D.C.: November
2000).
[10] P.L. 103-382, Oct. 20, 1994, 108 Stat. 3518.
[11] P.L. 105-185, June 23, 1998, 112 Stat. 523.
[End of section]
GAO's Mission:
The General Accounting Office, the investigative arm of Congress,
exists to support Congress in meeting its constitutional
responsibilities and to help improve the performance and accountability
of the federal government for the American people. GAO examines the use
of public funds; evaluates federal programs and policies; and provides
analyses, recommendations, and other assistance to help Congress make
informed oversight, policy, and funding decisions. GAO‘s commitment to
good government is reflected in its core values of accountability,
integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through the Internet. GAO‘s Web site [hyperlink,
http://www.gao.gov] contains abstracts and fulltext files of current
reports and testimony and an expanding archive of older products. The
Web site features a search engine to help you locate documents using
key words and phrases. You can print these documents in their entirety,
including charts and other graphics.
Each day, GAO issues a list of newly released reports, testimony, and
correspondence. GAO posts this list, known as ’Today‘s Reports,“ on its
Web site daily. The list contains links to the full-text document
files. To have GAO e-mail this list to you every afternoon, go to
[hyperlink, http://www.gao.gov] and select ’Subscribe to daily E-mail
alert for newly released products“ under the GAO Reports heading.
Order by Mail or Phone:
The first copy of each printed report is free. Additional copies are $2
each. A check or money order should be made out to the Superintendent
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or
more copies mailed to a single address are discounted 25 percent.
Orders should be sent to:
U.S. General Accounting Office:
441 G Street NW, Room LM:
Washington, D.C. 20548:
To order by Phone: Voice: (202) 512-6000:
TDD: (202) 512-2537:
Fax: (202) 512-6061:
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]:
E-mail: fraudnet@gao.gov:
Automated answering system: (800) 424-5454 or (202) 512-7470:
Public Affairs:
Jeff Nelligan, managing director, NelliganJ@gao.gov, (202) 512-4800:
U.S. General Accounting Office:
441 G Street NW, Room 7149:
Washington, D.C. 20548:
