Tax Administration
IRS Needs Better Strategic Planning and Evaluation of Taxpayer Assistance Training Gao ID: GAO-05-782 July 11, 2005Millions of taxpayers ask IRS questions about tax law each year. While the accuracy of IRS's answers has improved in some cases, it is still not always what taxpayers or Congress expect. Concerns about accuracy have raised questions about the adequacy of the training IRS provides to its taxpayer assistance staff. Because of these questions, GAO was asked to assess the extent to which IRS's planning and evaluation of its taxpayer assistor training conformed to guidance published by GAO and others. Planning and evaluation are part of a feedback loop whereby lessons from one year can be applied to making improvements in future years.
IRS devotes significant resources to training its tax law assistors to answer questions, by telephone and at walk-in sites, and prepare tax returns. Although IRS cannot separate the costs of training tax law assistors from other assistance staff, the thousands of staff devoted to providing tax law assistance receive training each year. The training includes classroom and computer-based training on such subjects as tax law and communication. While IRS has some data on travel and course development costs associated with training, it does not have data on what is likely the largest cost component, the value of staff time devoted to tax law training. Responsibility for training IRS taxpayer assistance staff is decentralized. IRS's Human Capital Office provides guidance and sets policy. The two divisions responsible for tax law assistance each have a human resources office with technical staff that are assigned to the various taxpayer assistance programs. Generally speaking, the taxpayer assistance programs share responsibility for planning training with human resource staff. The human resource staff are responsible for training evaluations. IRS's planning for taxpayer assistor training could be enhanced by a more strategic approach. To their credit, some taxpayer assistance programs clearly communicated the importance of training to staff and had knowledge and skills inventories. All the units had analyses of some of the individual factors that affect accuracy, such as the quality and use of their taxpayer assistance guidance. However, none of the programs had long-term goals for either accuracy or training or had benchmarked their training efforts against those of other organizations. Nor had they done a combined analysis of the major factors that affect accuracy in order to determine their relative importance. Setting long-term goals and analyzing training needs and relative impacts are important steps in strategic planning. Goals can provide a yardstick for measuring progress and benchmarking can help identify best practices. Analyses of relative impacts can help IRS make informed decisions about a strategy for improving accuracy, including the importance of training compared to other factors in that strategy. The taxpayer assistance programs routinely conducted evaluations of their training efforts. However, with one exception, the evaluations did not include analyses of the impact of training on the accuracy of assistance. Instead, the units conducted less sophisticated analyses of more immediate impacts, such as trainees' satisfaction. Given the importance of accurate answers to taxpayers' questions and the resources spent on training, the four assistance programs would benefit from more sophisticated evaluations of the effectiveness of training. One program had recognized the potential value of a more sophisticated evaluation of training and pilot tested an analysis in 2004. The value of evaluation is that it provides feedback about the effectiveness of one year's training that can be used to plan improvements to future training.
RecommendationsOur recommendations from this work are listed below with a Contact for more information. Status will change from "In process" to "Open," "Closed - implemented," or "Closed - not implemented" based on our follow up work.
Director: Team: Phone:GAO-05-782, Tax Administration: IRS Needs Better Strategic Planning and Evaluation of Taxpayer Assistance Training
This is the accessible text file for GAO report number GAO-05-782
entitled 'Tax Administration: IRS Needs Better Strategic Planning and
Evaluation of Taxpayer Assistance Training' which was released on
August 12, 2005.
This text file was formatted by the U.S. Government Accountability
Office (GAO) to be accessible to users with visual impairments, as part
of a longer term project to improve GAO products' accessibility. Every
attempt has been made to maintain the structural and data integrity of
the original printed product. Accessibility features, such as text
descriptions of tables, consecutively numbered footnotes placed at the
end of the file, and the text of agency comment letters, are provided
but may not exactly duplicate the presentation or format of the printed
version. The portable document format (PDF) file is an exact electronic
replica of the printed version. We welcome your feedback. Please E-mail
your comments regarding the contents or accessibility features of this
document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
Report to Congressional Requesters:
United States Government Accountability Office:
GAO:
July 2005:
Tax Administration:
IRS Needs Better Strategic Planning and Evaluation of Taxpayer
Assistance Training:
GAO-05-782:
GAO Highlights:
Highlights of GAO-05-782, a report to congressional requesters:
Why GAO Did This Study:
Millions of taxpayers ask IRS questions about tax law each year. While
the accuracy of IRS‘s answers has improved in some cases, it is still
not always what taxpayers or Congress expect. Concerns about accuracy
have raised questions about the adequacy of the training IRS provides
to its taxpayer assistance staff. Because of these questions, GAO was
asked to assess the extent to which IRS‘s planning and evaluation of
its taxpayer assistor training conformed to guidance published by GAO
and others. Planning and evaluation are part of a feedback loop whereby
lessons from one year can be applied to making improvements in future
years.
What GAO Found:
IRS devotes significant resources to training its tax law assistors to
answer questions, by telephone and at walk-in sites, and prepare tax
returns. Although IRS cannot separate the costs of training tax law
assistors from other assistance staff, the thousands of staff devoted
to providing tax law assistance receive training each year. The
training includes classroom and computer-based training on such
subjects as tax law and communication. While IRS has some data on
travel and course development costs associated with training, it does
not have data on what is likely the largest cost component, the value
of staff time devoted to tax law training.
Responsibility for training IRS taxpayer assistance staff is
decentralized. IRS‘s Human Capital Office provides guidance and sets
policy. The two divisions responsible for tax law assistance each have
a human resources office with technical staff that are assigned to the
various taxpayer assistance programs. Generally speaking, the taxpayer
assistance programs share responsibility for planning training with
human resource staff. The human resource staff are responsible for
training evaluations.
IRS‘s planning for taxpayer assistor training could be enhanced by a
more strategic approach. To their credit, some taxpayer assistance
programs clearly communicated the importance of training to staff and
had knowledge and skills inventories. All the units had analyses of
some of the individual factors that affect accuracy, such as the
quality and use of their taxpayer assistance guidance. However, none of
the programs had long-term goals for either accuracy or training or had
benchmarked their training efforts against those of other
organizations. Nor had they done a combined analysis of the major
factors that affect accuracy in order to determine their relative
importance. Setting long-term goals and analyzing training needs and
relative impacts are important steps in strategic planning. Goals can
provide a yardstick for measuring progress and benchmarking can help
identify best practices. Analyses of relative impacts can help IRS make
informed decisions about a strategy for improving accuracy, including
the importance of training compared to other factors in that strategy.
The taxpayer assistance programs routinely conducted evaluations of
their training efforts. However, with one exception, the evaluations
did not include analyses of the impact of training on the accuracy of
assistance. Instead, the units conducted less sophisticated analyses of
more immediate impacts, such as trainees‘ satisfaction. Given the
importance of accurate answers to taxpayers‘ questions and the
resources spent on training, the four assistance programs would benefit
from more sophisticated evaluations of the effectiveness of training.
One program had recognized the potential value of a more sophisticated
evaluation of training and pilot tested an analysis in 2004. The value
of evaluation is that it provides feedback about the effectiveness of
one year‘s training that can be used to plan improvements to future
training.
What GAO Recommends:
GAO is making several recommendations to IRS. Regarding planning, our
recommendations include:
* establishing long-term goals;
* determining the relative importance of the factors that affect
accuracy, including training; and
* benchmarking training against other organizations.
Regarding evaluation, we recommend that IRS continue to pursue
evaluations of the impact of its taxpayer assistance training efforts
on accuracy.
IRS agreed with five of our eight recommendations, including those
summarized above, and partially responded to the remaining three.
www.gao.gov/cgi-bin/getrpt?GAO-05-782.
To view the full product, including the scope and methodology, click on
the link above. For more information, contact James R. White at (202)
512-9110 or whitej@gao.gov.
[End of section]
Contents:
Letter:
Results in Brief:
Background:
Scope and Methodology:
Long-term Goals and Analyses of Needs and Impacts Could Improve IRS's
Planning for Training and Development:
Evaluating the Impact of Training on Accuracy Could Provide a Basis for
Future Improvements:
Conclusions:
Recommendations:
Agency Comments and Our Evaluation:
Appendix I: What GAO Looked For: Examples of Planning Practices That
Would Conform to Strategic Guidance for Training:
Appendix II: What GAO Looked For: Examples of Evaluation Practices That
Would Conform to Strategic Guidance:
Appendix III: Assessments of Planning: Less Complex Tax Law Telephone
Service by W&I CAS:
Appendix IV: Assessments of Planning: More Complex Tax Law Telephone
Service by SB/SE TEC:
Appendix V: Assessments of Planning: Tax Law Questions Walk-In Service
by W&I CARE:
Appendix VI: Assessments of Planning: Return Preparation Walk-In
Service by W&I CARE:
Appendix VII: Assessments of Evaluation: Less Complex Tax Law Telephone
Service by W&I CARE:
Appendix VIII: Assessments of Evaluation: More Complex Tax Law
Telephone Service by SB/SE TEC:
Appendix IX: Assessments of Evaluation: Tax Law Walk-in Service by W&I
CARE:
Appendix X: Assessments of Evaluation: Return Preparation Walk-in
Service by W&I CARE:
Appendix XI: Information on the Kirkpatrick Model of Training and
Development Evaluation:
Appendix XII: Comments from IRS:
Appendix XIII: GAO Contact and Staff Acknowledgments:
Tables:
Table 1: Assessments of IRS's Planning Practices in Training and
Developing Staff to Provide Accurate Taxpayer Assistance:
Table 2: Assessments of IRS Practices in Evaluating Training and
Development of Staff to Provide Accurate Taxpayer Assistance:
Figures:
Figure 1: Planning and Evaluation as Part of Four Components of the
Training and Development Process:
Figure 2: Example Agency's Training and Development Programs Assessed
Using Each Level of Evaluation:
United States Government Accountability Office:
Washington, DC 20548:
July 11, 2005:
The Honorable Max Baucus:
Ranking Minority Member:
Committee on Finance:
United States Senate:
The Honorable Byron L. Dorgan:
United States Senate:
Taxpayers expect timely and accurate assistance from the Internal
Revenue Service (IRS) when they have tax law questions or have tax
returns prepared. The quality of IRS's assistance can reduce the time
and aggravation of preparing tax returns and increase taxpayers'
compliance with the tax laws. In 2004, IRS answered almost 9 million
tax law questions by telephone and prepared almost half a million tax
returns for low income taxpayers.
IRS's past performance has shown that taxpayers cannot always rely on
it to provide accurate information. While the accuracy of IRS's
taxpayer assistance has improved in some cases, it has been
inconsistent or below expectations in others. After 2 years of decline,
in the first weeks of the 2005 filing season IRS telephone assistance
accuracy was estimated at 87, percent compared to 76 percent for the
same time period in 2004. Although data on the accuracy of assistance
at IRS's walk-in sites are limited by not being representative,
Treasury Inspector General for Tax Administration (TIGTA) reports have
raised concerns about the accuracy of the returns IRS prepares and that
IRS had not met its annual tax law accuracy goal.[Footnote 1]
This performance has raised questions about the adequacy of the
training IRS provides its taxpayer assistance staff. Although a number
of factors can affect the accuracy of assistance IRS provides
taxpayers, effective training and development programs can enhance an
organization's ability to achieve its mission and goals, such as
improving accuracy. At the same time training and developing staff is
costly, making it important that such investments are targeted
strategically and not wasted on efforts that are irrelevant,
duplicative, or ineffective.
Because of your interest in ensuring that taxpayers receive accurate
information when they contact IRS for assistance, and the contribution
that training makes to that end, you asked that we review how IRS plans
and evaluates its primary taxpayer assistance training and development
efforts. We focused on planning and evaluation from a strategic
perspective, that is, how planning and evaluation of training can help
improve accuracy. As discussed with your offices, our objectives were
to assess whether IRS's processes for planning and evaluating the
training and development of taxpayer assistance staff conform to
published guidance.
In conducting our work we developed detailed criteria to assess how IRS
plans and evaluates its training and development of taxpayer assistance
staff based on guidance we published for assessing strategic training
and development efforts in the federal government.[Footnote 2] We
developed 27 separate criteria for planning and evaluation, such as
establishing goals, conducting knowledge and skills needs analyses,
benchmarking against other organizations, systematically collecting
data, and comparing benefits and costs. We shared these criteria with
officials in IRS's Human Capital Office and they said the criteria are
appropriate and consistent with their policy guidance. We assessed
training and development for four types of assistance: less complex tax
law questions answered by phone, more complex tax law questions
answered by phone, tax law questions answered at IRS walk-in sites, and
return preparation at IRS walk-in sites. We collected documents
describing IRS's planning and evaluation process for the four taxpayer
assistance programs, interviewed officials in IRS's Wage and Investment
(W&I) and Small Business/Self-employed (SB/SE) divisions to get more
detail, and compared IRS's practices to our criteria. The scope and
methodology section provides more details.
We conducted our work from May 2004 through May 2005 in accordance with
generally accepted government auditing standards.
Results in Brief:
IRS's planning for taxpayer assistance training could be enhanced by
long-term goals and analyses of the relative importance of the factors
that affect accuracy, other organizations' experiences, and gaps
between long-term needs and existing skills. To their credit, all four
taxpayer assistance programs had efforts that conformed, at least in
part, to many of our planning criteria. They had efforts to involve
stakeholders in key annual planning decisions, communicate the
importance of training to staff, and plan changes to their training to
address annual tax law changes. The programs also had analyses of some
of the individual factors that affect accuracy, such as the quality and
use of their taxpayer assistance guidance. Two programs had annual
goals for accuracy. However, the programs did not conform to other
criteria. None of the programs had long-term accuracy goals, training
goals, or measures suitable for an assessment of the impact of training
on accuracy. Nor had they determined the relative importance of the
various factors that impact accuracy, benchmarked the training
practices of other organizations, or conducted assessments of long-term
skill needs. Setting goals and conducting these analyses could set a
direction and provide a more informed basis for planning improvements
to training in order to improve accuracy.
The four taxpayer assistance programs routinely conducted evaluations
of their training efforts, but these efforts did not fully comply with
criteria for strategic evaluations. IRS's Human Capital Office has
policy guidance urging such evaluations. One program met some of the
criteria for evaluating the impact of training on accuracy. Given the
importance of accurate answers to taxpayers' questions and the
resources spent on training, the four assistance programs would benefit
from more sophisticated evaluations of the effectiveness of training.
One program had recognized the potential value of a more sophisticated
evaluation of training and pilot tested such an analysis in 2004 and
plans to conduct a similar analysis in 2005.
We are making recommendations to the Commissioner of Internal Revenue
to improve IRS's planning and evaluations of taxpayer assistance
training and development efforts. The recommendations include
establishing long-term goals, determining the importance of the various
factors that affect accuracy, and conducting long-term skills and
knowledge gap analyses.
In commenting on a draft of this report (see app. XII), the
Commissioner of Internal Revenue agreed with five of our eight
recommendations and partially responded to the remaining three
recommendations.
Background:
IRS provides tax law assistance to taxpayers through IRS's toll-free
telephone lines and tax law and return preparation assistance in person
at IRS taxpayer assistance centers (formerly known as "walk-in sites")
nationwide, in addition to other means. The four taxpayer assistance
programs we reviewed were:
* W&I Customer Accounts Services (CAS) general tax law assistance by
telephone. Many taxpayers contact IRS by calling its toll-free
telephone number which is operated and staffed by CAS.[Footnote 3]
According to IRS fiscal year 2004 data, 3,420 CAS staff[Footnote 4]
handled more than 8.7 million telephone calls from taxpayers with
general, less complex tax law questions. We did not verify these IRS
data or other IRS data on taxpayer assistance programs' workload and
staffing.
* SB/SE Taxpayer Education Communication (TEC) complex tax law
assistance by phone. According to IRS officials, during the 2005 filing
season and in previous years, TEC staff supported telephone service by
answering taxpayers' questions on selected, more complex tax topics.
When taxpayers called IRS's toll-free number about these topics, W&I
CAS staff recorded the taxpayers' contact information and questions so
that TEC staff could call the taxpayers back later to provide answers.
According to IRS, in the 2004 filing season, TEC handled about 320,000
telephone calls for complex tax law assistance. According to IRS, TEC
trained approximately 400 staff and used 272,212 staff hours to provide
telephone assistance in 2004. In 2006 these calls will be handled by
W&I CAS staff.
* W&I Customer Assistance, Relationships, and Education (CARE) walk-in
tax law assistance. According to IRS, about 1.4 million staff hours
were devoted to providing walk-in assistance in fiscal year 2004. Also
according to IRS, 1,654 staff were trained to provide tax law
assistance in 2004. In 2004, there were about 7.7 million contacts with
taxpayers at IRS's approximately 400 taxpayer assistance centers. IRS
did not have information on how many of these contacts were for tax law
assistance. Many of the contacts were for other services, such as tax
forms, publications, or accounts issues.
* W&I Customer Assistance, Relationships, and Education (CARE) walk-in
return preparation assistance. According to IRS data, IRS staff
prepared 476,813 tax returns in fiscal year 2004. IRS did not have data
on the number of staff that prepared tax returns at taxpayer assistance
centers.
IRS does not have data on the amount of time its assistors spend
annually being trained on tax law.[Footnote 5] However, all staff
providing tax law assistance receive some training each year. IRS has
continuing professional education and refresher training requirements
for all its taxpayer assistance staff.
Tax law assistance staff receive a variety of training. According to an
IRS official, the staff get primarily classroom training but also
receive computer-based training. On-the-job training, managerial
coaching, and workshops are also part of training. Taxpayer assistance
staff receive training on technical tax law topics, how to use IRS
systems and guidance to answer questions, and communication.
Responsibility for training and developing IRS's tax law assistance
staff is decentralized. IRS's Human Capital Office provides guidance
and sets policy and standards on training and development for the
agency. W&I and SB/SE each have a human resources office with Learning
and Education (L&E) staff who are assigned to the taxpayer assistance
programs. L&E staff provide program staff advice and analysis on
related policies and issues and formulate strategies, procedures, and
practices to address the programs' human capital needs, including
training. Generally speaking, the taxpayer assistance program offices
identify training needs, L&E staff work with program staff to develop
and fund annual training plans, and the program offices administer
training. According to IRS policy, L&E staff are responsible for
evaluating training.
According to fiscal year 2004 IRS data, IRS invested about $7 million
in training W&I CAS and CARE staff to provide taxpayer assistance,
including such expenses as travel, supplies, contractor fees, and
development costs. IRS could not separate these costs into amounts
spent on tax law training and other topics. According to SB/SE TEC
fiscal year 2004 data, training costs were $325,072 in student and
instructor travel. However, the cost data IRS provided did not include
the costs of assistors' time associated with training. As with the
staffing and workload data, we did not verify the accuracy of IRS's
training cost data.
In March 2004, we issued an assessment guide that introduced a
framework for evaluating a federal agency's training and development
efforts.[Footnote 6] This assessment guide consists of a set of
principles and key questions that federal agencies can use to ensure
that their training and development investments are targeted
strategically and are not wasted on efforts that are irrelevant,
duplicative, or ineffective. As detailed in our assessment guide, the
training and development process can loosely be segmented into four
broad, interrelated components: (1) planning/front-end analysis, (2)
design/development, (3) implementation, and (4) evaluation. Figure 1
depicts the general relationships between the four components,
including the feedback loop between evaluation and planning. Planning
and evaluation are highlighted because they are the focus of this
report. Although these components can be discussed separately, they are
not mutually exclusive and encompass subcomponents that may blend with
one another. For instance, evaluation is part of the planning as
organizations should reach agreement up front on how the success of
training and development efforts will be assessed.
Figure 1: Planning and Evaluation as Part of Four Components of the
Training and Development Process:
[See PDF for image]
[End of figure]
TIGTA is conducting a review that covers some aspects of the design and
implementation of training.
Scope and Methodology:
Our work examined the training and development of employees who provide
tax law and return preparation assistance to taxpayers over the
telephone and at walk-in centers, and covered both seasonal and full-
time employees in IRS's W&I and SB/SE divisions. Our assessment of
IRS's training and development program for taxpayer assistance
employees was based on analyses of IRS data and interviews with IRS
officials. We also obtained background information from sources outside
IRS such as TIGTA, the IRS Oversight Board, and the IRS National
Taxpayer Advocate.
To assess how IRS plans for and evaluates the training and development
of IRS employees who provide tax law and return preparation assistance
to taxpayers over the telephone and at walk-in centers, we used our
guide for assessing training and development programs in the federal
government[Footnote 7] as a framework.
We used the parts of the GAO guide dealing with the planning and
evaluation components, along with supplemental guidance from the Office
of Personnel Management, to identify the detailed strategic training
and development criteria applicable to IRS and organized them into
chronological phases. We focused our criteria on strategic planning and
evaluation, that is, planning and evaluation intended to help achieve
IRS's accuracy goals. We developed 27 criteria, listed in appendices I
and II. We also developed examples of evidence that would demonstrate
conformance with each of the criteria. For example, for the criterion
"conduct a knowledge and skills inventory to identify employees'
current skills and knowledge," we looked for evidence such as a
completed knowledge and skills surveys and proficiency tests (see app.
I for the planning phase criteria and further evidence examples and
app. II for the same information on evaluation.) We shared the criteria
with officials in IRS's Human Capital Office and the taxpayer
assistance programs and assigned L&E staff. The officials from the
Human Capital Office said the criteria are appropriate and consistent
with their policy guidance. Some program and L&E officials expressed
concerns about their need and ability to satisfy the criteria. Their
concerns are discussed in the planning and evaluation sections of this
report.
In applying the criteria to IRS, we collected documents describing
IRS's planning and evaluation processes for the four types of
assistance. We also interviewed officials responsible for the taxpayer
assistance programs in W&I CAS and CARE units and SB/SE TEC unit and
associated L&E staff responsible for training planning and evaluation,
to get more details where needed. We reviewed all the evidence and made
a judgment about the extent to which IRS's practices conformed to the
criteria. We then discussed our initial assessments with IRS officials
responsible for planning and evaluating the taxpayer assistance
training and development programs who, for some of our preliminary
assessments, provided additional written evidence for us to consider in
making our final assessments. We then revised our initial assessments
based on the evidence they provided. We made our assessments in two
steps. First, an analyst reviewed all the evidence and made a judgment
about the extent to which it conformed to the criteria. Then a second
analyst independently reviewed the assessments. The evidence supporting
our assessments is provided in appendices III through X.
Much of the information we relied on was descriptive. We determined
that the information was reliable for our purposes. To the extent
possible we corroborated interview evidence with documentation. Where
not possible, the description is attributable to IRS officials. Where
relevant we corroborated that policy guidance, such as Internal Revenue
Manual guidance, was being implemented by collecting documentation and
reports showing implementation. With respect to controls over
databases, we reviewed documentation of the controls but did not assess
their adequacy or test data in the databases.
We conducted our work at the Wage and Investment Division headquarters
in Atlanta, Ga; the Small Business/Self-Employed Division offices in
Philadelphia, Pa; and IRS headquarters Human Capital Office in
Washington, D.C. from May 2004 through May 2005 in accordance with
generally accepted government auditing standards.
Long-term Goals and Analyses of Needs and Impacts Could Improve IRS's
Planning for Training and Development:
Summarizing table 1, IRS's planning for taxpayer assistance training
primarily focuses on meeting short-term needs, such as the challenge of
training staff about tax law changes in preparation for the next year's
filing season. Planning could be enhanced by long-term goals and
analyses of the relative importance of the factors that affect
accuracy, other organizations' experiences, and gaps between long-term
needs and existing skills. Table 1 shows our assessments of planning
for training by IRS's four primary taxpayer assistance programs for
each phase of the planning process: goals and priorities, information
gathering and assessments, skills and knowledge analysis, and strategy
development and selection. The evidence supporting our assessments is
shown in appendices III through VI.
Table 1: Assessments of IRS's Planning Practices in Training and
Developing Staff to Provide Accurate Taxpayer Assistance:
Phase: goals and priorities:
Criteria: Establish quantitative long-term accuracy goals that link to
IRS's strategic goals;
Extent to which IRS's practices conform to strategic guidance by type
of service: Telephone tax law: General (CAS): Conforms to guidance to
some extent;
Extent to which IRS's practices conform to strategic guidance by type
of service: Telephone tax law: More complex (TEC): Conforms to guidance
to little or no extent;
Extent to which IRS's practices conform to strategic guidance by type
of service: Walk-in assistance: Tax law (CARE): Conforms to guidance to
some extent;
Extent to which IRS's practices conform to strategic guidance by type
of service: Walk-in assistance: Return preparation (CARE): Conforms to
guidance to little or no extent.
Criteria: Involve key stakeholders, including human capital
professionals, managers, and employees, in key long-term planning
decisions;
Extent to which IRS's practices conform to strategic guidance by type
of service: Telephone tax law: General (CAS): Conforms to guidance to
some extent;
Extent to which IRS's practices conform to strategic guidance by type
of service: Telephone tax law: More complex (TEC): Conforms to guidance
to some extent;
Extent to which IRS's practices conform to strategic guidance by type
of service: Walk-in assistance: Tax law (CARE): Conforms to guidance to
some extent;
Extent to which IRS's practices conform to strategic guidance by type
of service: Walk-in assistance: Return preparation (CARE): Conforms to
guidance to some extent.
Criteria: Establish goals for, and measures to assess the effectiveness
of, training and development on accuracy;
Extent to which IRS's practices conform to strategic guidance by type
of service: Telephone tax law: General (CAS): Conforms to guidance to
little or no extent;
Extent to which IRS's practices conform to strategic guidance by type
of service: Telephone tax law: More complex (TEC): Conforms to guidance
to little or no extent;
Extent to which IRS's practices conform to strategic guidance by type
of service: Walk-in assistance: Tax law (CARE): Conforms to guidance to
little or no extent;
Extent to which IRS's practices conform to strategic guidance by type
of service: Walk-in assistance: Return preparation (CARE): Conforms to
guidance to little or no extent.
Criteria: Affirm, through upper-level management communication, the
importance of training and development to improve accuracy;
Extent to which IRS's practices conform to strategic guidance by type
of service: Telephone tax law: General (CAS): Conforms to guidance to
some extent;
Extent to which IRS's practices conform to strategic guidance by type
of service: Telephone tax law: More complex (TEC): Conforms to guidance
to some extent;
Extent to which IRS's practices conform to strategic guidance by type
of service: Walk-in assistance: Tax law (CARE): Conforms to guidance to
a great extent;
Extent to which IRS's practices conform to strategic guidance by type
of service: Walk- in assistance: Return preparation (CARE): Conforms to
guidance to a great extent.
Phase: information gathering and assessments:
Criteria: Analytically determine and track the strategic and
operational factors, including training, that affect accuracy;
Extent to which IRS's practices conform to strategic guidance by type
of service: Telephone tax law: General (CAS): Conforms to guidance to
some extent;
Extent to which IRS's practices conform to strategic guidance by type
of service: Telephone tax law: More complex (TEC): Conforms to guidance
to some extent;
Extent to which IRS's practices conform to strategic guidance by type
of service: Walk-in assistance: Tax law (CARE): Conforms to guidance to
some extent;
Extent to which IRS's practices conform to strategic guidance by type
of service: Walk- in assistance: Return preparation (CARE): Conforms to
guidance to some extent.
Criteria: Incorporate evaluation results into training and development
planning;
Extent to which IRS's practices conform to strategic guidance by type
of service: Telephone tax law: General (CAS): See table 2 for our
assessments;
Extent to which IRS's practices conform to strategic guidance by type
of service: Walk-in assistance: Tax law (CARE): [Empty];
Extent to which IRS's practices conform to strategic guidance by type
of service: Walk-in assistance: Return preparation (CARE): [Empty].
Criteria: Benchmark the training and development program against high-
performing organizations;
Extent to which IRS's practices conform to strategic guidance by type
of service: Telephone tax law: General (CAS): Conforms to guidance to
little or no extent;
Extent to which IRS's practices conform to strategic guidance by type
of service: Telephone tax law: More complex (TEC): Conforms to guidance
to little or no extent;
Extent to which IRS's practices conform to strategic guidance by type
of service: Walk-in assistance: Tax law (CARE): Conforms to guidance to
little or no extent;
Extent to which IRS's practices conform to strategic guidance by type
of service: Walk-in assistance: Return preparation (CARE): Conforms to
guidance to little or no extent.
Criteria: Assess whether human resource information systems provide
needed data in a timely manner;
Extent to which IRS's practices conform to strategic guidance by type
of service: It is too early for an assessment on the human resource
information. For details see appendices III through VI.
Phase: skills and knowledge analysis:
Criteria: Perform a needs assessment to determine the knowledge and
skills needed now and in the future;
Extent to which IRS's practices conform to strategic guidance by type
of service: Telephone tax law: General (CAS): Conforms to guidance to
some extent;
Extent to which IRS's practices conform to strategic guidance by type
of service: Telephone tax law: More complex (TEC): Conforms to guidance
to some extent;
Extent to which IRS's practices conform to strategic guidance by type
of service: Walk-in assistance: Tax law (CARE): Conforms to guidance to
some extent;
Extent to which IRS's practices conform to strategic guidance by type
of service: Walk-in assistance: Return preparation (CARE): Conforms to
guidance to some extent.
Criteria: Conduct a knowledge and skills inventory to identify
employees' current skills and knowledge;
Extent to which IRS's practices conform to strategic guidance by type
of service: Telephone tax law: General (CAS): Conforms to guidance to a
great extent;
Extent to which IRS's practices conform to strategic guidance by type
of service: Telephone tax law: More complex (TEC): Conforms to guidance
to little or no extent;
Extent to which IRS's practices conform to strategic guidance by type
of service: Walk-in assistance: Tax law (CARE): Conforms to guidance to
a great extent;
Extent to which IRS's practices conform to strategic guidance by type
of service: Walk-in assistance: Return preparation (CARE): Conforms to
guidance to a great extent.
Criteria: Perform a gap analysis to determine the differences between
current and needed skills and knowledge;
Extent to which IRS's practices conform to strategic guidance by type
of service: Telephone tax law: General (CAS): Conforms to guidance to
some extent;
Extent to which IRS's practices conform to strategic guidance by type
of service: Telephone tax law: More complex (TEC): Conforms to guidance
to little or no extent;
Extent to which IRS's practices conform to strategic guidance by type
of service: Walk-in assistance: Tax law (CARE): Conforms to guidance to
some extent;
Extent to which IRS's practices conform to strategic guidance by type
of service: Walk-in assistance: Return preparation (CARE): Conforms to
guidance to some extent.
Phase: strategy development and selection:
Criteria: Develop and apply criteria to determine when to use training
and development strategies to fill skills and knowledge gaps;
Extent to which IRS's practices conform to strategic guidance by type
of service: Telephone tax law: General (CAS): Conforms to guidance to
little or no extent;
Extent to which IRS's practices conform to strategic guidance by type
of service: Telephone tax law: More complex (TEC): Conforms to guidance
to little or no extent;
Extent to which IRS's practices conform to strategic guidance by type
of service: Walk-in assistance: Tax law (CARE): Conforms to guidance to
little or no extent;
Extent to which IRS's practices conform to strategic guidance by type
of service: Walk-in assistance: Return preparation (CARE): Conforms to
guidance to little or no extent.
Criteria: Ensure training and development efforts target factors that
affect accuracy and are linked to needed skills and knowledge;
Extent to which IRS's practices conform to strategic guidance by type
of service: Telephone tax law: General (CAS): Conforms to guidance to
some extent;
Extent to which IRS's practices conform to strategic guidance by type
of service: Telephone tax law: More complex (TEC): Conforms to guidance
to some extent;
Extent to which IRS's practices conform to strategic guidance by type
of service: Walk-in assistance: Tax law (CARE): Conforms to guidance to
some extent;
Extent to which IRS's practices conform to strategic guidance by type
of service: Walk-in assistance: Return preparation (CARE): Conforms to
guidance to some extent.
Criteria: Consider the anticipated costs and benefits of alternative
training and development efforts, ways to mitigate associated risks,
and the appropriate level of investment for training and development;
Extent to which IRS's practices conform to strategic guidance by type
of service: Telephone tax law: General (CAS): Conforms to guidance to
little or no extent;
Extent to which IRS's practices conform to strategic guidance by type
of service: Telephone tax law: More complex (TEC): Conforms to guidance
to little or no extent;
Extent to which IRS's practices conform to strategic guidance by type
of service: Walk-in assistance: Tax law (CARE): Conforms to guidance to
little or no extent;
Extent to which IRS's practices conform to strategic guidance by type
of service: Walk-in assistance: Return preparation (CARE): Conforms to
guidance to little or no extent.
Criteria: Establish a detailed plan for evaluating training, including
performance measures, data, planned analyses, and how the analyses
would be used;
Extent to which IRS's practices conform to strategic guidance by type
of service: Telephone tax law: General (CAS): See table 2 for our
assessments;
Extent to which IRS's practices conform to strategic guidance by type
of service: Walk-in assistance: Tax law (CARE): [Empty];
Extent to which IRS's practices conform to strategic guidance by type
of service: Walk-in assistance: Return preparation (CARE): [Empty].
Criteria: Establish process to ensure that strategic and tactical
changes can be incorporated into T&D efforts;
Extent to which IRS's practices conform to strategic guidance by type
of service: Telephone tax law: General (CAS): Conforms to guidance to
some extent;
Extent to which IRS's practices conform to strategic guidance by type
of service: Telephone tax law: More complex (TEC): Conforms to guidance
to some extent;
Extent to which IRS's practices conform to strategic guidance by type
of service: Walk-in assistance: Tax law (CARE): Conforms to guidance to
some extent;
Extent to which IRS's practices conform to strategic guidance by type
of service: Walk-in assistance: Return preparation (CARE): Conforms to
guidance to some extent.
Source: GAO analysis.
[End of table]
Goals and Priorities:
In the goals and priorities phase, the four units were relatively
strong in involving key stakeholders and communicating the importance
of training. The programs involved key stakeholders in annual training
planning decisions, shown by half-circles in table 1. However, they did
not have long-term planning processes in which to involve stakeholders.
All four programs had some upper-level management efforts to
communicate to staff the importance of training and development in
improving accuracy. CARE's was a model of communicating to all levels
of staff the importance of training. For example, CARE's management
guidance included managers responsible, communication vehicles, key
dates, and the message to be conveyed. The other programs communicated
with managers, but did not communicate more widely through the
organization.
With respect to goals, IRS does not have long-term goals, as opposed to
annual goals, for accuracy; nor do the four programs have goals for
training and development or measures of the impact of training on
accuracy. This observation about the lack of goals for accuracy is
consistent with other recent reports where we cited a lack of IRS long-
term goals, as well as reports and other assessments by OMB and
TIGTA.[Footnote 8] Two of the programs had annual goals, shown by half-
circles.
Some taxpayer assistance officials that we talked with questioned the
need for long-term accuracy goals or training and development goals and
measures. For example, W&I CAS telephone assistance officials said
setting long-term accuracy goals is not necessary because their annual
accuracy goals would move IRS toward improved performance. Program
officials also said that since staff training is considered a part of
doing business and is not managed as a program, training goals are
unnecessary. Further, some said that, given the number of factors that
affect accuracy, developing measures of training effectiveness would
not be possible.
However, without long-term accuracy goals, IRS, Congress, and others
are hampered in evaluating whether IRS is making satisfactory progress
improving taxpayers' service. As we said in a prior report on IRS's
telephone assistance program, a long-term, results-oriented goal would
provide a meaningful sense of direction as well as a yardstick for
measuring the results of operations and evaluating the extent of
improvements resulting from changes in resources, new technology, or
management of human capital.[Footnote 9] Similarly, goals and
performance measures for training would provide direction and help
measure progress. We recognize that collecting performance data can
sometimes be challenging, as discussed in the evaluation section. One
IRS taxpayer assistance program has been collecting performance data
suitable for determining the effectiveness of training.
Information Gathering and Assessments:
In this phase, the four programs determined and tracked selected
factors that affect accuracy, but would benefit from a more complete
analysis as well as benchmarking. IRS has identified, and tracked on an
individual basis, selected factors that affect accuracy. One
nontraining factor that IRS has tracked and analyzed is the use and
quality of the Probe and Response Guide, a manual provided to taxpayer
assistance staff intended to guide them in responding to taxpayers'
questions. IRS blamed decreases in accuracy in 2003 and 2004 on
problems with the Probe and Response Guide. According to IRS officials,
they used their analysis of the Probe and Response Guide to make the
guidance more usable by taxpayer assistance staff. Officials attribute
improvements in accuracy in 2005, at least in part, to this effort.
However, IRS has not conducted an analysis of the factors that affect
accuracy, including training--shown by half-circles in Table 1.
Specifically, IRS has not done an analysis to determine the relative
importance of the various factors, including training, that affect
accuracy. Determining the relative importance of the factors that
affect accuracy is important because there are multiple factors. Some
factors that affect accuracy are strategic and outside of IRS's direct
control such as tax law changes. Other factors are operational and
subject to IRS control such as manuals, information systems, and
training. Without an understanding of the relative importance each
factor has on accuracy, it is difficult for IRS to make informed
decisions about a strategy for improving accuracy, including training's
role in that strategy. Because of the purpose of the analysis and the
number of factors involved, the analysis might not give a precise
measure of each factor's impact. Furthermore, such analyses may be
conducted on an occasional basis.
None of the taxpayer assistance programs collected another type of
information--best practices of other organizations learned through
benchmarking. Some officials told us it would not be possible to
benchmark training programs because IRS has a unique mission. However,
the telephone taxpayer assistance programs have benchmarked other
aspects of their operations such as performance measures. Also, as we
have stated in another report, many processes that seem unique to the
government actually have counterparts in the private sector.[Footnote
10] Looking at processes in dissimilar organizations can lead to
rewarding improvements because it stimulates new thinking about
traditional approaches to doing work. For example, in the report cited
above, we noted that Xerox benchmarked L.L. Bean to improve order
fulfillment.
Still another type of information useful for planning can be obtained
from evaluations of training efforts. As we discuss in the background,
evaluation can provide useful feedback about the impact of existing
training to help plan improvements to training. Evaluation of IRS's
training efforts is the subject of our second section.
Skills and Knowledge Analysis:
All four assistance taxpayer assistance programs conducted annual needs
assessments. These assessments identified the knowledge and skills
assistors needed for the following year. None of them had longer term
analyses to project future needs. Determining skills and knowledge
needed is challenging for IRS, especially when IRS must react quickly
to tax law changes. For example, Congress passed a law in early January
2005 allowing taxpayers to deduct on either their 2004 or 2005 tax
returns contributions made during January 2005 for relief of the
victims of the December 2004 Indian Ocean Tsunami.[Footnote 11]
According to an IRS official, IRS had to react to this change, which
took place after the filing season had already begun, by quickly
alerting staff and providing them the information necessary to assist
taxpayers who had questions about the deduction.
However, none of the four programs had analyses of long-term needs,
such as improved proficiency in doing research, which is the basis for
the half-circles in table 1. Longer-term analyses of needs could help
the four programs better plan strategies to meet future needs. Such
planning could help ensure that the programs acquire, train, and retain
the needed staff.
Three of the four programs had knowledge and skills inventories based
on actual testing to determine employees' proficiency and knowledge
levels. TEC was the exception. TEC assumes the employees have an
underlying technical foundation because of their background in IRS.
However, in a recent TEC survey of 133 employees, the percentage of
employees who self-reported having pre-existing skills or being fully
proficient in several technical categories, such as depreciation, sale
of property, and trusts and fiduciaries, ranged from 31 percent to 62
percent.
Three of the four programs performed gap analyses to determine the
difference between current and needed skills and knowledge. Because of
the above limitations in needs analyses, the gap analysis was
necessarily annual not long term. Again, TEC was the exception. Because
TEC had not done a needs analysis, it could not do a gap analysis.
Strategy Development and Selection:
Weaknesses in analyses, described in the previous sections, hampered
the four programs' ability to develop criteria for when to use
training, as opposed to other strategies, for filling knowledge and
skills gaps, such as hiring and retention efforts. In addition, the
four programs lacked information about the benefits and costs of their
training efforts. As noted in the background section, IRS trains
several thousand tax law assistance staff annually but does not have
data on the cost of this significant effort. Without an understanding
of the usefulness of other strategies, and without an understanding of
the benefits and costs of training, IRS lacked information that would
be useful for making resource allocation decisions. This matters
because the resources devoted to training are significant. With better
information to select and develop a strategy, IRS might be able to
improve accuracy and save resources.
Evaluating the Impact of Training on Accuracy Could Provide a Basis for
Future Improvements:
Given the importance of accurate answers to taxpayers' questions and
the resources spent on training, the four assistance programs would
benefit from more sophisticated evaluations of the effectiveness of
training. The four programs all conduct some evaluations of their
training efforts. However, only TEC attempted an evaluation of the
impact of training on accuracy. In table 2, the lack of evaluation
plans in the other three programs had a cascading effect that left the
programs generally unable to fully satisfy our assessment criteria in
subsequent evaluation phases--data collection, data analysis, and
application of evaluation results. Appendices VII through X show the
evidence supporting our assessments in table 2.
Table 2: Assessments of IRS Practices in Evaluating Training and
Development of Staff to Provide Accurate Taxpayer Assistance:
Evaluation Phase: evaluation plan:
Criteria: Establish an overall approach for evaluating the impact of
training on accuracy;
Extent to which IRS's practice conforms to strategic guidance by type
of service: Telephone tax law: General: (CAS): Conforms to guidance to
some extent;
Extent to which IRS's practice conforms to strategic guidance by type
of service: Telephone tax law: Complex (TEC): Conforms to guidance to a
great extent;
Extent to which IRS's practice conforms to strategic guidance by type
of service: Walk-in assistance: Tax law (CARE): Conforms to guidance to
some extent;
Extent to which IRS's practice conforms to strategic guidance by type
of service: Walk-in assistance: Return preparation (CARE): Conforms to
guidance to some extent.
Criteria: Systematically analyze the feasibility and cost effectiveness
of alternative methodologies for evaluating the impact of training and
development on accuracy;
Extent to which IRS's practice conforms to strategic guidance by type
of service: Telephone tax law: General: (CAS): Conforms to guidance to
little or no extent;
Extent to which IRS's practice conforms to strategic guidance by type
of service: Telephone tax law: Complex (TEC): Conforms to guidance to
little or no extent;
Extent to which IRS's practice conforms to strategic guidance by type
of service: Walk-in assistance: Tax law (CARE): Conforms to guidance to
little or no extent;
Extent to which IRS's practice conforms to strategic guidance by type
of service: Walk-in assistance: Return preparation (CARE): Conforms to
guidance to little or no extent.
Criteria: Before implementing training, develop analysis plans,
including data to be collected, analyses to be conducted, and how the
analyses will be used;
Extent to which IRS's practice conforms to strategic guidance by type
of service: Telephone tax law: General: (CAS): Conforms to guidance to
some extent;
Extent to which IRS's practice conforms to strategic guidance by type
of service: Telephone tax law: Complex (TEC): Conforms to guidance to a
great extent;
Extent to which IRS's practice conforms to strategic guidance by type
of service: Walk-in assistance: Tax law (CARE): Conforms to guidance to
some extent;
Extent to which IRS's practice conforms to strategic guidance by type
of service: Walk-in assistance: Return preparation (CARE): Conforms to
guidance to some extent.
Evaluation Phase: data collection:
Criteria: Ensure that data are collected systematically and in a timely
manner;
Extent to which IRS's practice conforms to strategic guidance by type
of service: Telephone tax law: General: (CAS): Conforms to guidance to
some extent;
Extent to which IRS's practice conforms to strategic guidance by type
of service: Telephone tax law: Complex (TEC): Conforms to guidance to a
great extent;
Extent to which IRS's practice conforms to strategic guidance by type
of service: Walk-in assistance: Tax law (CARE): Conforms to guidance to
some extent;
Extent to which IRS's practice conforms to strategic guidance by type
of service: Walk-in assistance: Return preparation (CARE): Conforms to
guidance to some extent.
Criteria: Ensure that data are accurate, valid, complete, and reliable;
Extent to which IRS's practice conforms to strategic guidance by type
of service: Telephone tax law: General: (CAS): Conforms to guidance to
some extent;
Extent to which IRS's practice conforms to strategic guidance by type
of service: Telephone tax law: Complex (TEC): Conforms to guidance to a
great extent;
Extent to which IRS's practice conforms to strategic guidance by type
of service: Walk-in assistance: Tax law (CARE): Conforms to guidance to
some extent;
Extent to which IRS's practice conforms to strategic guidance by type
of service: Walk-in assistance: Return preparation (CARE): Conforms to
guidance to some extent.
Evaluation Phase: data analysis:
Criteria: Analyze data collected to assess the impact of training on
accuracy;
Extent to which IRS's practice conforms to strategic guidance by type
of service: Telephone tax law: General: (CAS): Conforms to guidance to
some extent;
Extent to which IRS's practice conforms to strategic guidance by type
of service: Telephone tax law: Complex (TEC): Conforms to guidance to
some extent;
Extent to which IRS's practice conforms to strategic guidance by type
of service: Walk-in assistance: Tax law (CARE): Conforms to guidance to
some extent;
Extent to which IRS's practice conforms to strategic guidance by type
of service: Walk-in assistance: Return preparation (CARE): Conforms to
guidance to some extent.
Criteria: Compare accuracy benefits to training costs;
Extent to which IRS's practice conforms to strategic guidance by type
of service: Telephone tax law: General: (CAS): Conforms to guidance to
little or no extent;
Extent to which IRS's practice conforms to strategic guidance by type
of service: Telephone tax law: Complex (TEC): Conforms to guidance to
little or no extent;
Extent to which IRS's practice conforms to strategic guidance by type
of service: Walk-in assistance: Tax law (CARE): Conforms to guidance to
little or no extent;
Extent to which IRS's practice conforms to strategic guidance by type
of service: Walk-in assistance: Return preparation (CARE): Conforms to
guidance to little or no extent.
Criteria: Benchmark training cost, training outcomes (accuracy), and
analytical methods against high-performing organizations;
Extent to which IRS's practice conforms to strategic guidance by type
of service: Telephone tax law: General: (CAS): Conforms to guidance to
little or no extent;
Extent to which IRS's practice conforms to strategic guidance by type
of service: Telephone tax law: Complex (TEC): Conforms to guidance to
little or no extent;
Extent to which IRS's practice conforms to strategic guidance by type
of service: Walk-in assistance: Tax law (CARE): Conforms to guidance to
little or no extent;
Extent to which IRS's practice conforms to strategic guidance by type
of service: Walk- in assistance: Return preparation (CARE): Conforms to
guidance to little or no extent.
Criteria: Analyze internal and external stakeholders' assessments of
training to include impact on accuracy;
Extent to which IRS's practice conforms to strategic guidance by type
of service: Telephone tax law: General: (CAS): Conforms to guidance to
some extent;
Extent to which IRS's practice conforms to strategic guidance by type
of service: Telephone tax law: Complex (TEC): Conforms to guidance to
some extent;
Extent to which IRS's practice conforms to strategic guidance by type
of service: Walk-in assistance: Tax law (CARE): Conforms to guidance to
some extent;
Extent to which IRS's practice conforms to strategic guidance by type
of service: Walk-in assistance: Return preparation (CARE): Conforms to
guidance to some extent.
Evaluation Phase: application of evaluation results:
Criteria: Evaluate training program as a whole, in addition to
individual course evaluations, and document the results;
Extent to which IRS's practice conforms to strategic guidance by type
of service: Telephone tax law: General: (CAS): Conforms to guidance to
little or no extent;
Extent to which IRS's practice conforms to strategic guidance by type
of service: Telephone tax law: Complex (TEC): Conforms to guidance to
some extent;
Extent to which IRS's practice conforms to strategic guidance by type
of service: Walk-in assistance: Tax law (CARE): Conforms to guidance to
little or no extent;
Extent to which IRS's practice conforms to strategic guidance by type
of service: Walk-in assistance: Return preparation (CARE): Conforms to
guidance to little or no extent.
Criteria: Incorporate evaluation results into the training and
development program to improve accuracy;
Extent to which IRS's practice conforms to strategic guidance by type
of service: Telephone tax law: General: (CAS): Conforms to guidance to
some extent;
Extent to which IRS's practice conforms to strategic guidance by type
of service: Telephone tax law: Complex (TEC): Conforms to guidance to
some extent;
Extent to which IRS's practice conforms to strategic guidance by type
of service: Walk-in assistance: Tax law (CARE): Conforms to guidance to
some extent;
Extent to which IRS's practice conforms to strategic guidance by type
of service: Walk-in assistance: Return preparation (CARE): Conforms to
guidance to some extent.
Source: GAO analysis.
[End of table]
Evaluation Plan Phase:
IRS adopted a four-level model based on the widely accepted Kirkpatrick
model[Footnote 12] for evaluating training. Under this model, the
sophistication of analysis increases as the numerical level of analysis
increases:
* Level 1 --Reaction. The goal is to measure participants' reaction to
training, usually through questionnaires.
* Level 2 - Learning. The goal is to determine what the training
participants learned through various kinds of tests administered
immediately after the training is completed.
* Level 3 - Behavior. The goal is to determine if the job performance
of the training participants changed in the aftermath of the training.
The most common means for making this determination is the
administration of a survey of trained staff and their supervisors 3
months, on average, after training is complete.
* Level 4 - Results. The goal is to determine if the training led to
the desired results, in this case, improved accuracy of taxpayer
assistance.
Options for level 4 analyses might include statistical correlations
between training and accuracy and controlled experiments where some
staff receive new training and others do not. Such ROI analyses are not
a part of IRS's evaluation approach. This level is sometimes split into
two levels with the fifth level--often referred to as return on
investment (ROI)--representing a comparison of costs and benefits
quantified in dollars.
Not all training and development programs are suitable for evaluations
of their effect on organizational results. The difficulty and costs of
analyzing the impact of training on accuracy need to be weighed against
the benefits of such an evaluation.
However, the more significant the activity targeted by the training and
development program, the greater the need for level 4 analysis. Factors
to consider when deciding the appropriate level of evaluation include
estimated costs of the training effort, size of the training audience,
management interest, program visibility, and anticipated life span of
the effort. As noted in the background, IRS devotes significant
resources to assisting taxpayers and training assistance staff.
Congress has also expressed great interest in improving taxpayer
service, such as accuracy. For example, much of the focus of the
Internal Revenue Service Reform and Restructuring Act of 1998[Footnote
13] was on in improving taxpayer service.
IRS's Human Capital Office recognizes the need to do level 4
evaluations. The office has a policy statement stating that level 4
evaluations should be done for all mission-critical training.
Despite the large investment of resources, significant congressional
attention, and Human Capital Office guidance, IRS officials involved in
three of the four taxpayer assistance training programs had not agreed
on whether to conduct a level 4 evaluation. They conducted analyses
only at lower levels, usually levels 1 or 2. By contrast, officials at
L&E and TEC agreed to conduct a pilot test of a level 4 analysis in
2004 for more complex tax law questions answered by telephone. Although
the pilot test was not successful in 2004--for reasons discussed below
in the data analysis phase--L&E and TEC intend to use the data
collection and analysis plan created for 2004 to conduct a similar
evaluation in 2005.
Data Collection Phase:
For three of the four taxpayer assistance programs, as shown in table
2, the lack of level 4 analyses had a cascading effect that left the
programs unable to fully satisfy our assessment criteria in the
subsequent evaluation phases. All four programs had controls in place
to help ensure systematic, timely data collection for level 1 and level
3 evaluations. TEC also had such controls in place for its level 4
evaluation. However, the database used to store level 2 data--the
Administrative Corporate Education System (ACES)--is no longer in
place, and there is no replacement system planned. With the exception
of TEC, the programs did not attempt to collect level 4 data. Database
controls and data collection plans for TEC's level 4 pilot evaluation
helped ensure the systematic and timely collection of data.
All four units had controls in place to help ensure that level 1 and
level 3 data were accurate, valid, reliable, and complete.[Footnote 14]
Because of the lack of a level 2 database, none of the four units had
controls in place for that data. TEC was concerned that the accuracy
data, which were based on test telephone calls, did not reflect the
types of calls that taxpayers actually made. TEC and IRS's quality
review staff are working together to try to address these concerns.
Data Analysis Phase:
All four assistance programs conducted some analyses of their training
efforts and analyzed stakeholders' assessments of training to identify
potential improvements to individual courses. A level 1 analysis,
usually surveys of trainees' opinions about a course, was conducted for
all courses. The officials also said level 2 analyses, often testing,
are generally done while level 3 analyses are infrequently done.
None of the four assistance programs successfully completed a level 4
evaluation of the impact of training on accuracy, compared benefits to
costs, benchmarked training evaluation, or analyzed stakeholders'
assessments of the impact of training on accuracy. The 2004 TEC level 4
pilot test was not completed because the level of proficiency of TEC
staff in the five technical categories reviewed was insufficient.
Application of Results Phase:
All the business units applied the results of their data analysis to
individual training courses, not their training efforts as a whole. For
example, they used the results of their level 1 and level 3 analyses to
make improvements in individual training courses and to identify
training needs for the upcoming year. However, because there had been
no successful evaluations of the impact of training on accuracy,
evaluation results could not be used to plan a strategy to improve
accuracy.
Conclusions:
Training has the potential to improve the service received from IRS by
millions of taxpayers. To its credit, IRS has planning processes in
place to address the challenges of training staff for each year's
filing season. The challenges include training staff to answer
questions about annual tax law changes--changes that are often very
complex. Although we do not know how much training has contributed,
IRS's taxpayer assistance accuracy has improved in recent years. On the
other hand, IRS's current level of accuracy remains a concern,
especially the accuracy of walk-in assistance and return preparation.
In addition, IRS devotes significant resources to training its tax law
assistance staff.
A more strategic approach to planning and evaluation would have several
benefits. Strategic planning could help managers better understand the
extent to which training, as opposed to other factors that affect
accuracy, could be used to improve accuracy. Evaluations of training's
impact on accuracy could help managers better understand which specific
training techniques are effective and which are not. While potentially
difficult to design and costly to conduct, IRS's Human Capital Office
has policy guidance urging more such analyses. In addition, one
taxpayer assistance program is pilot testing such an analysis. Gap
analyses, evaluations, and cost-benefit comparisons might also
contribute to providing training at lower cost by distinguishing
between effective and ineffective training.
Recommendations:
We recommend that the Commissioner of Internal Revenue take appropriate
action to ensure that IRS's planning and evaluations of its taxpayer
assistance training and development efforts better conform to guidance
for strategically planning and evaluating the training efforts.
Specifically, in the area of planning, we recommend that IRS:
* Establish a long-term goal for the accuracy of taxpayer assistance.
* Establish goals and measures for training and development logically
linked to accuracy.
* Determine and track the relative importance of the various factors,
including training, that affect accuracy.
* Benchmark training and development programs against high-performing
organizations.
* Conduct skills and knowledge gap analyses for all taxpayer assistance
programs, to include identifying and comparing current skills to long-
term skill needs.
* Consider costs, benefits, ways to mitigate risks, and the appropriate
level of investment for training and development efforts.
In the area of evaluation, we recommend that IRS continue to pursue the
level 4 pilot in TEC and, if that analysis is shown to be feasible,
conduct level 4 evaluations for its other taxpayer assistance training
and development programs. The evaluations should include the following:
* an analysis of the feasibility and cost effectiveness of alternative
level 4 methodologies and a data collection and analysis plan,
* a comparison of the accuracy benefits to the costs of the training,
* benchmarking of the analytical methods and the results of the data
analysis against high-performing organizations, and:
* an analysis of stakeholder assessments of the impact of training on
accuracy.
We also recommend that IRS replace the defunct ACES database, which had
been used to store level 2 data, with another database for this
purpose.
Agency Comments and Our Evaluation:
The Commissioner of Internal Revenue provided written comments on a
draft of this report in a letter dated July 6, 2005 (see app. XII). He
said the report offers valuable insight, is timely, and has been shared
with the project manager for IRS's recently-initiated effort to
reengineer its learning and education processes. Of the eight
recommendations we made, the Commissioner agreed with five
recommendations: (1) establish long-term accuracy goals, (2) determine
and track the relative importance of factors that affect accuracy, (3)
benchmark training and development programs, (4) conduct level 4
evaluations, and (5) replace IRS's system for storing level 2
evaluation data.
The Commissioner partially responded to the remaining three
recommendations. In commenting on our recommendation to establish
training goals and measures linked to accuracy, the Commissioner
focused on evaluation. Although related to after-the-fact evaluation,
setting clear training goals and measures to ascertain progress toward
those goals--consistent with agency goals which, in the case of
taxpayer assistance, would include accuracy--is an important up-front
planning step by which key stakeholders should agree on what training
success is and how it will be assessed.
The Commissioner said that IRS recognizes the value of conducting
skills and knowledge gap analyses. He summarized the programs' efforts
to identify short-term skills needs, which we recognized in this
report. However, he did not identify how short-term gap analyses would
be conducted for the program for responding to more complex tax
questions. Nor did the Commissioner discuss gap analyses to identify
long-term skills needed to reach accuracy goals. Failing to conduct gap
analyses, including analyses of strategic changes--such as economic,
technological, and demographic developments--can hinder performance and
the development of strategies that integrate new capabilities and
provide flexibility to meet new challenges and improve service.
In commenting on our recommendation that IRS consider the costs and
benefits of training efforts, the Commissioner's comments did not
specifically mention benefits and costs but did mention unfunded needs.
Given the resources dedicated to training staff and providing taxpayer
assistance, and the impact that assistance accuracy can have on
taxpayers, the taxpayer assistance programs would benefit from
analytically-based assurance that training efforts focus in a cost
beneficial way on achieving accuracy goals.
As we agreed with your offices, unless you publicly announce the
contents of this report earlier, we plan no further distribution of it
until 30 days from the date of this letter. We will then send copies of
this report to the Chairman of the Senate Committee on Finance, the
Chairman and Ranking Minority Member of the House Committee on Ways and
Means, and the Chairman and Ranking Minority Member, Subcommittee on
Oversight, House Committee on Ways and Means. We will also send copies
to the Secretary of the Treasury; the Commissioner of Internal Revenue;
the Director, Office of Management and Budget; and other interested
parties. We will also make copies available to others on request. In
addition, the report will be available at no charge on the GAO Web site
at http://www.gao.gov.
If you or your staff have any questions about this report, please
contact me at (202) 512-9110 or whitej@gao.gov. Contact points for our
Offices of Congressional Relations and Public Affairs may be found on
the last page of this report. GAO staff who made major contributions to
this report are listed in appendix XIII.
Signed by:
James R. White:
Director, Tax Issues:
[End of section]
Appendix I: What GAO Looked For: Examples of Planning Practices That
Would Conform to Strategic Guidance for Training:
Phase: Goals and priorities:
Criterion: Establish quantitative long-term accuracy goals that link to
IRS's strategic goals;
Conforming examples: Authoritative quantitative targets for the level
of accuracy to be achieved at a future point in time across multiple
fiscal years; A logical link between the accuracy goals and higher-
level strategic goals e.g., such as to improve customer service.
Criterion: Involve key stakeholders, including human capital
professionals, managers, and employees, in key long-term training
planning decisions;
Conforming examples: Documented management processes to ensure that
stakeholders' views are taken into account in key planning milestones
and related decisions, such as developing the training and development
budget request and what measures will be used to evaluate training and
development; Surveys or interview employees to determine their views
and perceptions on training and development in general and more
specifically on competencies and skills needed for the future.
Criterion: Establish goals for, and measures to assess the
effectiveness of, training and development on accuracy;
Conforming examples: Authoritative quantitative targets related to
training and development program activities to be achieved at a future
point in time that are logically linked to accuracy goals; A documented
set of established measures used to assess the impacts of the training
and development programs, such as on accuracy or employee skills and
knowledge; A list of methods, tools, and measures used to assess the
impact of training and development, such as control groups, surveys, or
a trend analysis; Targets and goals in strategic and performance plans
that establish how training and development strategies are expected to
contribute to improved organizational and programmatic results.
Criterion: Affirm, through upper-level management communication, the
importance of training and development to improve accuracy;
Conforming examples: Demonstrated efforts to communicate throughout the
organization the importance that upper-level management attachés to
training and development, such as memos, letters, statements found in
strategic plans, or other authoritative documents that articulate
management emphasis on training and development to improve accuracy;
Comments from midlevel mangers and employees on the extent to which
management communicates the importance of training, such as statements
captured in an employee survey.
Phase: Information gathering and assessments:
Criterion: Analytically determine and track the strategic and
operational factors, including training, that affect accuracy;
Conforming examples: Management processes to systematically collect and
analyze data that will allow IRS to comprehensively identify and report
on the major factors affecting the level of accuracy achieved such as:
Strategic factors (factors largely outside IRS's immediate control)
that might include:
* Complexity/changes of the tax law;
* Levels of funding or budget limitations;
* Economic trends;
* Available pools of staff to select from;
Operational factors (factors within IRS's immediate control) that might
may include:
* Workload placed upon assistors;
* Changes in CSR guidance;
* Available learning tools and job aides;
* Community of practice & communication of lessons learned;
* Manager/employee relations;
* Incentive/rewards & recognition program; Documentation that outlines
the analytical methods used to identify factors affecting accuracy;
Periodic, scheduled, systematic studies to determine the key factors
affecting accuracy; Periodic regular efforts to systemically monitor
the impacts strategic and operational factors have had on accuracy,
such as studies of factors over an extended amount of time, such as 2-3
years, to gauge whether IRS has experienced increased or decreased
levels of accuracy due to these factors.
Criterion: Incorporate evaluation results into training and development
planning;
Conforming examples: See appendix II.
Criterion: Benchmark the training and development program against high-
performing organizations;
Conforming examples: Periodic studies conducted of the training and
development programs of high-performing customer service organizations
providing similar services compared to identify potential improvements,
to include such elements as, for example:
* Employee skill level;
* Resources invested in training and development;
* Training and development curriculum;
* Geographic and demographic trends.
Criterion: Assess whether human resource information systems provide
needed data in a timely manner;
Conforming examples: Management processes to identify the user
requirements for its human resource information system used in planning
training and development, such as:
* Identifying the user needs;
* Surveying managers to determine if the information provided met their
needs in a timely manner.
Phase: Skills and knowledge analysis:
Criterion: Perform a needs assessment to determine the knowledge and
skills needed now and in the future;
Conforming examples: Management processes that include steps that
require IRS to identify the critical knowledge and skills its employees
need now and in the future to provide accurate tax law assistance,
including evidence such as documentation that explicitly outlines the
knowledge and skills requirements for taxpayer assistors to reach
strategic accuracy goals.
Criterion: Conduct a knowledge and skills inventory to identify
employees' current skills and knowledge;
Conforming examples: Management processes designed to ensure that IRS
has determined the current knowledge and skills of its employees.
Methods used to obtain this information may include:
* Completed knowledge and skills surveys;
* Proficiency tests;
* Manager assessment.
Criterion: Perform a gap analysis to determine the differences between
skills and knowledge needed now and in the future;
Conforming examples: A documented analysis that compares IRS's current
knowledge and skills to the knowledge and skills needed to effectively
assist taxpayers now and in the future to reach accuracy goals.
Phase: Strategy development and selection:
Criterion: Develop and apply criteria to determine when to use training
and development strategies to fill skills and knowledge gaps;
Conforming examples: As part of its process for determining how to fill
skill gaps and address factors that affect accuracy, or as part of its
process for determining whether to implement a given training and
development program, steps to:
* Require that approaches other than training be considered to fill the
skill gap or address the factor and;
* Systematically analyze and consider the relative costs and benefits
of the alternative(s) to training;
* Develop a documented action plan or similar document that outlines
when training and development interventions should take place once a
knowledge and or skill gap has been identified.
Criterion: Ensure training and development efforts target factors that
affect accuracy and are linked to needed skills and knowledge;
Conforming examples: As part of its process for deciding which training
and development programs to implement, steps to ensure efforts will
target needed improvements and enhance needed skills, which would yield
such items as; A logical or explicit link between the training and
development programs offered and (1) the knowledge and skills
identified in the skill needs assessment or (2) the strategic and
operational factors identified as affecting accuracy (to the extent
that the needs and the identified factors could be addressed through
training); A documented audit trail showing that training programs
offered now and planned for the future resulted from identified needed
skills and knowledge.
Criterion: Consider the anticipated cost and benefits of alternative
training and development efforts, ways to mitigate associated risks,
and the appropriate level of investment for training and development;
Conforming examples: Management processes to establish and apply
criteria to select the level of funding invested for training and
development, such a process would yield such items as:
* A comprehensive systematic effort that considers all costs related to
training and development and the associated level of return linked to
the investment;
* Historical data that show IRS reallocates training and development
resources based on data derived from a cost- benefit analysis;
* An analysis of training and development investments from prior years
and the outcomes achieved;
* An analysis documenting various training and development investments
scenarios and the respective expected outcomes;
As part of its process for deciding which training and development
efforts to implement, steps to require that the relative costs and
benefits of alternative efforts be compared to current training and
development efforts which would include such components as:
* The relative projected impact of the efforts on needed skills,
factors that need to be addressed, and accuracy levels;
* Training and development delivery mechanisms;
* Staff time involved/FTE's;
* Logistical options for staff travel;
* Products used to train and develop employees; As part of its
processes for deciding which training and development efforts to
implement, steps to minimize risks by requiring that risks of
alternative investments be identified and that alternative ways of
mitigating those risks are identified and considered, to yield such
items as a documented analysis for each proposed investment describing
the details of the risks and how they could be addressed.
Criterion: Establish a detailed plan for evaluating training, including
performance measures, data, planned analyses, and how the analyses will
be used;
Conforming examples: See appendix II.
Criterion: Establish a process to ensure that strategic and tactical
changes can be incorporated into training and development efforts;
Conforming examples: As part of IRS's process for planning for training
and development, steps to ensure decision makers incorporate a forward
looking approach to planning which lay out steps to address potential
strategic and operational changes the could impact IRS's training and
development programs such as:
* A decrease in the training and education budget;
* Legislative changes affecting the tax law;
* Unexpected staffing shortages;
* Technological innovations;
* Restructuring or a reorganization.
Source: GAO.
[End of table]
[End of section]
Appendix II: What GAO Looked For: Examples of Evaluation Practices That
Would Conform to Strategic Guidance:
Phase: Evaluation plan:
Criterion: Establish an overall approach for evaluating the impact of
training on accuracy (a level 4 evaluation);
Conforming examples: An authoritative document, or set of documents,
that describes how IRS will evaluate taxpayer assistance training and
development over time, to include evaluations of the impact of training
on accuracy.
Criterion: Systematically analyze the feasibility and cost
effectiveness of alternative methodologies for evaluating the impact of
training and development on accuracy;
Conforming examples: For each line of effort to evaluate training and
development impact, and for each major training and development
program, a documented comparison of alternative analytical approaches
to include comparisons of their practicality and potential costs and
effectiveness.
Criterion: Before implementing training, develop analysis plans,
including data to be collected, analyses to be conducted, and how the
analyses will be used;
Conforming examples: For each line of effort to evaluate training and
development impact, and for each major training and development
program, a document that describes in detail the specific data that
will be collected and the analyses that will be conducted.
Phase: Data collection:
Criterion: Ensure that data are collected systematically and in a
timely manner;
Conforming examples: Documentation of protocols and systematic
procedures for data collection specified in a data collection plan that
includes details of methods to ensure timely collection of data.
Criterion: Ensure that data are accurate, valid, complete, and
reliable;
Conforming examples: For each line of effort and for each set of data
collected to evaluate training and development, spot checks for
accuracy and completeness, validity reports, confirmation for each set
of data, tests to ensure data reliability, and/or data collection
procedures designed to ensure data reliability.
Phase: Data analysis:
Criterion: Analyze data collected to assess the impact of training on
accuracy;
Conforming examples: Steps to ensure that analyses will help determine
the impact of training and development on accuracy performance, such
as;
* Analyses have considered and/or used an array of approaches (both
qualitative and quantitative);
* Analyses that separate training from other strategic and operational
factors that might affect accuracy (to the extent that it is cost-
effective and/or feasible);
* Analyses enable relating processes to outputs and outcomes;
* Analyses allow for comparisons before and after training is taken;
* Analyses contain consistent qualities, allowing for tracking and
comparison of processes and results over time.
Criterion: Compare accuracy benefits to training costs;
Conforming examples: Steps to ensure that there are documented
comparisons of the costs and benefits for each key training and
development program effort to the extent deemed feasible, such as;
* Comparisons of benefits (including qualitative, estimated, and in
some cases monetized benefits) to the costs of the training and
development program;
* Use of forward-looking analytical approaches, such as forecasting and
trend analysis, to aid in estimating and comparing future performance
with and without the training intervention.
Criterion: Benchmark training cost, training outcomes (accuracy), and
analytical methods against high-performing organizations;
Conforming examples: Efforts to compare IRS's approach to that of high-
performing organizations in such areas as training for customer
assistance, including telephone and live assistance, spending on
training, ways to analyze the impact of training, and the effectiveness
of training.
Criterion: Analyze internal and external stakeholders' assessments of
training to include impact on accuracy;
Conforming examples: Efforts to analyze and consider stakeholder
feedback, including feedback from internal and external obtained from
such methods as:
* survey results used as data in various analytical reports on training
to improve accuracy;
* Panel studies, task forces, etc. aimed at accuracy-specific training
guidance;
* Regular employee satisfaction surveys and focus groups focusing
directly on training for accuracy.
Phase: Application of evaluation results:
Criterion: Evaluate training program as a whole, in addition to
individual course evaluations, and document the results;
Conforming examples: Reports detailing the results of analyses
performed to evaluate accuracy training and development with
assessments of quantitative and qualitative data pulled together in a
comprehensive way to integrate conclusions from each set of data.
Criterion: Incorporate evaluation results into the training and
development program to improve accuracy;
Conforming examples: Authoritative documents stating how evaluation
results will be used to inform, modify, and improve planning, such as;
* Annual performance reports comparing actual to target performance;
Memos, minutes, etc. from budget and training planning sessions that
describe decision making based on evaluations of training initiatives
aimed at specific accuracy goals; Indications that the agency is making
fact-based determinations of the impact of its training and development
programs by using these assessments to refine or redesign training and
development efforts as needed.
Source: GAO.
[End of table]
[End of section]
Appendix III: Assessments of Planning: Less Complex Tax Law Telephone
Service by W&I CAS:
Phase: goals and priorities:
Criteria: Establish quantitative long-term accuracy goals that link to
IRS's strategic goals;
Extent: Conforms to guidance to some extent;
Evidence summary: CAS reported an annual accuracy goal linked to IRS's
strategic goal of improving taxpayer service. The accuracy goal was not
long-term. CAS had accuracy projections through 2010, but according to
IRS officials, these projections were only internal and subject to
change. In addition, OMB stated that IRS did not have long-term goals.
Criteria: Involve key stakeholders, including human capital
professionals, managers, and employees, in key long-term training
planning decisions;
Extent: Conforms to guidance to some extent;
Evidence summary: CAS involved stakeholders in its annual planning
decisions. For example, CAS sought input from employees and managers on
training priorities and needed changes to training. However, CAS did
not have a long-term training planning process for involving key
stakeholders in such strategic issues as establishing training program
goals and long-term employee developmental needs.
Criteria: Establish goals for, and measures to assess the effectiveness
of, training and development on accuracy;
Extent: Conforms to guidance to little or no extent;
Evidence summary: CAS had no such goals or measures.
Criteria: Affirm, through upper-level management communication, the
importance of training and development to improve accuracy;
Extent: Conforms to guidance to some extent;
Evidence summary: CAS management had efforts that conveyed the
importance of training, such as an annual letter sent to field office
directors that referred to the need for specific types of training.
However, management did not communicate more widely throughout the
organization the strategic value of training and development and its
importance in achieving long-term accuracy goals.
Phase: information gathering and assessments.
Criteria: Analytically determine and track the strategic and
operational factors, including training, that affect accuracy;
Extent: Conforms to guidance to some extent;
Evidence summary: CAS collected and analyzed accuracy data by using a
quality review system to identify operational factors affecting
accuracy, such as the proper use of IRS guidance to assistors, and the
use of a pareto analysis to determine the correlation between the
causes and effects of key errors. However, CAS did not analyze and
track the strategic factors such as tax law changes. As a result, CAS
did not have information on the impact of training on accuracy, while
holding other factors such as the quality of assistors' guidance or tax
law changes, constant.
Criteria: Incorporate evaluation results into training and development
planning;
Extent: N/A;
Evidence summary: For our assessment, see "Application of Evaluation
Results" phase in appendix VII.
Criteria: Benchmark the training and development program against high-
performing organizations;
Extent: Conforms to guidance to little or no extent;
Evidence summary: CAS had not benchmarked its training and development
program. IRS had benchmarked other nontraining practices related to
customer service such as performance measures.
Criteria: Assess whether human resource information systems provide
needed data in a timely manner;
Extent: N/A;
Evidence summary: It was too early for an assessment because the
Enterprise Learning Management System (ELMS), which would be CAS's
primary system used for providing human resource information, is being
implemented in fiscal year 2005.
Phase: skills and knowledge analysis:
Criteria: Perform a needs assessment to determine the knowledge and
skills needed now and in the future;
Extent: Conforms to guidance to some extent;
Evidence summary: CAS identified the knowledge and skills assistors
needed for the following year. For example, CAS projected the number of
questions expected by tax topic and took into account recent tax law
changes. However, CAS had not identified the specific knowledge and
skills needed to achieve long-term accuracy goals.
Criteria: Conduct a knowledge and skills inventory to identify
employees' current skills and knowledge;
Extent: Conforms to guidance to a great extent;
Evidence summary: CAS had an inventory of assistors' current knowledge
and skills based on testing.
Criteria: Perform a gap analysis to determine the differences between
skills and knowledge needed now and in the future;
Extent: Conforms to guidance to some extent;
Evidence summary: CAS did an annual gap analysis, but because CAS had
not performed a needs assessment for a period that extended beyond the
next year, CAS was unable to determine the differences between current
skills and knowledge and skills and knowledge needed in the future.
Phase: strategy development and selection:
Criteria: Develop and apply criteria to determine when to use training
and development strategies to fill skills and knowledge gaps;
Extent: Conforms to guidance to little or no extent;
Evidence summary: CAS had no criteria to guide decisions on using
training and development strategies, as opposed to other strategies,
such as improving guidance or hiring practices, to address skills and
knowledge gaps.
Criteria: Ensure training and development efforts target factors that
affect accuracy and are linked to needed skills and knowledge;
Extent: Conforms to guidance to some extent;
Evidence summary: CAS had practices such as employee testing, and
studies conducted to identify the most frequent errors made on an
annual basis, that allowed CAS to link training efforts to identified
short-term skills and knowledge gaps. However, because of limitations
in long-term gap analyses, CAS had limited ability to ensure training
and development efforts were linked to long- term needed skills and
knowledge.
Criteria: Consider the anticipated cost and benefits of alternative
training and development efforts, ways to mitigate associated risks,
and the appropriate level of investment for training and development;
Extent: Conforms to guidance to little or no extent;
Evidence summary: CAS did not have these analyses.
Criteria: Establish a detailed plan for evaluating training, including
performance measures, data, planned analyses, and how the analyses will
be used;
Extent: N/A;
Evidence summary: For our assessment, see "Evaluation Plan" phase in
appendix VII.
Criteria: Establish a process to ensure that strategic and tactical
changes can be incorporated into training and development efforts;
Extent: Conforms to guidance to some extent;
Evidence summary: CAS had efforts to anticipate and react to potential
short-term changes that could affect training, such as tax law changes
or lack of experienced assistors due to staff turnover. However, CAS
did not have a training planning process to identify potential long-
term changes such as technological innovations or changes in the
economy that might impact training and development.
Source: GAO analysis.
[End of table]
[End of section]
Appendix IV: Assessments of Planning: More Complex Tax Law Telephone
Service by SB/SE[Footnote 15] TEC:
Phase: goals and priorities:
Criteria: Establish quantitative long-term accuracy goals that link to
IRS's strategic goals;
Extent: Conforms to guidance to little or no extent;
Evidence summary: TEC did not have such goals.
Criteria: Involve key stakeholders, including human capital
professionals, managers, and employees, in key long-term planning
decisions;
Extent: Conforms to guidance to some extent;
Evidence summary: TEC involved stakeholders in its annual training
planning. For example, TEC surveyed employees to obtain input on
training needs for the upcoming year. However, TEC did not have a long-
term training planning process for involving key stakeholders in such
strategic issues as establishing training program goals and long-term
employee developmental needs.
Criteria: Establish goals for, and measures to assess the effectiveness
of, training and development on accuracy;
Extent: Conforms to guidance to little or no extent;
Evidence summary: TEC had no such goals or measures.
Criteria: Affirm, through upper-level management communication, the
importance of training and development on accuracy;
Extent: Conforms to guidance to some extent;
Evidence summary: TEC management had efforts that conveyed the
importance of training, such as an annual letter sent to field office
directors that referred to the need for specific types of training.
However, management did not communicate more widely throughout the
organization the strategic value of training and development and its
importance in achieving long-term accuracy goals.
Phase: information gathering and assessments:
Criteria: Analytically determine and track the strategic and
operational factors, including training, that affect accuracy;
Extent: Conforms to guidance to some extent;
Evidence summary: TEC collected and analyzed accuracy data to identify
factors, such as an analysis of quality review system data. However,
TEC did not analyze and track the strategic factors such as changes to
tax law. As a result, TEC did not have information on the impact of
training on accuracy, while holding other factors such as the quality
of assistors' guidance or tax law changes, constant.
Criteria: Incorporate evaluation results into training and development
planning;
Extent: N/A;
Evidence summary: For our assessment, see "Application of Evaluation
Results" phase in appendix VIII.
Criteria: Benchmark the training and development program against high-
performing organizations;
Extent: Conforms to guidance to little or no extent;
Evidence summary: TEC had not benchmarked its training and development
program.
Criteria: Assess whether human resource information systems provide
needed data in a timely manner;
Extent: N/A;
Evidence summary: It was too early for an assessment because the
Enterprise Learning Management System (ELMS), which would be TEC's
primary system used for providing human resource information, is being
implemented in fiscal year 2005.
Phase: skills and knowledge analysis:
Criteria: Perform a needs assessment to determine the knowledge and
skills needed now and in the future;
Extent: Conforms to guidance to some extent;
Evidence summary: Annually, TEC identified the types of tax law topics
it would be handling and identified the knowledge and skills assistors
needed. This process included taking account of recent tax law changes.
However, TEC had not identified the specific knowledge and skills
needed to achieve long-term accuracy goals.
Criteria: Conduct a knowledge and skills inventory to identify
employees' current skills and knowledge;
Extent: Conforms to guidance to little or no extent;
Evidence summary: TEC did not conduct an inventory of its employees'
current skills and knowledge. Instead, TEC assumed that its employees
had an underlying technical foundation because of their prior IRS
experience, and the annual training they received was to supplement
previously acquired skills.
Criteria: Perform a gap analysis to determine the differences between
current and needed skills and knowledge now and in the future;
Extent: Conforms to guidance to little or no extent;
Evidence summary: Because TEC had not done a knowledge and skills
inventory, a gap analysis could not be done.
Phase: strategy development and selection:
Criteria: Develop and apply criteria to determine when to use training
and development strategies to fill skills and knowledge gaps;
Extent: Conforms to guidance to little or no extent;
Evidence summary: TEC had no criteria to guide decisions on using
training and development strategies, as opposed to other strategies
such as improving guidance or hiring practices, to address skills and
knowledge gaps.
Criteria: Ensure training and development efforts target factors that
affect accuracy and are linked to needed skills and knowledge;
Extent: Conforms to guidance to some extent;
Evidence summary: In 2004, TEC matched its training courses to the
general skills and knowledge assistors needed. However, because of
limitations in long-term gap analyses, TEC had limited ability to
ensure training and development efforts were linked to long-term needed
skills and knowledge.
Criteria: Consider the anticipated cost and benefits of alternative
training and development efforts, ways to mitigate associated risks,
and the appropriate level of investment for training and development;
Extent: Conforms to guidance to little or no extent;
Evidence summary: TEC did not have these analyses.
Criteria: Establish a detailed plan for evaluating training, including
performance measures, data, planned analyses, and how the analyses will
be used;
Extent: N/A;
Evidence summary: For our assessment, see "Evaluation Plan" phase in
appendix VIII.
Criteria: Establish a process to ensure that strategic and tactical
changes can be incorporated into training and development efforts;
Extent: Conforms to guidance to some extent;
Evidence summary: TEC had informal processes for anticipating and
reacting to potential short-term changes that could affect training.
However, TEC did not have a training planning process to identify
potential long-term changes such as technological innovations or
changes in the economy that might impact training and development.
Source: GAO analysis.
[End of table]
[End of section]
Appendix V: Assessments of Planning: Tax Law Questions Walk-In Service
by W&I CARE:
Phase: goals and priorities:
Criteria: Establish quantitative long-term accuracy goals that link to
IRS's strategic goals;
Extent: Conforms to guidance to some extent;
Evidence summary: CARE reported an annual accuracy goal linked to IRS's
strategic goal of improving taxpayer service. The goal was not long-
term. CARE had accuracy projections through 2010, but according to IRS
officials, these projections were internal and subject to change. In
addition, OMB stated that IRS did not have long-term goals.
Criteria: Involve key stakeholders, including human capital
professionals, managers, and employees, in key long-term planning
decisions;
Extent: Conforms to guidance to some extent;
Evidence summary: CARE involved stakeholders in its annual planning.
For example, human capital training staff developed an annual training
plan based on needs identified by program staff. However, CARE did not
have a long-term training planning process in which to involve
stakeholders in such strategic issues as identifying measures to assess
training progress or identifying long-term employee developmental
needs.
Criteria: Establish goals for, and measures to assess the effectiveness
of, training and development on accuracy;
Extent: Conforms to guidance to little or no extent;
Evidence summary: CARE had no such goals or measures.
Criteria: Affirm, through upper-level management communication, the
importance of training and development to improve accuracy;
Extent: Conforms to guidance to a great extent;
Evidence summary: CARE implemented a plan for communicating to various
levels of staff, including front-line employees, management's vision
for training. The plan included the vehicle and product to be used to
communicate this message, along with key dates activities should occur
to deliver the message of how training and development plays a role in
providing quality taxpayer service and enhancing employee development
to achieve future goals.
Phase: information gathering and assessments.
Criteria: Analytically determine and track the strategic and
operational factors, including training, that affect accuracy;
Extent: Conforms to guidance to some extent;
Evidence summary: CARE collected and analyzed data, including quality
review system data, to identify operational factors affecting accuracy
to identify the most frequent errors. CARE also had studies to
determine why those errors occurred, such as why staff did not properly
use IRS guidance to answer questions. However, CARE did not analyze and
track the strategic factors such as changes to the tax law or
attrition. As a result, CARE did not have information on the impact
these factors had on accuracy holding other factors such as changes in
the tax law or change in guidance constant.
Criteria: Incorporate evaluation results into training and development
planning;
Extent: N/A;
Evidence summary: For our assessment see the "Application of evaluation
results" phase in appendix IX.
Criteria: Benchmark the training and development program against high-
performing organizations;
Extent: Conforms to guidance to little or no extent;
Evidence summary: CARE had not benchmarked its training and development
program. IRS had benchmarked other nontraining practices related to
customer service such as performance measures.
Criteria: Assess whether human resource information systems provide
needed data in a timely manner;
Extent: N/A;
Evidence summary: It was too early for an assessment because the
Enterprise Learning Management System (ELMS), which would be CARE's
primary system used for providing human resource information, is being
implemented in fiscal year 2005.
Phase: skills and knowledge analysis:
Criteria: Perform a needs assessment to determine the knowledge and
skills needed now and in the future;
Extent: Conforms to guidance to some extent;
Evidence summary: CARE had identified the knowledge and skills
assistors needed for the following year. One example is ensuring that
assistors have the skills and knowledge needed to respond to questions
about recent tax law changes. However, CARE had not identified the
specific knowledge and skills needed to achieve long-term accuracy
goals.
Criteria: Conduct a knowledge and skills inventory to identify
employees' current skills and knowledge;
Extent: Conforms to guidance to a great extent;
Evidence summary: CARE had an inventory of assistors' current knowledge
and skills based on a prescreening testing process.
Criteria: Perform a gap analysis to determine the differences between
current and needed skills and knowledge now and in the future;
Extent: Conforms to guidance to some extent;
Evidence summary: CARE did an annual gap analysis, but because CARE had
not performed a needs assessment for a period that extended beyond the
next year, it had not performed a gap analysis to determine the
differences between current skills and knowledge and skills and
knowledge needed in the future.
Phase: strategy development and selection.
Criteria: Develop and apply criteria to determine when to use training
and development strategies to fill skills and knowledge gaps;
Extent: Conforms to guidance to little or no extent;
Evidence summary: CARE had no criteria to guide decisions on using
training and development strategies, as opposed to other strategies,
such as improving guidance, or hiring practices to address skill gaps.
Criteria: Ensure training and development efforts target factors that
affect accuracy and are linked to needed skills and knowledge;
Extent: Conforms to guidance to some extent;
Evidence summary: CARE has practices such as employee testing, and
studies conducted to identify the most frequent errors made on an
annual basis, that allow CARE to link training efforts to identified
short-term skills and knowledge gaps; However because of limitations in
long term gap analyses, staff have limited ability to ensure training
and development efforts are linked to long term needed skills and
knowledge.
Criteria: Consider the anticipated costs and benefits of alternative
training and development efforts, ways to mitigate associated risks,
and the appropriate level of investment for training and development;
Extent: Conforms to guidance to little or no extent;
Evidence summary: CARE did not have these analyses.
Criteria: Establish a detailed plan for evaluating training, including
performance measures, data, planned analyses, and how the analyses will
be used;
Extent: N/A;
Evidence summary: For our assessment, see "Evaluation Plan" phase in
appendix IX.
Criteria: Establish a process to ensure that strategic and tactical
changes can be incorporated into training and development efforts;
Extent: Conforms to guidance to some extent;
Evidence summary: CARE had efforts to anticipate and react to potential
short-term changes that could affect training, such as tax law changes
or lack of experienced assistors due to staff turnover. However, CARE
did not have a training planning process to identify potential long-
term changes, such as technological innovations or changes in the
economy, that might impact training and development.
Source: GAO analysis.
[End of table]
[End of section]
Appendix VI: Assessments of Planning: Return Preparation Walk-In
Service by W&I CARE:
Phase: goals and priorities:
Criteria: Establish quantitative long-term accuracy goals that link to
IRS's strategic goals;
Extent: Conforms to guidance to little or no extent;
Evidence summary: CARE did not have quantitative long-term accuracy
goals for return preparation. Fiscal year 2005 will be the baseline
year for a new measure of tax preparation accuracy, so therefore there
are no annual or long-term goals.
Criteria: Involve key stakeholders, including human capital
professionals, managers, and employees, in key long-term planning
decisions;
Extent: Conforms to guidance to some extent;
Evidence summary: CARE involves stakeholders in annual planning. For
example, human capital training staff developed an annual training plan
based on needs identified by program staff. However, CARE did not have
a long-term training planning process in which to involve stakeholders
in such strategic issues as identifying measures to assess training
progress or identifying long- term employee developmental needs.
Criteria: Establish goals for, and measures to assess the effectiveness
of, training and development on accuracy;
Extent: Conforms to guidance to little or no extent;
Evidence summary: CARE had no such goals or measures.
Criteria: Affirm, through upper-level management communication, the
importance of training and development to improve accuracy;
Extent: Conforms to guidance to a great extent;
Evidence summary: CARE implemented a plan for communicating to various
levels of staff, including front-line employees, management's vision
for training. The plan included the vehicle and product to be used to
communicate this message, along with key dates activities should occur
to deliver the message of how training and development play a role in
providing quality taxpayer service and enhancing employee development
to achieve future goals.
Phase: information gathering and assessments:
Criteria: Analytically determine and track the strategic and
operational factors, including training, that affect accuracy;
Extent: Conforms to guidance to some extent;
Evidence summary: CARE collected and analyzed accuracy data, including
quality review system data, to identify operational factors affecting
accuracy to identify the most frequent errors. CARE also had studies to
determine why those errors occurred, such as why staff did not properly
use IRS guidance to answer questions. However, CARE did not analyze and
track the strategic factors such as changes to the tax law or
attrition. As a result, CARE did not have information on the impact
these factors will have on accuracy holding other factors such as
changes in the tax law or change in guidance constant.
Criteria: Incorporate evaluation results into training and development
planning;
Extent: N/A;
Evidence summary: For our assessment see the "Application of evaluation
results" phase in appendix X.
Criteria: Benchmark of the training and development program against
high-performing organizations;
Extent: Conforms to guidance to little or no extent;
Evidence summary: CARE had not benchmarked its training and development
program. IRS had benchmarked other nontraining practices related to
customer service such as performance measures.
Criteria: Assess whether human resource information systems provide
needed data in a timely manner;
Extent: N/A;
Evidence summary: It was too early for an assessment because the
Enterprise Learning Management System (ELMS), which would be CARE's
primary system used for providing human resource information, is being
implemented in fiscal year 2005.
Phase: skills and knowledge analysis:
Criteria: Perform a needs assessment to determine the knowledge and
skills needed now and in the future;
Extent: Conforms to guidance to some extent;
Evidence summary: CARE identified the knowledge and skills assistors
needed for the following year. One example is ensuring that assistors
have the skills and knowledge needed to respond to questions about
recent tax law changes. However, CARE had not identified the specific
knowledge and skills enhancements needed to achieve long-term accuracy
goals.
Criteria: Conduct a knowledge and skills inventory to identify
employees' current skills and knowledge;
Extent: Conforms to guidance to a great extent;
Evidence summary: CARE had an inventory of assistors' current knowledge
and skills based on a prescreening testing process.
Criteria: Perform a gap analysis to determine the differences between
current and needed skills and knowledge now and in the future;
Extent: Conforms to guidance to some extent;
Evidence summary: CARE did an annual gap analysis, but because CARE had
not performed a needs assessment for a period that extended beyond the
next year, it had not performed a gap analysis to determine the
differences between current skills and knowledge and skills and
knowledge needed in the future.
Phase: strategy development and selection:
Criteria: Develop and apply criteria to determine when to use training
and development strategies to fill skills and knowledge gaps;
Extent: Conforms to guidance to little or no extent;
Evidence summary: CARE had no criteria to guide decisions on using
training and development strategies, as opposed to other strategies,
such as improving guidance, or hiring practices to address skill gaps.
Criteria: Ensure training and development efforts target factors that
affect accuracy and are linked to needed skills and knowledge;
Extent: Conforms to guidance to some extent;
Evidence summary: CARE had practices such as employee testing, and
studies conducted to identify the most frequent errors made on an
annual basis, that allowed CARE to link training efforts to identified
short-term skills and knowledge gaps; However, because of limitations
in long-term gap analyses, CARE had limited ability to ensure training
and development efforts were linked to long- term needed skills and
knowledge.
Criteria: Consider the anticipated costs and benefits of alternative
training and development efforts, ways to mitigate associated risks,
and the appropriate level of investment for training and development
efforts;
Extent: Conforms to guidance to little or no extent;
Evidence summary: CARE did not have these analyses.
Criteria: Establish a detailed plan for evaluating training, including
performance measures, data, planned analyses, and how the analyses will
be used;
Extent: N/A;
Evidence summary: For our assessment, see "Evaluation Plan" phase in
appendix X.
Criteria: Establish a process to ensure that strategic and tactical
changes can be incorporated into training and development efforts;
Extent: Conforms to guidance to some extent;
Evidence summary: CARE had efforts to anticipate and react to potential
short-term changes that could affect training, such as tax law changes
or lack of experienced assistors due to staff turnover. However, CARE
did not have a training process to identify potential long-term changes
such as technological innovations or changes in the economy that might
impact training and development.
Source: GAO analysis.
[End of table]
[End of section]
Appendix VII: Assessments of Evaluation: Less Complex Tax Law Telephone
Service by W&I CARE:
Evaluation Phase: evaluation plan:
Criteria: Establish an overall approach for evaluating the impact of
training on accuracy (a level 4 evaluation);
Extent: Conforms to guidance to some extent;
Evidence summary: L&E had adopted a four-level model for evaluating
training, the Evaluation Monitoring System-Integrated Training
Evaluation and Measurement Services (EMS-ITEMS), which was based on the
widely-accepted Kirkpatrick model. IRS's Human Capital Office L&E
officials concluded that level 4 evaluations were appropriate. However,
CAS and L&E officials had not agreed on whether to conduct level 4
evaluations. In addition, L&E officials had not documented an analysis
of what level of evaluation was appropriate.
Criteria: Systematically analyze the feasibility and cost effectiveness
of alternative methodologies for evaluating the impact of training and
development on accuracy;
Extent: Conforms to guidance to little or no extent;
Evidence summary: Because L&E and CAS officials had not agreed to do a
level 4 evaluation, L&E had not analyzed the feasibility and cost
effectiveness of alternative level 4 methodologies.
Criteria: Before implementing training, develop analysis plans,
including data to be collected, analyses to be conducted, and how the
analyses will be used;
Extent: Conforms to guidance to some extent;
Evidence summary: Because L&E and CAS officials had not agreed to do a
level 4 evaluation, L&E did not have a level 4 data analysis plan.
However, EMS-ITEMS did provide general guidance for different levels of
evaluation.
Evaluation Phase: data collection:
Criteria: Ensure that data are collected systematically and in a timely
manner;
Extent: Conforms to guidance to some extent;
Evidence summary: EMS-ITEMS had controls to help ensure systematic and
timely collection of data for level 1 and 3 evaluations. EMS-ITEMS had
guidance on the collection of data including responsible parties and
timing. However, the Administrative Corporate Education System (ACES)
database used to collect level 2 data was no longer in place, and as a
result, there was no system to consistently collect and store level 2
data. In addition, because L&E and CAS officials had not agreed to do a
level 4 evaluation, L&E had not collected the data necessary to do a
level 4 evaluation.
Criteria: Ensure that data are accurate, valid, complete, and reliable;
Extent: Conforms to guidance to some extent;
Evidence summary: EMS-ITEMS had level 1 and some level 3 data and the
system had controls in place to help ensure that data were accurate,
valid, and reliable. However, because L&E and CAS had not agreed to do
a level 4 evaluation, L&E did not collect level 4 data. The ACES
database used to collect level 2 data no longer exists and there is no
replacement system planned.
Evaluation Phase: data analysis:
Criteria: Analyze data collected to assess the impact of training on
accuracy;
Extent: Conforms to guidance to some extent;
Evidence summary: L&E conducted some level 1, 2, and 3 evaluations of
participants' opinions, learning, and subsequent job performance. For
example, L&E analyzed course evaluations, tests, and supervisory
evaluations after employees completed coursework to identify needed
improvements to training. However, L&E had done no level 4 evaluations.
Criteria: Compare accuracy benefits to training costs;
Extent: Conforms to guidance to little or no extent;
Evidence summary: Because L&E and CAS had not agreed to do a level 4
evaluation, L&E did not do a comparison of benefits to cost.
Criteria: Benchmark training cost, training outcomes (accuracy), and
analytical methods against high-performing organizations;
Extent: Conforms to guidance to little or no extent;
Evidence summary: None.
Criteria: Analyze internal and external stakeholders' assessments of
training to include impact on accuracy;
Extent: Conforms to guidance to some extent;
Evidence summary: L&E and CAS analyzed stakeholder assessments of
training to identify needed changes. For example, management councils
and focus groups were used to determine if changes were needed to
improve course material, training environment, timing, and objectives
of training. However, because L&E and CAS had not agreed to do a level
4 evaluation, there were no efforts to collect and analyze assessments
from internal and external stakeholders to assess training in terms of
its impact on accuracy.
Evaluation Phase: application of evaluation results:
Criteria: Evaluate training program as a whole, in addition to
individual course evaluations, and document the results;
Extent: Conforms to guidance to little or no extent;
Evidence summary: Evaluations were for individual courses, not for the
program as a whole.
Criteria: Incorporate evaluation results into the training and
development program to improve accuracy;
Extent: Conforms to guidance to some extent;
Evidence summary: L&E and CAS reported using evaluation results in
planning to make improvements to training courses or to identify
training needs for the upcoming year. L&E and CAS used the available
level 1 through 3 evaluations to make planned improvements on
individual courses. However, because L&E and CAS had not agreed to do a
level 4 evaluation or cost-benefit comparisons, L&E and CAS's ability
to make informed decisions on improving training to improve accuracy
was limited.
Source: GAO analysis.
[End of table]
[End of section]
Appendix VIII: Assessments of Evaluation: More Complex Tax Law
Telephone Service by SB/SE TEC:
Evaluation phase and practice: evaluation plan:
Evaluation phase and practice: Establish an overall approach for
evaluating the impact of training on accuracy (a level 4 evaluation);
Extent: Conforms to guidance to a great extent;
Evidence summary: L&E had adopted a four-level model for evaluating
training (EMS-ITEMS) that was based on the widely accepted Kirkpatrick
model. In 2004, L&E and TEC planned a pilot test of a level 4
evaluation. L&E and TEC officials stated in an official document that
they intend to conduct a level 4 evaluation in 2005.
Evaluation phase and practice: Systematically analyze the feasibility
and cost effectiveness of alternative methodologies for evaluating the
impact of training and development on accuracy;
Extent: Conforms to guidance to little or no extent;
Evidence summary: L&E had not analyzed the feasibility and cost
effectiveness of alternative methodologies.
Evaluation phase and practice: Before implementing training, develop
analysis plans, including data to be collected, analyses to be
conducted, and how the analyses will be used;
Extent: Conforms to guidance to a great extent;
Evidence summary: L&E had a plan to conduct a level 4 evaluation in
2004. L&E planned to conduct a level 4 evaluation in 2005 similar to
that done in 2004. Officials said they would use the 2004 plan as the
basis for the 2005 evaluation effort.
Evaluation phase and practice: data collection:
Evaluation phase and practice: Ensure that data are collected
systematically and in a timely manner;
Extent: Conforms to guidance to a great extent;
Evidence summary: L&E had controls to ensure data were collected
systematically and in a timely manner. Data collection practices for
all four levels of evaluation in the level 4 pilot test evaluation
included specific steps to be accomplished, status updates, and due
dates. In addition, for the 2005 pilot, data on TEC accuracy in five
technical categories and staff proficiency in those technical
categories were available before the end of 2004.
Evaluation phase and practice: Ensure that data are accurate, valid,
complete, and reliable;
Extent: Conforms to guidance to some extent;
Evidence summary: In 2004, TEC was concerned that the accuracy data in
the five technical categories, based on test calls, did not reflect the
types of calls taxpayers actually made. TEC and IRS's quality review
staff subsequently worked together in an effort to fix the problem. EMS-
ITEMS had level 1 and some level 3 data, and the system had controls in
place to ensure that data were accurate, valid, and complete. However,
the database used to collect level 2 data no longer existed, and there
was no replacement system planned.
Evaluation phase and practice: data analysis:
Evaluation phase and practice: Analyze data collected to assess the
impact of training on accuracy;
Extent: Conforms to guidance to some extent;
Evidence summary: In the 2004 pilot level 4 evaluation, L&E did an
analysis but was unable to assess the impact of training on accuracy,
because the level of proficiency of TEC staff in the five technical
categories (capital gains, depreciation and sale of business property,
rentals, trust and fiduciaries, and international and alien) was
insufficient. L&E decided not to measure the impact of training on
accuracy until 80 percent of the employees were proficient in the five
technical categories. In a survey of 133 TEC employees, the percent
reported having preexisting skills or being fully proficient in the
five technical categories ranged from 31 to 62 percent.
Evaluation phase and practice: Compare accuracy benefits to training
costs;
Extent: Conforms to guidance to little or no extent;
Evidence summary: Comparing the value of training to the costs of
training was to be the third phase of the uncompleted 2004 pilot test.
Evaluation phase and practice: Benchmark training cost, training
outcomes (accuracy), and analytical methods against high-performing
organizations;
Extent: Conforms to guidance to little or no extent;
Evidence summary: None.
Evaluation phase and practice: Analyze internal and external
stakeholders' assessments of training to include impact on accuracy;
Extent: Conforms to guidance to some extent;
Evidence summary: L&E and TEC analyzed stakeholder assessments of
training to identify needed changes. For example, TEC used information
from employee surveys to target training materials on the types of
calls assistors reported receiving. However, they did not successfully
collect and analyze assessments from internal and external stakeholders
to assess training in terms of its impact on accuracy.
Evaluation phase and practice: application of evaluation results:
Evaluation phase and practice: Evaluate training program as a whole, in
addition to individual course evaluations, and document the results;
Extent: Conforms to guidance to some extent;
Evidence summary: As discussed above, although L&E attempted to do a
pilot test of a level 4 evaluation of the training program as a whole,
the evaluation was unsuccessful.
Evaluation phase and practice: Incorporate evaluation results into the
training and development program to improve accuracy;
Extent: Conforms to guidance to some extent;
Evidence summary: L&E and TEC reported using evaluation results in
planning to make improvements to training courses or to identify
training needs for the upcoming year. However, because there was no
successful level 4 evaluation or benefit cost comparison, L&E and TEC's
ability to make informed decisions on improving training to improve
accuracy was limited.
Source: GAO analysis.
[End of table]
[End of section]
Appendix IX: Assessments of Evaluation: Tax Law Walk-in Service by W&I
CARE:
Evaluation phase and practice: evaluation plan:
Evaluation phase and practice: Establish an overall approach for
evaluating the impact of training on accuracy (a level 4 evaluation);
Extent: Conforms to guidance to some extent;
Evidence summary: L&E had adopted a four-level model for evaluating
training, the Evaluation Monitoring System-Integrated Training
Evaluation and Measurement Services (EMS-ITEMS), based on the widely-
accepted Kirkpatrick model. IRS's Human Capital Office L&E officials
concluded that level 4 evaluations were appropriate. However, CARE and
L&E officials had not agreed on whether to conduct level 4 evaluations.
In addition, L&E officials had not documented an analysis of what level
of evaluation was appropriate.
Evaluation phase and practice: Systematically analyze the feasibility
and cost effectiveness of alternative methodologies for evaluating the
impact of training and development on accuracy;
Extent: Conforms to guidance to little or no extent;
Evidence summary: Because L&E and CARE officials had not agreed to do a
level 4 evaluation, L&E had not analyzed the feasibility and cost
effectiveness of alternative level 4 methodologies. In addition, L&E
and CARE officials did not look at methodologies specific to questions
on tax law assistance. The approach for training and evaluating staff
who provided assistance by answering walk-in customers' tax law
questions and staff preparing returns were the same.
Evaluation phase and practice: Before implementing training, develop
analysis plans, including data to be collected, analyses to be
conducted, and how the analyses will be used;
Extent: Conforms to guidance to some extent;
Evidence summary: Because L&E and CARE officials had not agreed to do a
level 4 evaluation, L&E did not have a level 4 data analysis plan.
However, EMS-ITEMS did provide general guidance for different levels of
evaluation.
Evaluation phase and practice: data collection:
Evaluation phase and practice: Ensure that data are collected
systematically and in a timely manner;
Extent: Conforms to guidance to some extent;
Evidence summary: EMS-ITEMS had controls to help ensure systematic and
timely collection of data for level 1 and 3 evaluations. EMS-ITEMS has
guidance on the collection of data including responsible parties and
timing. However, the Administrative Corporate Education System (ACES)
database used to collect level 2 data was no longer in place, and as a
result, there was no system to consistently collect and store level 2
data. In addition, because L&E and CARE officials had not agreed to do
a level 4 evaluation, L&E had not collected the data necessary to do a
level 4 evaluation.
Evaluation phase and practice: Ensure that data are accurate, valid,
complete, and reliable;
Extent: Conforms to guidance to some extent;
Evidence summary: EMS-ITEMS had level 1 and some level 3 data and the
system had controls in place to help ensure that data were accurate,
valid, and reliable. However, because L&E and CARE had not agreed to do
a level 4 evaluation, L&E did not collect level 4 data. The ACES
database used to collect level 2 data no longer exists and there was no
replacement system planned.
Evaluation phase and practice: data analysis:
Evaluation phase and practice: Analyze data collected to assess the
impact of training on accuracy;
Extent: Conforms to guidance to some extent;
Evidence summary: L&E conducted some level 1, 2, and 3 evaluations of
participants' opinions, learning, and subsequent job performance. For
example, L&E analyzed course evaluations, tests, and supervisory
evaluations after employees completed coursework to identify needed
improvements to training. However, L&E had done no level 4 evaluations.
In addition, there was no distinction between analysis of walk-in tax
law and return preparation assistance.
Evaluation phase and practice: Compare accuracy benefits to training
costs;
Extent: Conforms to guidance to little or no extent;
Evidence summary: Because L&E and CARE had not agreed to do a level 4
evaluation, L&E did not do a comparison of benefits to cost.
Evaluation phase and practice: Benchmark training cost, training
outcomes (accuracy), and analytical methods against high-performing
organizations;
Extent: Conforms to guidance to little or no extent;
Evidence summary: None.
Evaluation phase and practice: Analyze internal and external
stakeholders' assessments of training to include impact on accuracy;
Extent: Conforms to guidance to some extent;
Evidence summary: L&E and CARE analyzed stakeholder assessments of
training to identify needed changes. For example, focus groups were
used to determine if changes were needed to improve course material,
training environment, timing, and objectives of training. However,
because L&E and CARE had not agreed to do a level 4 evaluation, there
were no efforts to collect and analyze assessments from internal and
external stakeholders to assess training in terms of its impact on
accuracy.
Evaluation phase and practice: application of evaluation results:
Evaluation phase and practice: Evaluate training program as a whole, in
addition to individual course evaluations, and document the results;
Extent: Conforms to guidance to little or no extent;
Evidence summary: Evaluations were for individual courses, not for the
program as a whole.
Evaluation phase and practice: Incorporate evaluation results into the
training and development program to improve accuracy;
Extent: Conforms to guidance to some extent;
Evidence summary: L&E and CARE reported using evaluation results in
planning to make improvements to training courses or to identify
training needs for the upcoming year. L&E and CARE used the available
level 1 through 3 evaluations to make planned improvements on
individual courses. However, because L&E and CARE had not agreed to do
a level 4 evaluation or cost-benefit comparisons, L&E and CARE's
ability to make informed decisions on improving training to improve
accuracy was limited.
Source: GAO analysis.
[End of table]
[End of section]
Appendix X: Assessments of Evaluation: Return Preparation Walk-in
Service by W&I CARE:
Evaluation phase and practice: evaluation plan:
Evaluation phase and practice: Establish an overall approach for
evaluating the impact of training on accuracy (a level 4 evaluation);
Extent: Conforms to guidance to some extent;
Evidence summary: L&E has adopted a four-level model for evaluating
training, the Evaluation Monitoring System-Integrated Training
Evaluation and Measurement Services (EMS-ITEMS), based on the widely-
accepted Kirkpatrick model. IRS's Human Capital Office L&E officials
had concluded that level 4 evaluations were appropriate. However, CARE
and L&E officials had not agreed on whether to conduct a level 4
evaluation. In addition, L&E officials had not documented an analysis
of what level of evaluation was appropriate.
Evaluation phase and practice: Systematically analyze the feasibility
and cost effectiveness of alternative methodologies for evaluating the
impact of training and development on accuracy;
Extent: Conforms to guidance to little or no extent;
Evidence summary: Because L&E and CARE officials had not agreed to do a
level 4 evaluation, L&E had not analyzed the feasibility and cost
effectiveness of alternative level 4 methodologies. In addition, L&E
and CARE officials did not look at methodologies specific to questions
on return preparation. The approaches for training and evaluating staff
that provided assistance by preparing returns and staff answering walk-
in customers' tax law questions were the same.
Evaluation phase and practice: Before implementing training, develop
analysis plans, including data to be collected, analyses to be
conducted, and how the analyses will be used;
Extent: Conforms to guidance to some extent;
Evidence summary: Because L&E and CARE officials had not agreed to do a
level 4 evaluation, L&E did not have a level 4 data analysis plan.
However, EMS-ITEMS did provide general guidance for different levels of
evaluation.
Evaluation phase and practice: data collection:
Evaluation phase and practice: Ensure that data are collected
systematically and in a timely manner;
Extent: Conforms to guidance to some extent;
Evidence summary: EMS-ITEMS had controls to help ensure systematic and
timely collection of data for level 1 and 3 evaluations. EMS-ITEMS had
guidance on the collection of data including responsible parties and
timing. However, the Administrative Corporate Education System (ACES)
database used to collect level 2 data was no longer in place, and as a
result, there was no system to consistently collect and store level 2
data. In addition, because L&E and CARE officials had not agreed to do
a level 4 evaluation, L&E had not collected the data necessary to do a
level 4 evaluation.
Evaluation phase and practice: Ensure that data are accurate, valid,
complete, and reliable;
Extent: Conforms to guidance to some extent;
Evidence summary: EMS-ITEMS had level 1 and some level 3 data and the
system had controls in place to help ensure that data were accurate,
valid, and reliable. However, because L&E and CARE had not agreed to do
a level 4 evaluation, L&E did not collect level 4 data. The ACES
database used to collect level 2 data no longer existed and there was
no replacement system planned.
Evaluation phase and practice: data analysis:
Evaluation phase and practice: Analyze data collected to assess the
impact of training on accuracy;
Extent: Conforms to guidance to some extent;
Evidence summary: L&E conducted some level 1, 2, and 3 evaluations of
participants' opinions, learning, and subsequent job performance. For
example, L&E analyzed course evaluations, tests, and supervisory
evaluations after employees completed coursework to identify needed
improvements to training. However, L&E had done no level 4 evaluations.
In addition, there was no distinction between analysis of walk-in tax
law and return preparation assistance.
Evaluation phase and practice: Compare accuracy benefits to training
costs;
Extent: Conforms to guidance to little or no extent;
Evidence summary: Because L&E and CARE had not agreed to do a level 4
evaluation, L&E did not do a comparison of benefits to cost.
Evaluation phase and practice: Benchmark training cost, training
outcomes (accuracy), and analytical methods against high-performing
organizations;
Extent: Conforms to guidance to little or no extent;
Evidence summary: None.
Evaluation phase and practice: Analyze internal and external
stakeholders' assessments of training to include impact on accuracy;
Extent: Conforms to guidance to some extent;
Evidence summary: L&E and CARE analyzed stakeholder assessments of
training to identify needed changes. For example, focus groups were
used to determine if changes were needed to improve course material,
training environment, timing, and objectives of training. However,
because L&E and CARE had not agreed to do a level 4 evaluation, there
were no efforts to collect and analyze assessments from internal and
external stakeholders to assess training in terms of its impact on
accuracy.
Evaluation phase and practice: application of evaluation results:
Evaluation phase and practice: Evaluate training program as a whole, in
addition to individual course evaluations, and document the results;
Extent: Conforms to guidance to little or no extent;
Evidence summary: Evaluations were for individual courses, not for the
program as a whole.
Evaluation phase and practice: Incorporate evaluation results into the
training and development program to improve accuracy;
Extent: Conforms to guidance to some extent;
Evidence summary: L&E and CARE reported using evaluation results in
planning to make improvements to training courses or to identify
training needs for the upcoming year. L&E and CARE used the available
levels 1 through 3 evaluations to make planned improvements on
individual courses. However, because L&E and CARE had not agreed to do
a level 4 evaluation or cost-benefit comparisons, L&E and CARE's
ability to make informed decisions on improving training to improve
accuracy was limited.
Source: GAO analysis.
[End of table]
[End of section]
Appendix XI: Information on the Kirkpatrick Model of Training and
Development Evaluation:
In recent years, a growing number of organizations have adopted a
balanced, multilevel approach to evaluating their training and
development efforts. Such an approach can help provide varied data and
perspectives on the effect that training efforts have on the
organization. One commonly accepted model consists of four levels of
assessment.[Footnote 16] The first level measures the training
participants' reaction to, and satisfaction with, the training program
or planned actions to use new or enhanced competencies. The second
level measures the extent to which learning has occurred because of the
training effort. The third level measures the application of this
learning to the work environment through changes in behavior that
trainees exhibit on the job because of the training or development
program. The fourth level measures the impact of the training program
on the agency's program or organizational results. The fourth level is
sometimes split into two levels with the fifth level representing a
comparison of costs and benefits quantified in dollars. This fifth
level--often referred to as return on investment (ROI)--compares the
benefits (quantified in dollars) to the costs of the training and
development program.[Footnote 17]
Not all training and development programs require, or are suitable for,
higher levels of evaluation. Indeed, higher levels of evaluation can be
challenging to conduct because of the difficulty and costs associated
with data collection and the complexity in directly linking training
and development programs to improved individual and organizational
performance. Figure 2 depicts an example gradation of the extent to
which an agency could use the various levels of evaluation to assess
its training and development programs. For example, an agency may
decide to evaluate participants' reactions for all (100 percent) of its
programs, while conducting an ROI analysis for 5 percent of its
programs.
Figure 2: Example Agency's Training and Development Programs Assessed
Using Each Level of Evaluation:
[See PDF for image]
[End of figure]
[End of section]
Appendix XII: Comments from IRS:
DEPARTMENT OF THE TREASURY:
INTERNAL REVENUE SERVICE:
WASHINGTON. D.C. 20224:
COMMISSIONER:
July 6, 2005:
Mr. James R. White:
Director, Tax Issues:
United States Government Accountability Office:
Washington, D.C. 20548:
Dear Mr. White:
Thank you for the opportunity to review your proposed report entitled
Tax Administration: IRS Needs Better Strategic Planning and Evaluation
of Taxpayer Assistance Training (GAO-05-782).
Your report provides an assessment of our training and development
programs for employees who provide taxpayers tax law and return
preparation assistance. Such assistance can be challenging given the
complexities of the tax code. Despite these challenges, I believe our
tax law and return preparation assistors conscientiously perform their
duties based on the facts and circumstances as presented to them by
taxpayers, and I am pleased that the accuracy of assistance provided to
taxpayers is improving. A 14.5 percent increase (after two years of
decline) in telephone assistance accuracy suggests we are on the right
track. This increase will be fully validated upon examination of the
relevant returns to ensure that data are complete and representative.
Your review of our training and development programs, and accompanying
recommendations, offer valuable insight. As the first agency to be
assessed against the Government Accountability Office's (GAO) criteria
in its Guide for Assessing Strategic Training and Development Efforts
in the Federal Government, we particularly appreciate the examples of
planning practices that conform to strategic guidance. Your timely
report will assist us in our current efforts to reengineer our learning
and education business processes. The Learning and Education Analysis
Project (LEAP) will result in a new training structure and achieve
significant cost savings. We expect to implement the new processes by
December 2006; this report has been shared with the LEAP manager.
The Internal Revenue Service's (IRS) training developers diligently
implement training programs for assistors. These programs must be
completed in time for the filing season, as well as incorporate
numerous and frequent tax law changes made subsequent to the previous
filing season. Generally, I believe our past training efforts in this
challenging area have been well and appropriately planned and executed.
Indeed, certain aspects have been recognized. For example, our Training
Development Quality Assurance System (TDQAS), which is our Servicewide
approach to training and development efforts, has been benchmarked by
the American Society for Training and Development and the International
Society for Performance Improvement. We believe that the TDQAS is an
educationally sound, comprehensive process to ensure that strategic and
tactical changes are incorporated into our training and development
efforts. It is incorporated into the Internal Revenue Manual (IRM) and
there are extensive guidelines that supplement the system. We
recognize, however, that our learning and education processes can be
improved; as noted above, we will take into account your
recommendations as we reengineer those processes.
If you have any questions, please contact Beverly Ortega Babers, Chief
Human Capital Officer, at (202) 622-7676.
Sincerely,
Signed by:
Mark W. Everson:
Enclosure:
Enclosure:
Our comments on the report's specific recommendations follow:
Recommendation: Establish a long-term goal for the accuracy of taxpayer
assistance.
We will continue efforts to achieve further improvements in the
accuracy of the assistance we provide taxpayers. We have begun the
process of establishing long-term goals based on the needs of
taxpayers, other benefits, and costs.
Recommendation: Establish goals and measures for training and
development logically linked to accuracy.
The true measure of training effectiveness is skillful on-the-job
application of knowledge. Training accuracy results in proficient
performance. We are committed to Level 3 Evaluation of our tax law and
return preparation assistance training programs. We developed the
Evaluation Monitoring System (EMS) based on the Kirkpatrick Model as
well as Integrated Training Evaluation and Measurement Services (ITEMS)
which include extensive online guidance and functionalities. This is
the EMS-ITEMS cited in the report; its use is covered by policy and the
IRM. While I believe our established processes conform to guidance to a
great extent, I agree that training evaluations should be in line with
benchmarked practices and the results used to plan further improvements
in the accuracy of tax law and return preparation assistance we provide
taxpayers.
Recommendation: Determine and track the relative importance of the
various factors, including training, that affect accuracy.
We will investigate all factors that affect the accuracy of assistance
provided to taxpayers. In an environment that requires changes on short
notice, our strategic approach must remain flexible.
Recommendation: Benchmark training and development programs against
high-performing organizations.
We agree that benchmarking is a beneficial process, albeit a resource-
intensive one. Nevertheless, we are committed to benchmarking with
regard to our taxpayer assistance training programs to the extent
practicable within resource limitations.
Recommendation: Conduct skills and knowledge gap analyses for all
taxpayer assistance programs, to include identifying and comparing
current skills to long-term skill needs.
We recognize the value of such assessments. For example, our walk-in
assistance program has developed the Technical Assessment Battery (TAB)
to conduct skills and:
knowledge gap analysis for its taxpayer assistance program. This
assessment tool was recently redesigned to compare employees' current
skill levels to the long-term skills needed to meet customers' needs.
The TAB includes competencies required for all tasks needed in the
Taxpayer Assistance Centers including tax law, return preparation,
account work, notice inquiries, and collection work. Our telephone
centers conduct skills and knowledge gap analysis as part of their
recurring planning cycles directed at the delivery of accurate customer
assistance. This analysis is conducted annually and for each planning
period during the year to align employee skills with customer needs.
Employee skill enhancements necessitated by the introduction of new
technologies or employee tools are incorporated in the implementation
process for the initiative. When organizational or legislative changes
occur, we conduct additional skills and knowledge gap assessments to
the extent practicable within resource limitations.
Recommendation: Consider costs, benefits, ways to mitigate risks, and
the appropriate level of investment for training and development
efforts.
Our annual training plan formulation process allows business divisions
to identify their training priorities based on strategic plans and the
level of funding appropriated for that year. Every business division
closely monitors training plan execution throughout the year to ensure
delivery of training programs and to identify unfunded needs and funds
that will not be utilized. This allows for maximum flexibility to meet
changing business priorities.
Recommendation: Continue to pursue the level 4 pilot in TEC and, if
that analysis is shown to be feasible, conduct level 4 evaluations for
its other taxpayer assistance training and development programs.
Level 4 Evaluation as embodied in EMS was carefully developed with the
assistance of expert consultants. Our approach comprises three phases:
time to capability, training impact on organizational performance
indicators, and establishing training worth. The approach is
innovative, yet reasonable. It is consistent with GAO's criteria,
except for Return on Investment, although we believe the methodology
developed by the Office of Personnel Management for establishing the
worth of training is more appropriate. The TEC Level 4 pilot should be
completed this year. However, because our Small Business/Self-Employed
Division will not be providing taxpayer assistance on tax law after
this year, our Wage and Investment Division will undertake future
efforts. The EMS Level 4 Evaluation approach is established as policy,
and our learning and education functions intend to follow it.
Recommendation: Replace the defunct ACES database, which had been used
to store level 2 data, with another database for this purpose.
We recognize the importance of Level 2 data in the training evaluation
process and are looking into available options, including our new
Enterprise Learning Management System, for storing such data.
[End of section]
Appendix XIII: GAO Contact and Staff Acknowledgments:
GAO Contact:
James R. White (202) 512-9110:
Acknowledgments:
In addition to the contact named above, Charlie Daniel, David Dornisch,
Chad Gorman, Jason Jackson, Ronald Jones, Veronica Mayhand, and Katrina
Taylor made key contributions to this report.
[End of section]
FOOTNOTES
[1] Treasury Inspector General for Tax Administration, Improvements Are
Needed to Ensure Tax Returns Are Correctly Prepared at Taxpayer
Assistance Centers, Reference No. 2004-40-025 (Washington, D.C.: 2003)
and Treasury Inspector General for Tax Administration, Customer Service
at the Taxpayer Assistance Centers Is Improving but Is Still Not
Meeting Expectations, Reference No. 2005-40-021 (Washington, D.C.:
2004).
[2] GAO, Human Capital: A Guide for Assessing Strategic Training and
Development Efforts in the Federal Government, GAO-04-546G (Washington,
D.C.: March 2004).
[3] Millions of taxpayers use IRS's Web site for taxpayer assistance.
[4] This is the total number of staff that were assigned to provide tax
law assistance by telephone. The staff also had other duties. IRS could
not provide data on the time staff spent providing telephone
assistance.
[5] In commenting on a draft of this report, an IRS official noted that
IRS tracks training time, but does not have time data specific to tax
law training.
[6] GAO-04-546G.
[7] GAO-04-546G.
[8] For example, see GAO, Internal Revenue Service: Assessment of
Fiscal Year 2006 Budget Request and Interim Results of the 2005 Filing
Season, GAO-05-416T (Washington D.C.: Apr. 14, 2005); and Treasury
Inspector General For Tax Administration, The Accounts Management
Program Has Annual Performance Goals But Should Develop Long-term
Performance Goals, Reference No. 2005-40-079 (Washington D.C: May 6,
2005).
[9] U.S. General Accounting Office, IRS Telephone Assistance:
Opportunities to Improve Human Capital Management, GAO-01-144
(Washington, D.C.: Jan. 30, 2001).
[10] GAO, Tax Administration: Planning for IRS's Enforcement Process
Changes Included Many Key Steps But Can Be Improved, GAO-04-287
(Washington, D.C.: Jan. 20, 2004).
[11] Pub. L. No. 109-1 (2005).
[12] Donald L. Kirkpatrick, Evaluating Training Programs: The Four
Levels (San Francisco: Berrett-Koehler, 1994).
[13] Pub. L. No. 105-206 (1998).
[14] Our review was to determine if some basic controls were in place
to ensure that data were collected timely and systematically and that
data were accurate, valid, reliable, and complete. We did not assess or
test the adequacy of the controls, data collection methods, or data
quality or completeness.
[15] In 2006, W&I will be handling these calls and training their staff
to provide answers to these more complex tax law questions.
[16] Donald L. Kirkpatrick (author of Evaluating Training Programs: The
Four Levels) conceived a commonly recognized four-level model for
evaluating training and development efforts.
[17] Jack J. Phillips conceived the fifth level in the model for
evaluating training and development efforts as discussed in the book,
Measuring ROI in the Public Sector.
GAO's Mission:
The Government Accountability Office, the investigative arm of
Congress, exists to support Congress in meeting its constitutional
responsibilities and to help improve the performance and accountability
of the federal government for the American people. GAO examines the use
of public funds; evaluates federal programs and policies; and provides
analyses, recommendations, and other assistance to help Congress make
informed oversight, policy, and funding decisions. GAO's commitment to
good government is reflected in its core values of accountability,
integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through the Internet. GAO's Web site ( www.gao.gov ) contains
abstracts and full-text files of current reports and testimony and an
expanding archive of older products. The Web site features a search
engine to help you locate documents using key words and phrases. You
can print these documents in their entirety, including charts and other
graphics.
Each day, GAO issues a list of newly released reports, testimony, and
correspondence. GAO posts this list, known as "Today's Reports," on its
Web site daily. The list contains links to the full-text document
files. To have GAO e-mail this list to you every afternoon, go to
www.gao.gov and select "Subscribe to e-mail alerts" under the "Order
GAO Products" heading.
Order by Mail or Phone:
The first copy of each printed report is free. Additional copies are $2
each. A check or money order should be made out to the Superintendent
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or
more copies mailed to a single address are discounted 25 percent.
Orders should be sent to:
U.S. Government Accountability Office
441 G Street NW, Room LM
Washington, D.C. 20548:
To order by Phone:
Voice: (202) 512-6000:
TDD: (202) 512-2537:
Fax: (202) 512-6061:
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Web site: www.gao.gov/fraudnet/fraudnet.htm
E-mail: fraudnet@gao.gov
Automated answering system: (800) 424-5454 or (202) 512-7470:
Public Affairs:
Jeff Nelligan, managing director,
NelliganJ@gao.gov
(202) 512-4800
U.S. Government Accountability Office,
441 G Street NW, Room 7149
Washington, D.C. 20548:
