Information Technology
Treasury Needs to Better Define and Implement Its Earned Value Management Policy Gao ID: GAO-08-951 September 22, 2008In 2008, the Department of Treasury (Treasury) plans to spend approximately $3 billion on information technology (IT) investments--the third largest planned IT expenditure among civilian agencies. To more effectively manage such investments, in 2005 the Office of Management and Budget required agencies to use earned value management (EVM). EVM is a project management approach that, if implemented appropriately, provides objective reports of project status, produces early warning signs of impending schedule delays and cost overruns, and provides unbiased estimates of a program's total costs. GAO was asked to assess whether the department and its key component agencies (1) have the policies in place to effectively implement EVM and (2) are adequately using EVM techniques to manage critical system investments. GAO compared agency policies to best practices identified in the Cost Assessment Guide and reviewed the implementation of key EVM practices for several investments.
The Department of Treasury's EVM policy is not fully consistent with best practices. Specifically, of seven best practices that leading organizations address in their policies, Treasury's policy fully addresses three, partially addresses three, and does not address the training component. According to the Director for Capital Planning and Investment Control, the department is currently working on revising its policy and according to Deputy Assistant Secretary for Information Systems and Chief Information Officer expects to finalize it by October 2008. Until Treasury develops a comprehensive policy to guide its efforts, it will be difficult for the department to optimize the effectiveness of EVM as a management tool. The department and its bureaus are not fully implementing key EVM practices needed to effectively manage their critical system investments. Specifically, the six programs at Treasury that GAO reviewed were not consistently implementing practices needed for establishing a comprehensive EVM system, ensuring that data from the system are reliable, and using the data to help manage the program. For example, when executing work plans and recording actual costs, a key practice for ensuring that the data resulting from the EVM system are reliable, only two of the six investments reviewed incorporated government costs with contractor costs. These weaknesses exist in part because Treasury's policy is not comprehensive and because the department does not have a process for ensuring effective EVM implementation. Unless the department consistently implements fundamental EVM practices, it may not be able to effectively manage its critical programs.
RecommendationsOur recommendations from this work are listed below with a Contact for more information. Status will change from "In process" to "Open," "Closed - implemented," or "Closed - not implemented" based on our follow up work.
Director: Team: Phone:GAO-08-951, Information Technology: Treasury Needs to Better Define and Implement Its Earned Value Management Policy
This is the accessible text file for GAO report number GAO-08-951
entitled 'Information Technology: Treasury Needs to Better Define and
Implement Its Earned Value Management Policy' which was released on
September 22, 2008.
This text file was formatted by the U.S. Government Accountability
Office (GAO) to be accessible to users with visual impairments, as part
of a longer term project to improve GAO products' accessibility. Every
attempt has been made to maintain the structural and data integrity of
the original printed product. Accessibility features, such as text
descriptions of tables, consecutively numbered footnotes placed at the
end of the file, and the text of agency comment letters, are provided
but may not exactly duplicate the presentation or format of the printed
version. The portable document format (PDF) file is an exact electronic
replica of the printed version. We welcome your feedback. Please E-mail
your comments regarding the contents or accessibility features of this
document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
Report to Subcommittee on Financial Services and General Government,
Committee on Appropriations, U.S. Senate:
United States Government Accountability Office:
GAO:
September 2008:
Information Technology:
Treasury Needs to Better Define and Implement Its Earned Value
Management Policy:
GAO-08-951:
GAO Highlights:
Highlights of GAO-08-951, a report to Subcommittee on Financial
Services and General Government, Committee on Appropriations, U.S.
Senate.
Why GAO Did This Study:
In 2008, the Department of Treasury (Treasury) plans to spend
approximately $3 billion on information technology (IT) investments”the
third largest planned IT expenditure among civilian agencies. To more
effectively manage such investments, in 2005 the Office of Management
and Budget required agencies to use earned value management (EVM). EVM
is a project management approach that, if implemented appropriately,
provides objective reports of project status, produces early warning
signs of impending schedule delays and cost overruns, and provides
unbiased estimates of a program‘s total costs.
GAO was asked to assess whether the department and its key component
agencies (1) have the policies in place to effectively implement EVM
and (2) are adequately using EVM techniques to manage critical system
investments. GAO compared agency policies to best practices identified
in the Cost Assessment Guide and reviewed the implementation of key EVM
practices for several investments.
What GAO Found:
The Department of Treasury‘s EVM policy is not fully consistent with
best practices. Specifically, of seven best practices that leading
organizations address in their policies, Treasury‘s policy fully
addresses three, partially addresses three, and does not address the
training component (see table below). According to the Director for
Capital Planning and Investment Control, the department is currently
working on revising its policy and according to Deputy Assistant
Secretary for Information Systems and Chief Information Officer expects
to finalize it by October 2008. Until Treasury develops a comprehensive
policy to guide its efforts, it will be difficult for the department to
optimize the effectiveness of EVM as a management tool.
The department and its bureaus are not fully implementing key EVM
practices needed to effectively manage their critical system
investments. Specifically, the six programs at Treasury that GAO
reviewed were not consistently implementing practices needed for
establishing a comprehensive EVM system, ensuring that data from the
system are reliable, and using the data to help manage the program. For
example, when executing work plans and recording actual costs, a key
practice for ensuring that the data resulting from the EVM system are
reliable, only two of the six investments reviewed incorporated
government costs with contractor costs. These weaknesses exist in part
because Treasury‘s policy is not comprehensive and because the
department does not have a process for ensuring effective EVM
implementation. Unless the department consistently implements
fundamental EVM practices, it may not be able to effectively manage its
critical programs.
Table: Seven Key Components of an Effective EVM Policy:
Policy component: Establish clear criteria for which programs are to
use EVM; Assessment of Treasury policy: Fully addressed.
Policy component: Require programs to comply with national standards;
Assessment of Treasury policy: Partially addressed.
Policy component: Require programs to use a standard structure for
defining the work products that enables managers to track cost and
schedule by defined deliverables (e.g., hardware or software
component); Assessment of Treasury policy: Partially addressed.
Policy component: Require programs to conduct detailed reviews of
expected costs, schedules, and deliverables (called an integrated
baseline review); Fully addressed.
Policy component: Require and enforce EVM training; Assessment of
Treasury policy: Not addressed.
Policy component: Define when programs may revise cost and schedule
baselines (called rebaselining); Assessment of Treasury policy: Fully
addressed.
Policy component: Require system surveillance”routine validation checks
to ensure that major acquisitions continue to comply with agency
policies and standards; Assessment of Treasury policy: Partially
addressed.
Source: GAO Cost Guide: Exposure Draft (GAO-07-1134SP) and analysis of
Treasury data.
[End of table]
What GAO Recommends:
GAO is recommending that the Secretary of Treasury define a
comprehensive EVM policy consistent with best practices and establish a
process for ensuring effective EVM implementation. In written comments
on a draft of the report, Treasury agreed with the report findings and
recommendations.
To view the full product, including the scope and methodology, click on
[hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-08-951]. For more
information, contact David Powner at 202-512-9286 or PownerD@gao.gov.
[End of section]
Contents:
Letter:
Results In Brief:
Background:
Treasury's EVM Policy Is Not Fully Consistent with Best Practices:
Treasury Is Not Fully Implementing Key Earned Value Management
Techniques to Manage Critical System Investments:
Conclusions:
Recommendations for Executive Action:
Agency Comments and Our Evaluation:
Appendix I: Comments from the Department of Treasury:
Appendix II: Objectives, Scope, and Methodology:
Appendix III: Overview of Industry Guidelines that Support Sound Earned
Value Management:
Appendix IV: Description of Selected Investments:
Appendix V: GAO Contact and Staff Acknowledgments:
Tables:
Table 1: Key Components of an Effective EVM Policy:
Table 2: Assessment of Treasury's EVM Policies, as of May 2008:
Table 3: Key EVM Practices for System Acquisition Programs:
Table 4: Assessment of EVM Use to Manage Treasury Investments:
Table 5: Management Functions Addressed by ANSI Guidance on Earned
Value Management Systems:
Figure:
Figure 1: Treasury's Organizational Chart (condensed):
Abbreviations:
ANSI: American National Standards Institute:
BPD: Bureau of the Public Debt:
CPIC: Capital Planning and Investment Control:
DO: Departmental Offices:
EDAS: Enterprise Data Access Strategy:
EIA: Electronic Industries Alliance:
EVM: earned value management:
FIRST: Financial Information and Reporting Standardization:
FMS: Financial Management Service:
IFS: Integrated Financial System/Core Financial System:
IRS: Internal Revenue Service:
IT: information technology:
OCIO: Office of the Chief Information Officer:
OMB: Office of Management and Budget:
STAR: System to Administer Retirement:
TAAPS: Treasury Automated Auction Processing System:
Treasury: Department of the Treasury:
[End of section]
United States Government Accountability Office:
Washington, DC 20548:
September 22, 2008:
The Honorable Richard J. Durbin:
Chairman:
The Honorable Sam Brownback:
Ranking Member:
Subcommittee on Financial Services and General Government:
Committee on Appropriations:
United States Senate:
The Department of the Treasury (Treasury) and its nine bureaus rely
extensively on information technology (IT) to carry out the
responsibility of promoting the economic and financial prosperity and
security of the United States. In fiscal year 2008, the department
plans to spend approximately $3.0 billion for IT investments. This is
the third largest planned IT expenditure among civilian agencies.
[Footnote 1]
To help federal departments more effectively manage their IT
investments, in 2005 the Office of Management and Budget (OMB) issued a
memo that requires departments to implement earned value management
(EVM).[Footnote 2] EVM is a project management approach that, if
implemented appropriately, provides objective reports of project
status, produces early warning signs of impending schedule delays and
cost overruns, and provides unbiased estimates of anticipated costs at
completion. Given the size and significance of Treasury's IT
investments, you asked us to determine whether the department and its
key component agencies (1) have the policies in place to effectively
implement EVM and (2) are adequately using EVM techniques to manage
critical system investments.
To address our objectives, we reviewed agency documentation including
Treasury-wide and bureau-level policies and plans governing the use of
EVM on IT acquisitions, documented EVM practices and performance
reports for six programs, and interviewed key agency officials. We
compared the department's policies and practices to federal standards
and best practices identified in the Cost Assessment Guide[Footnote 3]
to determine the effectiveness of Treasury's use of earned value data
in managing its IT investments.
We conducted this performance audit from August 2007 to July 2008 in
accordance with generally accepted government auditing standards. Those
standards require that we plan and perform the audit to obtain
sufficient, appropriate evidence to provide a reasonable basis for our
findings and conclusions based on our audit objectives. We believe that
the evidence obtained provides a reasonable basis for our findings and
conclusions based on our audit objectives. Further details on our
objectives, scope, and methodology are provided in appendix II.
Results In Brief:
The Department of Treasury's EVM policy does not fully address best
practices that leading organizations include in their policies. Leading
organizations establish EVM policies consisting of seven key
components:
* establishing clear criteria defining which programs are to use EVM;
* requiring programs to comply with a national standard (called the
ANSI standard[Footnote 4]);
* requiring programs to use a standard structure for defining work
products;
* requiring programs to conduct detailed reviews of expected costs,
schedules, and deliverables (called an integrated baseline review);
* requiring and enforcing EVM training;
* defining when programs may revise cost and schedule baselines (called
rebaselining); and:
* requiring system surveillance--routine validation checks to ensure
that major acquisitions are complying with agency policies and
standards.
Treasury's EVM policy fully addresses three of these components,
partially addresses three, and does not address the training
requirement. However, the department is working on revising its policy.
Until Treasury develops a comprehensive policy to guide its efforts, it
will be difficult for the department to optimize the effectiveness of
EVM as a management tool.
The department and its bureaus are not fully implementing key EVM
practices to manage their critical system investments. Specifically,
the six programs at Treasury that we reviewed were not consistently
implementing practices needed for establishing a comprehensive EVM
system, ensuring that data from the system are reliable, and using the
data to help manage the program. For example, when executing work plans
and recording actual costs, a key practice for ensuring that the data
resulting from the system are reliable, four of six investments we
reviewed did not incorporate government costs with contractor costs. In
addition, five out of six did not adequately analyze performance data
and record the variances from the baseline. These weaknesses exist
partly due to the weaknesses in Treasury's policy and because the
department does not have a process for ensuring effective EVM
implementation. Unless the department consistently implements
fundamental EVM practices, it may not be able to manage its programs
effectively.
We are making recommendations to the Secretary of Treasury to direct
the Assistant Secretary for Management and Chief Financial Officer, in
collaboration with the Chief Information Officer, to define a policy
that is fully consistent with best practices and ensure effective EVM
implementation such that all eligible investments address the practices
associated with establishing a comprehensive EVM system, ensuring data
reliability, and effectively making decisions using the data.
In written comments on a draft of this report, the Department of
Treasury's Deputy Assistant Secretary for Information Systems and Chief
Information Officer generally agreed with our findings. He stated that
Treasury will issue a revised version of the EVM policy that will
address our nine recommendations by October 2008, and will work with
the bureaus to establish mechanisms and tools to ensure compliance with
the policy. Treasury also provided technical comments which we have
addressed as appropriate.
Background:
Treasury is the primary federal agency responsible for the economic and
financial prosperity and security of the United States, and as such is
responsible for a wide range of activities, including advising the
President on economic and financial issues, promoting the President's
growth agenda, and enhancing corporate governance in financial
institutions.
To accomplish its mission, Treasury is organized into departmental
offices and operating bureaus. The departmental offices are primarily
responsible for the formulation of policy and management of the
department as a whole, while the nine operating bureaus--including the
Internal Revenue Service and the Bureau of Public Debt--carry out
specific functions assigned to Treasury. Figure 1 shows the
organizational structure of the department.
Figure 1: Treasury's Organizational Chart (condensed):
[See PDF for image]
This figure is an illustration of Treasury's Organizational Chart
(condensed), as follows:
Secretary;
Deputy Secretary;
* Office of the Comptroller of the Currency (Treasury Bureau);
* Office of Thrift Supervision (Treasury Bureau);
* United States Mint (Treasury Bureau);
* Bureau of Engraving and Printing (Treasury Bureau);
* Internal Revenue Service (Treasury Bureau);
* Under Secretary for Domestic Finance; Fiscal Assistant Secretary;
- Financial Management Service (Treasury Bureau);
- Bureau of Public Debt (Treasury Bureau);
* Under Secretary for Terrorism and Financial Intelligence;
- Financial Crimes Enforcement Network (Treasury Bureau);
* Assistant Secretary (Tax Policy);
- Alcohol, Tobacco, Tax and Trade Bureau (Treasury Bureau);
* Assistant Secretary (Management and CFO);
- Deputy Assistant Secretary (Chief Information Officer); Associate
Chief Information Officer (Planning and Management);
- Deputy Assistant Secretary (Departmental Offices Operations);
- Office of the Procurement Executive;
- Deputy Assistant Secretary (Human Resources and Chief Human Capital
Officer); Director, Office of DC Pensions.
Source: Department of the Treasury.
[End of figure]
Treasury Relies on Information Technology to Carry Out Its Mission:
Information technology plays a critical role in helping Treasury meet
its mission. For example, the Internal Revenue Service relies on a
number of information systems to process tax returns, account for tax
revenues collected, send bills for taxes owed, issue refunds, assist in
the selection of tax returns for audit, and provide telecommunications
services for business activities, including the public's toll-free
access to tax information. To assist with delinquent debt collections,
Treasury is engaged in the development of the FedDebt system. In fiscal
year 2008, Treasury plans to spend approximately $3 billion for 234 IT
investments--including about $2 billion (about 71 percent) for 60 major
investments.
Prior Reviews of IT Management Issues at Treasury Identified
Weaknesses:
In 2004,[Footnote 5] we identified weaknesses in Treasury's IT
investment management processes. For example, Treasury did not describe
or document work and decision-making processes for agencywide board(s).
Additionally, it did not use the IT asset inventory as part of
managerial decision making. As a result of these and the other
identified weaknesses, we made recommendations to the Secretary of the
Treasury to improve the department's IT investment management
processes.
In 2007,[Footnote 6] we reported that Treasury had made progress in
establishing many of the practices needed to build an investment
foundation and manage its products as a portfolio. However, we
identified additional investment management weaknesses. Specifically,
the department lacked an executive investment review board that was
actively engaged in the investment management process. As a result of
these weaknesses, we made recommendations to Treasury for strengthening
their investment management capability. In response, Treasury stated
that it would take steps to strengthen its investment board operations
and oversight of IT resources and programs. For example, the department
recently established an executive-level investment review board.
In July 2008, we reported that Treasury's rebaselining policy fully
addressed one of five practices leading organizations include in their
policies and partially addressed the remaining practices.[Footnote 7]
Since the time of our review, Treasury has improved its rebaselining
policies and procedures to be more consistent with those of leading
organizations.
Several of Treasury's projects have been deemed to be poorly planned
and managed by the OMB and have warranted inclusion on OMB's Management
Watch and High Risk Lists. In recent testimony summarizing our analysis
of projects on these lists,[Footnote 8] we reported that Treasury had 4
projects on the Management Watch List as of July 2008, including one on
the list for the fourth consecutive year. We also reported that the
department had 21 high-risk projects determined to be poorly
performing, most of them because of cost and schedule variances
exceeding 10 percent.
Earned Value Management Provides Insight on Program Cost and Schedule:
Pulling together essential cost, schedule, and technical information in
a meaningful, coherent fashion is a challenge for most programs. In
addition to comparing budgeted to actual costs, EVM measures the value
of work accomplished in a given period. This technique compares the
earned value with the planned value of work scheduled and with the
actual cost of work accomplished for that period.
Differences in these values are measured in both cost and schedule
variances. Cost variances compare the earned value of the completed
work with the actual cost of the work performed. For example, if a
contractor completed $5 million worth of work and the work actually
cost $6.7 million, there would be a -$1.7 million cost variance.
Schedule variances are also measured in dollars, but they compare the
earned value of the work completed with the value of work that was
expected to be completed. For example, if a contractor completed $5
million worth of work at the end of the month but was budgeted to
complete $10 million worth of work, there would be a -$5 million
schedule variance. Positive variances indicate that activities are
costing less or are completed ahead of schedule, whereas negative
variances indicate activities are costing more or are falling behind
schedule. These cost and schedule variances can be used to estimate the
cost and time needed to complete a program.
Without knowing the planned cost of completed work (that is, the earned
value), it is difficult to determine a program's true status. Earned
value provides information necessary for understanding the health of a
program; it provides an objective view of program status. As such, it
can alert program managers to potential problems sooner than
expenditures alone can, thereby reducing the chance and magnitude of
cost overruns and schedule delays. Moreover, EVM directly supports the
institutionalization of key processes for acquiring and developing
systems and the ability to effectively manage investments--areas which
are often found to be inadequate based on our assessments of major IT
investments.
Because of the importance of ensuring quality earned value data, in May
1998 the American National Standards Institute (ANSI) and the
Electronics Industries Alliance (EIA) jointly established a national
standard for EVM systems. This standard delineates 32 guidelines on how
to establish a sound EVM system, ensure that the data coming from the
system are reliable, and use the earned value data to manage the
program. See appendix III for details on the 32 guidelines.
Federal Guidance Calls for Using Earned Value Management to Improve IT
Management:
In June 2002, OMB's Circular A-11 included the requirement that
agencies use a performance-based acquisition management system based on
the May 1998 ANSI/EIA Standard to obtain timely information regarding
the progress of capital investments.
This requirement was restated in subsequent versions of the circular
and, in August 2005, OMB issued a memorandum[Footnote 9] that outlined
steps that agencies must take for all major and high-risk development
projects to better ensure improved execution and performance and to
promote more effective oversight through the implementation of EVM.
Specifically, this guidance directs agencies to:
1. develop comprehensive policies to ensure that agencies are using EVM
to plan and manage development activities for major IT investments,
2. include a provision and clause in major acquisition contracts or
agency in-house project charters directing the use of an EVM system
that is compliant with the ANSI standard,
3. provide documentation demonstrating the EVM system complies with the
national standard,
4. conduct periodic surveillance reviews, and:
5. conduct integrated baseline reviews[Footnote 10] on individual
programs to finalize the cost, schedule, and performance goals.
Building on OMB's guidance, in July 2007, we issued an exposure draft
on best practices for estimating and managing program costs.[Footnote
11] This draft highlights policies and practices adopted by leading
organizations to implement an effective EVM program. Specifically, the
guidance identifies the need for organizational policies to require
clear criteria for which programs are required to use EVM, compliance
with the ANSI standard, a standard product-oriented structure for
defining work products, integrated baseline reviews, specialized
training, criteria and conditions for rebaselining programs, and an
ongoing surveillance function. In addition, the guidance identifies key
practices that individual programs can use to ensure that they
establish a sound EVM system, that the earned value data are reliable,
and that they are used to support decision making. OMB refers to this
guide as a key reference manual for agencies in its 2006 Capital
Programming Guide.[Footnote 12]
Treasury's Approach to Earned Value Management:
Treasury's approach to EVM involves several entities, including the
Office of the Chief Information Officer (OCIO), the Office of the
Procurement Executive--both of which are under the Assistant Secretary
for Management and Chief Financial Officer, and Capital Planning and
Investment Control (CPIC) desk officers. Responsibility for the
administration and maintenance of Treasury's EVM policy lies with the
OCIO. Specifically, the CPIC group within that office supports the
department's investment management oversight process.[Footnote 13] CPIC
desk officers are responsible for oversight of one or more bureaus and
serve as the bureau CPIC coordinator's primary point of contact,
responsible for scoring exhibit 300s[Footnote 14] and coordinating
information sharing with the departmental budget office and other
critical partners. Further, they develop bureau-level IT portfolio
expertise and provide input and recommendations to the bureaus,
Treasury's CIO, and Treasury's Investment Review Board.
Working with the OCIO to identify acquisitions which require earned
value management, the Office of the Procurement Executive is
responsible for ensuring that the identified acquisitions throughout
Treasury and its bureaus contain EVM requirements that are consistent
with the Federal Acquisition Regulation. According to agency officials,
40 investments are currently using EVM.
Project managers and contractors are required to gather the monthly
costs and progress associated with each of their investments. The
information gathered includes the planned value, actual costs, and
earned value. This information is analyzed and used for corrective
actions at the bureau level. Quarterly, the bureaus forward investment
performance reports to the OCIO's CPIC office, which reviews them and
forwards summaries to Treasury's Technical Investment Review Board.
In January 2008, Treasury convened an EVM working group, which has
representation from every bureau. According to the CPIC Director, the
working group has several objectives including establishing (1) level
of reporting for contractors and government employees based on
thresholds; (2) rule sets, processes, and procedures for the
development of work breakdown structures, integrated baselines,
standard roll-up into milestones, and the use of EVM systems at the
bureaus; (3) bureau monthly recordkeeping requirements; (4) standard
procedures for quarterly uploading of data from the bureaus into
Treasury's automated investment management tool; and (5) requirements
for maintaining documentation to support the project manager
validations and bureau CIO certifications of cost, schedule, and
performance data for major and nonmajor investments. According to the
CPIC Director, the working group is also working on revising the
department's EVM policy.
Treasury's EVM Policy Is Not Fully Consistent with Best Practices:
While Treasury has established policy to guide its implementation of
EVM, key components of this policy are not fully consistent with best
practices. Without a comprehensive policy, the department risks
implementing policies inconsistently and using inaccurate cost and
schedule performance data.
We recently reported[Footnote 15] that leading organizations establish
EVM policies that:
* establish clear criteria defining which programs are to use EVM;
* require programs to comply with a national ANSI standard;
* require programs to use a standard structure for defining work
products;
* require programs to conduct detailed reviews of expected costs,
schedules, and deliverables (called an integrated baseline review);
* require and enforce EVM training;
* define when programs may revise cost and schedule baselines (called
rebaselining); and:
* require system surveillance--routine validation checks to ensure that
major acquisitions are complying with agency policies and standards.
Table 1 further describes these seven key components of an effective
EVM policy.
Table 1: Key Components of an Effective EVM Policy:
Component: Criteria for implementing EVM on all major investments;
Description: OMB requires agencies to implement EVM on all major IT
investments and to ensure that the corresponding contracts include
provisions for using EVM systems. Each agency is responsible for
establishing its own definition of a "major" IT investment and clearly
defining the conditions under which a new or ongoing acquisition
program is required to implement EVM.
Component: Compliance with the ANSI standard;
Description: OMB requires agencies to use EVM systems that are
compliant with a national standard developed by ANSI and the EIA
(ANSI/EIA 748-B). This standard consists of 32 guidelines that an
organization can use to establish a sound EVM system, ensure that the
data resulting from the EVM system are reliable, and use earned value
data for decision-making purposes (see app. III).
Component: Standard structure for defining the work products;
Description: The work breakdown structure defines the work necessary to
accomplish a program's objectives. It is the first criterion stated in
the ANSI standard and the basis for planning the program baseline and
assigning responsibility for the work. It is a best practice to
establish a product-oriented work breakdown structure because it allows
a program to track cost and schedule by defined deliverables, such as a
hardware or software component. This allows a program manager to more
precisely identify which components are causing cost or schedule
overruns and to more effectively mitigate the root cause of the
overruns. Standardizing the work breakdown structure is also considered
a best practice because it enables an organization to collect and share
data among programs.
Component: Integrated baseline review;
Description: An integrated baseline review is an evaluation of the
performance measurement baseline--the foundation for an EVM system--to
determine whether all program requirements have been addressed, risks
have been identified, mitigation plans are in place, and available and
planned resources are sufficient to complete the work. The main goal of
an integrated baseline review is to identify potential program risks,
including risks associated with costs, management processes, resources,
schedules, and technical issues.
Component: Training requirements;
Description: EVM training should be provided and enforced for all
personnel with investment oversight and program management
responsibilities. Executive personnel with oversight responsibilities
need to understand EVM terms and analysis products in order to make
sound investment decisions. Program managers and staff need to be able
to interpret and validate earned value data to effectively manage
deliverables, costs, and schedules.
Component: Rebaselining criteria;
Description: At times, management may conclude that the remaining
budget and schedule targets for completing a program (including the
contract) are significantly insufficient and that the current baseline
is no longer valid for realistic performance measurement. They may
decide that a revised baseline for the program is needed to restore
management's control of the remaining work effort. An agency's
rebaselining criteria should define acceptable reasons for rebaselining
and require programs to explain why the current plan is no longer
feasible and what measures will be implemented to prevent recurrence,
and develop a realistic cost and schedule estimate for remaining work
that has been validated and spread over time to the new plan.
Component: System surveillance;
Description: Surveillance is the process of reviewing a program's
(including contractor's) EVM system as it is applied to one or more
programs. The purpose of surveillance is to focus on how well a program
is using its EVM system to manage cost, schedule, and technical
performance. Two goals are associated with EVM system surveillance: to
ensure that the program is following corporate processes and
procedures, and to confirm that the program's processes and procedures
continue to satisfy ANSI guidelines.
Source: GAO, Cost Assessment Guide: Best Practices for Estimating and
Managing Program Costs, GAO-07-1134SP Exposure Draft (Washington, D.C.:
July 2007).
[End of table]
In December 2005, Treasury developed the EVM Policy Guide, which
provides an approach for implementing EVM requirements for the
department's major investments.[Footnote 16] The policy currently in
place fully addresses three of the seven components, partially
addresses three, and does not address one (see table 2).
Table 2: Assessment of Treasury's EVM Policies, as of May 2008:
Policy component: Criteria for implementing EVM on all major
investments;
Assessment of Treasury policy: Fully addressed.
Policy component: Compliance with the ANSI standard;
Assessment of Treasury policy: Partially; addressed.
Policy component: Standard structure for defining the work products;
Assessment of Treasury policy: Partially addressed.
Policy component: Integrated baseline review;
Assessment of Treasury policy: Fully addressed.
Policy component: Training requirements;
Assessment of Treasury policy: Not addressed.
Policy component: Rebaselining criteria;
Assessment of Treasury policy: Fully addressed.
Policy component: System surveillance;
Assessment of Treasury policy: Partially addressed.
Source: GAO analysis of Treasury data.
[End of table]
Specifically, Treasury has policies and guidance that fully address
criteria for implementing EVM on all major investments, for the conduct
of integrated baseline reviews, and for rebaselining.
* Criteria for implementing EVM on all major investments: The
department's policy requires all of its major development,
modernization, and enhancement investments to use EVM. Investments in
steady-state (i.e., those with no development, modernization, or
enhancement milestones) and those ending prior to September 2007 were
not required by the department to implement the EVM requirement.
* Integrated baseline review: In order to verify whether the
performance measurement baseline is realistic and to ensure that the
government and contractor mutually understand program scope, schedule,
and risks, Treasury's policy calls for an integrated baseline review.
According to the policy, this review should be completed as soon as
possible but no later than 6 months after the contract is awarded.
Furthermore, another review may be required following any significant
contract modifications.
* Rebaselining criteria: Treasury developed a rebaselining policy which
specifies that a valid reason for requesting a new baseline must be
clearly understood and documented. The policy also specifies acceptable
reasons for an investment team to request a rebaseline. Further, to
submit a rebaseline request, investment teams are required to explain
why the current plan is no longer feasible and develop realistic cost
and schedule estimates for remaining work that has been validated and
spread over time to the new plan.
However, Treasury's policy and guidance do not fully address the best
practices represented by the following three key components: addressing
compliance with the ANSI standard, defining a meaningful structure for
defining work products, and conducting system surveillance reviews.
Training is not addressed by the Treasury policy.
* Compliance with the ANSI standard: Treasury policy states that major
investments are to comply with ANSI standards. Further, it outlines
processes and guidelines to assist its bureau in achieving ANSI-
compliant processes. However, the policy lacks sufficient detail for
addressing some of the criteria defined in the standard, including the
use of standard methods for EVM data collection across the department
and cost performance reporting. For example, the policy does not
discuss the use of templates or tools to help ensure that EVM data are
collected consistently and reliably. Furthermore, the policy does not
discuss what cost performance report formats are to be used. Until
Treasury's policy includes a methodology that standardizes data
collection and reporting, data integrity and reliability may be in
jeopardy and management may not be able to make informed decisions
regarding the investments and its next steps.
* Standard structure for defining the work products: Treasury's EVM
policy calls for a product-oriented work breakdown structure that
identifies and documents all activities associated with the investment.
However, it does not require the use of common elements[Footnote 17] in
its development. According to the CPIC Director, Treasury's EVM working
group plans to establish rule sets, processes, and procedures for the
development of work breakdown structures. Until Treasury's policy
provides more guidance on the systematic development and documentation
of work breakdown structures including the incorporation of
standardized common elements, it will be difficult to ensure that the
entire effort is consistently included in the work structure and that
investments will be planned and managed appropriately.
* System surveillance: According to Treasury's policy, the contractor's
EVM system is to be validated using the industry surveillance approach
identified by the National Defense Industrial Association's
Surveillance Guide.[Footnote 18] Additionally, Treasury is to require
clear evidence that the system continues to remain compliant or that
the contractor has brought the system back into compliance. However,
the policy lacks guidance on conducting surveillance reviews on the
government's (i.e., the department's) EVM system. Until Treasury's
policy specifies reviews of the government's systems, Treasury risks
not being able to effectively manage cost, schedule, and technical
performance of its major investments.
* Training requirements: Treasury's policy does not specify EVM
training requirements for program management team members or senior
executives. Furthermore, the policy does not require the agency to
maintain training logs confirming that all relevant staff have been
appropriately trained. Until the department establishes policy for EVM
training requirements for relevant personnel, it cannot effectively
ensure that its program staff have the appropriate skills to validate
and interpret EVM data and that its executives fully understand the
data they are given in order to ask the right questions and make
informed decisions.
According to the CPIC Director, Treasury's EVM working group, which was
established in January 2008, is working on the development of a revised
EVM policy, which, according to Deputy Assistant Secretary for
Information Systems and Chief Information Officer, is expected to be
finalized by October 2008. Addressing these weaknesses could help
Treasury optimize the effective use of EVM.
Treasury Is Not Fully Implementing Key Earned Value Management
Techniques to Manage Critical System Investments:
While the six programs we reviewed were all using EVM, none had fully
implemented any of the practices for establishing a comprehensive EVM
system, ensuring that the data resulting from the system are reliable,
or using earned value data for decision-making purposes. These
weaknesses exist in part because, as previously noted, Treasury's
policy does not fully address key elements and because the department
does not have a mechanism to enforce its implementation. Until Treasury
adequately implements EVM, it faces an increased risk that some
programs will experience cost and schedule overruns or deliver less
capability than planned.
In our work on best practices, we identified three key management areas
that leading organizations use to manage their acquisitions:
establishing a comprehensive EVM system, ensuring reliable data, and
using earned value data to manage the investment (see table 3).
Table 3: Key EVM Practices for System Acquisition Programs:
Program management area: Establish a comprehensive EVM system;
EVM practice:
* Define the scope of effort using a work breakdown structure.
* Identify who in the organization will perform the work.
* Schedule the work.
* Estimate the labor and material required to perform the work and
authorize the budgets, including management reserve.
* Determine objective measure of earned value.
* Develop the performance measurement baseline.
Program management area: Ensure that the data resulting from the EVM
system are reliable;
EVM practice:
* Execute the work plan and record all costs.
* Analyze EVM performance data and record variances from the
performance measurement baseline plan.
* Forecast estimates at completion.
Program management area: Ensure that the program management team is
using earned value data for decision-making purposes;
EVM practice:
* Take management action to mitigate risks.
* Update the performance measurement baseline as changes occur.
Source: GAO, Cost Assessment Guide.
[End of table]
Table 4 provides a summary of how each investment is using EVM in the
key practices areas and is followed by our analysis of these areas. The
investments we reviewed are the Financial Management Service's FedDebt
and Financial Information and Reporting Standardization (FIRST); the
Departmental Office's DC Pension System to Administer Retirement
(STAR); the Bureau of Public Debt's Treasury Automated Auction
Processing System (TAAPS); and the Internal Revenue Service's
Integrated Financial System/Core Financial System (IFS) and Enterprise
Data Access Strategy (EDAS). These investments were identified by the
department as major investments and all had milestones in development,
modernization, or enhancement at the time of our review. Appendix II
includes information regarding the selection of these investments and
appendix IV provides a description of each.
Table 4: Assessment of EVM Use to Manage Treasury Investments:
Program management key practices: Establish a comprehensive EVM system;
EDAS: partially implemented;
FedDebt: partially implemented;
FIRST: partially implemented;
IFS: partially implemented;
STAR: partially; implemented;
TAAPS: partially; implemented.
Program management key practices: Ensure that the data resulting from
the EVM system are reliable;
EDAS: not implemented;
FedDebt: partially; implemented;
FIRST: partially; implemented;
IFS: partially; implemented;
STAR: partially; implemented;
TAAPS: partially; implemented.
Program management key practices: Ensure that the program management
team is using earned value data for decision-making;
EDAS: not implemented;
FedDebt: partially; implemented;
FIRST: partially; implemented;
IFS: partially; implemented;
STAR: partially; implemented;
TAAPS: partially; implemented.
Source: GAO analysis of Treasury data.
Key:
Fully implemented: The investment addresses all of the practices
identified in the corresponding program management area.
Partially implemented: The investment addresses some, but not all, of
the practices identified in the corresponding program management area.
Not implemented: The investment does not address any of the practices
identified in the corresponding program management area.
[End of table]
Investments All Partially Established a Comprehensive Earned Value
Management System:
Comprehensive EVM systems were not consistently established to manage
the six investments. Although aspects of a comprehensive system were
present, none of the investments fully met all the best practices
comprising this management area.
For example, of the six investments, only IFS and STAR adequately
defined the scope of effort using a work breakdown structure. Three
investments developed a work breakdown structure; however, the work
packages could not be traced back to EVM project management documents,
such as the project management baseline, the work breakdown structure,
and the statement of work or project charter. For example, although
EDAS had detailed work breakdown structures, correlation could not be
established among the work breakdown structure elements, the contract
deliverables, and the elements being reported in the contract
performance reports. Officials for the remaining investment--TAAPS--
stated that there was a documented work structure; however, they did
not provide evidence of this.
As another example, performance measurement baselines were developed
for five of six investments. However, the baselines had noted
weaknesses. Specifically, four investments--FIRST, IFS, STAR, and
TAAPS--had a baseline, but some elements were not included, such as
planned costs for STAR. Further, for TAAPS, independent validation of
the investment's baseline was not conducted. FedDebt had a performance
measurement baseline which underwent integrated baseline validation in
March 2006. However, the validation indicated that there was no time-
phased planned value at the individual contract level, nor was there a
roll-up at the program level. No explanation was provided of how the
monthly performance data on individual FedDebt projects were rolled up
to the investment level as required by OMB.[Footnote 19] Further, EDAS
did not have a time-phased budget baseline or a performance measurement
baseline.
Investments Did Not Fully Implement Steps to Ensure Data Reliability:
None of the six investments fully implemented the steps to ensure data
reliability from their EVM systems. Five partially implemented the
steps, and one investment--EDAS--did not meet any of the steps.
When executing work plans and recording actual costs, two of the six
investments incorporated government costs with contractor costs. For
example, FedDebt included both government and contractor costs in their
quarterly reporting. However, while IFS had a mechanism for recording
monthly government costs, it did not have a method that combined both
contractor and government costs for review on a monthly basis. Also,
few if any checks are performed to measure the quality of EVM data and,
according to agency officials, Treasury currently focuses more on
reporting the data than on their reliability.
In addition, five of the six investments did not adequately analyze
performance data and record the variances from the baseline. The IFS
investment included monthly reviews of performance reports and included
cost and schedule variances. The remaining investments conducted
analyses of performance data, but did not all provide documentation to
show cost and schedule updates and variances. For example, according to
officials, TAAPS' cost and schedule variances were calculated at the
project and program levels, but evidence of this could not be provided.
Further, as part of its performance reporting, STAR did not calculate
the cost variance and incorrectly calculated the schedule variance.
Most Investments Did Not Fully Implement Practices for Using Earned
Value Data for Decision Making:
None of the six investments fully implemented the two practices needed
to ensure the use of EVM data for decision-making purposes.
Specifically, EDAS did not take management action to mitigate risks
identified through their EVM performance data, or update the
performance measurement baseline as changes occurred; IFS addressed one
of these practices, and the remaining investments only partially
addressed them.
In order to support management action to mitigate risks identified
through EVM performance data--variance analysis, corrective action
planning, and reviewing estimates at completion--the IFS project
manager was provided with monthly performance report that indicated
when cost and/or schedule variances exceeded acceptable tolerances.
Further, investment-level status was provided to bureau-level and
agency-level management to allow them to make capital planning and
investment control decisions. However, the remaining five investments
did not fully take action to mitigate risks for a variety of reasons.
For example, for FedDebt, although some monthly EVM data were included
in quarterly reports, no documentation was provided on how such data
were being used to manage at the project or investment level. A similar
situation exists for the TAAPS investment where, although agency
officials stated that meetings were routinely held to discuss
performance issues, no evidence was provided that a systematic method
existed to use EVM metrics for decision-making purposes.
Regarding the update of performance measurement baselines as changes
occur, one investment team stated that it did not have any baseline
changes; however, documentation showed that the schedule for the
investment had been changed three times. In addition, although IFS
maintained a log for tracking changes, we could not determine that
these changes had been incorporated into the baseline. Further,
according to officials, EDAS had a scope change in fiscal year 2007;
however, the investment team was not able to provide documentation
reflecting the corresponding change in the performance measurement
baseline.
Inconsistent Implementation Is Due in Part to Weaknesses in Policy and
Level of Oversight:
The inconsistent application of EVM across investments exists in part
because the department does not have a policy that fully addresses key
components including training and system surveillance and because the
department is leaving the implementation of the policy largely up to
bureaus. For example, project management staff had not consistently
received training, an item which is not addressed in the policy, and
surveillance reviews, which are partially addressed in the policy, had
not been performed for any of the investments.
Furthermore, the department does not have a process for ensuring
effective EVM implementation. However, in comments on a draft of this
report, the Deputy Assistant Secretary for Information Systems and
Chief Information Officer stated that the department is working with
the bureaus to establish mechanisms and tools to ensure full compliance
with the provisions of the updated EVM policy, which is to be finalized
by October 2008. These mechanisms and tools would help address the
implementation gaps we have identified.
Conclusions:
Treasury has established a policy that addresses criteria for
implementing EVM, integrated baseline reviews, and project rebaselining
consistent with best practices. However, it does not fully address
other elements including compliance with the ANSI standard and system
surveillance, which are necessary for effective implementation. In
regards to implementation, the department is not fully addressing key
practices needed to effectively manage its critical investments.
Specifically, none of the six programs we reviewed were fully
implementing any of the practices associated with establishing a
comprehensive EVM system, ensuring the reliability of the data
resulting from the system, or using earned value data to make
decisions. The gaps in implementation are due in part to the weaknesses
with the policy and to the low level of oversight provided by the
department. Until the department defines a comprehensive policy and
establishes a process for ensuring effective EVM implementation, it
will be difficult for Treasury to optimize the effectiveness of EVM as
a management tool and consistently implement the fundamental practices
needed to effectively manage its critical programs.
Recommendations for Executive Action:
To improve Treasury's ability to effectively implement EVM on its IT
acquisition programs, we recommend that the Secretary of Treasury
direct the Assistant Secretary for Management, in collaboration with
the Chief Information Officer, to take the following nine actions:
Define a comprehensive EVM policy that specifies:
* a methodology that standardizes EVM data collection and reporting
compliant with the ANSI standard;
* a systematic approach to the development and documentation of work
breakdown structures including the incorporation of standardized common
elements;
* guidance on conducting surveillance reviews on the government's EVM
system; and:
* training requirements for relevant personnel.
Implement a process for ensuring effective implementation of EVM
throughout the department by:
* establishing a comprehensive EVM system by, among other things,
* defining the scope of effort using a work breakdown structure that
allows for traceability across EVM project management documents;
* ensuring the development of validated performance measurement
baselines that includes planned costs and schedules;
* ensuring that the data resulting from the EVM system are reliable,
including:
* executing the work plan and recording both government and contractor
costs;
* ensuring that the program management team is using earned value data
for decision-making by:
* systematically using EVM performance metrics in making the ongoing
monthly decisions required to effectively manage the investment; and:
* properly documenting updates to the performance measurement baseline
as changes to the cost and schedule occur.
Agency Comments and Our Evaluation:
In written comments on a draft of this report, the Department of
Treasury's Deputy Assistant Secretary for Information Systems and Chief
Information Officer generally agreed with our findings and stated that
the department will issue a revised version of the EVM policy that will
address our nine recommendations by October 2008. He also noted that
the department is working with the bureaus to establish mechanisms and
tools including processes for conducting system surveillance and
monitoring of EVM data to ensure compliance with the policy. Treasury
also provided technical comments which we have addressed as
appropriate. Treasury's written comments are reprinted in appendix I.
We will be sending copies of this report to interested congressional
committees, the Secretary of Treasury, and other interested parties. In
addition, the report will be available at no charge on our Web site at
[hyperlink, http://www.gao.gov]. If you or your staffs have any
questions on matters discussed in this report, please contact me at
(202) 512-9286 or by e-mail at pownerd@gao.gov. Contact points for our
Offices of Congressional Relations and Public Affairs may be found on
the last page of this report. GAO staff who made major contributions to
this report are listed in appendix V.
Signed by:
David A. Powner:
Director, Information Technology Management Issues:
[End of section]
Appendix I: Comments from the Department of Treasury:
Department Of The Treasury:
Washington, D.C. 20220:
September 16, 2008:
Mr. David A. Powner:
Director, Information Technology Management Issues:
United States Government Accountability Office:
Washington, DC 20548:
Dear Mr. Powner:
The Department of the Treasury has reviewed the draft Government
Accountability Office (GAO) report and is in general agreement with its
findings and appreciates the opportunity to review this Report. We
respectfully request that the GAO consider our specific comments and
clarifications in the attached prior to finalizing the report.
The Department is committed to ensuring that our information technology
investments are properly managed and appropriately follow prescribed
Earned Value Management (EVM) principles and best practices. We
appreciate GAO's view that successful application and management use of
EVM in the Department of the Treasury requires comprehensive policy
guidance, and we recognize there are areas within our current EVM
policy that need clarification and more detailed description of
procedures.
The Office of the Chief Information Officer (OCIO) is working closely
with the Treasury Bureaus to finalize a revised version of the EVM
policy that will address the nine recommendations included in the draft
GAO report. We anticipate this guidance will be issued in final form by
October 2008.
Beyond providing expanded policy guidance, OCIO is also focusing on
ensuring Departmental and Bureau staffs receive adequate training for
program and project managers including the use of EVM. Training is
already completed or underway in several Treasury components, including
the Bureau of Public Debt, the Financial Management Service, and the
Internal Revenue Service. Treasury is also actively implementing the
Federal Acquisition Council's Program and Project Management
certification requirements as required by the Office of Federal
Procurement Policy (OFPP) Memorandum for Chief Acquisition Officers of
April 25, 2007. The review of qualifications for approximately 60
program and project managers was conducted in August 2008. It is
anticipated that certifications or waivers (for new Program/Project
Managers) will be issued before the end of Fiscal Year 2008.
The OCIO is also working closely with our procurement office and the
Bureaus to establish mechanisms and tools to ensure full compliance
with the provisions of the updated EVM policy. Such measures include
specification of processes for conducting system surveillance and
monitoring of EVM data.
If you have any additional questions, please let me know, or contact
Diane Litman, Associate CIO for Planning and Management on 202-622-
7704.
Sincerely,
Signed by:
Michael D. Duffy:
Deputy Assistant Secretary for Information Systems and Chief
information Officer:
Attachment:
[End of section]
Appendix II: Objectives, Scope, and Methodology:
Our objectives were to determine whether the Department of the Treasury
and its key component agencies (1) have the policies in place to
effectively implement earned value management (EVM) and (2) are
adequately using EVM techniques to manage critical system investments.
To assess whether Treasury has policies in place to effectively
implement EVM, we analyzed Treasury and its component bureaus' policies
and guidance that support EVM implementation departmentwide as well as
on capital planning and investment control guidance. Specifically, we
compared these policies and guidance documents to both Office of
Management and Budget requirements and key best practices recognized
within the federal government and industry for the implementation of
EVM. These best practices are contained in an exposure draft version of
our cost guide.[Footnote 20] We also interviewed key agency officials,
including the Director for Capital Planning and Investment Control, to
obtain information on the agency's ongoing and future EVM plans.
To determine whether Treasury is adequately using EVM techniques to
manage critical system investments, we reviewed 6 of the 40 systems the
department required to use EVM. Specifically, we selected investments
from each of the four component agencies identified as having eligible
investments. We selected one investment from the Bureau of Public Debt,
another from Departmental Offices, and two from the Financial
Management Service and the Internal Revenue Service since they had a
greater percentage of investments using EVM. With the exception of the
Bureau of Public Debt which had only one major investment, we selected
investments based on (1) size, (2) EVM history (i.e., use of EVM for a
long enough period of time to have some history of EVM data), and (3)
completion date (i.e., those that would not end during the course of
our review). The 6 projects selected were FedDebt and Financial
Information and Reporting Standardization from the Financial Management
Service, DC Pension System to Administer Retirement (STAR) from the
Departmental Offices, Treasury Automated Auction Processing System
(TAAPS) from the Bureau of Public Debt, and Integrated Financial
System/Core Financial System and Enterprise Data Access Strategy from
the Internal Revenue Service. Our review was not intended to be
generalizable, but instead to illustrate the status of a variety of
programs.
To determine the extent of each program's implementation of sound EVM,
we compared program documentation to the 11 fundamental EVM practices
implemented on acquisition programs of leading organizations, as
identified in the Cost Assessment Guide.[Footnote 21] We determined
whether the program fully implemented, partially implemented, or did
not implement each of the practices.[Footnote 22] Finally, we
interviewed program officials to obtain clarification on how EVM
practices are implemented and how the data are validated and used for
decision-making purposes. Regarding the reliability of cost data, we
did not test the adequacy of agency or contractor cost-accounting
systems. Our evaluation of these cost data was based on what we were
told by the agency and the information they could provide.
We conducted this performance audit from August 2007 to July 2008 in
Washington, D.C., in accordance with generally accepted government
auditing standards. Those standards require that we plan and perform
the audit to obtain sufficient, appropriate evidence to provide a
reasonable basis for our findings and conclusions based on our audit
objectives. We believe that the evidence obtained provides a reasonable
basis for our findings and conclusions based on our audit objectives.
[End of section]
Appendix III: Overview of Industry Guidelines that Support Sound Earned
Value Management:
Organizations must be able to evaluate the quality of an EVM system in
order to determine the extent to which the cost, schedule, and
technical performance data can be relied on for program management
purposes. In recognition of this, the American National Standards
Institute (ANSI) and the Electronics Industries Alliance (EIA) jointly
established a national standard for EVM systems--ANSI/EIA 748-B
(commonly referred to as the ANSI standard). This standard consists of
32 guidelines addressing organizational structure; planning,
scheduling, and budgeting; accounting considerations; analysis and
management reports; and revisions and data maintenance. These standards
comprise three fundamental management functions for effectively using
EVM: establishing a sound earned value management system, ensuring that
the EVM data are reliable, and using earned value data for decision-
making purposes. Table 5 lists the management functions and the
guidelines.
Table 5: Management Functions Addressed by ANSI Guidance on Earned
Value Management Systems:
Management function: Establish a sound EVM system;
ANSI guidelines:
1. Define the authorized work elements for the program. A work
breakdown structure, tailored for effective internal management
control, is commonly used in this process.
2. Identify the program organizational structure including the major
subcontractors responsible for accomplishing the authorized work, and
define the organizational elements in which work will be planned and
controlled.
3. Provide for the integration of the planning, scheduling, budgeting,
work authorization, and cost accumulation processes with each other
and, as appropriate, with the program work breakdown structure and the
program organizational structure.
4. Identify the organization or function responsible for controlling
overhead (indirect costs).
5. Provide for integration of the program work breakdown structure and
the program organizational structure in a manner that permits cost and
schedule performance measurement by elements of either or both
structures as needed.
6. Schedule the authorized work in a manner that describes the sequence
of work and identifies significant task interdependencies required to
meet the requirements of the program.
7. Identify physical products, milestones, technical performance goals,
or other indicators that will be used to measure progress.
8. Establish and maintain a time-phased budget baseline, at the control
account level, against which program performance can be measured.
Initial budgets established for performance measurement will be based
on either internal management goals or the external customer negotiated
target cost, including estimates for authorized but undefinitized work.
Budget for far-term efforts may be held in higher level accounts until
an appropriate time for allocation at the control account level. If an
over target baseline is used for performance measurement reporting,
prior notification must be provided to the customer.
9. Establish budgets for authorized work with identification of
significant cost elements (labor, material, etc.) as needed for
internal management and for control of subcontractors.
10. To the extent it is practicable to identify the authorized work in
discrete work packages, establish budgets for this work in terms of
dollars, hours, or other measurable units. Where the entire control
account is not subdivided into work packages, identify the far-term
effort in larger planning packages for budget and scheduling purposes.
11. Provide that the sum of all work package budgets plus planning
package budgets within a control account equals the control account
budget.
12. Identify and control "level of effort" activities by time-phased
budgets established for this purpose. Only efforts that are
unmeasurable or for which measurement is impractical may be classified
as "level of effort" activities.
13. Establish overhead budgets for each significant organizational
component of the company for expenses that will become indirect costs.
Reflect in the program budgets, at the appropriate level, the amounts
in overhead pools that are planned to be allocated to the program as
indirect costs.
14. Identify management reserves and undistributed budget.
15. Provide that the program target cost goal is reconciled with the
sum of all internal program budgets and management reserves.
Management function: Ensure that the EVM data are reliable;
ANSI guideline:
16. Record direct costs in a manner consistent with the budgets in a
formal system controlled by the general books of account.
17. When a work breakdown structure is used, summarize direct costs
from control accounts into the work breakdown structure without
allocation of a single control account to two or more work breakdown
structure elements.
18. Summarize direct costs from the control accounts into the
contractor's organizational elements without allocation of a single
control account to two or more organizational elements.
19. Record all indirect costs that will be allocated to the program
consistent with the overhead budgets.
20. Identify unit costs, equivalent units costs, or lot costs when
needed.
21. For the earned value management system, the material accounting
system will provide for (1) accurate cost accumulation and assignment
of costs to control accounts in a manner consistent with the budgets
using recognized, acceptable, costing techniques; (2) cost recorded for
accomplishing work performed in the same period that earned value is
measured and at the point in time most suitable for the category of
material involved, but no earlier than the actual receipt of material;
and (3) full accountability of all material purchased for the program
including the residual inventory.
22. At least on a monthly basis, generate the following information at
the control account and other levels as necessary for management
control using actual cost data from, or reconcilable with, the
accounting system: (1) Comparison of the amount of planned budget and
the amount of budget earned for work accomplished. This comparison
provides the schedule variance. (2) Comparison of the amount of the
budget earned and the actual (applied where appropriate) direct costs
for the same work. This comparison provides the cost variance.
23. Identify, at least monthly, the significant differences between
both planned and actual schedule performance and planned and actual
cost performance, and provide the reasons for the variances in the
detail needed by program management.
24. Identify budgeted and applied (or actual) indirect costs at the
level and frequency needed by management for effective control, along
with the reasons for any significant variances.
25. Summarize the data elements and associated variances through the
program organization and/or work breakdown structure to support
management needs and any customer reporting specified in the contract.
Management function: Ensure that the program management team is using
earned value data for decision-making purposes;
ANSI guideline:
26. Implement managerial actions taken as the result of earned value
information.
27. Develop revised estimates of cost at completion based on
performance to date, commitment values for material, and estimates of
future conditions. Compare this information with the performance
measurement baseline to identify variances at completion important to
company management and any applicable customer reporting requirements
including statements of funding requirements.
28. Incorporate authorized changes in a timely manner, recording the
effects of such changes in budgets and schedules. In the directed
effort before negotiation of a change, base such revisions on the
amount estimated and budgeted to the program organizations.
29. Reconcile current budgets to prior budgets in terms of changes to
the authorized work and internal replanning in the detail needed by
management for effective control.
30. Control retroactive changes to records pertaining to work performed
that would change previously reported amounts for actual costs, earned
value, or budgets. Adjustments should be made only for correction of
errors, routine accounting adjustments, and effects of customer or
management directed changes, or to improve the baseline integrity and
accuracy of performance measurement data.
31. Prevent revisions to the program budget except for authorized
changes.
32. Document changes to the performance measurement baseline.
Source: Excerpts from "Earned Value Management Systems" (ANSI/EIA 748-
B) ©2007, Information Technology Association of America. All Rights
Reserved. Reprinted by permission; GAO analysis.
[End of table]
[End of section]
Appendix IV: Description of Selected Investments:
Below is a description of the six investments we reviewed to assess
whether the department is adequately using EVM techniques to manage
critical system investments.
Investment Name: FedDebt;
Investment Description: FedDebt supports the federal government's
delinquent debt collection programs, which were centralized in the
Financial Management Service (FMS) pursuant to the Debt Collection
Improvement Act of 1996. FedDebt also supports Treasury's strategic
goal to manage the U.S. Government's finances effectively and the FMS
strategic goal to maximize collection of government delinquent debt by
providing efficient and effective centralized debt collection services.
FedDebt plans to integrate the collection services that FMS provides to
Federal Program Agencies through its other programs.
Investment Name: Financial Information and Reporting Standardization
(FIRST);
Investment Description: FIRST is intended to automate the maintenance
and distribution of the U.S. Standard General Ledger accounting rules
and guidance. It also plans to integrate the general ledger guidance
with the collection of all accounting trail balance data, thus
providing a standardized method of collecting, storing, reporting, and
analyzing such data. Furthermore, the investment is expected to
facilitate accounting validations of the agency trial balance data to
provide better feedback to agencies concerning the accuracy and
consistency of these data.
Investment Name: DC Pension System to Administer Retirement (STAR);
Investment Description: STAR is to assist Treasury and the District of
Columbia Government by automating the determination of eligibility,
calculating pension benefits, and delivery of payments, therefore
allowing for (1) increased accuracy of pension benefit calculations and
(2) improved customer service. Key functionality for this investment
includes serving annuitants and survivors of the Judges Pension Plan;
making benefit payments to 11,000 teachers, police, and firefighters
who retired before July 1997, as well as their survivors; and
automatically calculating the gross annuity and split benefit payment
for teachers, police, and firefighters, to service those annuitants who
retired after June 1997.
Investment Name: Treasury Automated Auction Processing System (TAAPS);
Investment Description: TAAPS is intended to ensure that all auction-
related operations are carried out flawlessly and securely. Key among
auction activities are the announcement of upcoming Treasury auctions;
bid submission and processing; calculation of awards; publication of
results; creation and dissemination of settlement wires; creation of
accounting reports and reports needed for auctions analysis; and the
storage of all securities-, bidder-, and auction-related information.
TAAPS is expected to make numerous intersystem interfaces and manual
processes obsolete by consolidating auction processing requirements
into one system and providing appropriate backup and disaster recovery
systems and services.
Investment Name: Integrated Financial System/Core Financial System
(IFS);
Investment Description: IFS is intended to operate as the Internal
Revenue Services' (IRS) new accounting system of record, replacing
IRS's core financial systems, including expenditure controls, accounts
payable, accounts receivable, general ledger, budget formulation, and
purchasing controls. IRS intends to upgrade to software that provides
federal accounting functionality. By migrating to federal accounting
practices, IFS is to provide benefits, such as eliminating current work-
around processes, improving project management capability, and
enhancing budget reports.
Investment Name: Enterprise Data Access Strategy (EDAS);
Investment Description: EDAS is intended to consolidate data from
multiple Business Systems Modernization applications and produce a
consolidated data repository source to be used for issue detection and
case selection. The goal is to develop integrated data solutions that
allow IRS to retire duplicative and costly data extracts. The first
major project is to develop an Integrated Production Model as a central
repository for corporate data and make those data available to projects
currently in development. Long-term benefits include the retirement of
multiple systems and efficiency gains from improved processes.
Source: GAO Analysis of Treasury Data.
[End of table]
[End of section]
Appendix V: GAO Contact and Staff Acknowledgments:
GAO Contact:
David A. Powner, (202) 512-9286 or pownerd@gao.gov:
Staff Acknowledgments:
In addition to the contact named above, Sabine Paul, Assistant
Director; Neil Doherty; Mary D. Fike; Nancy Glover; Sairah R. Ijaz;
Rebecca LaPaze; and Paul B. Middleton made key contributions to this
report.
[End of section]
Footnotes:
[1] Office of Management and Budget, Report on Information Technology
(IT) Spending for the Federal Government for Fiscal Years 2007, 2008,
2009 (Washington, D.C.: April 2008).
[2] OMB Memorandum, M-05-23 (Aug. 4, 2005).
[3] GAO, Cost Assessment Guide: Best Practices for Estimating and
Managing Program Costs, Exposure Draft, [hyperlink,
http://www.gao.gov/cgi-bin/getrpt?GAO-07-1134SP] (Washington, D.C.:
July 2007).
[4] American National Standards Institute/Electronic Industries
Alliance Standard, Earned Value Management Systems, ANSI/EIA-748-B-
2007 approved July 9, 2007.
[5] GAO, Information Technology Management: Governmentwide Strategic
Planning, Performance Measurement, and Investment Management Can Be
Further Improved, [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-04-
49] (Washington, D.C.: January 2004).
[6] GAO, Information Technology: Treasury Needs to Strengthen Its
Investment Board Operations and Oversight, [hyperlink,
http://www.gao.gov/cgi-bin/getrpt?GAO-07-865] (Washington, D.C.: July
2007).
[7] GAO, Information Technology: Agencies Need to Establish
Comprehensive Policies to Address Changes to Projects' Cost, Schedule,
and Performance Goals, [hyperlink, http://www.gao.gov/cgi-
bin/getrpt?GAO-08-925] (Washington, D.C.: July 31, 2008).
[8] GAO, Information Technology: OMB and Agencies Need to Improve
Planning, Management, and Oversight of Projects Totaling Billions of
Dollars, [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-08-1051T]
(Washington, D.C.: July 31, 2008).
[9] OMB Memorandum, M-05-23 (Aug. 4, 2005).
[10] An integrated baseline review is an evaluation of a program's
baseline plan to determine whether all program requirements have been
addressed, risks have been identified, mitigation plans are in place,
and available and planned resources are sufficient to complete the
work.
[11] [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-07-1134SP].
[12] OMB, Capital Programming Guide, Supplement to Circular A-11, Part
7, version 2.0 (June 2006).
[13] The Director for the CPIC group reports to the Associate Chief
Information Officer for Planning and Management, which in turn reports
to the Chief Information Officer.
[14] OMB requires agencies to submit justification packages for major
IT investments on an annual basis. This justification package is called
the exhibit 300.
[15] [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-07-1134SP].
[16] Four bureaus (the Internal Revenue Service, Departmental Offices,
Financial Management Services, and the Office of the Comptroller of the
Currency) also have EVM policies supplementing the department's policy.
[17] The Cost Assessment Guide identifies 11 common elements that apply
to all programs and should be included in every work breakdown
structure. These elements include: (1) integration, assembly, test, and
checkout; (2) system engineering; (3) program management; (4) training;
(5) data; (6) system test and evaluation; (7) peculiar support
equipment; (8) common support equipment; (9) operational and site
activation; (10) facilities; and (11) initial spares and repair parts.
[18] National Defense Industrial Association, Surveillance Guide
(October 2004).
[19] As part of their exhibit 300 budget submissions to OMB, agencies
must provide investment-level EVM data, including cost and schedule
baselines.
[20] GAO, Cost Assessment Guide: Best Practices for Estimating and
Managing Program Costs, Exposure Draft, [hyperlink,
http://www.gao.gov/cgi-bin/getrpt?GAO-07-1134SP] (Washington, D.C.:
July 2007).
[21] [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-07-1134SP].
[22] The program was considered to have fully implemented each program
management area if it addressed all EVM practices identified in the
corresponding program management area. It partially implemented each
program management area if the investment addressed some EVM practices
identified in the corresponding program management area, but not all.
Further, the investment was deemed as not implemented if it did not
address any EVM practices identified in the corresponding program
management area.
[End of section]
GAO's Mission:
The Government Accountability Office, the audit, evaluation and
investigative arm of Congress, exists to support Congress in meeting
its constitutional responsibilities and to help improve the performance
and accountability of the federal government for the American people.
GAO examines the use of public funds; evaluates federal programs and
policies; and provides analyses, recommendations, and other assistance
to help Congress make informed oversight, policy, and funding
decisions. GAO's commitment to good government is reflected in its core
values of accountability, integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each
weekday, GAO posts newly released reports, testimony, and
correspondence on its Web site. To have GAO e-mail you a list of newly
posted products every afternoon, go to [hyperlink, http://www.gao.gov]
and select "E-mail Updates."
Order by Mail or Phone:
The first copy of each printed report is free. Additional copies are $2
each. A check or money order should be made out to the Superintendent
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or
more copies mailed to a single address are discounted 25 percent.
Orders should be sent to:
U.S. Government Accountability Office:
441 G Street NW, Room LM:
Washington, D.C. 20548:
To order by Phone:
Voice: (202) 512-6000:
TDD: (202) 512-2537:
Fax: (202) 512-6061:
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]:
E-mail: fraudnet@gao.gov:
Automated answering system: (800) 424-5454 or (202) 512-7470:
Congressional Relations:
Ralph Dawn, Managing Director, dawnr@gao.gov:
(202) 512-4400:
U.S. Government Accountability Office:
441 G Street NW, Room 7125:
Washington, D.C. 20548:
Public Affairs:
Chuck Young, Managing Director, youngc1@gao.gov:
(202) 512-4800:
U.S. Government Accountability Office:
441 G Street NW, Room 7149:
Washington, D.C. 20548:
