Department of Energy
Further Actions Are Needed to Strengthen Contract Management for Major Projects
Gao ID: GAO-05-123 March 18, 2005
The Department of Energy (DOE) pays its contractors billions of dollars each year to implement its major projects--those costing more than $400 million each. Many major projects have experienced substantial cost and schedule overruns, largely because of contract management problems. GAO was asked to assess, for major departmental projects, (1) DOE's use of performance incentives to effectively control costs and maintain schedules, (2) the reliability of the data DOE uses to monitor and assess contractor performance, and (3) the reliability of the Project Assessment and Reporting System (PARS) data that senior managers use for project oversight.
DOE could use performance incentives more effectively for controlling costs and schedules if it developed performance incentive guidance and assigned responsibility for reviewing a contract's project management provisions prior to award. DOE has awarded contracts for 15 of 33 major projects that use a schedule or other performance incentive without an associated cost incentive or constraint; thus a contractor could receive full fees by meeting all schedule baselines while substantially overrunning costs. DOE has relied on unvalidated contractor data to monitor contractors' progress in executing major projects and to award fees for performance. In particular, DOE's self-assessment of contract administration in 2002 found that field personnel overly relied on contractors' accounting systems and contractor-collected data in assessing performance, without significant validation of those data. No subsequent self-assessment has been conducted to determine if this problem continues. Furthermore, DOE has not required that its contracting officers receive the training needed to assess the adequacy of contractors' project management systems that generate data used to monitor progress. A lthough development of PARS is a positive step, the reliability of the project performance data that PARS provides to senior DOE managers is limited by problems with accuracy, completeness, and timeliness. Regarding accuracy, DOE has not assessed the reliability of contractors' project management systems that feed data into PARS for 31 of 33 major projects, even though DOE believes that some systems are deficient. Regarding completeness, GAO identified 3 major projects that are not in PARS. As to timeliness, cost and schedule data for 6 major projects in the June 2004 PARS report were significantly out of date because DOE has not required contractors to submit timely performance data. These contract management problems limit DOE's ability to effectively manage its major projects and avoid further cost and schedule slippages.
Recommendations
Our recommendations from this work are listed below with a Contact for more information. Status will change from "In process" to "Open," "Closed - implemented," or "Closed - not implemented" based on our follow up work.
Director:
Team:
Phone:
GAO-05-123, Department of Energy: Further Actions Are Needed to Strengthen Contract Management for Major Projects
This is the accessible text file for GAO report number GAO-05-123
entitled 'Department of Energy: Further Actions Are Needed to
Strengthen Contract Management for Major Projects' which was released
on April 19, 2005.
This text file was formatted by the U.S. Government Accountability
Office (GAO) to be accessible to users with visual impairments, as part
of a longer term project to improve GAO products' accessibility. Every
attempt has been made to maintain the structural and data integrity of
the original printed product. Accessibility features, such as text
descriptions of tables, consecutively numbered footnotes placed at the
end of the file, and the text of agency comment letters, are provided
but may not exactly duplicate the presentation or format of the printed
version. The portable document format (PDF) file is an exact electronic
replica of the printed version. We welcome your feedback. Please E-mail
your comments regarding the contents or accessibility features of this
document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
Report to the Committee on Government Reform, House of Representatives:
March 2005:
Department of Energy:
Further Actions Are Needed to Strengthen Contract Management for Major
Projects:
[Hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-05-123]:
GAO Highlights:
Highlights of GAO-05-123, a report to the Committee on Government
Reform, House of Representatives
Why GAO Did This Study:
The Department of Energy (DOE) pays its contractors billions of dollars
each year to implement its major projects”those costing more than $400
million each. Many major projects have experienced substantial cost and
schedule overruns, largely because of contract management problems. GAO
was asked to assess, for major departmental projects, (1) DOE‘s use of
performance incentives to effectively control costs and maintain
schedules, (2) the reliability of the data DOE uses to monitor and
assess contractor performance, and (3) the reliability of the Project
Assessment and Reporting System (PARS) data that senior managers use
for project oversight.
What GAO Found:
DOE could use performance incentives more effectively for controlling
costs and schedules if it developed performance incentive guidance and
assigned responsibility for reviewing a contract‘s project management
provisions prior to award. DOE has awarded contracts for 15 of 33 major
projects that use a schedule or other performance incentive without an
associated cost incentive or constraint; thus a contractor could
receive full fees by meeting all schedule baselines while substantially
overrunning costs.
DOE has relied on unvalidated contractor data to monitor contractors‘
progress in executing major projects and to award fees for performance.
In particular, DOE‘s self-assessment of contract administration in 2002
found that field personnel overly relied on contractors‘ accounting
systems and contractor-collected data in assessing performance, without
significant validation of those data. No subsequent self-assessment has
been conducted to determine if this problem continues. Furthermore, DOE
has not required that its contracting officers receive the training
needed to assess the adequacy of contractors‘ project management
systems that generate data used to monitor progress.
Although development of PARS is a positive step, the reliability of the
project performance data that PARS provides to senior DOE managers is
limited by problems with accuracy, completeness, and timeliness.
Regarding accuracy, DOE has not assessed the reliability of
contractors‘ project management systems that feed data into PARS for 31
of 33 major projects, even though DOE believes that some systems are
deficient. Regarding completeness, GAO identified 3 major projects that
are not in PARS. As to timeliness, cost and schedule data for 6 major
projects in the June 2004 PARS report were significantly out of date
because DOE has not required contractors to submit timely performance
data.
These contract management problems limit DOE‘s ability to effectively
manage its major projects and avoid further cost and schedule
slippages.
Contractors‘ Performance Awards Relied on Unvalidated Data for 30 of
the 33 Major Projects GAO Reviewed:
[See PDF for image]
[End of figure]
What GAO Recommends:
GAO recommends that DOE strengthen its contract management for major
projects by, among other things, (1) developing a chapter in DOE‘s
Acquisition Guide that specifies a systematic contracting approach for
major projects; (2) reducing DOE‘s overreliance on unvalidated
contractor data in awarding contract fees; and (3) developing a
schedule for assessing the reliability of contractors‘ project
management systems, giving priority to systems that DOE believes are
deficient. In commenting on the draft report, DOE generally concurred
with all of the recommendations.
www.gao.gov/cgi-bin/getrpt? GAO-05-123.
To view the full product, including the scope and methodology, click on
the link above. For more information, contact Robin Nazzaro at (202)
512-3841 or nazzaror@gao.gov.
[End of section]
Contents:
Letter:
Results in Brief:
Background:
DOE Could Use Performance Incentives More Effectively for Controlling
Costs and Schedules:
DOE Has Relied on Unvalidated Contractor Data to Monitor and Assess
Contractors' Performance for Major Projects:
The Reliability of PARS Data Is Limited:
Conclusions:
Recommendations for Executive Action:
Agency Comments and Our Evaluation:
Appendixes:
Appendix I: The Department of Energy's 33 Major Projects That We
Reviewed:
Appendix II: Scope and Methodology:
Appendix III: Comments from the Department of Energy:
Related GAO Products:
Table:
Table 1: PARS Reports Assess a Project's Cost and Schedule Performance
against Only the Current DOE-Approved Baselines:
Abbreviations:
DOE: Department of Energy:
FAR: Federal Acquisition Regulation:
NNSA: National Nuclear Security Administration:
OMB: Office of Management and Budget:
PARS: Project Assessment and Reporting System:
Letter March 18, 2005:
The Honorable Tom Davis:
Chairman:
The Honorable Henry A. Waxman:
Ranking Minority Member:
Committee on Government Reform:
House of Representatives:
The Department of Energy (DOE) pays its contractors billions of dollars
each year to implement projects costing more than $400 million each,
which the department designates as major projects. These major projects
include environmental cleanup at its current and former nuclear weapons
production facilities, refurbishment of nuclear weapons, and
construction of specialized scientific facilities. Our previous reports
have found that many of DOE's major projects have experienced
substantial cost overruns and delays. As a result, since 1990, we have
designated DOE's contract management--broadly defined to include
contract administration and project management--as a high-risk area for
fraud, waste, abuse, and mismanagement because of the department's
history of inadequate management and oversight and failure to hold its
contractors accountable. In recent years, we have continued to find
problems with DOE's contract management of its major projects. (See the
list of related GAO products at the end of this report.) For example,
our 2002 assessment of DOE's contract reform initiatives found that 5
of the 16 major projects we examined had more than doubled in cost--for
billions of dollars in total cost overruns--and experienced more than 5
years in delays.[Footnote 1]
In response to these problems, DOE has instituted contracting and
project management reforms. To better align DOE functions with
challenges to improve the planning, execution, and management of its
contracts, the department established the Office of Contract Management
in 2000 with responsibility for assisting the field and other DOE
offices in the planning, implementation, and oversight of (1) the
contract award process, which involves those front-end activities
necessary to ensure that contracts are structured in a way that best
fulfills DOE's needs, and (2) the contract administration process,
which encompasses all dealings between DOE and its contractors from the
time a contract is awarded until the work has been completed and
accepted or the contract has been terminated, payment made, and any
disputes resolved. To improve the contract award process, the Office of
Contract Management has, among other things, developed updates to DOE's
Acquisition Guide, which amplifies upon the requirements contained in
the Federal Acquisition Regulation (FAR) that governs governmentwide
federal procurement activities. Each year, the Office of Contract
Management reviews a limited number of pending contract actions,
including any matters affecting a contract's deliverables, schedule,
and cost. To improve the contract administration process, the Office of
Contract Management issued, in May 2000, a Reference Book for Contract
Administrators, a consolidated reference tool for DOE's contract
administrators. The office also has performed periodic self-assessments
of contract administration practices.
As part of its project management reforms, DOE established the Office
of Engineering and Construction Management in 1999 to oversee the
project management process by developing project management policy,
improving project oversight systems, and implementing a career
development program for DOE project managers. In particular, the Office
of Engineering and Construction Management in 2001 implemented the
Project Assessment and Reporting System (PARS), a Web-based system for
collecting and analyzing current performance data for projects costing
more than $5 million. Although the department has other ways to
communicate project status information, including quarterly reports for
some projects, PARS monthly project status reports for the Deputy
Secretary of Energy (hereafter referred to as PARS reports) are DOE's
primary tool for keeping senior managers apprised of a project's
performance. In August 2003, the office initiated a certification
program for contractors' project management systems that assesses the
accuracy of the cost and schedule performance data that contractors
generate by examining, in particular, whether each contractor's system
complies with private industry's standard for earned value management-
-a systematic approach for integrating and measuring cost, schedule,
and technical (scope) accomplishments on a project.
You asked us to examine DOE's contract management actions designed to
control cost growth and schedule slippage for its major projects.
Specifically, you asked us to assess, for major departmental projects,
(1) DOE's use of performance incentives to effectively control costs
and maintain schedules, (2) the reliability of the data DOE uses to
monitor and assess contractor performance, and (3) the reliability of
the PARS data that senior managers use for project oversight. Our
review focused primarily on the 33 major projects that had passed, as
of March 2004, DOE's Critical Decision 2 milestone--the point at which
the department approves a project's cost, schedule, and scope baselines
on the basis of an approved conceptual design report and acquisition
strategy. (See app. I for the 33 major projects that we reviewed.) The
projects we reviewed include 28 projects that cost more than $400
million each and 5 projects that our 2002 assessment defined as major
projects because their total costs exceeded $100 million. DOE's Office
of Environmental Management (Environmental Management), Office of
Science, and National Nuclear Security Administration (NNSA) are
funding the 33 major projects.
To determine DOE's guidance for using different performance incentives
for major projects, we examined the contract provisions for the 33
major projects that have DOE-approved performance baselines, reviewed
various recent contract-related documents associated with these 33
projects, and discussed the contents of these documents with key DOE
contracting officials. We then compared the contract provisions with
requirements in the FAR; the DOE Acquisition Regulation; and DOE's
contract-award guidance, including the Performance-Based Contracting
Guide and the Acquisition Guide. To assess DOE's oversight of contract
administration activities, we reviewed DOE's Reference Book for
Contract Administrators, compared the findings made in DOE's various
contract administration self-assessment reports with the actions the
department has taken, and interviewed contracting officials at the
Department of Defense and administrators at the National Aeronautics
and Space Administration because DOE officials acknowledged their
leadership in implementing such project and contract management reforms
as earned value management principles. To evaluate the reliability of
PARS data, we assessed the data's accuracy, completeness, and
timeliness by examining the underlying data and the way data are
reported to senior managers. For example, we compared the information
in monthly PARS reports from January through September 2004 with
project-specific data obtained from various DOE program offices. When
we identified discrepancies, we assessed whether the discrepancies
limited the reliability of data reported in PARS. Given our review of
the documentation provided by DOE and our discussions with DOE
officials, we have reservations about the reliability of PARS data.
These issues are discussed in this report. We conducted our work
between January 2004 and January 2005 in accordance with generally
accepted government auditing standards, which included an assessment of
data reliability and internal controls. (See app. II for further
information about the scope and methodology of our review.)
Results in Brief:
DOE could use performance incentives more effectively for controlling
costs and schedules if it developed performance incentive guidance and
assigned responsibility for reviewing a contract's project management
provisions. DOE has awarded contracts for 15 of 33 major projects that
use a technical, schedule, or other performance incentive without an
associated cost incentive or cost constraint, so a contractor could
receive full fees by meeting all schedule baselines even though the
contractor substantially overran baseline costs. For 13 additional
major projects, DOE has used cost-plus-incentive-fee contracts without
certifying that contractors' project management systems generate
reliable cost and schedule data for measuring performance and awarding
incentive fees. In addition, for 11 major projects that are components
of the environmental cleanup of a DOE facility, no award fee amount is
directly linked to the successful completion of the major project,
generally because the project is part of the contractor's larger
cleanup responsibility. As a result, the contractor may not give
sufficient attention to the completion of a complex major project and
focus, instead, on less critical activities that are easier to achieve.
Furthermore, an Environmental Management review found that the award
fee incentives in its contracts were not sufficiently focused on site
cleanup activities. As a result, Environmental Management decided that,
beginning in fiscal year 2003, award fee determinations would consider
only contractor activities directly related to cleanup work, while
excluding such indirect work-related activities as providing timely and
accurate reports to DOE because they are basic expectations of any
contractor. In contrast, NNSA has not conducted a review similar to
Environmental Management's and, for at least 1 of its major projects,
NNSA has considered a contractor's indirect work-related activities in
awarding incentive fees. DOE has not issued guidance that specifies the
indirect work-related activities to be included in determining
incentive fee awards.
DOE has relied on unvalidated contractor data to monitor contractors'
progress in executing major projects and to award fees for performance.
This reliance on unvalidated data limits the department's ability to
ensure it gets what it is paying for. Specifically, DOE's self-
assessments of contract administration in 1997 and 2002 found that
field personnel overly relied on contractor accounting systems and
contractor-collected project data in awarding fees, without significant
validation of those data. However, the 2002 self-assessment made no
recommendation for fixing the problem, and no broad self-assessment of
contract administration has been conducted since 2002. Without a
specific recommendation, implementation of that recommendation, and
periodic self-assessments, DOE lacks a mechanism for ensuring that
overreliance on contractor data and other contract administration
problems are corrected. DOE has begun to certify the reliability of
contractors' project management systems that generate the performance
data used to monitor contractors' progress; however, the department has
no time table for the completion of this certification program.
Furthermore, with the exception of NNSA, DOE has not required its
contracting officers to receive training in earned value management,
even though contracting officers are required to determine whether the
contractor's project management system, after contract award, meets
private industry's earned value management standard. In contrast,
administrators at the National Aeronautics and Space Administration
told us that their contracting officers need training to adequately
assess contractors' use of earned value management.
Although the development of PARS is a positive step toward improving
DOE's project oversight, the reliability of the project performance
data that PARS provides to senior managers is limited by problems with
the data's accuracy, completeness, and timeliness.
* The accuracy of the PARS cost and schedule data is uncertain for
three reasons. First, even though DOE believes that some of its
contractors' project management systems are deficient, the department
has assessed the reliability of systems for only 2 of the 33 major
projects we reviewed. Additionally, despite the department's goal of
assessing all systems by the end of fiscal year 2006, it has not
developed a schedule for reviewing other contractors' project
management systems. As a result, senior DOE managers cannot rely on
PARS performance data for many major projects to determine whether the
projects are on cost and on schedule. For example, DOE officials told
us that they do not know if the $5.7-billion Tank Waste Treatment and
Immobilization Plant project at Hanford, Washington, is meeting its
performance baselines, even though PARS data show that the project is
on track. Second, the cost and schedule variances reported in PARS for
many projects are small and do not accurately reflect the considerable
cost growth and schedule slippage that have occurred in the past
because DOE updates the PARS baselines when project changes are
approved. DOE officials told us that departmental project management
guidance does not require the reporting of life-of-project cost and
schedule variances; however, PARS cannot reflect prior cost overruns
and schedule slippages without such reporting. Third, most of DOE's
project directors have not received earned value management training,
which is needed to ensure that contractors' project management systems
are providing accurate performance data to PARS, and DOE does not
expect to complete this training until May 2006.
* Regarding the completeness of these data, we identified 3 major
projects, as well as 2 smaller projects, that are not included in PARS.
In particular, while DOE stated in July 2003 that it intended to treat
each of its nuclear weapons refurbishment programs as projects, none of
these refurbishments have been added to the PARS database. Although DOE
recently asked its program offices to identify and enter additional
projects into PARS, implementation has been slow in part because key
program office officials lack project management training, which is
necessary for determining what activities are subject to PARS
reporting, according to Office of Engineering and Construction
Management officials. For projects that are being reported, PARS lacks
forward-looking trend data, such as the projects' estimated cost at
completion that, according to numerous officials, are critical for PARS
to effectively report possible performance challenges. Although these
data are available, they are not included in PARS reports because
Office of Engineering and Construction Management officials want to
minimize the amount of information that senior managers review.
* Regarding timeliness, the June 2004 PARS report's performance data
for 6 major projects were significantly out of date primarily because
contractors had not submitted more current performance data, according
to Office of Engineering and Construction Management officials. As a
result, the PARS monthly reports did not show senior DOE managers the
need for corrective actions in response to cost and schedule slippages.
While some minor lag is to be expected in reporting project
performance, Office of Engineering and Construction Management
officials noted they are open to exploring options to improve the
timeliness of PARS data in those cases where data is significantly out
of date.
We are making several recommendations to the Secretary of Energy to
ensure the use of effective performance incentives, strengthen
oversight of contract administration, and improve the reliability of
the project performance data provided by PARS. DOE generally concurred
with all of the recommendations. DOE also provided a number of comments
to improve the report's accuracy, which we incorporated where
appropriate.
Background:
DOE is the largest civilian contracting agency in the federal
government; about 90 percent of its annual budget is spent on contracts
for carrying out its activities and operating its facilities. In
fulfilling their missions, DOE's program offices are responsible for
contracting for and overseeing the execution of the department's major
projects, many of which are first-of-a-kind efforts and thus involve
substantial risk and may also be separate line items in DOE's budget.
For example:
* Environmental Management's mission is to accelerate risk reduction
and cleanup of the environmental legacy of the nation's nuclear weapons
program and government-sponsored nuclear energy research. Environmental
Management has used a single sitewide contract that involves several
major projects costing billions of dollars for cleaning up some of its
former facilities. In addition, Environmental Management has undertaken
many large-scale individual projects. For example, the Hanford Tank
Waste Treatment and Immobilization Plant project is an important part
of the cleanup effort at Hanford, Washington. The project, which was
initiated in December 2000, is intended to treat and prepare for
disposal 55 million gallons of high-level radioactive waste by July
2011 at an estimated cost of $5.7 billion.
* NNSA's mission is to meet national security requirements by, among
other things, maintaining and enhancing the safety, reliability, and
performance of the U.S. nuclear weapons stockpile, which includes
maintaining the capability to design, produce, and test nuclear
weapons. To fulfill this mission, NNSA undertakes such projects as
refurbishing W-80 nuclear warheads to extend their operational lives.
The W-80 refurbishment project was initiated in September 1998 and is
expected to be completed in fiscal year 2017 at an estimated cost of
about $2.45 billion.
* The Office of Science's mission is to deliver the remarkable
discoveries and scientific tools that transform our understanding of
energy and matter and advance the national, economic, and energy
security of the United States. To fulfill this mission, the Office of
Science has constructed specialized scientific research facilities,
such as the Spallation Neutron Source at the Oak Ridge National
Laboratory. This project consists of an accelerator system that
delivers short (microsecond) pulses to a target/moderator system where
neutrons are produced by a nuclear reactor process called spallation.
This project is designed to provide the next-generation spallation
neutron source for neutron scattering and related research in broad
areas of the physical, chemical, materials, biological, and medical
sciences. The Spallation Neutron Source project began in October 1998
and is expected to be completed in June 2006 at an estimated cost of
about $1.4 billion.
DOE's principal official responsible for the execution of a major
project is the federal project director, who is located at the project
site and is supported by project managers. The project director is
responsible for overseeing a project's design, execution, budgeting,
and performance. For contracts with award fee provisions, senior DOE
program office managers consult with contracting and project officers
to assess a contractor's performance and determine the appropriate
award fees.
In addition to the contract management problems our prior reports have
identified, a recent series of reports by the National Research Council
of the National Academies identified weaknesses in DOE's project
management. The council's 2004 report cited several factors that have
contributed to the slow pace of project management improvements and
resulted in inconsistent project performance.[Footnote 2] These factors
include the desire of DOE site office personnel and contractors to be
independent of oversight from DOE headquarters, insufficient support
for training, inadequate numbers of DOE project managers to oversee
contractors' performance, and the absence of a champion for project
managers and process improvement who has the authority to ensure both
adherence to policies and procedures and the availability of necessary
funding and personnel resources.
During the past year, DOE has continued to implement contracting and
project management reforms. In particular, in December 2003, the
Secretary of Energy appointed an Associate Deputy Secretary with
responsibility, among other things, for both contract and project
management, addressing a key National Research Council concern. DOE
also entered into an agreement with the Defense Contract Management
Agency, within the Department of Defense, to support the certification
of contractors' project management systems. More recently, DOE is
developing an action plan in response to the Civil Engineering Research
Foundation's assessment of departmental project management that
recommended that DOE, among other things, develop a core group of
highly qualified project directors, require peer reviews for first-of-
a-kind and technically complex projects when the projects' preliminary
baselines are approved, and enhance PARS by making the data more
timely.[Footnote 3] Furthermore, to improve its contract award process,
DOE revised its Acquisition Guide by adding chapter 16, which lists the
various contract types available and discusses their respective
advantages and constraints. To address future skill gaps in its
procurement organization, DOE established an acquisition career
development program and has certified 90 percent of its procurement
professionals as attaining mandatory training and experience standards
under this program. Within the Office of Environmental Management, a
series of contract and project management improvements have occurred
consisting of, but not limited to, providing additional training and
managing more of the cleanup work as projects. Within the Office of
Contract Management, a series of contract award and administration
initiatives have been completed. These initiatives include, among other
things, strengthening contract competition policies and practices,
improving acquisition workforce effectiveness, increasing small
business utilization throughout DOE, and strengthening DOE management
and fiscal effectiveness. For fiscal year 2005, the Office of Contract
Management has multiple initiatives planned, including identifying and
implementing follow-on actions related to the DOE management challenge
pertaining to contract competition.
DOE Could Use Performance Incentives More Effectively for Controlling
Costs and Schedules:
Because many of DOE's major projects are first-of-a-kind and thus
involve substantial risk, DOE's contracting decisions can be critical
to the successful completion of its major projects. However, DOE could
use performance incentives more effectively for controlling costs and
schedules for its major projects if the department developed criteria
for using different performance incentives and assigned responsibility
for reviewing a contract's project management provisions prior to
award. For example, DOE has used contracts that have a technical,
schedule, or other performance incentive without an associated cost
incentive or cost constraint (other than the annual funding level for
the contract). DOE also has used cost-plus-incentive-fee contracts
without certifying that contractors' project management systems
generate reliable cost and schedule data for measuring performance and
awarding fees. In addition, we found that the contract incentives for
most of the 25 major environmental restoration projects substantially
differ from the "Gold Chart" performance metrics that Environmental
Management uses to assess its performance and report its progress to
the Congress. Furthermore, for 11 major projects that are components of
the environmental cleanup of a DOE facility, Environmental Management
has not directly linked incentive fees to the successful completion of
the project, generally because the project is part of the contractor's
larger cleanup responsibility. Finally, while Environmental Management
has decided that incentive fee determinations would consider only
contractor activities directly related to cleanup work, NNSA has, for
at least 1 of its major projects, considered a contractor's indirect
work-related activities in awarding incentive fees.
DOE Has Not Fully Developed Performance Incentive Guidance to
Effectively Control Costs and Maintain Schedules:
Despite efforts in recent years to improve contract and project
management, DOE has not fully developed performance incentive guidance
to effectively control costs and maintain schedules. DOE has issued the
following guidance, order, and manual that are applicable to the
contract award process for major projects and that supplement the FAR
and the DOE Acquisition Regulation:
* In the late 1990s, DOE issued its Acquisition Guide to, among other
things, supplement the FAR and the DOE Acquisition Regulation and be a
repository of best practices found throughout the department. Chapter
16 of the guide discusses contract types; however, the chapter notes
that it was not intended to provide a template for matching a contract
type to given contracting situations. While the guide's index shows
that chapter 34 is reserved for guidance to contracting officials
related to major projects, DOE has never drafted the chapter, according
to the DOE official responsible for maintaining revisions to the
Acquisition Guide.
* In October 2000, DOE issued Order 413.3, "Program and Project
Management for the Acquisition of Capital Assets," to ensure that
capital assets, including major projects, would be delivered on
schedule, within budget, and fully capable of meeting mission needs. To
accomplish these goals, the order states, in part, that DOE officials
are to develop an acquisition plan during the acquisition process that
includes such elements as contracting options and a contractor
incentive process. The order, however, does not elaborate on the
possible contracting and performance incentive options whatsoever.
* In March 2003, DOE issued manual 413.3-1, "Project Management for the
Acquisition of Capital Assets," to improve the implementation of DOE
Order 413.3. The manual addresses various activities, including a
chapter on contracting that contains no direct reference to major
projects. The chapter states that the type of contract and incentives
proposed should be based on an overall view of the principal risks to
the project and provides a limited discussion of the types of contracts
available. For example, it states that fixed-price contracts are not
appropriate for research and development efforts or other complex
projects where there is a high degree of uncertainty in the execution
or DOE requirements. While the chapter mentions that DOE generally uses
a cost-plus-award-fee contract for contractors managing and operating
DOE sites, it does not address the other available types of contract.
Furthermore, DOE has not used its Acquisition Guide to identify best
practices, or lessons learned, based on its major project contracting
experiences. In our view, given DOE's long history with major projects,
considerable information could be added to this guide detailing those
major project contracting approaches that worked and those that did
not. Improved guidance could help DOE better control costs and maintain
schedules for its major projects.
DOE Has Not Always Reviewed the Project Management Provisions of Its
Major Contracts Prior to Award:
Neither the Office of Contract Management nor the Office of Engineering
and Construction Management always reviews the project management
provisions of major project contracts prior to award to ensure that the
performance incentives are appropriately used. At the heart of this
problem is confusion over responsibility. The Director of the Office of
Contract Management and the Director of the Office of Engineering and
Construction Management each believe that the other office has
headquarters responsibility for reviewing the project management
provisions of contracts prior to approval.
The confusion exists because the chapter in DOE's Acquisition Guide on
the headquarters review of contract and financial assistance actions is
silent on the role of the Office of Engineering and Construction
Management in the review process. This chapter indicates that packages
pertaining to contract actions will be sent to nine different DOE
offices for review, none of which is the Office of Engineering and
Construction Management. As a consequence, if this office has a role in
the contract review process, it has not been clearly defined.
According to the Director, Office of Contract Management, the Office of
Engineering and Construction Management should be responsible for
reviewing the project management provisions in major project contracts
because of its responsibility for project management matters. The
director told us that his office typically reviews from 60 to 70
pending contract actions each year, and these reviews follow a general
approach looking at any matters that might affect timing, delivery, and
cost--but no specific, formalized list is followed.
According to the Director of the Office of Engineering and Construction
Management, his office reviews certain documentation that could affect
which company is selected for a contract, but his office has no role in
reviewing the actual provisions of the contract. While the Office of
Contract Management sends contract proposals to the Office of
Engineering and Construction Management for review, the director noted
that his office has only one staff person with contracting experience.
The director believes the solution to improving the review of major
project contracts is for contracting officials within the Office of
Contract Management to become more familiar with earned value
management, a DOE contracting requirement for integrating and measuring
a contractor's performance.
Problems Have Developed Because DOE Has Not Effectively Used
Performance Incentives:
For many of the 33 major projects we reviewed, DOE has used performance
incentives that limit its ability to effectively control cost and
schedule performance. For example, almost all of DOE's cost-plus-award-
fee contracts for major projects have included a performance incentive
without also using an associated cost incentive or cost constraint
(other than the annual funding level for the contract). Also, DOE has
used cost-plus-incentive-fee contracts without certifying that
contractors' project management systems generate reliable cost and
schedule data for measuring performance and awarding fees. We also
found that (1) Environmental Management's contracts included
environmental cleanup performance incentives that differed
substantially from its new Gold Chart performance metrics; (2) DOE did
not always link its fee awards to contractors' performance on major
projects; and (3) DOE's program offices have treated indirect work-
related activities, such as providing timely and accurate reports to
DOE, differently in determining the contractors' incentive award fees.
DOE's Implementation of Performance Incentive Provisions Does Not
Enable DOE to Effectively Control Costs:
For 15 of the 17 major projects that use a cost-plus-award-fee
contract, the contract contained a technical, schedule, or other
performance incentive without including an associated cost incentive or
cost constraint (other than the annual funding level for the contract).
Under such circumstances, the potential exists that a contractor could
meet all incentives and overrun baseline costs but still receive full
fees. The other 2 major projects used a cost-plus-award-fee contract
that included an associated cost incentive or cost constraint for each
technical, schedule, or other performance incentive.
The FAR, the DOE Acquisition Regulation, and DOE guidance preclude the
inclusion of a schedule or other performance incentive without also
including a cost incentive or cost constraint. FAR § 16.402-1 states
that no incentive contract may provide for other incentives without
also providing for a cost incentive or cost constraint. Similarly, DOE
Acquisition Regulation § 970.5215-3 provides that requirements
incentivized by other than cost incentives must be performed within
their specified cost constraint. DOE's Performance-Based Contracting
Guide, dated October 2003, states that (1) cost incentives should be
included if other incentives are included because a schedule or other
performance incentive may result in the contractor paying little
attention to the cost of achieving those incentives unless cost is also
a consideration and (2) DOE contracts, in developing incentives and
incentive programs, must comply with the incentive contract provisions
of the FAR and the DOE Acquisition Regulation.
The Director of the Office of Contract Management told us that to
implement the FAR requirement to include a cost incentive or cost
constraint whenever a noncost incentive is in the contract, each
noncost incentive does not necessarily need an associated cost
constraint dedicated to that noncost incentive. According to the
director, a single cost constraint, which could be equivalent to the
project's annual funding level, would fulfill the FAR
requirement.[Footnote 4]
However, DOE contracting officials at Oak Ridge, West Valley, and
Savannah River believe that to implement the FAR and DOE Acquisition
Regulation requirements in a way that effectively controls costs, a
contract with a technical, schedule, or other noncost incentive should
also have an associated cost incentive to function as a constraint on
the expenditure of funds. One of these officials added that as the
noncost incentives become more objective and measurable, the cost
constraint should be more clearly defined in relation to each noncost
incentive. Similarly, another one of these officials told us that using
the annual funding level or the project's cost baseline as the
constraint is too vague and unworkable, and that some funding levels
and cost baselines do not track down to the performance incentive
level. As a result, neither the funding level nor the cost baseline
would indicate whether the performance incentive was accomplished
within the cost constraint.
These views are consistent with the findings from DOE's 1997 assessment
of performance-based incentives, which found that DOE's and
contractors' financial systems generally are budget-based and do not
segregate and track costs at the performance incentive level. The
assessment added that this limits DOE's ability to establish meaningful
cost baselines and to monitor the cost of performance under specific
incentivized work efforts in relation to the total cost of the
contract.
DOE Has Used Cost-Plus-Incentive-Fee Contracts without Certifying the
Reliability of Contractors' Performance Data:
For 13 of the 33 major projects we reviewed, DOE used a cost-plus-
incentive-fee contract that provides the contractor with an initially
negotiated fee that is subsequently adjusted by a formula based on the
relationship of total allowable costs to total target costs. The
formula provides, within limits, for fee increases when total allowable
costs are less than target costs. In recent years, DOE has made a major
effort to move toward the use of cost-plus-incentive-fee contracts.
Because a cost-plus-incentive-fee contract provides higher fee awards
to the extent that actual costs are lower than anticipated, it depends
upon reliable cost estimating at the outset in the form of a target
cost and reliable cost reporting later. In July 1997, the Office of
Management and Budget (OMB) issued requirements regarding the
acceptability of contractors' project management systems. However, DOE
has not certified the reliability of contractors' project management
systems that generate the target cost data for the 13 major
projects.[Footnote 5] As a result, a contractor might receive a high
fee payment because its project management system generated an
unreliable high initial cost estimate and subsequently reported lower
actual costs. A U.S. Army Corps of Engineers' report, issued in May
2004, concluded that it was not appropriate to use a cost-plus-
incentive-fee contract for the Hanford Tank Waste Treatment and
Immobilization Plant project, in part because reliable cost data could
not be generated in advance.[Footnote 6]
Furthermore, DOE site personnel may not provide adequate surveillance
of the contractors' cost records for these 13 projects. According to
DOE's Performance-Based Contracting Guide, it is inappropriate to use a
cost-plus-incentive-fee contract if there is an overreliance on
contractor accounting systems and contractor-collected data without
significant validation of those data.[Footnote 7] In such situations,
the guide states, any potential cost savings reported might be the
result of a poor estimate of the amount of labor or material required,
the approach planned, or the associated costs. The Office of Contract
Management's self-assessment of contract administration in 2002 found
that most of the DOE field locations visited relied almost exclusively
on the contractors' data because they did not have the staff resources
capable of validating cost or technical baselines. The report, however,
did not identify the DOE field locations visited, and, according to an
Office of Contract Management official, no individual field location
reports were prepared.
Contracts' Performance Incentives Differed from the Gold Chart
Performance Metrics:
For 16 of the 25 major environmental restoration projects that we
reviewed, the contracts' performance incentives differed substantially
from the Gold Chart performance metrics that Environmental Management
uses to assess its performance and report its progress to the
Congress.[Footnote 8] Environmental Management developed the Gold Chart
performance metrics in October 2002 as a basis for clearly and
objectively showing the progress being made in the environmental
cleanup program. We found, however, that these Gold Chart metrics were
not being used to measure contractors' performance or award fees.
Instead, DOE measures performance and awards fees on the basis of
information from the contractors' project management systems, which DOE
has not yet certified as capable of producing reliable information.
For 4 projects at the Fernald Closure Site in Ohio, a lower performance
fee might have been appropriate if the Gold Chart metric had been used.
For fiscal year 2003, DOE awarded the contractor about $7.7 million of
the $8 million in available fee, or 97 percent, on the basis of
acceptable cost and schedule performance toward closure of the entire
site during fiscal year 2003. However, according to the fiscal year
2003 Gold Chart metrics, the goal for the Fernald Closure Project was
to accomplish four radioactive facility completions and dispose of
2,568 cubic meters of radioactive waste.[Footnote 9] According to
Environmental Management information, the contractor did not fully
complete one of these tasks. Because the contractor accomplished only
three of the four radioactive facility completions, Environmental
Management might have given a different fee amount if the two Gold
Chart metrics had been used to determine award fee.
Conversely, a different fee amount might have been warranted for the
Solid Waste Stabilization and Disposition project at Hanford,
Washington. For fiscal year 2003, DOE awarded the contractor about $2.2
million of about $3 million in available fee, or 73 percent, on the
basis of the contractor's disposal of radioactive waste in accordance
with an approved schedule that DOE determined the contractor had met.
In contrast, Environmental Management data for fiscal year 2003, using
Gold Chart metrics, show that the contractor actually disposed of 3,634
cubic meters of waste as compared with a goal of disposing 2,320 cubic
meters of waste, or about 157 percent of the work intended. If the Gold
Chart metrics had been used to determine the award fee, the contractor
might have received a different fee amount.
For the Spent Nuclear Fuels project, at Hanford, Washington, the Gold
Chart metric and the contract's performance incentive were so
dissimilar that it was difficult to determine how to gauge the
contractor's performance. For fiscal year 2003, DOE awarded the
contractor about $2.8 million of about $3.3 million in available fee,
or 85 percent, on the basis of the contractor's removing 777 metric
tons, or 87 percent, of the 890 metric tons that had been planned.
However, Environmental Management data for fiscal year 2003, using the
Gold Chart metrics, show the contractor removed 805 units, or 94
percent, of the goal's 855 units. Because the Gold Chart metric and the
contract's performance incentive were so dissimilar, we could not
reconcile the information.
Environmental Management officials told us that the performance
incentives contained in environmental cleanup contracts and the Gold
Chart metrics should be aligned. In commenting on the draft report,
Environmental Management officials stated that the new Savannah River
cleanup contract incorporates Gold Chart metrics. They added, however,
that the contract renewals for the Oak Ridge, Fernald, and Rocky Flats
facilities do not contain the Gold Chart metrics because each is a cost-
plus-incentive-fee contract that awards fee based on the final closure
costs and date for the site. It is unclear whether these cost-plus-
incentive-fee contracts will more effectively track contractors'
performance because they rely on contractors' project management
systems that DOE has yet to certify. In contrast, the Gold Chart
metrics assess the accomplishment of discrete amounts of work that is
verifiable.
Incentive Fees Paid to Contractors Were Not Directly Tied to
Performance for Some Major Projects:
In 1996, we reported that a key factor inhibiting the successful
completion of DOE's major projects was the lack of effective
incentives.[Footnote 10] To the extent that incentives are properly
applied, they can help achieve agency goals. On the other hand, if
incentives are nonexistent or not effectively applied, a project may
not be successfully completed.
Sixteen of the 33 major projects we reviewed had no incentive fees
directly associated with the successful completion of work. Nine of
these 16 projects involve closure work at the Fernald and Rocky Flats
sites, where the payment of incentive fees is based on an overall
average of the cost and schedule status for all site closure
activities, including major projects and other site activities.
Environmental Management officials told us that rather than awarding
incentive fees specifically for completing any of the 9 major projects,
or for other key interim milestones, the Fernald and Rocky Flats
contracts award provisional incentive fees for meeting or exceeding
overall targets for a fiscal year, provided the contractors
successfully achieve site closure on schedule.
However, it remains to be seen whether this approach will be effective
in completing major projects on time and within cost. For example,
although a major project at the Fernald site that we reviewed was
experiencing cost growth to the point where it was expected to exceed
its cost baseline--the total cost estimate to accomplish the project--
DOE considered the overall average of the cost and schedule status for
all site activities at Fernald to be acceptable and paid the contractor
provisional incentive fees for fiscal year 2003. Similarly, a major
project at the Rocky Flats site had overrun its estimated cost by about
$42 million through fiscal year 2003. However, this overrun was offset
by an underrun of about $46 million in activities such as general
counsel work and planning and integration that, according to DOE
information, had historically been understaffed. The net effect was
that DOE paid the contractor provisional incentive fees because the
contractor's overall cost and schedule status for fiscal year 2003 was
considered to be acceptable.
DOE Program Offices Have Treated Indirect Work-Related Activities
Differently in Awarding Incentive Fees:
In addition to these other contracting problems, we found that DOE
program offices treated indirect work-related activities differently in
awarding incentive fees. In late 2002, Environmental Management decided
that award fee determinations will consider only contractor activities
directly related to cleanup work, while excluding such indirect work-
related activities as providing timely and accurate reports to DOE,
providing support services to the government, and complying with the
contract because these activities are basic expectations of any
contractor. Environmental Management made this determination after its
review of contractors' authorized fee incentives identified numerous
examples of incentive fee payments for indirect work-related
activities. The review also found that Environmental Management was
paying some contractors additional fees for performing work safely that
the review concluded was a basic expectation, and not exceptional
performance worthy of additional fee.
NNSA has not conducted a review similar to Environmental Management's
assessing what, if any, indirect work-related activities are worthy of
incentive payments. The contractor for one NNSA major project received
incentive fee payments for providing timely and accurate reports to DOE
and other indirect work-related activities during fiscal year 2003.
Discrepancies in the treatment of various indirect work-related
activities have occurred because DOE's guidance does not address the
appropriateness of including a contractor's performance of indirect
work-related activities in determining incentive fee awards.
In commenting on the draft report, Environmental Management expressed
concern that it would be virtually impossible to develop meaningful
guidance that could be applied universally to DOE's diverse programs.
We disagree. We believe that all DOE programs should use incentive fees
to reward contractors for achieving work-related activities, as opposed
to such indirect activities as providing the DOE programs with timely
reports.
DOE Has Relied on Unvalidated Contractor Data to Monitor and Assess
Contractors' Performance for Major Projects:
Because most of DOE's operations are carried out through contracts,
contract administration is a significant part of DOE's work. DOE has
relied on unvalidated contractor data to monitor contractors' progress
in executing major projects and awarding fees for performance.[Footnote
11] This reliance on unvalidated data limits the department's ability
to ensure it gets what it is paying for. Specifically, DOE's self-
assessments of its contract administration in 1997 and 2002 both found
that field personnel overly relied on contractor accounting systems and
contractor-collected project data without significant validation of
these data. However, unlike the 1997 self-assessment, the one in 2002
made no recommendation to fix this problem, and no subsequent self-
assessment has been initiated to determine if the problem has
continued. DOE has begun to certify the reliability of contractors'
project management systems that generate the performance data used to
monitor contractors' progress; however, the department has no time
table for the completion of this certification program. In addition,
DOE has not required its contracting officers and contracting officer
representatives to receive training in earned value management--a
systematic approach for integrating and measuring cost, schedule, and
technical (scope) accomplishments on a project or task--even though
these officials are required to determine whether contractors' project
management systems meet the private industry's earned value management
standard.
DOE Has Not Used Self-assessments to Correct an Overreliance on
Contractor Data:
Self-assessment is an important tool for evaluating organizational
effectiveness. By taking a comprehensive look at itself, an
organization can identify weaknesses and plot a course of corrective
action. DOE performed comprehensive self-assessments of its contract
administration practices in 1997, 1999, and 2002.
In 1997, DOE assessed 20 contracts to ensure that financial incentives
contained in those contracts were rational, linked to well-defined
performance objectives and measures, and properly administered. The
self-assessment reported both positive and negative findings. For
example, it found that the use of performance-based objectives
generally had been effective in directing contractors' management
attention to desired performance outcomes. However, it also found that
field personnel overly relied on contractor accounting systems and
contractor-collected data without significant validation of these data,
and that DOE's approval of fees earned by the contractors relied upon
contractor-generated documents. To correct this deficiency, the self-
assessment recommended (1) that the cognizant DOE heads of contracting
at each field location, as part of their overall contract
administration plan, identify the mechanisms, responsibilities, and
authorities for ensuring that contractor performance against
established objectives is appropriately monitored and (2) that
performance achievements are verified.
In 1999, DOE's follow-up assessment of the effectiveness of the actions
taken in response to the 1997 self-assessment found that the
recommendation that contractor performance be monitored and
achievements verified had been implemented. Specifically, field offices
reported that their plans for administering contracts had been
appropriately modified and instituted. In addition, the follow-up
assessment stated that (1) early results indicated a substantial
improvement in the way incentives were being managed from DOE
headquarters and administered at DOE field contracting offices and (2)
anecdotal evidence suggested that contractor performance had improved.
In 2002, the Contract Administration Division again performed a self-
assessment that examined, in part, how contract administration planning
and execution was conducted at various DOE field locations. The
findings and conclusions of this review were somewhat inconsistent with
those of the 1999 follow-up assessment. The 2002 review, like the 1997
assessment, determined that few sites had the resources capable of
validating contractor cost or technical information and most sites must
rely almost exclusively on the contractor's data. The review noted, in
one instance, that financial data provided by the contractor were
generally accepted by DOE, not on the basis of reasonableness and
allowability, but on the basis of the contractor's "acceptable" self-
assessment of the procedures used to collect those data. However,
unlike the 1997 assessment, the 2002 review contained no specific
recommendation to correct this overreliance on contractor data.
According to the Director of DOE's Contract Administration Division,
because of funding constraints and other factors, no broad self-
assessment of contractor administration has been done since 2002. The
director added that DOE now conducts individual site assessments as
necessary rather than conducting more comprehensive assessments.
According to information provided to us in April 2004, the last
individual site assessment was made in August 2003 and documented in
December 2003. This site assessment identified problems similar to
those reported in the 2002 self-assessment. Specifically, the site
assessment noted that, with respect to one contract reviewed, there was
no evidence of effective cost controls and/or contract management. The
site assessment contained no formal recommendation to fix this problem.
On the other hand, the site assessment contained a recommendation to
address the high rate of expenditure on this contract over the
remaining 2-year-option period. The assessment recommended that the DOE
site office review the scope and cost of its current task orders for
prioritization and inclusion in the remaining option term.
In August 2003, DOE began to certify the reliability of contractors'
project management systems that generate the performance data used to
monitor contractors' progress. However, as of December 2004, the
department has assessed and certified project management systems for
only 2 of the 33 major projects we reviewed and does not have a time
table for completing this certification program.
In commenting on the draft report, DOE noted that both Environmental
Management and the Office of Engineering and Construction Management
have been validating contractors' cost and schedule performance
baselines for several years. In our view, DOE validation of contractor
baselines will not fully address the problems that have been
identified. Validating baselines is just the first step in performing
adequate contractor oversight. After baselines have been validated, DOE
must not overly rely on contractor accounting systems in reporting
costs and on contractor-collected project data in awarding fees. That
is the message from two DOE self-assessments of performance-based
contracting. With respect to DOE's experience in baseline validation,
the Civil Engineering Research Foundation's July 2004 report for the
Office of Engineering and Construction Management found that some
improvements in baseline validation were needed. This report noted that
many of the DOE projects it reviewed were formulated with inadequate
baseline estimates. In addition, the report stated that periodic
baseline changes were occurring that masked the true status of certain
projects. The report recommended that DOE develop guidelines that
appropriately control the rebaselining of projects.
DOE further stated that the promulgation of contract management
planning guidance and the requirement for a contract management plan
addressed many of the issues that the 2002 self-assessment identified.
However, in our view, until a subsequent assessment is done, it remains
unclear whether this DOE action has adequately resolved the issues
identified in the 2002 self-assessment. For fiscal year 2005, DOE is
planning to examine the contract management plans and contractors'
purchasing systems.
DOE Has Not Required Its Contracting Officers and Contracting Officer
Representatives to Receive Earned Value Management Training:
During the early 1990s, OMB issued several reports on civilian
agencies' contract administration practices that found that agencies
frequently experienced cost overruns and delays in receiving goods and
services because their contracting officials allocated more time to
awarding contracts than to administering existing ones. In response,
OMB revised its Circular A-11 to require that federal agencies assess
and certify contractors' project management systems for proper use of
earned value management principles. OMB also identified several other
deficiencies, including a lack of proper training for agency officials
performing contract oversight. According to administrators at the
National Aeronautics and Space Administration, earned value management
training is essential for their contracting officers to adequately
assess whether a contractor's project management system complies with
the private industry's standard.
We found that, with the exception of NNSA, DOE has not required its
contracting officers or contracting officer representatives to receive
earned value management training, even though they are responsible for
determining whether the contractor's project management system complies
with the private industry's earned value management standard after the
contract is awarded. The following three DOE documents contain the
contracting officer's responsibilities, the standards against which
those responsibilities are to be discharged, and the training
requirements for contracting officers:
* Chapter 1 of DOE's Reference Book for Contract Administrators, issued
in 2000 and in effect through October 2004, outlines the contracting
officers' many responsibilities, including a review of the adequacy of
the contractor's project management system.[Footnote 12] The reference
book states that the system's adequacy must be confirmed by the
contracting officer with the support of other DOE headquarters and
field office personnel, as appropriate. The reference book also
indicates that corrective action plans resulting from DOE reviews of
contractor project management systems are to be tracked until the DOE
contracting officer confirms that all open issues are closed.
* DOE Order 413.3, "Program and Project Management for the Acquisition
of Capital Assets," also issued in 2000, specifies that contractors'
project management systems must comply with the American National
Standards Institute's standard on earned value management. The order
states that this requirement applies only to systems involved in
controlling the performance of projects costing more than $20 million
in total. The order also requires that contractors' systems provide
cost and schedule performance, milestone status, and financial status
to DOE on a monthly basis.
* DOE Order 361.1A, "Acquisition Career Development Program," issued in
April 2004, outlines the training and certification requirements for
DOE contracting officers and contracting officer representatives. The
order identifies a training curriculum for these officers by functional
area--including, among others, procurement contracts; interagency
agreements and sales contracts; grants and cooperative agreements;
loans and loan guarantees; and the government purchase card. The order,
however, does not require either the contracting officer or the
contracting officer representative to receive earned value management
training.
The Director of the Contract Administration Division corroborated our
assessment of DOE's order for acquisition career development. The
director noted that the only reference to earned value management
training in DOE Order 361.1A requires that DOE project directors, not
contracting officers, complete a course on earned value management
systems. Without this training, however, it is unclear how DOE
contracting officers and contracting officer representatives can
fulfill their responsibilities and properly assess the adequacy of the
project management systems of departmental contractors. In providing us
with exit conference comments, DOE Office of Contract Management
officials acknowledged that contracting officers do have a
responsibility in the area of earned value management and will be
receiving training on that subject in the future. Subsequently, in
December 2004, DOE provided contracting professionals at DOE
headquarters with a 1-hour course on earned value management. DOE said
that this training session, which was video recorded, is being required
nationwide for all DOE contracting officials. As opposed to this 1-hour
course, we noted that NNSA requires its contracting officials to
participate in a 48-hour course on the fundamentals of earned value
management.[Footnote 13]
The Reliability of PARS Data Is Limited:
The reliability of the project cost and schedule data that PARS
provides to senior DOE managers is limited by problems with the data's
accuracy, completeness, and timeliness. In general, the accuracy of
PARS report data is uncertain because DOE (1) has assessed the
reliability of contractors' project management systems for only 2 of
the 33 major projects we reviewed, (2) generally measures projects'
cost and schedule performance in PARS against the current DOE-approved
cost and schedule baselines without also tracking performance against
the original targets, and (3) has not provided most of its major
project directors with the training needed to ensure contractors are
generating accurate performance data. PARS report data are not complete
because DOE program offices have not submitted performance data to PARS
for 3 major projects, as well as at least 2 smaller projects, and PARS
reports do not provide each project's estimated cost at completion or
other helpful, forward-looking data. In addition, the Office of
Engineering and Construction Management stated that the June 2004 PARS
report's performance data for 6 major projects and 5 smaller projects
were significantly out of date, primarily because contractors did not
provide updated project performance information. Senior managers have
used PARS data to take actions that averted cost increases for certain
projects that were experiencing cost or schedule challenges. Without
reliable data, however, PARS has not provided senior managers with
information about cost increases and schedule slippages for many
projects, and the status of many other projects is uncertain.
The Accuracy of Most PARS Data Is Uncertain:
Three factors impair the accuracy of cost and schedule data reported in
PARS. First, DOE officials told us they have little assurance that the
cost and schedule data for most projects in PARS are accurate because
DOE has not assessed the reliability of contractors' project management
systems that generate such data for data reliability, particularly
those systems believed to be using incorrect methods. Second, for
almost all projects, PARS reports compare cost and schedule performance
against DOE's current baselines, without identifying the extent of cost
or schedule slippages that previously occurred. Third, most DOE project
directors lack the necessary training to evaluate and verify the
accuracy of the performance data that contractors generate, according
to DOE officials.
DOE Has Assessed Few Contractors' Systems That Generate Project Data:
OMB Circular A-11 and DOE Order 413.3 require that DOE assess and
certify contractors' project management systems for proper use of
earned value management principles in generating cost and schedule
performance data before the department approves a project's cost and
schedule baseline at its Critical Decision 2 milestone. Earned value
management, when used correctly, produces data that reflect a
contractor's progress toward completing a project within cost and
schedule targets. In essence, earned value management measures the
value of work completed against the cost and schedule of work planned,
as opposed to comparing actual with planned expenditures. To
illustrate, assume a contract calls for 4 miles of railroad track to be
laid in 4 weeks at a cost of $4 million. After 3 weeks of work, assume
$2 million has been spent. By analyzing planned versus actual
expenditures, it appears the project is underrunning the estimated
costs. However, an earned value analysis reveals that the project is in
trouble because even though only $2 million has been spent, only 1 mile
of track has been laid; thus, the contract is only 25 percent complete.
On the basis of the value of work done, the project will cost $8
million ($2 million to complete each mile of track), and the 4 miles of
track will take a total of 12 weeks (3 weeks for each mile of track) to
complete instead of the originally estimated 4 weeks.
To ensure correct application of earned value management principles,
contractors must develop budgets and schedules based on measurable
components of a project, which include defined start points, end
points, and scopes of work. In addition, contractors must calculate the
value of work performed against the budgets and schedules for the
measurable project components. Experts in earned value management told
us that without defined start and end points and other measurable
project components, project performance data give little insight as to
whether cost and schedule performance are on track, and the data might
mask more serious problems.
DOE's Office of Engineering and Construction Management and the Defense
Contract Management Agency assess whether a given contractor's project
management system properly uses earned value management principles by
examining whether the contractor's system complies with the industry
standards and verifying that the contractor is using the system to
manage the project.[Footnote 14] Once a contractor has fully addressed
the concerns identified by the assessment, DOE is to certify the
project management system, attesting that project performance data--
data that convey progress toward the approved cost and schedule
targets--are generated reliably. Assessment and certification of
contractors' earned value management systems are critical components of
DOE's management of its performance-based contracting, according to DOE
earned value management training documents. While only three systems
have been assessed since August 2003, Office of Engineering and
Construction Management officials told us that they and the Defense
Contract Management Agency, working together, could assess the project
management systems for about 10 contractors in a given year now that
they are becoming more familiar with the process.
In August 2003, the Office of Engineering and Construction Management
and the Defense Contract Management Agency began the process of
assessing contractors' project management systems as a basis for
certifying that they properly use earned value management principles.
In September 2004, DOE certified Sandia National Laboratories' project
management system for 1 major project, the Microsystems and Engineering
Sciences Applications project, and 6 smaller projects. DOE also plans
to certify Oak Ridge National Laboratory's project management system
for another major project, the Spallation Neutron Source, once minor
deficiencies are corrected.[Footnote 15] Overall, however, DOE has
assessed project management systems for only 2 of the 33 major projects
we reviewed--and 8 of the 73 projects in PARS--that have passed
Critical Decision 2 with DOE-approved cost and schedule baselines. (The
remaining 65 projects in PARS whose systems have not been assessed have
baseline costs of nearly $75 billion.) According to an Office of
Engineering and Construction Management official, the first three
contractors' systems were selected for assessment on the basis of
visibility, significance, and criticality to the Department's success,
but also because cognizant DOE officials were confident that the
contractors' project management systems would meet certification
criteria.
The National Research Council's 2004 report on DOE's project management
found that the quality of earned value management across the
department's projects was inconsistent and stated that senior DOE
managers do not know whether the reported data on cost and schedule
performance are accurate unless contractors' systems are assessed and
certified. Because DOE has only recently begun to assess contractors'
project management systems that feed data into PARS, DOE officials
acknowledged to us that they lack assurance regarding the accuracy of
PARS performance data, adding that they believe some of the project
management systems not yet assessed have important deficiencies. For
example, a DOE expert in earned value management noted that contractors
for most Environmental Management projects--about half of the projects
in PARS that have passed Critical Decision 2--have not properly
implemented earned value management principles because, among other
things, many of the projects' components lack defined start and end
points. For example, the earned value management expert believes, on
the basis of his assessment of work breakdown structures and other
project components, that the contractor's project management system for
the $10-billion Yucca Mountain Nuclear Waste Repository project does
not properly use earned value management principles and generates
performance data that cannot be regarded as accurate. Consequently,
senior DOE managers have no assurance that cost and schedule targets
will be met, even if the data suggest they will.
Similarly, for several major projects we examined, the contractors'
project management systems do not seem to properly implement earned
value management principles to measure cost and schedule performance.
For example, the $2-billion East Tennessee Technology Park project at
Oak Ridge lacks measurable project components. In some instances, work
is categorized into activities such as "general operations" and
"contractor operations" that have no apparent defined start and end
points. According to the expert in earned value management, the
categories of work for this project make it difficult to accurately
measure project performance because there is no clear activity or time
frame against which to measure costs incurred or time spent. Instead,
PARS data for this project seem to measure only the project's
expenditures, which can conceal information on the project's cost and
schedule status and progress toward completion. In addition, the $5.7-
billion Tank Waste Treatment and Immobilization Plant at Hanford,
Washington, lacks discrete, measurable project components because work
is categorized into activities such as "providing technology" and
"providing infrastructure" that lack defined start and end points.
While we recognize that it is appropriate, according to industry
standards, to categorize a small amount of work in this fashion, DOE
project management officials said the particular categories of work in
these instances reflected a poor comprehension of earned value
management and limited their confidence in the assessment of project
performance.
Two Office of Engineering and Construction Management officials
acknowledged that the accuracy of data for these projects is uncertain
because DOE has not assessed whether the contractors' project
management systems properly applied earned value management principles.
One of these officials suggested that the contractors' project
management systems for such projects should be assessed as soon as
possible to correct deficiencies and improve the reliability of project
performance data provided to senior managers to oversee progress toward
cost and schedule targets. The Director of the Office of Engineering
and Construction Management agreed that DOE should develop a schedule
that would give priority to assessing these and other high-risk and
high-cost systems. As of January 2005, a schedule had not been
developed, but the director told us that he was in the process of doing
so.
PARS Reports Generally Do Not Show Total Cost Overruns and Schedule
Slippages:
The accuracy of the PARS report data is further impaired because PARS
reports generally do not show total cost overruns and schedule
slippages, even though DOE requires each project team to estimate life-
cycle costs and assess project performance against established cost and
schedule baselines. Instead, a project's DOE project director updates
the cost and schedule baselines in PARS when DOE approves a contract
modification. As a result, PARS reports show relatively small variances
between a project's actual performance and its approved baselines, so
that many of the projects we reviewed appear not to have experienced
problems when, in fact, they did. For almost all projects, PARS reports
do not provide data that would enable senior DOE managers to assess (1)
a contractor's performance against the project's original DOE-approved
baselines to identify total cost overruns and schedule slippages or (2)
the effect of any DOE initiatives to control a project's costs. The
Civil Engineering Research Foundation's July 2004 report similarly
found that PARS cost and schedule data often do not convey the actual
status of projects since their inception because of periodic revisions
of cost or schedule baselines. Furthermore, for most Environmental
Management projects, PARS measures project performance from arbitrary
dates, such as the beginning of the fiscal year, which do not
necessarily correspond to progress toward DOE-approved targets. The
following examples illustrate how PARS has masked problems with
projects by giving an incomplete picture of project costs or project
performance:
* The January 2004 PARS report showed that the $1.6-billion Spent
Nuclear Fuels Stabilization and Disposition project at Hanford,
Washington, was on track to meet cost and schedule performance
targets.[Footnote 16] However, by April, total costs for the project
increased by nearly $150 million. DOE officials acknowledged that
because the January 2004 PARS report to senior DOE managers measured
only project performance from the beginning of the fiscal year, instead
of against the DOE-approved baselines, the PARS report concealed longer
term problems that threatened the project's completion within costs.
* In October 2002, the Tritium Extraction Facility at Savannah River,
South Carolina, had an approved total cost of about $400 million. Costs
for the project increased more than $100 million by September 2003, and
subsequent PARS reports showed that costs were on track to meet cost
targets, despite the 25 percent increase in the project's costs.
* In June 2004, Environmental Management restructured the PARS
reporting for 4 projects at Oak Ridge, Tennessee, by combining their
respective costs and schedules with those of other Oak Ridge
projects.[Footnote 17] As a result, Environmental Management stopped
reporting project performance data for each project, masking the fact
that 2 of them, totaling about $300 million, were significantly behind
schedule. Two Office of Engineering and Construction Management
officials believe the projects should be reported separately because
combining projects' respective cost and schedule data can inhibit the
correct use of earned value management.
* The April 2004 PARS report showed that the total cost of the Soil and
Water Remediation project at Ashtabula, Ohio, would be $45 million,
although the performance data indicated the project would not likely
meet its baselines. However, this amount does not include about $109
million in expenditures on this project by October 1, 2003.
Environmental Management reports this project's total costs to be about
$157 million--more than three times the amount reported in PARS.
* PARS reports that total project costs for the Nuclear Facility
Deactivation and Decommissioning project at Columbus, Ohio, will be
about $31.5 million. However, this amount does not include about $106
million in expenditures prior to 2004. Environmental Management
estimates that this project's total cost will exceed $163 million--more
than five times the amount reported in PARS.
The June 2004 PARS report showed that 90 percent of the 63 projects
with approved baselines were expected to meet their cost and schedule
baselines.[Footnote 18] However, this percentage may reflect project
managers' efforts to keep the projects' baselines up to date rather
than improvements in project management performance because PARS
generally measures projects' performance against the most current DOE-
approved baselines. For example, as shown in table 1, the October 2002
PARS report's assessment of 2 major projects was red because both
projects were expected to breach their cost/schedule performance
baselines. However, the September 2003 PARS report's assessment of
these major projects was green because total project costs were within
the revised baseline that DOE had subsequently approved. The September
2003 PARS report did not indicate the extent to which each project's
total costs had exceeded the costs that DOE approved at Critical
Decision 2 on the basis of an approved conceptual design report and
acquisition strategy.
Table 1: PARS Reports Assess a Project's Cost and Schedule Performance
against Only the Current DOE-Approved Baselines:
Dollars in millions:
Project name: Waste Treatment and Immobilization Plant Project
(Hanford):
October 2002 PARS report;
Total cost: $4,746.9;
Overall assessment: Red[A];
PARS comments: Proposed baseline approval is scheduled. Unable to
validate baseline;
cannot recommend approval.
September 2003 PARS report;
Total cost: $5,781.0;
Overall assessment: Green[B];
PARS comments: No comment.
Project name: Tritium Extraction Facility Project (Savannah River).
October 2002 PARS report;
Total cost: $401.0;
Overall assessment: Red[A];
PARS comments: This project has a baseline breach. Baseline Change
Proposal is being prepared for Secretarial approval.
September 2003 PARS report;
Total cost: $506.4;
Overall assessment: Green[B];
PARS comments: No comment.
Source: GAO compilation of data for selected projects from fiscal year
2003 PARS reports.
[A] Red indicates that a project is expected to breach its
cost/schedule performance baseline.
[B] Green indicates that a project is expected to meet its
cost/schedule performance baseline.
[End of table]
In addition to these projects, the 90 percent figure includes many
Environmental Management projects, whose performance is measured over
time frames that do not necessarily reflect performance against DOE-
approved baselines. Further, the 90 percent figure does not reflect the
4 Oak Ridge projects whose performance data showed imminent performance
problems before being combined with the performance data of other
projects at the site.
DOE officials told us that the monthly PARS reports are the primary
tool for communicating project performance information to senior
management. However, for many projects--particularly those overseen by
Environmental Management--PARS does not report projects' life-cycle
costs or performance against original baselines, even though DOE
requires each project team to estimate life-cycle costs and assess
project performance against established cost baselines and schedule
milestones. Office of Engineering and Construction Management officials
acknowledged that reporting life-cycle costs and project performance
against original cost and schedule baselines in PARS would make cost or
schedule challenges easier to identify, and Environmental Management
officials told us they plan to report life-cycle costs and project
performance against original baselines in PARS reports beginning by
December 2004.
In addition to Environmental Management's plans for PARS reporting, the
Office of Engineering and Construction Management intends to make
several upgrades to the PARS database, such as making the process for
entering monthly data more efficient and easier for users to understand
and ensuring that the correct data are being entered. Office of
Engineering and Construction Management officials reported that they
are in the process of implementing these improvements. However, these
upgrades do not address the limitations to reporting accurate data that
we identified. Furthermore, these improvements do not address
limitations in the reliability of data stemming from contractor's
project management systems that have not been assessed or data that
have not been reviewed.
Most DOE Project Directors Lack Certification in Earned Value
Management:
Project directors are DOE's focal point for assessing the contractors'
cost and schedule performance data that feed into PARS. However, most
of DOE's project directors have not been certified in earned value
management, further reducing assurances that PARS data are accurate.
Because DOE believes that it is critical for project directors to
understand earned value management, the department informally
designates its project directors as "acting directors" if they have not
completed the project manager career development program, which
includes training in earned value management. Office of Engineering and
Construction Management officials told us that while some acting
project directors are proficient in earned value management and capable
of evaluating the reliability of contractor-generated data, other
acting project directors are not.
DOE recently implemented the project management career development
program through which project directors are being trained in, among
other things, earned value management. However, DOE had trained only
about 25 percent of them through this program as of July 2004, with
plans to train the remaining 75 percent by May 2006. A DOE official
told us that the appropriate level of earned value management training
for acting project directors depends on their experience in using
earned value management. While DOE aims to assess project directors'
capabilities in earned value management to ensure that they are
competent, validating the adequacy of prior earned value management
experience for acting project directors has been time consuming. The
lack of trained projects directors reviewing the accuracy of a
project's performance data may, in some cases, adversely affect the
ability of senior DOE managers to properly assess the status of major
projects.
PARS Lacks Complete Information:
In addition to reporting data of questionable accuracy, PARS provides
incomplete data, therefore senior DOE managers may not be aware of the
need to implement corrective actions to prevent cost overruns or
schedule slippages. We identified the following 5 projects--3 major
projects to refurbish nuclear weapons and 2 projects costing more than
$100 million each--that are not in the PARS database, despite DOE's
requirement that projects costing more than $5 million provide monthly
reporting:[Footnote 19]
* W80 Life Extension Program. NNSA recently increased the total cost of
this program, designed to extend the service life of the W80 nuclear
warhead by replacing components, from $1.3 billion to about $2.45
billion.
* W76 Trident Missile Life Extension Program. NNSA expects this
project, designed to extend the service life of the W76 nuclear warhead
by replacing components, to cost about $680 million over the next 4
years.
* B61 Alteration 357 Life Extension Program. NNSA expects this project,
designed to extend the service life of the B61 bomb, to cost nearly
$500 million. Our July 2003 report recommended that DOE improve its
oversight of the life extension program's cost and schedule status.
* Purple and BlueGene/L Supercomputers under the Advanced Simulation
and Computing Program. NNSA expects this project, to cost about $290
million and be completed in 2005.
* Enterprise Project. NNSA increased the total cost of this project,
which will replace the accounting and management systems at Los Alamos
National Laboratory, from about $70 million when it was initiated in
2001 to nearly $160 million.
The National Research Council's 2004 report found that DOE has not
acted in a timely fashion to include all projects costing more than $5
million in PARS. Office of Engineering and Construction Management
officials told us DOE is still in the process of applying project
management principles to many of the department's operational
activities. While DOE's program offices are responsible for converting
these activities to projects, many of the program office personnel
responsible for applying project management principles do not have the
necessary training, according to an Office of Engineering and
Construction Management official. While project management training is
available, DOE has required only project directors and other senior-
level employees to take this training. An Office of Engineering and
Construction Management official told us this training would help
expedite the application of project management principles to DOE's
operational activities.
In addition, for many projects included in the PARS database, PARS
reports do not provide important performance information that senior
DOE managers need to assess the projects' status. In some cases,
project performance data are not reported because the project is
incorrectly listed as being in the design phase when, in fact, it has
passed Critical Decision 2. For example, contractors have spent almost
half of the approved funds for 2 projects at the Idaho National
Engineering and Environmental Laboratory projected to cost $4.3 billion
without reporting performance data in PARS.[Footnote 20] The PARS
reports show that these projects are still in the design phase and,
therefore, are not subject to reporting performance data, but a DOE
official acknowledged that both projects have, in fact, passed Critical
Decision 3 and other subsequent milestones. As a result, senior DOE
managers cannot rely on PARS for accurate and current performance
information for these projects, nor can they rely on PARS to determine
whether these projects require corrective actions.
For these and other projects, PARS also lacks forward-looking data,
such as scheduled work to be performed, the projects' upcoming
milestones, and the projects' estimated cost at completion. Without
such data, PARS cannot provide information on projects' cost or
schedule challenges and DOE management does not have a basis for
projecting progress or identifying trends. While not in PARS, this
information is available from acting project directors. For example,
although early cost savings for the Microsystems and Engineering
Sciences Applications project at Sandia National Laboratories led to
favorable performance data, DOE's project director identified supply
imbalances in the steel market that would increase the estimated
construction costs. Using this information, the project director
revised the project's estimated total cost. Currently, PARS reports to
senior DOE managers lack such forward-looking data that could alert
them to future cost or schedule challenges. The National Research
Council's 2004 report stated that PARS reports should display forward-
looking data to notify senior managers of upcoming milestones. In
addition, several acting project directors told us that forward-looking
data, such as data on estimated costs at completion, should be included
in PARS to identify project performance challenges for senior DOE
managers.
To further illustrate this need, the total costs of some DOE projects
are projected to increase dramatically in the future, despite PARS
reports showing that they are expected to be completed on time and
within budget. For example, PARS report data show that the Hanford's
Tank Waste Treatment and Immobilization Plant is projected to meet the
DOE-approved baseline of $5.78 billion. However, PARS does not show
that DOE approved a $1.4-billion increase above the project's original
contract estimate of $4.35 billion in April 2003, nor does it show that
the U.S. Army Corps of Engineers, in a May 2004 report, stated that
project costs would probably exceed the $5.78-billion cost baseline by
$720 million.[Footnote 21]
Even though the DOE project management teams knew of cost and schedule
performance problems for the Tank Waste Treatment and Immobilization
Plant project, PARS reports have shown that this project was on track
for meeting cost and schedule targets. An Office of Engineering and
Construction Management official told us that PARS monthly reports do
not include forward-looking data and trend data to minimize the amount
of time necessary for senior managers' review. As a result, PARS did
not provide senior DOE managers for this and other projects with
important information to analyze potential future challenges. Forward-
looking performance information, such as scheduled work to be performed
and estimated cost at completion, would better enable senior managers
to address project management challenges and minimize cost overruns or
schedule slippages.
PARS Lacks Timely Information:
Further compounding reliability concerns, we identified problems with
the timeliness of PARS data that may limit the ability of senior DOE
managers to effectively identify and apply corrective actions.
Specifically, we found that cost and schedule performance data were
significantly out of date at some time during our review for 8 of the
33 major projects we reviewed and 20 smaller projects in PARS that had
passed Critical Decision 2. In these instances, data were out of date
because DOE has not effectively enforced requirements that contractors
produce updated monthly cost and schedule performance data, and that
project directors ensure current performance data are reported into
PARS. For some projects, the lack of up-to-date data masked problems
that resulted in cost overruns and schedule slippages. For instance:
* The September 2003 PARS report showed that the Spent Nuclear Fuels
project at Hanford, Washington, was on track to meet its DOE-approved
total project cost of about $1.6 billion and its schedule completion
date of 2007; however, these data were 3 months out of date.
Subsequently, the April 2004 PARS report (1) showed that total project
costs had exceeded the project's cost baseline by nearly $150 million
and (2) indicated that the project would exceed this revised total cost
and the scheduled completion date would slip. In June 2004, the
contractor requested additional funding from DOE because both cost and
schedule performance continued to worsen.
* The September 2003 PARS report showed that the K25/27 Buildings
Deactivation and Decommissioning Removal project at Oak Ridge,
Tennessee, was on track to meet its DOE-approved total project cost of
about $265 million and its schedule completion date of 2008. However,
the contractor did not update the project's performance data until
April 2004, when the PARS report showed the project would still meet
its cost baseline. Environmental Management officials told us that
although they knew for several months that the K25/27 project's total
cost would exceed its baseline, the PARS cost data were not updated
because the project was being combined with 5 other Oak Ridge projects.
The total cost of the K25/27 project could exceed $400 million--more
than 50 percent above the DOE-approved total project cost.
* In June 2004, the Soil and Water Remediation project at Pantex,
Texas, had a DOE-approved total project cost of about $175 million, but
the Office of Engineering and Construction Management could not assess
the project's performance because data were not provided. Subsequently,
the September 2004 PARS report showed that the project was at risk of
exceeding its DOE-approved schedule target.
In addition to these timeliness problems, the monthly data in PARS
reports typically lag a project's actual performance by 2 to 3 months
because of the time contractors need to generate the data and the time
DOE project managers need to review and incorporate the summary data
into the PARS database. The 2004 National Research Council report
stated that the lack of timely data prevents senior managers from using
PARS to assess the performance of projects in real time. Similarly,
Department of Defense officials familiar with project management have
said that using such data to assess project performance is like
"overseeing by looking through a rear view mirror" because performance
problems have usually gotten worse by the time departmental managers
become aware of them.
We found that the Department of Defense requires all of its newer
contracts to use electronic data interchange to provide more timely
information to department program managers.[Footnote 22] In addition,
some acting project directors told us that electronically linking PARS
to contractors' project management systems would improve timeliness
because manually entering cost and schedule data into the PARS database
had often resulted in delays of 2 to 3 months to complete the process.
In some instances, data were entered incorrectly, although in each
instance the data were corrected before being reported to senior
managers. While the DOE project directors we contacted uniformly agree
that manually entered data are correctly entered by the time PARS
monthly reports are delivered to senior managers, electronically
linking PARS to contractor systems could eliminate the potential for
such errors and enhance senior managers' ability to address potential
cost or schedule challenges in real time. Alternatively, DOE might
include a provision requiring timely monthly reporting in all
applicable contracts.
When data can be relied upon, DOE senior managers have taken corrective
actions to address cost or schedule challenges while minimizing costs
to the government. For example, NNSA terminated the Sandia Underground
Reactor Facility project, which was intended to reduce the future
operational costs associated with securing a reactor, when management
learned that cost estimates had increased by more than 150 percent
between project conception and the final design phase. The project was
terminated before costs were incurred. In another instance,
Environmental Management approved a contractor's recovery plan to
complete the Melton Valley Closure project at Oak Ridge, Tennessee,
whose schedule performance had slipped dramatically and required
corrective actions. The contractor lengthened work hours and modified
its approach for constructing a subproject. As a result, the recovery
plan showed that the scope of work could be accomplished without
increasing project schedule.
Conclusions:
Since 1990, we have designated DOE's contract management, which we have
broadly defined to include contract administration and project
management, as a high-risk area for fraud, waste, abuse, and
mismanagement. Although DOE has implemented important contract
administration and project management reforms, problems persist and
many major projects continue to experience millions of dollars in cost
overruns and years of delays. Two deficiencies--the lack of contracting
criteria for major projects and the lack of reviews of the project
management terms in major project contracts--have resulted in
questionable DOE contracting decisions that limit its ability to
effectively control cost and schedule performance. For example, many of
DOE's contracts for major projects have used performance incentives
that have used a technical, schedule, or performance incentive without
an associated cost incentive or cost constraint, thereby giving
contractors an incentive to pay limited attention to costs when working
toward meeting technical or performance levels in order to earn a
higher award fee.
Furthermore, for major projects, DOE has given insufficient emphasis to
the oversight of contract administration, which begins after contracts
are awarded and helps ensure that the department gets what it pays for.
DOE needs to give increased emphasis to reviewing how it administers
contracts; correcting previously identified weaknesses, such as
overreliance on contractor data; and providing training to its
contracting officers. Without such actions, the department is totally
dependent on its contractors' self-reports on their performance.
Because of problems with the accuracy, completeness, and timeliness of
the PARS data, senior DOE managers lack key project performance
information for assessing the progress of many major projects and
making decisions about corrective actions. In particular, because DOE
has assessed the reliability of only three contractors' project
management systems that feed data into PARS, senior managers cannot be
certain that the contractor systems are producing reliable data. Such
data are critical to good project management and affect DOE's
assessment of contractor performance. Absent reliable data from the
contractor systems, DOE lacks assurance that the fees it awards for a
contractor's project management actions are well deserved.
Recommendations for Executive Action:
To ensure the use of effective performance incentives for major
projects, we recommend that the Secretary of Energy direct the
Associate Deputy Secretary with responsibility for contract and project
management to take the following two actions:
* develop a major projects chapter in the DOE Acquisition Guide that
specifies a systematic contracting approach, including, for example,
criteria for (1) ensuring that incentive fee awards are based on
reliable performance data, (2) using appropriate cost and schedule
incentives, (3) better linking fee awards to performance for major
projects that are part of larger site cleanups, and (4) determining
which indirect work-related activities should and should not be
considered in awarding contractors' fees, and:
* clarify roles and responsibilities for reviewing contracts prior to
award to ensure project management consistency.
To strengthen departmental oversight of contract administration for
major projects, we recommend that the Secretary of Energy direct the
Associate Deputy Secretary with responsibility for contract and project
management to take the following three actions:
* conduct comprehensive self-assessments of contract administration at
least every 3 years,
* identify corrective actions to reduce the overreliance on unvalidated
contractor data in awarding contract fees that was identified in
previous self-assessments, and:
* train contracting officials in earned value management.
To improve the reliability and usefulness of project performance data
in PARS, we recommend that the Secretary of Energy direct the
appropriate managers to take the following seven actions:
* develop a schedule for assessing the reliability of the contractors'
project management systems, giving priority to major projects and those
projects with systems believed to be using incorrect methods to
generate PARS data;
* revise DOE manual 413.3-1 to provide guidance that enhances the
accurate reporting of total cost and project performance data into
PARS, such as the reporting of life-of-project cost and schedule
variances;
* expedite training for major project directors in earned value
management concepts;
* ensure that program office officials receive currently available
project management training so that they can better identify the
elements of a project, and apply the project management concepts
necessary for them to report performance data in PARS;
* incorporate forward-looking trend data into PARS reports so that
senior managers can better identify negative trends and potentially
take corrective action;
* explore options for ensuring that contractors provide cost and
schedule performance data to PARS on a monthly basis, such as making
monthly submissions a requirement in all applicable contracts; and:
* explore options for providing senior DOE managers with more timely
project performance data by, for example, electronically linking
contractors' project management systems to PARS.
Agency Comments and Our Evaluation:
We provided DOE with a draft of this report for its review and comment.
In written comments, DOE generally concurred with our recommendations
but provided clarifying comments on four of the recommendations. (See
app. III.) First, concerning our recommendation that DOE develop a
major projects chapter in its Acquisition Guide, DOE stated that the
department has already developed an extensive body of material that
constitutes a "systematic contracting approach" for the acquisition and
management of departmental major projects, but added that the
department will develop an overview and summary of this information in
a major projects chapter in its Acquisition Guide. We believe this
chapter will further enhance DOE's guidance, particularly if the
department provides criteria that address each of the four issues
identified in our first recommendation. Second, concerning our
recommendation on DOE's comprehensive assessment of contract
administration, DOE stated that the department did not stop conducting
comprehensive assessments. In response, we have revised our
recommendation to state that DOE should conduct these assessments at
least every 3 years. Third, concerning our recommendation that DOE
identify corrective actions for reducing overreliance on unvalidated
contractor data, DOE stated that the department had already taken
positive steps to reduce its overreliance on contractor data by, for
example, reviewing and validating such data and project baselines. DOE
added that the department would continue to identify any corrective
actions necessary to reduce overreliance on contractors' data in
awarding fees. While we agree that validating project baselines is an
important first step, we believe that DOE's efforts to ensure that
contractor performance data are reliable by certifying contractors'
project management systems is vital. Fourth, concerning our
recommendation that DOE link PARS and contractors' project management
systems, DOE stated that our recommendation is too narrowly focused,
particularly in light of DOE's efforts to implement a departmentwide
enterprise architecture solution. We agree, and we have revised our
recommendation accordingly. In addition, DOE stated that it believes
the draft report contained a number of inaccuracies and provided
detailed comments. We have revised the report, where appropriate, in
response to these comments.
As arranged with your offices, unless you publicly announce its
contents earlier, we plan no further distribution of this report until
30 days after the date of this letter. At that time, we will send
copies to the Secretary of Energy and other interested parties. We will
also make copies available to others upon request. In addition, the
report will be available at no charge on the GAO Web site at
[Hyperlink, http://www.gao.gov].
If you or your staff have any questions on this report, please contact
me at (202) 512-3841. Key contributors to this report were Richard
Cheston, Robert Baney, Nathan Anderson, Bernice Dawson, Cynthia Norris,
Judy Pagano, and Doreen Feldman.
Signed by:
Robin M. Nazzaro:
Director, Natural Resources and Environment:
[End of section]
Appendixes:
[End of section]
Appendix I: The Department of Energy's 33 Major Projects That We
Reviewed:
Fernald, Ohio:
Project name and construction line number: Non-Nuclear Facility
Decontamination and Decommissioning (OH-FN-0050);
Contract type: Fernald, Ohio: Cost plus incentive fee;
Cost incentive or cost constraint for individual project in contract?
Yes;
Schedule performance incentive for individual project in contract? Yes;
Fee available for individual project? No[A].
Project name and construction line number: Silos (OH-FN-07)[B];
Contract type: Cost plus incentive fee;
Cost incentive or cost constraint for individual project in contract?
Yes;
Schedule performance incentive for individual project in contract? Yes;
Fee available for individual project? No[A].
Project name and construction line number: Soil & Water Remediation (OH-
FN-0030);
Contract type: Cost plus incentive fee;
Cost incentive or cost constraint for individual project in contract?
Yes;
Schedule performance incentive for individual project in contract? Yes;
Fee available for individual project? No[A].
Project name and construction line number: Solid Waste Stabilization &
Disposition (OH-FN-0013);
Contract type: Cost plus incentive fee;
Cost incentive or cost constraint for individual project in contract?
Yes;
Schedule performance incentive for individual project in contract? Yes;
Fee available for individual project? No[A].
Hanford Reservation: River Protection:
Project name and construction line number: Hanford Tank Waste Treatment
and Immobilization Plant (01-D-416);
Contract type: Cost plus incentive fee;
Cost incentive or cost constraint for individual project in contract?
Yes;
Schedule performance incentive for individual project in contract? Yes;
Fee available for individual project? Yes.
Project name and construction line number: Radioactive Liquid Tank
Waste Stabilization & Disposition (ORP-0014);
Contract type: Cost plus award fee;
Cost incentive or cost constraint for individual project in contract?
No;
Schedule performance incentive for individual project in contract? Yes;
Fee available for individual project? Yes.
Project name and construction line number: Interim Tank Retrieval
System (94-D-407)[C];
Contract type: Cost plus award fee;
Cost incentive or cost constraint for individual project in contract?
No;
Schedule performance incentive for individual project in contract? Yes;
Fee available for individual project? Yes.
Project name and construction line number: Tank Farm Restoration and
Safe Operations (97-D-402)[B];
Contract type: Cost plus award fee;
Cost incentive or cost constraint for individual project in contract?
No;
Schedule performance incentive for individual project in contract? Yes;
Fee available for individual project? Yes.
Hanford Reservation at Richland, Washington:
Project name and construction line number: Nuclear Facility
Decontamination and Decommissioning--Fast Flux Test Facility Project
(RL-0042);
Contract type: Cost plus award fee;
Cost incentive or cost constraint for individual project in contract?
No;
Schedule performance incentive for individual project in contract? Yes;
Fee available for individual project? Yes.
Project name and construction line number: Nuclear Facility
Decontamination and Decommissioning--Remainder of Hanford (RL-0040);
Contract type: Cost plus award fee;
Cost incentive or cost constraint for individual project in contract?
No;
Schedule performance incentive for individual project in contract? Yes;
Fee available for individual project? Yes.
Project name and construction line number: Nuclear Facility
Decontamination and Decommissioning--River Corridor Closure Project (RL-
0041);
Contract type: Cost plus award fee;
Cost incentive or cost constraint for individual project in contract?
No;
Schedule performance incentive for individual project in contract? Yes;
Fee available for individual project? Yes.
Project name and construction line number: Nuclear Material
Stabilization & Disposition-PFP (RL-0011);
Contract type: Cost plus award fee;
Cost incentive or cost constraint for individual project in contract?
No;
Schedule performance incentive for individual project in contract? Yes;
Fee available for individual project? Yes.
Project name and construction line number: Soil and Water Remediation-
Vadose Zone (RL-0030);
Contract type: Cost plus award fee;
Cost incentive or cost constraint for individual project in contract?
No;
Schedule performance incentive for individual project in contract? No;
Fee available for individual project? Yes.
Project name and construction line number: Solid Waste Stabilization &
Disposition--200 Area (RL-0013);
Contract type: Cost plus award fee;
Cost incentive or cost constraint for individual project in contract?
No;
Schedule performance incentive for individual project in contract? Yes;
Fee available for individual project? Yes.
Project name and construction line number: Spent Nuclear Fuels (RL-
0012);
Contract type: Cost plus award fee;
Cost incentive or cost constraint for individual project in contract?
No;
Schedule performance incentive for individual project in contract? Yes;
Fee available for individual project? Yes.
Idaho National Engineering and Environmental Laboratory:
Project name and construction line number: Advanced Mixed Waste
Treatment Facility (97-PVT-2);
Contract type: Fixed price;
Cost incentive or cost constraint for individual project in contract?
No;
Schedule performance incentive for individual project in contract? No;
Fee available for individual project? No.
Project name and construction line number: Spent Nuclear Fuel Dry
Storage (98-PVT-2)[B];
Contract type: Fixed price;
Cost incentive or cost constraint for individual project in contract?
No;
Schedule performance incentive for individual project in contract? No;
Fee available for individual project? No.
Lawrence Livermore National Laboratory:
Project name and construction line number: National Ignition Facility
(96-D-111);
Contract type: Cost plus award fee;
Cost incentive or cost constraint for individual project in contract?
Yes;
Schedule performance incentive for individual project in contract? Yes;
Fee available for individual project? No[D].
Oak Ridge, Tennessee:
Project name and construction line number: East Tennessee Technology
Park Three-Building D&D and Recycle Project (OR-0040);
Contract type: Cost plus incentive fee;
Cost incentive or cost constraint for individual project in contract?
Yes;
Schedule performance incentive for individual project in contract? Yes;
Fee available for individual project? Yes.
Project name and construction line number: Facilities Capability
Assurance Program (88-D-122-27 & 88-D-122-42)[C];
Contract type: Cost plus award fee;
Cost incentive or cost constraint for individual project in contract?
No;
Schedule performance incentive for individual project in contract? Yes;
Fee available for individual project? Yes.
Project name and construction line number: Spallation Neutron Source
(99-E-334);
Contract type: Cost plus incentive fee;
Cost incentive or cost constraint for individual project in contract?
Yes;
Schedule performance incentive for individual project in contract? Yes;
Fee available for individual project? Yes.
Rocky Flats Facility at Denver, Colorado:
Project name and construction line number: Nuclear Facility D&D/North
Side Facility Closures (RF-0040);
Contract type: Cost plus incentive fee;
Cost incentive or cost constraint for individual project in contract?
Yes;
Schedule performance incentive for individual project in contract? Yes;
Fee available for individual project? No[A].
Project name and construction line number: Nuclear Facility D&D/South
Site Facility Closures (RF-0041);
Contract type: Cost plus incentive fee;
Cost incentive or cost constraint for individual project in contract?
Yes;
Schedule performance incentive for individual project in contract? Yes;
Fee available for individual project? No[A].
Project name and construction line number: Nuclear Material
Stabilization & Disposition (RF-0011);
Contract type: Cost plus incentive fee;
Cost incentive or cost constraint for individual project in contract?
Yes;
Schedule performance incentive for individual project in contract? Yes;
Fee available for individual project? No[A].
Project name and construction line number: Soil & Water Remediation (RF-
0030);
Contract type: Cost plus incentive fee;
Cost incentive or cost constraint for individual project in contract?
Yes;
Schedule performance incentive for individual project in contract? Yes;
Fee available for individual project? No[A].
Project name and construction line number: Solid Waste Stabilization &
Disposition (RF-0013);
Contract type: Cost plus incentive fee;
Cost incentive or cost constraint for individual project in contract?
Yes;
Schedule performance incentive for individual project in contract? Yes;
Fee available for individual project? No[A].
Sandia National Laboratories:
Project name and construction line number: Microsystems and Engineering
Science Application (01-D-108);
Contract type: Fixed price;
Cost incentive or cost constraint for individual project in contract?
Yes;
Schedule performance incentive for individual project in contract? Yes;
Fee available for individual project? No[D].
Savannah River Site:
Project name and construction line number: High-Level Waste Removal
from Filled Waste Tanks (SR-0014C);
Contract type: Cost plus award fee;
Cost incentive or cost constraint for individual project in contract?
No;
Schedule performance incentive for individual project in contract? Yes;
Fee available for individual project? Yes.
Project name and construction line number: Tritium Extraction Facility
(98-D-125);
Contract type: Cost plus incentive fee;
Cost incentive or cost constraint for individual project in contract?
Yes;
Schedule performance incentive for individual project in contract? Yes;
Fee available for individual project? Yes.
West Valley, New York:
Project name and construction line number: Nuclear Facility
Decontamination and Decommissioning (OH-WV-0040);
Contract type: Cost plus award fee;
Cost incentive or cost constraint for individual project in contract?
Yes;
Schedule performance incentive for individual project in contract? Yes;
Fee available for individual project? Yes.
Nuclear Weapons Refurbishment:
Project name and construction line number: Life Extension Program--
B61[E];
Contract type: Cost plus award fee;
Cost incentive or cost constraint for individual project in contract?
No;
Schedule performance incentive for individual project in contract? Yes;
Fee available for individual project? No[D].
Project name and construction line number: Life Extension Program--
W76[E];
Contract type: Cost plus award fee;
Cost incentive or cost constraint for individual project in contract?
No;
Schedule performance incentive for individual project in contract? Yes;
Fee available for individual project? No[D].
Project name and construction line number: Life Extension Program--
W80[E];
Contract type: Cost plus award fee;
Cost incentive or cost constraint for individual project in contract?
No;
Schedule performance incentive for individual project in contract? Yes;
Fee available for individual project? No[D].
Source: GAO compilation of DOE data.
[A] Fee is based on reaching site closure, rather than on completing
individual projects.
[B] Although the September 2004 PARS report showed that this project
would cost less than $400 million, we included it in our review because
it was included in our 2002 review of DOE's major projects. (GAO,
Contract Reform: DOE Has Made Progress, but Actions Needed to Ensure
Initiatives Have Improved Results, [Hyperlink, http://www.gao.gov/cgi-
bin/getrpt?GAO-02-798] (Washington, D.C.: Sept. 13, 2002).
[C] This project was designated as a major project when DOE's threshold
was $100 million.
[D] NNSA factors performance on these projects into each responsible
management and operating contractor's annual performance evaluation and
decisions on the amount of the fee awards.
[E] NNSA stated that each of these life extension projects involved
multiple management and operating contractors (not 1 contract) in
multiple locations, which is different from every other project that is
listed in this appendix.
[End of table]
[End of section]
Appendix II: Scope and Methodology:
Our review focused primarily on 33 major projects that had passed, as
of March 2004, the Department of Energy's (DOE) Critical Decision 2
milestone--the point at which the department approves a project's cost,
schedule, and scope baselines on the basis of an approved conceptual
design report and acquisition strategy. The projects we reviewed
include 28 projects that cost more than $400 million each and 5
projects that our 2002 assessment defined as major projects because
their total costs exceeded $100 million each. Our review did not
include 46 major projects that, as of March 2004, had not passed the
Critical Decision 2 milestone. Since March 2004, at least 6 major
projects have passed the Critical Decision 2 milestone and now have
approved baselines. The remaining major projects do not have approved
baselines for measuring performance.
To assess DOE's use of performance incentives in contracts to
effectively control cost and maintain schedules, we reviewed relevant
requirements in the Federal Acquisition Regulation (FAR) and the DOE
Acquisition Regulation, as well as DOE Order 413.3, DOE manual 413.3-1,
and DOE's Acquisition Guide, to obtain information on the factors that
should be used in determining a contractor's fee. Through this effort,
we identified whether the department provided guidance on the
appropriate circumstances for using each contract type and the
appropriate factors for determining a contractor's fee. In particular,
we examined requirements regarding contract provisions for award fees;
cost, schedule, and performance incentives; and fee determination
plans.
We then compared government and departmental requirements with project-
specific elements found in the contracts for each of the 33 major
projects that have DOE-approved cost, schedule, and scope baselines to
determine whether DOE has used appropriate (1) types of performance
incentives, such as cost or schedule incentives, and (2) fee
determination plans and fee payments. For instance, to assess whether
DOE's contracts used the appropriate incentives for each of three types
of contracts, we compared the types of incentives that DOE's contracts
and relevant modifications used for each of the 33 major projects with
the types of incentives that the FAR and the DOE Acquisition
Regulation, as well as departmental orders and guidance, require. We
then interviewed cognizant DOE officials to discuss reasons for the
inconsistencies we found. In addition, we examined various contract-
related documents associated with the 33 major projects we reviewed,
such as the "Gold Chart" metrics that Environmental Management uses to
measure its progress in DOE's annual budget submission to the Congress.
Specifically, we compared the Gold Chart's performance metrics for each
of Environmental Management's 25 major projects with the performance
measures in each projects' contract. Where differences were identified,
we discussed the contents of the Gold Chart and the associated
projects' contract with appropriate DOE contracting officials.
Furthermore, we interviewed officials in DOE's Office of Contract
Management and officials in the Office of Engineering and Construction
Management to determine the extent to which DOE had reviewed, prior to
award, the contracts for the 33 major projects to ensure that they
included appropriate project management provisions.
To assess the reliability of the data DOE uses to monitor and assess
contractor performance, we reviewed the Office of Management and
Budget's (OMB) directives, DOE's Reference Book for Contract
Administrators, and other DOE documents and studies to identify
relevant requirements and departmental guidance. We identified the
roles and responsibilities of contract administration officials and
examined the extent to which these officials adhered to their
responsibilities. More specifically, we reviewed the department's
recent contract administration self-assessments and the frequency with
which they were conducted. In so doing, we examined the department's
recommendations for improving contract administration and determined
whether the recommendations were followed. If they were not followed,
we discussed the reasons with cognizant officials in the Contract
Administration Division. We also examined DOE's order for acquisition
career development, and other related DOE directives, to assess
training requirements for DOE's contracting officers and contracting
officer representatives, particularly regarding training in earned
value management principles.
To determine the reliability of Project Analysis and Reporting System
(PARS) data used by senior managers for project oversight, we assessed
the accuracy, completeness, and timeliness of PARS data. To assess the
accuracy of the project performance data in PARS, we did the following:
* Reviewed DOE Order 413.3, "Program and Project Management for the
Acquisition of Capital Assets," and its implementing guidance; OMB
Circular A-11, part 7, "Planning, Budgeting, Acquisition, and
Management of Capital Assets"; and various documents outlining the
requirements in American National Standards Institute/Electronic
Industries Association-748-1998, which defines the requirements for
earned value management--the component of contractors' project
management systems critical for producing reliable project performance
data.
* Interviewed cognizant DOE officials in the Office of Engineering and
Construction Management, the Office of Environmental Management, the
Office of Science, and the National Nuclear Security Administration on
the extent to which the performance data that DOE contractors' project
management systems produced for PARS met earned value management
requirements. These officials included a DOE expert in earned value
management, who is responsible for assessing the accuracy of the data
that various projects' systems produce. Where specific deficiencies in
a contractor's project management system were identified, we obtained
relevant documents from the appropriate acting DOE project director and
analyzed whether the contractor generated project performance data in
accordance with the industry standard. We also interviewed officials in
two other major contracting agencies--the Department of Defense and the
National Aeronautics and Space Administration--about their experience
in implementing earned value management requirements.
* Compared data in monthly PARS reports provided to senior DOE managers
from January through September 2004 with project-specific cost and
schedule data obtained from earlier PARS reports, cognizant program
offices, project status reports, Inspector General reports, and
external reviews. When we identified total cost or project performance
data discrepancies between PARS and these other sources, we contacted
relevant project officials to determine their cause.
* Identified the extent to which contractor-generated data in PARS were
sufficiently reviewed and verified by DOE by (1) identifying
requirements in DOE Order 413.3 and its implementing guidance for the
departmental review and verification of contractor project performance
data and (2) interviewing DOE project management officials to determine
whether the current breadth of review was adequate and what plans, if
any, DOE had for increasing the rigor of its review and verification of
contractor data.
To assess the completeness of PARS data, we determined whether the PARS
database included major DOE activities--those costing more than $400
million or that DOE management had designated--identified in our prior
reports, Inspector General reports, DOE press releases, and printouts
from DOE's Management Accounting and Reporting System. For projects
that were not included in PARS, we contacted headquarters project
management officials to determine if the projects met the criteria for
PARS reporting. For projects that were included in PARS, we examined
the completeness of reported data in various data fields by reviewing
printouts from the PARS database and by reviewing the reports of the
National Academies' National Research Council and the Civil Engineering
Research Foundation, which also examined the completeness of PARS data.
In addition, we reviewed a 2004 report by the U.S. Army Corps of
Engineers on a major project at Hanford, Washington. Moreover, we
discussed options with DOE officials for reporting additional data that
would improve PARS' ability to enable senior DOE managers to identify
potential cost or schedule challenges.
To assess the timeliness of PARS data, we reviewed PARS monthly reports
to senior DOE managers and identified projects whose performance data
were out-of-date. For many of these projects, we talked to headquarters
and project officials to determine the reasons for delay and explored
options with them on how timeliness could be improved. We also
interviewed numerous acting DOE project directors to learn how data
from their project management systems were summarized and incorporated
into the PARS database. In addition, we explored options with DOE
headquarters and project officials for improving the timeliness of all
data reported in PARS.
Given our review of the documentation provided by DOE and our
discussions with DOE officials, we have reservations about the
reliability of PARS data. These issues are discussed in this report.
We conducted our work from January 2004 through January 2005 in
accordance with generally accepted government auditing standards, which
included an assessment of data reliability and internal controls.
[End of section]
Appendix III: Comments from the Department of Energy:
Department of Energy:
Washington, DC 20585:
FEB 02 2005:
Ms. Robin M. Nazzaro:
Director, Natural Resources and Environment:
U.S. Government Accountability Office:
441 G Street, NW Room 2964:
Washington, DC 20548:
Dear Ms. Nazzaro:
We have reviewed the draft Government Accountability Office (GAO)
Report entitled "Department of Energy: Further Actions Are Needed to
Strengthen Contract Management for Major Projects" (GAO-05-123). We
appreciated the opportunity to review prior informal drafts of the
report to eliminate any inaccuracies and questionable conclusions. We
also appreciate that the report recognizes the Department of Energy's
(DOE) many improvements in a number of areas related to project and
contract management.
The GAO report specifically recognizes a number of improvements in
DOE's management of major projects. For example, it notes: the
establishment of the Office of Engineering and Construction Management
(OECM) to oversee project management; the issuance of a comprehensive
project management policy; implementation of a career development
program for DOE project managers; implementation of the Project
Assessment and Reporting System (PARS); required use of earned value
management techniques; a certification program for contractors' project
management systems to assure compliance with private industry's
standard for earned value management; improved contract processes
through the issuance of updates to DOE's Acquisition Guide and the
Reference Book for Contract Administrators; and periodic assessments of
its contract administration practices.
Because of the importance of the subject matter, the beneficial nature
of its suggestions, and the Department's commitment to continuous
improvement in this area, DOE concurs generally in all of the draft
report recommendations. Nonetheless, despite its prior revisions, the
draft report still contains a number of factual inaccuracies and
inappropriate conclusions which affect the report's credibility.
Although these inaccuracies were addressed in the Department's comments
on the draft "statement of facts," they remain an issue in the report.
The DOE's responses to the technical and factual inaccuracies,
including the report's misrepresentations of certain conversations, are
provided to you as an attachment to this letter. We request that the
GAO revise its draft report to correct these inaccuracies. If this
course of action is not acceptable, DOE requests that the full text of
its response (including the attachment to this letter) be published
with the final GAO report.
The report provides three sets of recommendations (a total of 13
recommendations in all) to the Secretary in order to improve: (1) the
use of effective performance incentives for major projects, (2)
departmental oversight of contract administration for major projects,
and (3) the reliability and usefulness of project performance data in
the PARS. Notwithstanding that the Department disagrees with a number
of the report's findings, we do agree with the general thrust of the
recommendations.
With respect to areas of disagreement, the Department offers the
following:
1. GAO Recommendation: Develop a major projects chapter in the DOE
Acquisition Guide that specifies a systematic contracting approach,
including, for example, criteria for (1) ensuring that incentive fee
awards are based on reliable performance data, (2) using appropriate
cost and schedule incentives, (3) better linking fee awards to
performance for major projects that are part of larger site cleanups,
and (4) determining which indirect work-related activities should and
should not be considered in awarding contractors' fees.
DOE Comment: The Department has already developed an extensive body of
material that constitutes a "systematic contracting approach" for the
acquisition and management of the Department's major projects. The
principles for determining the contract type, the nature of the
incentive structure, etc. for contracts are already well established in
the Federal Acquisition Regulation and the Department of Energy
Acquisition Regulation. Unlike other Federal agencies, DOE has expanded
on this information in the form of more detailed internal guidance on
contract types and the effective use of performance incentives. The
regulatory coverage when taken with the additional requirements of the
DOE Acquisition Regulation, the DOE Acquisition Guide, the DOE
Performance-Based Contracting Guide, DOE Order 413.3, DOE Manual 413.3-
1, the various workshops that the Department has conducted and the
various professional training courses made available (both project
management and acquisition courses) provides a solid basis for
procurement and project management professionals to make informed
decisions regarding appropriate types of contracts, incentives, and
other pertinent components of properly executed major projects and
contracts. Both the Office of Contract Management (OCM) and the OECM
work together on acquisition strategies for the Department's major
contracts. DOE Manual 413.3-1 requires, prior to approval of CD-1, that
the Integrated Project Team develop an acquisition strategy for each
project.
Based on the Department's previous review of materials made available
by other agencies for the management of major projects, we know of no
other agency which has provided as extensive and detailed a body of
guidance to its contracting professionals in order to improve the state
of contract and project management. Nonetheless, DOE will develop an
overview and summary of the various information already extant in this
major projects chapter in the DOE Acquisition Guide in accordance with
the GAO's recommendation. Additionally, we will update our benchmarking
of other agencies to identify other areas of information guidance, and
direction that may be useful to the DOE project and contract management
practitioners.
2. GAO Recommendation: Reinstitute comprehensive self-assessments of
contract administration.
DOE Comment: This recommendation implies that the Department has ceased
conducting comprehensive assessments of contract administration.
Nothing could be further from the truth. As stated in the attached
comments, the Department conducted assessments in 1997, 1999, 2002 and
2003. That OCM did not conduct an assessment in 2004 does not indicate
that the Department no longer values such assessments or that they will
not be performed in the future. Assessments are, and will be, conducted
on a periodic basis. For example, during fiscal year 2005, two separate
sets of assessments are planned. One set of assessments will examine
the effective use by DOE sites of contract management plans (CMP) to
manage contracts. A second set of assessments will be conducted on
contractor purchasing systems. The Department concurs in the GAO's
recommendation "to continue such assessments."
3. GAO Recommendation: Identify corrective actions to reduce the over-
reliance on contractors' data in awarding contract fees that was
identified in previous self-assessments.
DOE Comment: Every procurement agency in the Federal Government relies
to some extent on contractor data, a necessity which the Federal
Acquisition Regulation recognizes. The Department fully acknowledges
the importance of ensuring that such data has been validated to the
extent necessary for the effective management of the Department's
contracts and projects.
Consequently, the Department has already taken positive steps to reduce
any over-reliance on contractor submitted data and the associated
baselines. For example, the Office of Environmental Management (EM)
regularly reviews and validates such data and baselines. Specifically,
EM review teams have performed independent reviews of the Integrated
Life Cycle Baselines and Contract Project Baselines of all EM sites
over the past two years. To date, all but six EM sites (West Valley,
Nevada, Stanford Linear Accelerator Center, Waste Isolation Pilot
Plant, Los Alamos National Laboratory, and Energy Technology
Engineering Center) have approved baselines which are under
configuration control. The criteria for the reviews included, but were
not limited to, cost estimation, scope definition, project schedule,
earned value management system, performance metrics, regulatory
requirements, and government furnished services and items. The goal is
to have the baselines for the remaining six EM sites under
configuration control by the end of fiscal year 2005.
Additionally, the OECM has been validating project baselines for
several years. This was previously commented to GAO at its exit
conference and in the Department's written comments on the draft
"statement of facts." OECM recently sent a report to Congress
documenting its efforts in validating EM project baselines. The
Subcommittee on Energy and Water Development Appropriations stated that
it supports OECM's efforts and that OECM "continues to provide a strong
focal point for the improvement of project management capabilities
throughout the Department." Furthermore, the Department is in the
process of certifying that its contractors' Earned Value Management
Systems comply with the industry standard. We are not aware that any
other civilian agency is doing this.
Clearly, DOE has been proactive in addressing the issue of ensuring
that data and baselines for the Department's major management contracts
and projects is validated and that the systems which track this data
are in compliance with the industry standard. Nonetheless, the
Department concurs with the general thrust of the GAO's recommendation
and will continue to identify any corrective actions necessary to
reduce over-reliance on contractors' data in awarding contract fees.
4. GAO Recommendation: Explore options for electronically linking
contractors' project management systems to PARS to provide senior DOE
managers with timely project performance data.
DOE Comment: The Department agrees with the need to ensure timely
project reporting. However, the GAO's linking of PARS and contractors'
project management systems is too narrowly focused, particularly in
light of the Department's efforts to implement a Department-wide
enterprise architecture solution. The recommendation would be
beneficial if it were not focused on the existing stand-alone system
(PARS), but rather addressed a more comprehensive exploration of
options that would improve the timeliness and accuracy of project
performance reporting. Consequently, the Department concurs with the
concept behind the GAO recommendation and will explore options to
improve the timeliness and accuracy of project performance reporting.
Once again, DOE appreciates GAO's identification of opportunities to
improve its project and contract management systems. Additional
comments and corrections on the draft report provided by the cognizant
DOE organizations are attached for your review. The Department
regularly reviews its approach to implementing and managing its
projects and their supporting contracts as evidenced by the numerous
internal reviews/reports examining the Department's implementation of
these contracts.
The Department continues to seek ways to expand the implementation of
what works and correct what does not. We appreciate the GAO's
recommendations and will incorporate them into our future improvement
efforts.
Sincerely,
Signed by:
Susan J. Grant:
Director, Office of Management, Budget and Evaluation/Chief Financial
Officer:
Attachments:
[End of section]
Related GAO Products:
[End of section]
Nuclear Waste: Absence of Key Management Reforms on Hanford's Cleanup
Project Adds to Challenges of Achieving Cost and Schedule Goals.
[Hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-04-611]
Washington, D.C.: June 9, 2004.
Nuclear Waste Cleanup: DOE Has Made Some Progress in Cleaning Up the
Paducah Site, but Challenges Remain.
[Hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-04-457]
Washington, D.C.: April 1, 2004.
Nuclear Weapons: Opportunities Exist to Improve the Budgeting, Cost
Accounting, and Management Associated with the Stockpile Life Extension
Program.
[Hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-03-583]
Washington, D.C.: July 28, 2003.
Nuclear Waste: Challenges and Savings Opportunities in DOE's High-Level
Waste Cleanup Program.
[Hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-03-930T]
Washington, D.C.: July 17, 2003.
Nuclear Waste: Challenges to Achieving Potential Savings in DOE's High-
Level Waste Cleanup Program.
[Hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-03-593]
Washington, D.C.: June 17, 2003.
Department of Energy: Status of Contract and Project Management
Reforms.
[Hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-03-570T]
Washington, D.C.: March 20, 2003.
Major Management Challenges and Program Risks: Department of Energy.
[Hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-03-100]
Washington, D.C.: January 1, 2003.
High-Risk Series: An Update.
[Hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-03-119]
Washington, D.C.: January 1, 2003.
Contract Reform: DOE Has Made Progress, but Actions Needed to Ensure
Initiatives Have Improved Results.
[Hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-02-798]
Washington, D.C.: September 13, 2002.
Nuclear Waste: Technical, Schedule, and Cost Uncertainties of the Yucca
Mountain Repository Project.
[Hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-02-191]
Washington, D.C.: December 21, 2001.
Department of Energy: Follow-Up Review of the National Ignition
Facility.
[Hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-01-677R]
Washington, D.C.: June 1, 2001.
Nuclear Cleanup: Progress Made at Rocky Flats, but Closure by 2006 Is
Unlikely, and Costs May Increase.
[Hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-01-284]
Washington, D.C.: February 28, 2001.
High-Risk Series: An Update.
[Hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-01-263]
Washington, D.C.: January 1, 2001.
Major Management Challenges and Program Risks: Department of Energy.
[Hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-01-246]
Washington, D.C.: January 1, 2001.
(360423):
FOOTNOTES
[1] GAO, Contract Reform: DOE Has Made Progress, but Actions Needed to
Ensure Initiatives Have Improved Results, GAO-02-798 (Washington, D.C.:
Sept. 13, 2002).
[2] National Research Council of the National Academies, Progress in
Improving Project Management at the Department of Energy: 2003
Assessment (Washington, D.C.: 2004).
[3] Civil Engineering Research Foundation, Independent Research
Assessment of Project Management Factors Affecting Department of Energy
Project Success (July 12, 2004).
[4] DOE's management and operating contracts include the DOE
Acquisition Regulation clause providing discretion to the fee
determination official to reduce the contractor's fee by up to 75
percent when the contractor fails to meet stipulated cost performance
levels. (See 48 C.F.R. § 970.5215-3.) In DOE's management and operating
contracts, the cost performance level for the contract, unless
otherwise stipulated, is typically the annual funding level for the
contract.
[5] DOE has assessed the contractor's project management system for the
Spallation Neutron Source project and plans to certify the system after
some minor changes are made.
[6] U.S. Army Corps of Engineers, Independent Cost & Schedule Baseline
Review Summary Report, Hanford Waste Treatment and Immobilization Plant
(Walla Walla, Washington: May 28, 2004).
[7] DOE developed the Performance-Based Contracting Guide as a
reference document for agency personnel involved in all aspects of
performance-based management contracting.
[8] The other 9 Environmental Management projects did not have
associated Gold Chart metrics.
[9] Environmental Management defines a radioactive facility completion
as a decommissioning, deactivation, dismantlement, demolishment, or
transfer of responsibility for the facility to another program or
owner.
[10] GAO, Department of Energy: Opportunity to Improve Management of
Major System Acquisitions, GAO/RCED-97-17 (Washington, D.C.: Nov. 26,
1996).
[11] In addition, DOE's project directors and support field staff
assess contractor performance by walking the site, evaluating work
performed, and periodically meeting with the contractor.
[12] In November 2004, DOE revised this chapter in the Reference Book
for Contract Administrators. Although the revised chapter does not
mention this specific responsibility, in at least 2 contracts we
reviewed, we found that the contracting officer had this
responsibility.
[13] In addition, during fiscal year 2004, the Office of Environmental
Management offered five 5-day classes on earned value management to DOE
project directors and other individuals involved in project management.
[14] The American National Standards Institute/Electronic Industries
Association-748-1998 established 32 criteria for use of earned value
management. Earned value management is widely employed by the private
sector as a means of ensuring reliable project performance information.
Contractor project management systems, which include earned value
management systems, may have been validated by other external
validation entities, but DOE's certification review in collaboration
with the Defense Contract Management Agency verifies not only that
project management systems have earned value management systems in
place but that they are used in a correct fashion.
[15] The Defense Contract Management Agency also has assessed the
Washington Group International's project management system for the
Elimination of Weapons Grade Plutonium project in Russia, which has not
passed the Critical Decision 2 milestone.
[16] The January 2004 total cost was about $1 billion more than DOE's
original projected total cost, as noted in our report entitled Nuclear
Waste: DOE's Hanford Spent Nuclear Fuel Storage Project--Cost,
Schedule, and Management Issues, GAO/RCED-99-267 (Washington, D.C.:
Sept. 20, 1999).
[17] These 4 projects are the East Tennessee Technology Park Three
Building Deactivation and Decommissioning, K25/27 Buildings
Deactivation and Decommissioning Removal, Molten Salt Reactor
Experiment, and Oak Ridge National Laboratory Burial Grounds.
[18] These 63 projects did not include 25 Environmental Management
projects. Nineteen, or 76 percent, of the 25 Environmental Management
projects were expected to meet cost and schedule baselines.
[19] Our report entitled Nuclear Weapons: Opportunities Exist to
Improve the Budgeting, Cost Accounting, and Management Associated with
the Stockpile Life Extension Program, GAO-03-583 (Washington, D.C.:
July 28, 2003) recommended that DOE manage its weapons refurbishment
programs as projects. Although DOE agreed with this recommendation,
NNSA has not implemented it. DOE defines a project as a specific
undertaking, with defined beginning and end points, that supports a
program mission.
[20] These are the Solid Waste Stabilization and Disposition and the
Radioactive Liquid Tank Waste Stabilization and Disposition projects.
[21] U.S. Army Corps of Engineers, Independent Cost & Schedule Baseline
Review Summary Report, Hanford Waste Treatment and Immobilization Plant
(Walla Walla, Washington: May 28, 2004). Also GAO, Nuclear Waste:
Absence of Key Management Reforms on Hanford's Cleanup Project Adds to
Challenges of Achieving Cost and Schedule Goals, GAO-04-611
(Washington, D.C.: June 9, 2004).
[22] GAO, Major Acquisitions: Significant Changes Underway in DOD's
Earned Value Management Process, GAO/NSIAD-97-108 (Washington, D.C.:
May 5, 1997).
GAO's Mission:
The Government Accountability Office, the investigative arm of
Congress, exists to support Congress in meeting its constitutional
responsibilities and to help improve the performance and accountability
of the federal government for the American people. GAO examines the use
of public funds; evaluates federal programs and policies; and provides
analyses, recommendations, and other assistance to help Congress make
informed oversight, policy, and funding decisions. GAO's commitment to
good government is reflected in its core values of accountability,
integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through the Internet. GAO's Web site ( www.gao.gov ) contains
abstracts and full-text files of current reports and testimony and an
expanding archive of older products. The Web site features a search
engine to help you locate documents using key words and phrases. You
can print these documents in their entirety, including charts and other
graphics.
Each day, GAO issues a list of newly released reports, testimony, and
correspondence. GAO posts this list, known as "Today's Reports," on its
Web site daily. The list contains links to the full-text document
files. To have GAO e-mail this list to you every afternoon, go to
www.gao.gov and select "Subscribe to e-mail alerts" under the "Order
GAO Products" heading.
Order by Mail or Phone:
The first copy of each printed report is free. Additional copies are $2
each. A check or money order should be made out to the Superintendent
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or
more copies mailed to a single address are discounted 25 percent.
Orders should be sent to:
U.S. Government Accountability Office
441 G Street NW, Room LM
Washington, D.C. 20548:
To order by Phone:
Voice: (202) 512-6000:
TDD: (202) 512-2537:
Fax: (202) 512-6061:
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Web site: www.gao.gov/fraudnet/fraudnet.htm
E-mail: fraudnet@gao.gov
Automated answering system: (800) 424-5454 or (202) 512-7470:
Public Affairs:
Jeff Nelligan, managing director,
NelliganJ@gao.gov
(202) 512-4800
U.S. Government Accountability Office,
441 G Street NW, Room 7149
Washington, D.C. 20548: