Department of Energy
Major Construction Projects Need a Consistent Approach for Assessing Technology Readiness to Help Avoid Cost Increases and Delays Gao ID: GAO-07-336 March 27, 2007The Department of Energy (DOE) spends billions of dollars on major construction projects that help maintain the nuclear weapons stockpile, conduct research and development, and process nuclear waste so that it can be disposed of. Because of DOE's long-standing project management problems, GAO determined the extent to which (1) DOE's major construction projects are having cost increases and schedule delays and the major factors contributing to these problems and (2) DOE ensures that project designs are sufficiently complete before construction begins to help avoid cost increases and delays. We examined 12 DOE major projects with total costs of about $27 billion, spoke with federal and contractor officials, and reviewed project management documents.
Of the 12 DOE major projects GAO reviewed, 9 exceeded their original cost or schedule estimates, principally because of ineffective DOE project oversight and poor contractor management. Specifically, 8 of the 12 projects experienced cost increases ranging from $79.0 million to $7.9 billion, and 9 of the 12 projects were behind schedule by 9 months to more than 11 years. Project oversight problems included, among other things, inadequate systems for measuring contractor performance, approval of construction activities before final designs were sufficiently complete, ineffective project reviews, and insufficient DOE staffing. Furthermore, contractors poorly managed the development and integration of the technology used in the projects by, among other things, not accurately anticipating the cost and time that would be required to carry out the highly complex tasks involved. Even though DOE requires final project designs to be sufficiently complete before beginning construction, it has not systematically ensured that the critical technologies reflected in these designs have been demonstrated to work as intended (technology readiness) before committing to construction expenses. Specifically, only one of the five DOE project directors with projects that have recently begun or are nearing construction had systematically assessed technology readiness. The other four directors also told us that they have or will have completed prior to construction, 85 to 100 percent of their projects' final design, but they had not systematically assessed technology readiness. Proceeding into construction without also demonstrating a technology's readiness can lead to cost increases and delays. For example, one technology to be used in DOE's Waste Treatment and Immobilization Plant was not sufficiently demonstrated--that is, shown to be technologically ready for its intended application--before construction began. Consequently, the technology did not perform as expected, which resulted in about $225 million in redesign costs and schedule delays of more than 1 year. To help avoid these problems, the National Aeronautics and Space Administration (NASA) pioneered and the Department of Defense (DOD) has adopted for its projects a method for measuring and communicating technology readiness levels (TRL). Using a scale from one (basic principles observed) through nine (total system used successfully in project operations), TRLs show the extent to which technologies have been demonstrated to work as intended in the project. DOE project directors agreed that such an approach would help make technology assessments more transparent and improve stakeholder communication prior to making critical project decisions, such as authorizing construction.
RecommendationsOur recommendations from this work are listed below with a Contact for more information. Status will change from "In process" to "Open," "Closed - implemented," or "Closed - not implemented" based on our follow up work.
Director: Team: Phone:GAO-07-336, Department of Energy: Major Construction Projects Need a Consistent Approach for Assessing Technology Readiness to Help Avoid Cost Increases and Delays
This is the accessible text file for GAO report number GAO-07-336
entitled 'Department of Energy: Major Construction Projects Need a
Consistent Approach for Assessing Technology Readiness to Help Avoid
Cost Increases and Delays' which was released on March 28, 2007.
This text file was formatted by the U.S. Government Accountability
Office (GAO) to be accessible to users with visual impairments, as part
of a longer term project to improve GAO products' accessibility. Every
attempt has been made to maintain the structural and data integrity of
the original printed product. Accessibility features, such as text
descriptions of tables, consecutively numbered footnotes placed at the
end of the file, and the text of agency comment letters, are provided
but may not exactly duplicate the presentation or format of the printed
version. The portable document format (PDF) file is an exact electronic
replica of the printed version. We welcome your feedback. Please E-mail
your comments regarding the contents or accessibility features of this
document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
Report to the Subcommittee on Energy and Water Development, and Related
Agencies, Committee on Appropriations, House of Representatives:
United States Government Accountability Office:
GAO:
March 2007:
Department of Energy:
Major Construction Projects Need a Consistent Approach for Assessing
Technology Readiness to Help Avoid Cost Increases and Delays:
GAO-07-336:
GAO Highlights:
Highlights of GAO-07-336, a report to the Subcommittee on Energy and
Water Development, and Related Agencies, Committee on Appropriations,
House of Representatives
Why GAO Did This Study:
The Department of Energy (DOE) spends billions of dollars on major
construction projects that help maintain the nuclear weapons stockpile,
conduct research and development, and process nuclear waste so that it
can be disposed of. Because of DOE‘s long-standing project management
problems, GAO determined the extent to which (1) DOE‘s major
construction projects are having cost increases and schedule delays and
the major factors contributing to these problems and (2) DOE ensures
that project designs are sufficiently complete before construction
begins to help avoid cost increases and delays. We examined 12 DOE
major projects with total costs of about $27 billion, spoke with
federal and contractor officials, and reviewed project management
documents.
What GAO Found:
Of the 12 DOE major projects GAO reviewed, 9 exceeded their original
cost or schedule estimates, principally because of ineffective DOE
project oversight and poor contractor management. Specifically, 8 of
the 12 projects experienced cost increases ranging from $79.0 million
to $7.9 billion, and 9 of the 12 projects were behind schedule by 9
months to more than 11 years. Project oversight problems included,
among other things, inadequate systems for measuring contractor
performance, approval of construction activities before final designs
were sufficiently complete, ineffective project reviews, and
insufficient DOE staffing. Furthermore, contractors poorly managed the
development and integration of the technology used in the projects by,
among other things, not accurately anticipating the cost and time that
would be required to carry out the highly complex tasks involved.
Even though DOE requires final project designs to be sufficiently
complete before beginning construction, it has not systematically
ensured that the critical technologies reflected in these designs have
been demonstrated to work as intended (technology readiness) before
committing to construction expenses. Specifically, only one of the five
DOE project directors with projects that have recently begun or are
nearing construction had systematically assessed technology readiness.
The other four directors also told us that they have or will have
completed prior to construction, 85 to 100 percent of their projects‘
final design, but they had not systematically assessed technology
readiness. Proceeding into construction without also demonstrating a
technology‘s readiness can lead to cost increases and delays. For
example, one technology to be used in DOE‘s Waste Treatment and
Immobilization Plant was not sufficiently demonstrated”that is, shown
to be technologically ready for its intended application”before
construction began. Consequently, the technology did not perform as
expected, which resulted in about $225 million in redesign costs and
schedule delays of more than 1 year. To help avoid these problems, the
National Aeronautics and Space Administration (NASA) pioneered and the
Department of Defense (DOD) has adopted for its projects a method for
measuring and communicating technology readiness levels (TRL). Using a
scale from one (basic principles observed) through nine (total system
used successfully in project operations), TRLs show the extent to which
technologies have been demonstrated to work as intended in the project.
DOE project directors agreed that such an approach would help make
technology assessments more transparent and improve stakeholder
communication prior to making critical project decisions, such as
authorizing construction.
Figure: Technology Readiness Levels:
[See PDF for Image]
Source: GAO analysis of DOD data.
[End of figure]
What GAO Recommends:
GAO recommends that DOE develop a consistent approach for measuring the
readiness of critical project technologies. DOE supports GAO‘s
recommendations but suggested revisions to allow it to first conduct a
pilot application on selected projects to better understand the process
and evaluate its potential use.
[Hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-07-336].
To view the full product, including the scope and methodology, click on
the link above. For more information, contact Gene Aloise at (202) 512-
3841 or aloisee@gao.gov.
[End of section]
Contents:
Letter:
Results in Brief:
Background:
Most Major Projects Have Exceeded Original Costs and Are Years Late,
Principally Because of Ineffective DOE Project Oversight and Contractor
Management:
DOE Does Not Consistently Measure Technology Readiness to Ensure That
Critical Technologies Will Work as Intended before Construction Begins:
Conclusions:
Recommendations for Executive Action:
Agency Comments and Our Evaluation:
Appendix I: Scope and Methodology:
Appendix II: Information on the 12 Department of Energy Major Projects
Reviewed:
Appendix III: Independent Studies Reviewed:
Appendix IV: Survey Results for Primary Factors Affecting Cost and
Schedule on Nine Projects with Cost or Schedule Changes:
Appendix V: Definitions of Technology Readiness Levels:
Appendix VI: Comparison of DOD's Product Development Process with DOE's
Project Management Process:
Appendix VII: Comments from the Department of Energy:
Appendix VIII: GAO Contact and Staff Acknowledgments:
Tables:
Table 1: Changes in Estimated Total Project Cost for DOE Major
Construction Projects:
Table 2: Changes in Estimated Project Schedules for DOE Major
Construction Projects:
Table 3: Reasons for Cost Increases and Schedule Delays:
Abbreviations:
DOD: Department of Defense:
DOE: Department of Energy:
EM: Office of Environmental Management:
ITP: In-Tank Precipitation:
NASA: National Aeronautics and Space Administration:
NNSA: National Nuclear Security Administration:
PDRI: Product Definition Rating Index:
TPC: total project cost:
TRL: technology readiness level:
United States Government Accountability Office:
Washington, DC 20548:
March 27, 2007:
The Honorable Peter J. Visclosky:
Chairman:
The Honorable David L. Hobson:
Ranking Member:
Subcommittee on Energy and Water Development, and Related Agencies:
Committee on Appropriations:
House of Representatives:
The Department of Energy (DOE) spends billions of dollars on major
construction projects that, among other things, are used to help
maintain the nuclear weapons stockpile, conduct research and
development in the areas of high-energy physics and nuclear physics,
and process nuclear waste into forms suitable for longer-term storage
or permanent disposal. DOE oversees the construction of facilities
primarily at government-owned, contractor-operated sites throughout the
nation. In July 2006, DOE revised its dollar threshold defining a
construction project as "major"; it is now at $750 million, up from
$400 million when we began our review in December 2005. The following
12 projects included in our review had estimated project costs
exceeding the original threshold.[Footnote 1] The total cost of these
projects is currently estimated at about $27 billion.[Footnote 2] These
12 projects and their locations are as follows:
* Chemistry and Metallurgy Research Facility Replacement--Los Alamos
National Laboratory, Los Alamos, New Mexico.
* Depleted Uranium Hexafluoride 6 Conversion Facility--Portsmouth,
Ohio, and Paducah, Kentucky.
* Highly Enriched Uranium Materials Facility--Y-12 National Security
Complex, Oak Ridge, Tennessee.
* Linac Coherent Light Source--Stanford Linear Accelerator Center,
Menlo Park, California.
* Microsystems and Engineering Sciences Applications--Sandia National
Laboratories, Albuquerque, New Mexico.
* Mixed Oxide Fuel Fabrication Facility--Savannah River Site, Aiken,
South Carolina.
* National Ignition Facility--Lawrence Livermore National Laboratory,
Livermore, California.
* Pit Disassembly and Conversion Facility--Savannah River Site, Aiken,
South Carolina.
* Salt Waste Processing Facility--Savannah River Site, Aiken, South
Carolina.
* Spallation Neutron Source--Oak Ridge National Laboratory, Oak Ridge,
Tennessee.
* Tritium Extraction Facility--Savannah River Site, Aiken, South
Carolina.
* Waste Treatment and Immobilization Plant--Hanford Site, Richland,
Washington.
These major projects require the construction of large building
complexes and the development of innovative cleanup and other
technologies. Many of these technologies are developed for the project
or are applied in a new way. DOE project directors are responsible for
managing these major projects and overseeing the contractors that
design and construct the facilities. In doing so, project directors
follow specific departmental directives, policies, and guidance for
project management. Among these are DOE Order 413.3A and Manual 413.3-
1, which establish protocols for planning and executing a project. The
protocols require DOE projects to go through a series of five critical
decisions as they enter each new phase of work. Two of the decisions
made before construction begins are key: (1) formally approving the
project's definitive cost and schedule estimates as accurate and
complete--an approval that is to be based on a review of the project's
completed preliminary design, and (2) reaching agreement that the
project's final design is sufficiently complete and that resources can
be committed toward procurement and construction. To oversee projects
and approve these critical decisions, DOE conducts its own reviews,
often with the help of independent technical experts. In addition,
projects are regularly subject to reviews by DOE's Office of
Engineering and Construction Management and it's Office of Inspector
General, the Department of Defense's (DOD) U.S. Army Corps of
Engineers, the Defense Nuclear Facilities Safety Board, and the
National Research Council, among others.
We and others have reported over the past decade that project
management weaknesses have impaired these projects. For example, for
the Waste Treatment and Immobilization Plant, we reported that DOE's
use of a "fast-track, design-build" approach to construction--in which
design and construction activities overlap--has been problematic for
highly complex, first-of-a-kind facilities. We found that the designs
for these facilities were not sufficiently complete for construction to
begin, which has resulted in significant cost increases and schedule
delays.[Footnote 3]
In this context, we determined the extent to which (1) DOE's active
major construction projects are experiencing cost increases and
schedule delays and the key factors contributing to these problems and
(2) DOE ensures that project designs are sufficiently complete before
construction begins to help avoid cost increases and schedule
delays.[Footnote 4]
To determine the extent to which DOE projects are experiencing cost
increases or schedule delays and factors contributing to these
problems, we sent a survey to the 12 DOE directors of major projects
and reviewed the project management documents for these projects--6
projects that were above the $750 million threshold, 2 estimated to
cost between the previous $400 million threshold and $750 million, and
4 estimated to cost between $300 million and $400 million. (App. II
describes these projects.) These 12 projects are managed by DOE's
Office of Science, Office of Environmental Management, or National
Nuclear Security Administration (NNSA). We conducted site visits and
analyzed independent project studies for the 9 projects that
experienced cost increases or schedule delays. During the course of our
review, we identified a method used by the DOE project director for the
Pit Disassembly and Conversion Facility to systematically assess the
extent to which a technology is sufficiently developed for its intended
purpose. The project director based this method on a system developed
by the National Aeronautics and Space Administration (NASA). We had
previously reported on the use of a similar assessment system--
technology readiness levels (TRL), which DOD has adopted for its major
projects.[Footnote 5] We obtained and reviewed documents regarding
these two assessment systems.
To determine the extent to which DOE ensures that project designs are
sufficiently complete before construction, we reviewed in detail 5 of
the 12 projects that were approaching or had recently begun
construction. Specifically, we obtained information on the extent to
which project designs were, or are expected to be, complete before
beginning construction, and the actions DOE has taken to ensure that
technologies used in these designs are sufficiently ready to begin
construction.
Because we and others have previously expressed concern about the data
reliability of a key DOE project management tracking database--the
Project Assessment and Reporting System--we did not develop conclusions
or findings on the basis of information generated through that
system.[Footnote 6] Instead, we collected information directly from
project site offices. In addition, we spoke with officials from DOE
program offices and DOE's Office of Engineering and Construction
Management in Washington, D.C. We provided interim briefings to the
Subcommittee on the status of our work in May and September, 2006. We
performed our work between December 2005 and January 2007, in
accordance with generally accepted government auditing standards.
Appendix I contains a detailed description of our scope and
methodology.
Results in Brief:
Nine of the 12 DOE major projects we reviewed have exceeded their
original cost estimates and/or experienced schedule delays, principally
because of ineffective DOE project oversight and poor contractor
management, according to independent studies we reviewed and interviews
we conducted with DOE and contractor project officials. Specifically, 8
of the 12 projects experienced cost increases ranging from $79.0
million to $7.9 billion, and 9 of the 12 projects are behind schedule
by 9 months to more than 11 years. Major factors cited for these cost
increases and delays included the following:
* Ineffective DOE project oversight. For all 9 projects experiencing
cost increases or schedule delays, poor DOE oversight was a key
contributing factor. Project oversight problems included inadequate
systems for measuring contractor performance, approval of construction
activities before final designs were sufficiently complete, ineffective
project reviews, and insufficient DOE staffing and project management
experience.
* Poor contractor management. Eight of the 9 major projects experienced
cost increases and/or schedule delays, in part because contractors did
not effectively manage the development and integration of the
technology used in the projects, including not accurately anticipating
the cost and time that would be required to carry out the highly
complex tasks involved. For example, the National Ignition Facility has
had over $1 billion in cost increases and years of schedule delays
owing in part to technology integration problems, according to the DOE
project director. Other examples of poor contractor performance
included inadequate quality assurance for the Highly Enriched Uranium
Materials Facility, which resulted in concrete work that did not meet
design specifications. The subsequent suspension of construction
activities and rework added to the project's estimated cost and
schedule.
DOE officials also explained that a now-defunct policy may have
contributed to increased costs and delays for several projects we
examined. Until 2000, DOE required contractors to prepare cost and
schedule estimates early in the project, before preliminary designs
were completed. These estimates were used to establish a baseline for
measuring contractor performance and tracking any cost increases or
schedule delays. However, these estimates often were based on early
conceptual designs and, thus, were subject to significant change as
more detailed designs were developed. To improve the reliability of
these estimates, DOE issued a new order in October 2000 that required
the preparation of a cost estimate range at the start of preliminary
design, and delayed the requirement for a definitive cost and schedule
baseline estimate until after preliminary design was completed.
Consequently, DOE officials explained, the new policy should result in
improved estimates and a more accurate measure of cost and schedule
performance.
Even though DOE requires final project designs to be sufficiently
complete before beginning construction, it has not systematically
ensured that the critical technologies reflected in these designs have
been demonstrated to work as intended (technology readiness) before
committing to construction expenses. Only one of the five DOE directors
with projects that have recently begun, or are nearing construction,
had systematically assessed technology readiness. The other four
directors also told us that they have or will have completed prior to
construction, 85 to 100 percent of their projects' final design, but
they had not systematically assessed for technology readiness. Lack of
technology readiness can result in cost overruns and schedule delays.
For example, technology used in a subsystem intended to prepare
radioactive material for processing in DOE's Waste Treatment and
Immobilization Plant was not fully developed and did not work as
expected after construction had already begun, resulting in redesign
costs of about $225 million and over 1 year in schedule delays.
To effectively assess technology readiness, NASA pioneered and DOD has
adopted a process for measuring and communicating technology readiness
for first-of-a-kind technology applications. This process uses a nine-
point scale for assessing TRLs. Using this scale, a technology would
receive a higher TRL value (e.g., TRL 7) if it has been successfully
demonstrated in an operational environment, compared with a technology
that has been demonstrated only in a laboratory test (e.g., TRL 4).
Several DOE project directors we spoke with agreed that a consistent,
systematic method for assessing technology readiness would help
standardize terminology, make technology assessments more transparent,
and help improve communication among project stakeholders before they
make critical project decisions.
To improve oversight and decision making for DOE's major construction
projects, we are recommending that the Secretary of Energy evaluate and
consider adopting a disciplined and consistent approach to assessing
TRLs for projects with critical technologies.
DOE provided comments to us based on a draft of the report. DOE agreed
with our recommendations but suggested revisions that would first allow
them to conduct a pilot application on selected projects to better
understand the technology readiness assessment process and evaluate its
potential use. We revised our recommendations as appropriate. DOE
suggested that our report is too narrowly focused on technology
assessment, and that we inappropriately calculated cost increases and
schedule delays using preliminary estimates that were only intended for
internal DOE planning. We believe that our recommendations were
justifiably based on our finding that DOE has not systematically
ensured that project designs, including critical technologies reflected
in these designs, have been demonstrated to work as intended prior to
construction. We also believe it was appropriate, when necessary, to
measure cost and schedule changes using the initial estimates that were
developed at the end of conceptual design, as specified in DOE's
project management policy in effect prior to 2000. We note that these
estimates were, in some instances, the only initial estimates available
and had been used by DOE to inform the Congress of the estimated cost
and schedule of the projects while it was seeking initial project
funding. We also incorporated technical changes in this report where
appropriate on the basis of detailed comments provided by DOE.
Background:
To meet its diverse missions, DOE pays its contractors billions of
dollars each year to implement hundreds of projects, ranging from
hazardous waste cleanups at sites in the weapons complex to the
construction of scientific facilities. Many of these complex, unique
projects are designed to meet defense, energy research, environmental,
and fissile materials disposition goals. They often rely on
technologies that are unproven in operational conditions. In recent
years, DOE's budget has been dominated by the monumental task of
environmental restoration and waste management to repair damage caused
by the past production of nuclear weapons.
DOE has long had a poor track record for developing designs and cost
estimates and managing projects. We reported in 1997 that from 1980 to
1996, 31 of DOE's 80 major projects were terminated prior to
completion, after expenditures of over $10 billion; 15 of the projects
were completed, but most of them were finished behind schedule and with
cost overruns; and the remaining 34 ongoing projects also were
experiencing schedule slippage or cost overruns.[Footnote 7] In
addition, for over a decade, DOE's Office of Inspector General, the
National Academy of Sciences, and others have identified problems with
DOE's management of major construction projects. Projects were late or
never finished; project costs increased by millions and sometimes
billions of dollars; and environmental conditions at the sites did not
significantly improve. According to the National Research
Council,[Footnote 8] DOE's construction and environmental remediation
projects take much longer and cost about 50 percent more than
comparable projects by other federal agencies or projects in the
private sector.[Footnote 9] A 2004 assessment of departmental project
management completed by the Civil Engineering Research Foundation
recommended, among other things, that DOE develop a core group of
highly qualified project directors and require peer reviews for first-
of-a-kind and technically complex projects when the projects'
preliminary baselines are approved.[Footnote 10]
To address project management issues, DOE began a series of reforms in
the 1990s that included efforts to strengthen project management
practices. To guide these reforms, the department formed the Office of
Engineering and Construction Management in 1999. The reforms instituted
to date have included planning, organizing, and tracking project
activities, costs, and schedules; training to ensure that federal
project managers had the required expertise to manage projects;
increasing emphasis on independent reviews; and strengthening project
reporting and oversight.
Most Major Projects Have Exceeded Original Costs and Are Years Late,
Principally Because of Ineffective DOE Project Oversight and Contractor
Management:
The estimated costs of many of the DOE major construction projects we
reviewed have significantly exceeded original estimates and schedules
have slipped. On the basis of our analysis of independent project
studies and interviews with project directors, cost growth and schedule
slippage occurred principally because of ineffective DOE project
oversight and poor contractor project management. Furthermore,
unreliable initial cost and schedule estimates resulting from a now-
defunct policy may have been a contributing factor, according to DOE
project officials. Although external factors, such as additional
security and safety requirements, contributed to cost growth and
delays, the management of these requirements was complicated by
ineffective and untimely DOE communication and decision making.
Eight of the 12 DOE projects we reviewed had increases in estimates of
total project cost (TPC) ranging from $79.0 million to $7.9 billion. As
table 1 shows, the percentage of cost increase for these 8 projects
ranged from 2 percent to over 200 percent.
Table 1: Changes in Estimated Total Project Cost for DOE Major
Construction Projects:
Dollars in millions.
Project: Mixed Oxide Fuel Fabrication Facility[C];
Initial total project cost (TPC) estimate[A]: $1,400;
Current TPC estimate: $4,699;
Percentage increase[B]: 205%.
Project: Waste Treatment and Immobilization Plant;
Initial total project cost (TPC) estimate[A]: 4,350;
Current TPC estimate: 12,263;
Percentage increase[B]: 143.
Project: Highly Enriched Uranium Materials Facility;
Initial total project cost (TPC) estimate[A]: 251;
Current TPC estimate: 549;
Percentage increase[B]: 102.
Project: National Ignition Facility;
Initial total project cost (TPC) estimate[A]: 1,199;
Current TPC estimate: $2,248;
Percentage increase[B]: 59.
Project: Salt Waste Processing Facility;
Initial total project cost (TPC) estimate[A]: 440;
Current TPC estimate: 680[D];
Percentage increase[B]: 50.
Project: Pit Disassembly and Conversion Facility[C];
Initial total project cost (TPC) estimate[A]: 1,700;
Current TPC estimate: 2,694;
Percentage increase[B]: 40.
Project: Tritium Extraction Facility;
Initial total project cost (TPC) estimate[A]: 384;
Current TPC estimate: 506;
Percentage increase[B]: 15.
Project: Spallation Neutron Source;
Initial total project cost (TPC) estimate[A]: 1,333;
Current TPC estimate: 1,412;
Percentage increase[B]: 2.
Project: Depleted Uranium Hexafluoride 6 Conversion Facility;
Initial total project cost (TPC) estimate[A]: 346;
Current TPC estimate: 346;
Percentage increase[B]: 0.
Project: Chemistry and Metallurgy Research Facility Replacement;
Initial total project cost (TPC) estimate[A]: 837;
Current TPC estimate: 837;
Percentage increase[B]: 0.
Project: Microsystems and Engineering Sciences Applications;
Initial total project cost (TPC) estimate[A]: 518;
Current TPC estimate: 518;
Percentage increase[B]: 0.
Project: Linac Coherent Light Source;
Initial total project cost (TPC) estimate[A]: 379;
Current TPC estimate: 379;
Percentage increase[B]: 0.
Source: GAO analysis of DOE data.
[A] In 2000, DOE changed its requirements for establishing initial cost
and schedule estimates. Prior to 2000, these estimates were established
at the end of conceptual design. After 2000, DOE required initial
estimates to be completed later in the project--at the end of
preliminary design. For projects beginning prior to 2000, and for
projects beginning after 2000 that had not yet completed preliminary
design, we used the TPC estimates prepared after conceptual design. For
additional details on our methodology, see appendix I.
[B] We calculated the percentages of cost increases on the basis of
constant 2007 dollars to make them comparable across projects and to
show real increases in cost while excluding increases due to inflation.
[C] NNSA officials, in commenting on our draft report, stated that
initial and current cost estimates for the Mixed Oxide Fuel Fabrication
Facility and the Pit Disassembly and Conversion Facility should not be
used in this analysis because neither project has an approved budget
quality baseline. Nevertheless, we included the estimates in this
analysis because both projects have been in an extended period of
project design, without an approved budget-quality baseline, for about
10 years, and the estimates provided here are the only estimates
available.
[D] Estimate may change when DOE approves contractor's revised TPC in
2007.
[End of table]
In addition, as shown in table 2, 9 of the 12 projects experienced
schedule delays ranging from 9 months to more than 11 years. Of the 9
projects, 7 had schedule delays of at least 2 years or more.
Table 2: Changes in Estimated Project Schedules for DOE Major
Construction Projects:
Project: Pit Disassembly and Conversion Facility;
Year mission need was approved: 1997;
Initial completion date estimate: 06/2005;
Current completion date estimate: 03/2017;
Schedule delay as of February 2007: 11 years, 9 months.
Project: Mixed Oxide Fuel Fabrication Facility;
Year mission need was approved: 1997;
Initial completion date estimate: 09/2004;
Current completion date estimate: 03/2016;
Schedule delay as of February 2007: 11 years, 6 months.
Project: Waste Treatment and Immobilization Plant;
Year mission need was approved: 1995;
Initial completion date estimate: 07/2011;
Current completion date estimate: 11/2019;
Schedule delay as of February 2007: 8 years, 4 months.
Project: National Ignition Facility;
Year mission need was approved: 1993;
Initial completion date estimate: 10/2003;
Current completion date estimate: 03/2009;
Schedule delay as of February 2007: 5 years, 5 months.
Project: Depleted Uranium Hexafluoride 6 Conversion Facility[A];
Year mission need was approved: 2000;
Initial completion date estimate: 03/ 2006;
Current completion date estimate: 06/2008;
Schedule delay as of February 2007: 2 years, 3 months.
Project: Salt Waste Processing Facility;
Year mission need was approved: 2001;
Initial completion date estimate: 07/2009;
Current completion date estimate: 09/2011[B];
Schedule delay as of February 2007: 2 years, 2 months.
Project: Tritium Extraction Facility;
Year mission need was approved: 1995;
Initial completion date estimate: 06/2005;
Current completion date estimate: 07/2007;
Schedule delay as of February 2007: 2 years, 1 month.
Project: Highly Enriched Uranium Materials Facility;
Year mission need was approved: 1999;
Initial completion date estimate: 04/2008;
Current completion date estimate: 03/2010;
Schedule delay as of February 2007: 1 year, 11 months.
Project: Spallation Neutron Source;
Year mission need was approved: 1996;
Initial completion date estimate: 09/2005;
Current completion date estimate: 06/2006[C];
Schedule delay as of February 2007: 9 months.
Project: Chemistry and Metallurgy Research Facility Replacement;
Year mission need was approved: 2002;
Initial completion date estimate: 03/ 2014;
Current completion date estimate: 03/2014;
Schedule delay as of February 2007: Not applicable.
Project: Microsystems and Engineering Sciences Applications;
Year mission need was approved: 1999;
Initial completion date estimate: 01/ 2009;
Current completion date estimate: 01/2009;
Schedule delay as of February 2007: Not applicable.
Project: Linac Coherent Light Source;
Year mission need was approved: 2001;
Initial completion date estimate: 03/2009;
Current completion date estimate: 03/2009;
Schedule delay as of February 2007: Not applicable.
Source: GAO analysis of DOE data.
[A] This project reported a schedule delay but did not report an
increase in the estimated total project cost (TPC). According to the
DOE project director, the original cost estimate was probably too high
and was not well supported.
[B] According to DOE officials, schedule may slip further when the
contractor submits its revised TPC to DOE in July 2007.
[C] Project was completed on this date. Transition to operations has
begun.
[End of table]
As table 3 shows, ineffective DOE project oversight and poor contractor
management were frequently cited reasons for cost increases and
schedule delays for the projects we reviewed, according to our review
of independent studies of the 9 projects experiencing cost growth and
schedule delays and our follow-up interviews with DOE project
directors. Project officials, in commenting on our draft report, were
concerned that table 3 might misrepresent the overall successful
execution and completion of some projects, such as the Spallation
Neutron Source, and that some problems may have already been addressed.
Nevertheless, to clarify our main purpose for table 3, our intent is to
show broad categories of major reasons for cost increases and schedule
delays, regardless of when they occurred or whether they have been
adequately addressed.
Table 3: Reasons for Cost Increases and Schedule Delays:
Project: Depleted Uranium Hexafluoride 6 Conversion Facility;
DOE project oversight: X;
Poor contractor management: X;
External factors (e.g., safety/security): X.
Project: Highly Enriched Uranium Materials Facility;
DOE project oversight: X;
Poor contractor management: X;
External factors (e.g., safety/security): X.
Project: Mixed Oxide Fuel Fabrication Facility;
DOE project oversight: X;
Poor contractor management: X;
External factors (e.g., safety/ security): X.
Project: National Ignition Facility;
DOE project oversight: X;
Poor contractor management: X;
External factors (e.g., safety/security): [Empty].
Project: Pit Disassembly and Conversion Facility;
DOE project oversight: X;
Poor contractor management: X;
External factors (e.g., safety/security): X.
Project: Salt Waste Processing Facility;
DOE project oversight: X;
Poor contractor management: [Empty];
External factors (e.g., safety/ security): X.
Project: Spallation Neutron Source;
DOE project oversight: X;
Poor contractor management: X;
External factors (e.g., safety/security): [Empty].
Project: Tritium Extraction Facility;
DOE project oversight: X;
Poor contractor management: X;
External factors (e.g., safety/security): X.
Project: Waste Treatment and Immobilization Plant;
DOE project oversight: X;
Poor contractor management: X;
External factors (e.g., safety/security): X.
Project: Total;
DOE project oversight: 9;
Poor contractor management: 8;
External factors (e.g., safety/security): 7.
Source: GAO analysis of independent project studies and interviews with
DOE project directors (a list of the project studies we reviewed is
included in app. III).
[End of table]
The DOE project oversight issues mentioned in table 3 include the
following:
* inadequate systems for measuring contractor performance;
* approval of construction activities before final designs were
sufficiently complete;
* ineffective project reviews;
* insufficient DOE staffing and experience;
* inadequate use of project management controls;
* lack of headquarters assistance and oversight support of field
project directors;
* failure to detect contractor performance problems, including
inadequate federal inspection activities; and:
* poor government cost estimates, including inadequate funding for
contingencies.
DOE's lack of adequate systems to measure contractor performance was
cited in a December 2005 DOE Inspector General review of the Mixed
Oxide Fuel Fabrication Facility. The Inspector General criticized DOE's
NNSA for failing to approve a baseline against which to measure
contractor performance and relying on outdated cost plans.[Footnote 11]
According to the report, NNSA relied on confusing and misleading
information detailed in the monthly project reports to monitor progress
and track costs--reports that the contractor acknowledged as being
"useless for evaluating performance or managing the project."
Furthermore, although the contractor reported unfavorable cost and
schedule variances for months, these variances were inaccurate and
meaningless because performance was being compared against a 2-year-old
plan. NNSA, in commenting on our draft report, stated that project
oversight and contractor management problems identified in previous
GAO, Inspector General, and other independent assessments have led to
extensive improvements to the project, and that major findings
identified during a recent independent review have been successfully
addressed.
Similarly, DOE's approval of construction activities before final
designs were sufficiently complete has contributed significantly to
project cost growth and schedule delays. As we have previously
reported, the accelerated fast-track, design-build approach used for
the Waste Treatment and Immobilization Plant, a highly complex first-
of-a-kind nuclear facility, resulted in significant cost increases and
schedule delays.[Footnote 12] DOE also allowed the contractor on
another project, the Tritium Extraction Facility, to begin construction
before the final design was completed to meet schedule commitments.
According to a 2002 DOE Inspector General report on the
project,[Footnote 13] this revised acquisition strategy of simultaneous
design and construction directly resulted in at least $12 million in
project overruns.
The contractor management issues mentioned in table 3 include the poor
management of technological challenges, among other contractor
performance issues, according to DOE project directors. Cost increases
and schedule delays for 6 of the 9 projects were due in part to
contractors' poor management of the development and integration of
technologies used in project designs by, among other things, not
accurately anticipating the cost and time that would be required to
carry out the highly complex tasks involved.[Footnote 14] For example:
* The National Ignition Facility had over $1 billion in cost overruns
and years of schedule delays, in large part because of technology
integration problems. The requirements for the National Ignition
Facility--the use of 192 high-power laser beams focused on a single
target in a "clean room" environment--had not been attempted before on
such a large scale. According to the DOE project director, early
incorrect assumptions about the original facility design and the amount
of work necessary to integrate the technologies and assemble the
technical components contributed to about half of the project's cost
increases and schedule delays.
* The design of the Mixed Oxide Fuel Fabrication Facility has presented
technical challenges in adapting the design of a similar plant in
France to the design needs of this project. Although the technological
challenge related to adopting the process designs from the French
designs was not the primary contributor to the project's cost increases
and schedule delays, according to NNSA officials, it has affected the
project's complexity. The basic technology--combining plutonium oxide
with depleted uranium to form fuel assemblies for use in commercial
power reactors--has been previously demonstrated in France. However,
the DOE project director told us that the DOE facility design must,
among other things, account for processing surplus weapon-grade
plutonium, a different type of material than processed in the French
facility, and must be adapted to satisfy U.S. regulatory and other
local requirements. In addition, the DOE facility faced the
technological challenge of reducing the scale of components used in the
French facility. Although definitive cost estimates are not yet
available, expected costs for completing this project have grown by
about $3.3 billion since 2002, and the schedule has been extended by
more than 11 years, in part because the contractor did not initially
understand the project's complexity and underestimated the level of
effort needed to complete the work. NNSA explained that the capability
of the reference plants currently in operation in France, and by
extension, the Mixed Oxide Fuel Fabrication Facility process design, is
currently being demonstrated by several prototype fuel assemblies
manufactured with weapon-grade plutonium oxide, which are currently
being successfully used in a reactor in South Carolina.
* For the Waste Treatment and Immobilization Plant, a technology
application used on the project had not been tested before
construction. Filters, widely used in the water treatment industry,
were being designed for the project to concentrate and remove
radioactive particles in liquid waste, a new application for the
filters. Although tests are currently under way to demonstrate the
effectiveness of this application, project officials conceded that
these filters may still not be appropriate for the project.
Other contractor performance problems are illustrated by two examples.
First, DOE cited the contractor working on the Highly Enriched Uranium
Materials Facility for inadequate quality assurance that resulted in
concrete work that did not meet design specifications. The subsequent
suspension of construction activities and rework added to the project's
estimated cost and schedule. Second, the DOE project director of the
Depleted Uranium Hexafluoride 6 Conversion Facility told us that the
project was delayed 2 years because the contractor (1) did not have
experience in government contracts, (2) underestimated the design
effort needed, and (3) failed to properly integrate the operations of
three separate organizations it managed.
As table 3 shows, external factors were cited as also contributing to
cost growth and schedule delays, such as additional work to implement
requirements for higher levels of safety and security in project
operations, among other things. For example, design rework for 4 of the
projects occurred in response to external safety oversight
recommendations by the Defense Nuclear Facilities Safety Board that
large DOE construction projects meet a certain level of personnel
safety, and that their designs be robust enough to withstand certain
seismic events. In addition, owing to new security requirements
implemented after September 11, 2001, project officials on the Highly
Enriched Uranium Materials Facility had to redesign some aspects of the
project to ensure that heightened security measures were addressed.
While DOE faced additional requirements for safety and security, it did
not always reach timely decisions on how to implement these
requirements, which contributed significantly to cost increases and
schedule delays for the Salt Waste Processing Facility. The DOE project
director for this project told us the Defense Nuclear Facilities Safety
Board had expressed concerns in June 2004, 5 months after the
preliminary design was started, that the facility design might not
ensure nuclear wastes would be adequately contained in the event of
earthquakes. However, DOE did not decide how to address this concern
until 17 months later, as the project continued to move forward with
the existing project design. According to the project director, better
and more timely discussions between site officials and headquarters to
decide on the actions needed to adequately address these safety and
security requirements might have hastened resolution of the problem,
and up to 1 year of design rework might have been avoided. The delay,
the director told us, added $180 million to the total project cost and
extended the schedule by 26 months. In commenting on our draft report,
EM officials noted that it is now requiring a more rigorous safety
analysis earlier in the decision-making process.
Other external factors also contributed significantly to cost increases
and delays for 2 interrelated projects we reviewed--the Mixed Oxide
Fuel Fabrication Facility and the Pit Disassembly and Conversion
Facility. Project officials for these projects told us that 25 to 50
percent of the cost increases and over 70 percent of the schedule
delays they experienced were the direct result of Office of Management
and Budget funding constraints and restrictions resulting from
international agreements with Russia. That is, work that is delayed to
a subsequent year because of funding constraints and other work
restrictions can delay project completion, which likely increases total
project costs. Similarly, Office of Science officials, commenting on
our draft report, stated that external factors caused the largest
percentage cost increase and schedule delay for the Spallation Neutron
Source, including a reduced level of funding appropriated at a time
when project activities and costs were increasing considerably.
However, congressional funding was reduced in fiscal year 2000 because
of concerns about poor project oversight and management in the early
stages of this project.
DOE officials also explained that a now-defunct policy may have
contributed to increased costs and delays for several projects we
examined. Until 2000, DOE required contractors to prepare cost and
schedule estimates early in the project, before preliminary designs
were completed. These estimates were used to establish a baseline for
measuring contractor performance and tracking any cost increases or
schedule delays. However, these estimates often were based on early
conceptual designs and, thus, were subject to significant change as
more detailed designs were developed. To improve the reliability of
these estimates, DOE issued a new order in October 2000 that required
the preparation of a cost estimate range at the start of preliminary
design, and delayed the requirement for a definitive cost and schedule
baseline estimate until after the preliminary design was completed.
Consequently, DOE officials explained, the new policy should result in
improved estimates and a more accurate measure of cost and schedule
performance.
We also sent a survey to DOE project directors for all 12 projects
asking them to identify key events that led to the greatest cost
increases or schedule delays, and the major factors contributing to
these key events. However, no individual factors were identified as
being major contributors to the cost increases or schedule delays. In
responding to our survey, DOE project directors cited several factors
that affected changes in cost and schedule. However, when asked to rate
the relative significance of these factors for their impact on cost and
schedule changes, the project directors generally did not judge them to
be significant contributors to the changes. The most frequently cited
factors were:
* an absence of open communication, mutual trust, and close
coordination;
* changes in "political will" during project execution (e.g., project
changes resulting from political decisions, both internal and external
to the project);
* interruptions in project funding; and:
* project managers' lack of adequate professional experience.
(For detailed survey results covering these four factors, see app. IV.)
In contrast to the cost increases and schedule delays incurred on most
of the projects we reviewed, 3 projects had not yet experienced cost
increases or schedule delays--Microsystems and Engineering Sciences
Applications, the Linac Coherent Light Source, and the Chemistry and
Metallurgy Research Facility Replacement. DOE project officials
identified key conditions that they believed helped avoid those cost
increases and delays. These conditions included:
* active oversight--that is, the DOE project directors were never
"blindsided" by contractor issues;
* a lack of technological complexity;
* an effective system to measure contractor performance;
* reliable cost estimates;
* effective communication with and integration of all stakeholders;
and:
* sustained leadership.
However, we observed that the Linac Coherent Light Source and the
Chemistry and Metallurgy Research Facility Replacement facilities are
still in a relatively early stage in the project development process,
and thus it may be too early to gauge the overall success of either
project. Additionally, because none of these 3 projects are highly
technologically complex, they may be less susceptible to the types of
problems associated with other projects we reviewed that experienced
cost increases and delays.
DOE Does Not Consistently Measure Technology Readiness to Ensure That
Critical Technologies Will Work as Intended before Construction Begins:
Although DOE requires its final designs to be sufficiently complete
before beginning construction, it has not systematically ensured that
the critical technologies reflected in project designs are
technologically ready. Recognizing that a lack of technology readiness
can result in cost overruns and schedule delays, other federal
agencies, such as NASA and DOD, have issued guidance for measuring and
communicating technology readiness.
DOE Does Not Consistently Assess Technology Readiness:
Only 1 of the 5 projects we reviewed to determine how DOE ensures that
project designs are sufficiently complete before construction--
projects that were approaching or had recently begun construction--had
a systematic assessment of technology readiness to determine whether
the project components would work individually or collectively as
expected in the intended design.[Footnote 15] Specifically, the DOE
project director for the Pit Disassembly and Conversion Facility
systematically measured and assessed readiness levels for each critical
component of the overall project.[Footnote 16] The assessment was based
on a method developed by NASA, that is, rating each technology from 0
to 10 in terms of relative maturity. Because the project has not yet
begun construction, we could not determine whether the technology
readiness assessment has helped project managers to avoid cost
increases or schedule delays during construction. However, according to
DOE and contractor officials responsible for the project, the
assessment helped focus management attention during project design on
critical technologies that may require additional resources to ensure
that they are sufficiently ready before construction begins. In
reviewing the assessment, however, we noted that project officials had
not updated the assessment tool for this project for over 3 years.
DOE's project director acknowledged the delay in updating the
assessment and responded that he plans to begin updating the assessment
annually.
The other 4 projects did not have systematic assessments of
technological readiness. Therefore, the risk associated with the
technology may not be clearly and consistently understood across all
levels of management. Formally approving the project's cost and
schedule estimates as accurate and complete, or proceeding into
construction, without having clearly assessed evidence of technology
readiness can result in cost overruns and schedule delays.
DOE's experience with the Waste Treatment and Immobilization Plant is a
case in point. Specifically, technology known as "pulse jet
mixers"[Footnote 17] was used in the design of a subsystem intended to
prepare radioactive material for processing. However, this technology
had not been used previously in this application, and it did not work
in tests as expected, even after construction had already begun.
Consequently, DOE incurred about $225 million in redesign costs and
over 1 year in schedule delays, according to the DOE project director.
Over the past several years, we and others have stressed the importance
of assessing technology readiness to complete projects successfully,
while avoiding cost increases and schedule delays. Specifically, by
1999, we reported that organizations using best practices recognize
that delaying the resolution of technology problems until production or
construction can result in at least a 10-fold cost increase.[Footnote
18] Furthermore, we reported that delaying the resolution until after
the start of production could increase costs by 100-fold. Reporting on
similar concerns, the National Research Council has identified factors
common to large construction projects--in the areas of cost, schedule,
and scope--that help to ensure projects are completed
successfully.[Footnote 19] Among key technical conditions for defining
project scope, the council stated, is a project plan that is based on
employing the best available, state-of-the-art technology, but not
experimental or unproven technology. As such, employing a consistent,
systematic method for measuring the extent to which technology is still
experimental or unproven is of critical importance.
An assessment of technology readiness is even more crucial at certain
points in the life of a project--particularly as DOE decides to accept
a project's (1) preliminary design and formally approve the project's
cost and schedule estimates as accurate and complete and (2) final
design as sufficiently complete so that resources can be committed
toward procurement and construction. Proceeding through these critical
decision points without a credible and complete technology readiness
assessment can lead to problems later in the project. Specifically, if
DOE proceeds with the project when technologies are not yet ready,
there is less certainty that the technologies specified in the
preliminary or final designs will work as intended. Project managers
may then need to modify or replace these technologies to make them work
properly, which can result in costly and time-consuming redesign work.
Moreover, modifying the design of a facility after construction has
already begun can be expensive and time consuming. First, changes to an
already designed work plan are not necessarily subject to competition
because the new work can occur through "change orders"--that is,
modifications to existing contracts. These change orders can be
expensive, according to DOE project directors. Second, worker
productivity can be lost if, for example, extra downtime results from
delays to interrelated construction work. Finally, tearing down and
rebuilding items already constructed, such as concrete floors, walls,
and doors, might be necessary to accommodate a design change.
DOE's experience in the predecessor project to the Salt Waste
Processing Facility--the In-Tank Precipitation (ITP) project process--
at the Savannah River Site illustrates the potential consequences of
proceeding with technology that is not sufficiently ready. As we
reported in 2000, the ITP project was selected in 1983 as the preferred
method for separating highly radioactive material from 34 million
gallons of liquid stored at the Savannah River site--a step considered
necessary to effectively handle this large quantity of waste.[Footnote
20] A 1983 test using the ITP technology on a tank containing 500,000
gallons of waste resulted in a significant buildup of benzene--a highly
explosive and hazardous compound. The buildup of benzene was more than
the tank instruments could register. Nevertheless, project managers
decided to proceed with the project. In 1985, DOE estimated that it
would take about 3 years and $32 million to construct the ITP facility.
After a number of delays, the ITP facility was constructed and began
start-up operations in 1995, which were halted because of safety
concerns about the amount of benzene that the facility generated. In
1998, after about a decade of delays and costs of almost $500 million,
DOE suspended the project because it did not work as safely and
efficiently as designed. This suspension put an effective remedy for
treating high-level waste at the Savannah River Site years behind
schedule. DOE then directed its contractor to begin a process to
identify and select an alternative technology, which has developed into
the current project intended to treat this waste--the Salt Waste
Processing Facility project.
In response to our concerns about the 4 projects without systematic
assessments of technology readiness, DOE project directors explained
that they have alternative methods for assessing readiness. They are
required to submit a project execution plan, which includes an
assessment of risks, including technological risks, and a plan for
mitigating risks. They also rely upon independent reviews, including
extensive design reviews, before making critical decisions to accept
designs, and cost and schedule estimates, or to proceed with
construction. For example, DOE's Office of Engineering and Construction
Management formally reviews major projects in an effort to ensure that
the designs are sufficiently complete to begin construction.
Specifically, an external independent readiness review is performed,
often using the services of various independent technical experts,
that, at a minimum, is intended to verify the readiness of the project
to proceed into construction or to identify remedial action. Finally,
several DOE project directors stated that they intentionally have
avoided using fast-track, design-build approaches because of the many
problems it posed for the Waste Treatment and Immobilization Plant
project. The DOE project directors of the 5 DOE projects that are
nearing, or have recently begun construction, told us they have
completed, or expect to complete prior to construction, 85 to 100
percent of their projects' final design.
In addition to following the more standard approaches for managing
projects, such as preparing risk assessment plans, some DOE offices
have developed their own tools for assessing the readiness of projects.
For example, DOE's Office of Environmental Management (EM) uses a
Product Definition Rating Index (PDRI) as a tool to assess how well a
project is planned, and whether it is ready to proceed to the next
project phase. Project elements rated include cost, schedule, scope/
technical, management planning and control, and external factors. Among
the 77 project elements rated, 2 involve technology--the identification
of technology development requirements, and the testing and evaluation
of the technology to be used. While the project technologies are
collectively given a ranking with this tool, the PDRI does not
represent a rigorous examination of the demonstrated readiness of each
critical technology for its application in the project. Furthermore,
not all EM projects we examined were using this tool.
DOE's design reviews, risk assessments, and other actions to monitor
design completion are extensive and certainly have merit. However, we
found that these actions alone do not provide consistent and
transparent assurance that all technologies are sufficiently ready
because they do not use a consistent and systematic method of
measurement. DOE's project design reviews, for example, do not always
clearly distinguish between technology that has been demonstrated to
work as expected in the intended design versus a judgment that the
technology has potential for reaching a specific level of readiness.
The external review of the technologies for the Mixed Oxide Fuel
Fabrication Facility illustrates the shortfalls in DOE's current
approach to assessing technology readiness and communicating the
results of those assessments.[Footnote 21] The report concluded, among
other things, that the method chosen by the contractor is the most
rigorous and comprehensive, and should result in the most successful
technology transfer possible. Furthermore, the review team was very
impressed with the rigor with which designs and design changes were
being managed, finding ample evidence verifying that the exact design
process used by the French was being transferred to the United States
facility. Although the external reviewers seemed to be impressed with
many aspects of the design transfer, and concluded that the
technologies should not be problematic, they had identified some key
concerns about technology readiness in the body of their final report.
The reviewers did not explain how they reconciled their conclusion with
their concerns. To reconcile these differences, we obtained several
clarifying statements from DOE's project director, technical experts,
and one of the study's authors. These clarifying statements appear to
support the reviewers' conclusions. However, without these statements,
the level of technological readiness was not readily evident because
the independent review lacked consistent, systematic criteria and a
method for measuring the degree of readiness or clearly communicating
assessment results, and the review was not transparent.
DOE does not consistently assess technology readiness of project
technologies because its project management guidance lacks
comprehensive standards for systematically measuring and communicating
the readiness of project technologies. Specifically, DOE lacks
consistent metrics for determining technology readiness departmentwide,
terminology to facilitate effective communication, and oversight
protocols for reporting and reviewing technology readiness levels. DOE
project management guidance is contained in two key documents--DOE
Order 413.3A and Manual 413.3-1. Although the manual requires final
designs to be sufficiently complete before beginning construction, it
does not specify how technologies reflected in project designs are to
be assessed for readiness--to determine that they have been
sufficiently demonstrated to work as intended. Consequently, critical
decisions made without standard measures are susceptible to varying
interpretations of the actual technology readiness attained and the
level needed for a project to proceed, which can easily vary among
projects and among officials within a single project.
Other Federal Agencies Use a Standard Method for Measuring and
Communicating Technology Readiness:
Other federal agencies have recognized the importance of ensuring that
technologies have been sufficiently demonstrated for their intended
purpose and have issued standard guidance for measuring and
communicating TRLs. In particular, recognizing the need to measure the
readiness level of project technologies, NASA began using a systematic
method of measurement in the mid-1990s. NASA incorporated a structured
TRL approach into guidance on integrated technology planning.
Similarly, to improve DOD management of risk and technology
development, the Deputy Under Secretary of Defense (Science and
Technology) officially endorsed, in a July 2001 memorandum, the use of
TRLs in new major programs. In 2002, DOD issued mandatory procedures
for major defense acquisition programs and major automated information
system acquisition programs, which identified technology readiness as a
principal element of program risk. The procedures require the military
services' science and technology officials to conduct a systematic
assessment of critical technologies that are identified in major weapon
systems programs before starting engineering and manufacturing
development and production. Using TRLs is the preferred method, and
approval must be obtained from the Deputy Under Secretary if an
equivalent alternative method is used, according to the Deputy Under
Secretary's memorandum. Importantly, the procedures stated that TRLs
are a measure of demonstrated technical maturity--they do not discuss
the probability of occurrence (i.e., the likelihood of attaining
required maturity) or the impact of not achieving technology maturity.
Both NASA and DOD use a nine-point scale to measure technology
readiness, from a low of TRL 1 (basic principles observed) to a high of
TRL 9 (total system used successfully in project operations). (App. V
contains the definitions of these nine TRLs.) For example, a subsystem
prototype that has been successfully demonstrated in an operational
environment would receive a higher TRL value (i.e., TRL 7) than a
technological component that has been demonstrated in a laboratory test
(i.e., TRL 4). In our previous work, we recommended to the Secretary of
Defense that key project technologies used in weapons systems be
demonstrated in an operational environment, reaching a high maturity
level--analogous to TRL 7--before deciding to commit to a cost,
schedule, and performance baseline for development and production of
the weapon system.[Footnote 22] In response to our recommendation, DOD
has agreed that if a technology does not achieve a score of TRL 6 or 7,
project managers must develop a plan to bring the technology to the
required readiness level before proceeding to the next project phase.
Use of TRLs is not by itself a cure-all for managing critical
technologies, but TRLs can be used in conjunction with other measures
to improve the way projects are managed. For example, according to
studies by NASA, DOD, and others, TRLs can:
* provide a common language among the technology developers, engineers
who will adopt/use the technology, and other stakeholders;
* improve stakeholder communication regarding technology development--
a by-product of the discussion among stakeholders that is needed to
negotiate a TRL value;
* reveal the gap between a technology's current readiness level and the
readiness level needed for successful inclusion in the intended
product;
* identify at-risk technologies that need increased management
attention or additional resources for technology development to
initiate risk-reduction measures; and:
* increase transparency of critical decisions by identifying key
technologies that have been demonstrated to work or by highlighting
still immature or unproven technologies that might result in high
project risk.
Two DOE headquarters offices have attempted to systematically assess
technology readiness. First, under the Office of Nuclear Energy, a DOE
contractor preparing a congressional report used a TRL method to
compare the maturity of advanced fuel cycle technologies. In addition,
in 2000, DOE's Office of Science and Technology, under EM, issued a
report that defined a process for assessing technology maturity of EM
projects.[Footnote 23] However, according to an EM official, the office
decided to discontinue using this assessment process because it was
considered overly burdensome. As a result, DOE devolved responsibility
for managing technology readiness to the contractor level.
According to several DOE project directors we spoke with, a consistent,
systematic method for assessing technology readiness would help achieve
a number of objectives: that is, standardize terminology, make
technology assessments more transparent, and improve communication
among project stakeholders before they make critical project decisions.
DOE project managers also acknowledged that TRLs could improve project
management departmentwide, and some managers are now attempting to use
this tool to assess technology maturity. The DOE project director for
the Waste Treatment and Immobilization Plant told us that a senior DOE
official encouraged him to begin using TRLs. He is consulting with DOD
officials knowledgeable about using the TRL method and expects to
develop a TRL tool and have TRL determinations for major parts of the
project in 2007. (App. VI compares DOD's product development process
with DOE's project management process for major projects.)
Conclusions:
The magnitude of the cost increases and schedule delays for DOE's major
projects is cause for serious rethinking of how DOE manages them. To
its credit, DOE has completed, or expects to complete prior to
construction, 85 to 100 percent of project design work for the 5
projects we reviewed that have recently begun or are nearing
construction. However, DOE has not systematically addressed another key
factor--the readiness level of the technologies it expects to use in
these projects. DOE lacks comprehensive standards in DOE Order 413.3A
and Manual 413.3-1 for systematically measuring and communicating the
readiness of project technologies. Specifically, the department lacks
consistent metrics for determining technology readiness departmentwide,
terminology, and oversight protocols for reporting and reviewing TRLs.
Without consistent measurement and communication of the readiness of
technologies, DOE does not have a basis for defining the acceptable
level of technological risk for each project, making critical decisions
on accepting the validity of a project's total estimated cost and
schedule, or proceeding with construction.
Other federal agencies have recognized the need to consistently measure
and communicate technology readiness to help avoid cost increases and
delays that result from relying on immature technologies. DOD, for
example, requires its managers to use a TRL process to measure
technology readiness and generally requires a TRL 7 (as we had
recommended) before system development and demonstration. In contrast,
as DOE's poor track record for managing the technological complexity of
major projects shows, DOE has not systematically measured the readiness
of critical project technologies before it approves definitive cost and
schedule estimates or begins construction. Furthermore, without a
systematic method for measuring technological readiness, DOE cannot
effectively communicate within the department and to the Congress
whether projects are at risk of experiencing cost increases and
schedule delays associated with technology problems.
Recommendations for Executive Action:
To improve decision making and oversight for major DOE construction
projects, including how project technology readiness is measured and
reported, we recommend that the Secretary of Energy evaluate and
consider adopting a disciplined and consistent approach to assessing
TRLs for projects with critical technologies that includes the
following three actions:
* Develop comprehensive standards for systematically measuring and
communicating the readiness of project technologies. At a minimum,
these standards should (1) specify consistent metrics for determining
technology readiness departmentwide, (2) establish terminology that can
be consistently applied across projects, and (3) detail the oversight
protocols to be used in reporting and reviewing TRLs. In preparing
these standards, DOE should consider lessons learned from NASA and DOD,
and its own experience in measuring technology readiness. If DOE's
evaluation results in the decision to adopt these standards, it should
incorporate them into DOE Order 413.3A and Manual 413.3-1, and provide
the appropriate training to ensure their proper implementation.
* Direct DOE Acquisition Executives to ensure that projects with
critical technologies reach a level of readiness commensurate with
acceptable risk--analogous to TRL 7--before deciding to approve the
preliminary design and commit to definitive cost and schedule
estimates, and at least TRL 7 or, if possible, TRL 8 before committing
to construction expenses.
* Inform the appropriate committees and Members of Congress of any DOE
decision to approve definitive cost and schedule estimates, or to begin
construction, without first having ensured that project technologies
are sufficiently ready (at TRL 7 or 8). This information should include
specific plans for mitigating technology risks, such as developing
backup technologies to offset the effects of a potential technology
failure, and appropriate justification for accepting higher
technological risk.
Agency Comments and Our Evaluation:
We provided a draft of this report to DOE for its review and comment.
DOE's written comments are reproduced in appendix VII. DOE agreed with
our recommendations but suggested revisions that would allow it to
first conduct a pilot application on selected projects to better
understand the technology readiness assessment process and evaluate its
potential use. We revised our recommendations to give DOE this
flexibility. DOE also provided detailed technical comments, which we
have incorporated into our report as appropriate.
DOE also expressed several specific concerns with our draft report.
First, DOE stated that while our draft broadly asserts that DOE project
management has led to increases in cost and schedule, our
recommendations are narrowly focused on technology assessment. We agree
that our draft states that DOE project management has led to cost
increases and schedule delays, a conclusion we reached on the basis of
our contact with DOE project directors and our review of numerous
studies and reports on DOE major projects. Our recommendations address
technology assessment, a critical project management activity, because
they were developed primarily on the basis of our specific finding that
DOE lacks a systematic approach to ensure that final project designs,
including critical technologies reflected in these designs, have been
demonstrated to work as intended prior to construction. This report
explains that delaying resolution of technology problems until
construction can potentially lead to significant cost increases and
schedule delays.
Second, DOE stated that our draft report inappropriately characterizes
cost and schedule growth from a small sample of projects by using
preliminary cost and schedule estimates that are intended for internal
DOE planning. To clarify, the scope of our review included an
evaluation of DOE's major construction projects. In addition, our
report explains that DOE changed its project management policy in 2000
to allow cost and schedule estimates to be prepared later in the
project--at the end of preliminary design. Prior to this new policy,
project directors submitted cost and schedule estimates earlier in the
project development phase--at the end of conceptual design. For
projects under way prior to the policy in 2000, we used post-2000
validated baseline estimates, if available. Otherwise, we used earlier
estimates since these were the only estimates available and had been
previously used by DOE to inform Congress of the total expected project
cost and schedule while seeking initial project funding. We also note
that for the five projects that were started after the new policy in
2000, we used the validated project baseline estimates recommended by
DOE, if available.
Third, DOE suggested we revise table 3 in our report to more clearly
identify the correlation between cost and schedule growth and
technology maturity. As our report states, the information in table 3
was drawn from the results of our review of independent studies
involving the projects we reviewed and the results of our interviews
with DOE project directors. Our report explains that cost increases and
schedule delays for 6 of the 9 projects shown in the table were due in
part to contractors' poor management of the development and integration
of technologies used in the project designs.
Finally, DOE stated that it is unclear how the factors cited in
appendix IV, such as communication, and changes in "political will,"
among other things, led to our recommendation to assess technology
readiness. Although not all of the factors cited in our survey have a
link to our recommendation on technology readiness, one factor in
particular--absence of communication--is addressed in our
recommendation. Specifically, we recommended that the Secretary of
Energy consider developing comprehensive standards for systematically
measuring and communicating the readiness of project technologies,
including the establishment of terminology that is to be consistently
applied across projects.
We are sending copies of the report to interested congressional
committees, the Secretary of Energy, and the Director of the Office of
Management and Budget. We will make copies available to others on
request. In addition, the report will also be available at no charge on
the GAO Web site at http://www.gao.gov.
If you or your staffs have any questions about this report, please
contact me at (202) 512-3841 or aloisee@gao.gov. Contact points for our
Offices of Congressional Relations and Public Affairs may be found on
the last page of this report. Other staff contributing to the report
are listed in appendix VIII.
Signed by:
Gene Aloise:
Director, Natural Resources and Environment:
[End of section]
Appendix I: Scope and Methodology:
To determine the extent to which the Department of Energy's (DOE) major
construction projects have experienced cost increases and schedule
delays and the factors that have contributed to these problems, we
identified (1) active DOE major line-item construction projects that
have current total project cost estimates above the $750 million
threshold--DOE's criteria for "major construction projects," and (2)
the projects with estimates above $400 million--the DOE threshold for
major projects until July 2006. We also identified those projects above
$300 million to account for any projects that may pass the $400 million
threshold.[Footnote 24] In all, we identified the following 12
projects:
* Five of these 12 projects began before DOE moved its requirement for
firm cost and schedule estimates to later in the project: the National
Ignition Facility, the Mixed Oxide Fuel Fabrication Facility, the Pit
Disassembly and Conversion Facility, the Spallation Neutron Source, and
the Tritium Extraction Facility. We used the estimates at the end of
conceptual design, as reported by project directors, for the initial
project cost and schedule estimates.
* Four of the remaining 7 projects had cost and schedule estimates
completed at the end of preliminary design, according to the new DOE
guidelines: the Highly Enriched Uranium Materials Facility,
Microsystems and Engineering Sciences Applications, the Depleted
Uranium Hexafluoride 6 Conversion Facilities, and the Linac Coherent
Light Source. For these projects, we considered the estimates as
reported by project directors to be the initial cost and schedule
estimates.
* One project, the Waste Treatment and Immobilization Plant, began
after DOE moved the requirement for firm cost and schedule estimates to
later in the project. However, DOE initially exempted the contractor
from submitting firm cost and schedule estimates. Therefore, we used
the estimates reported by the project director to be the initial cost
and schedule estimates.
* The final 2 projects, although falling under the new DOE
requirements, had yet to complete their preliminary design at the time
of our review: the Chemistry and Metallurgy Research Facility
Replacement and the Salt Waste Processing Facility. For these projects,
we considered the cost and schedule estimates at the end of conceptual
design reported by project directors to be the initial project cost and
schedule estimates.
Because we and others have previously expressed concern about the data
reliability of a key DOE project management tracking database--the
Project Assessment and Reporting System--we did not develop conclusions
or findings based on information generated through that
system.[Footnote 25] Instead, we collected information directly through
surveys and interviews with project site officials.
To identify cost increases and schedule delays, and the factors that
may have contributed to these changes, we surveyed DOE project
directors, interviewed DOE and contractor project personnel, and
reviewed project management documents for 12 major projects. These 12
projects are managed by DOE's Office of Science, Office of
Environmental Management (EM), or National Nuclear Security
Administration (NNSA). (See app. II for information on these projects.)
Our survey asked DOE project directors of the 12 projects to identify
the degree to which cost and schedule estimates may have changed and
the reasons for these changes, and to describe the events and
conditions that led to any changes. Eight of the 12 project directors
responded that their projects had experienced cost increases and
schedule delays, and 1 project director reported only a schedule delay.
For these 9 projects, we asked project directors to (1) identify the
top three events that led to the cost and schedule delays and (2)
indicate to what extent certain factors may have contributed to the
event that led to the largest percentage cost increase or schedule
delay. The factors included in the survey instrument were based on the
results of a National Research Council study that listed essential or
important conditions needed for the successful completion of major
projects.[Footnote 26] We asked project directors to identify the
extent to which the lack of these conditions may have contributed to
any cost and schedule delays. (App. IV shows key survey results for
these 9 projects.)
In addition to reviewing project documentation, we conducted site
visits for the 9 projects that had experienced cost and schedule
changes, and we analyzed (1) studies of these projects completed by
DOE's Office of Inspector General and (2) external independent project
reviews conducted under the direction of DOE's Office of Engineering
and Construction Management in Washington, D.C. We interviewed federal
project directors of the 3 projects that had not experienced cost
increases or schedule delays to obtain information on factors they
believe are important in avoiding such increases.
To determine the extent to which DOE ensures that project designs are
sufficiently complete before construction, we obtained additional
information from project directors on 5 projects that were approaching,
or had recently begun, construction. During our review, we obtained
information on the extent project designs were, or are expected to be,
complete before beginning construction, and the actions DOE had taken
to ensure technologies used in these designs are sufficiently ready to
begin construction. For 2 of these 5 projects, we applied a tool we
previously had used to assess DOD programs--the tool enables project
directors to characterize the readiness level of each technology being
developed for use in aircraft and other military applications. In
addition, we spoke with officials from DOE program offices and DOE's
Office of Engineering and Construction Management in Washington, D.C.
We provided interim briefings to the Subcommittee on Energy and Water
Development, House Committee on Appropriations, on the status of our
work in May and September, 2006. We performed our work between December
2005 and January 2007, in accordance with generally accepted government
auditing standards.
[End of section]
Appendix II: Information on the 12 Department of Energy Major Projects
Reviewed:
Table:
Project: Chemistry and Metallurgy Research Facility Replacement;
DOE program office: National Nuclear Security Administration;
Project purpose/objectives: Relocate and consolidate mission-critical
analytical chemistry, material characterization, and research and
development capabilities to ensure continuous national security mission
support beyond 2010.
Project: Depleted Uranium Hexafluoride 6 Conversion Facility;
DOE program office: Office of Environmental Management;
Project purpose/ objectives: Design and construct facilities at
Portsmouth, Ohio, and Paducah, Kentucky, to convert the Department of
Energy's existing inventory of depleted uranium hexafluoride into a
more stable form for disposal or beneficial reuse.
Project: Highly Enriched Uranium Materials Facility;
DOE program office: National Nuclear Security Administration;
Project purpose/ objectives: Project will construct a highly secure,
state-of-the-art facility for consolidating and storing highly enriched
uranium, resulting in cost savings and an increased security posture.
Project: Linac Coherent Light Source;
DOE program office: Science;
Project purpose/objectives: Provide laser-like radiation in the X-ray
region of the spectrum that is 10 billion times greater in peak
brightness than any existing X-ray light source. The project will apply
these high-brightness X-rays to experiments in the chemical, material,
and biological sciences.
Project: Microsystems and Engineering Sciences Applications;
DOE program office: National Nuclear Security Administration;
Project purpose/objectives: Provide state-of-the-art national complex
that will provide for the design, integration, prototyping, and
qualification of microsystems into components, subsystems, and systems
within the nuclear weapons stockpile.
Project: Mixed Oxide Fuel Fabrication Facility;
DOE program office: National Nuclear Security Administration;
Project purpose/objectives: Facility will combine surplus weapon-grade
plutonium oxide with depleted uranium to form mixed oxide fuel
assemblies that will be irradiated in United States commercial nuclear
reactors. Once irradiated and converted into spent fuel, the resulting
plutonium can no longer be readily used for nuclear weapons.
Project: National Ignition Facility;
DOE program office: National Nuclear Security Administration;
Project purpose/objectives: Provide experimental capability to assess
nuclear weapons physics, providing critical data that will allow the
United States to maintain its technical capabilities in nuclear weapons
in the absence of underground testing, and to advance fusion as an
energy source.
Project: Pit Disassembly and Conversion Facility;
DOE program office: National Nuclear Security Administration;
Project purpose/objectives: Eliminate surplus Russian and United States
plutonium and highly enriched uranium by disassembling surplus nuclear
weapons pits and converting the resulting plutonium metal to a powder
form that can later be fabricated into mixed oxide fuel to produce
nuclear fuel assemblies for use in commercial nuclear reactors.
Project: Salt Waste Processing Facility;
DOE program office: Office of Environmental Management;
Project purpose/objectives: Meet site cleanup goals and reduce
significant environmental and health/safety risk by construction of a
facility to treat large quantities of waste from reprocessing and
nuclear materials production operations at the Savannah River Site.
Process will separate waste, solidify it in glass, and send it to
federal repositories for disposal.
Project: Spallation Neutron Source;
DOE program office: Science;
Project purpose/objectives: Provide next generation, short-pulse
spallation neutron source for neutron scattering, to be used by
researchers from academia, national and federal labs, and industry for
basic and applied research and technology development in the fields of
condensed matter physics, materials sciences, magnetic materials,
polymers and complex fluids, chemistry, biology, earth sciences, and
engineering.
Project: Tritium Extraction Facility;
DOE program office: National Nuclear Security Administration;
Project purpose/objectives: To replenish the tritium needs of the
nuclear weapons stockpile, the facility will extract tritium produced
in a commercial nuclear reactor for use in nuclear weapons development.
Project: Waste Treatment and Immobilization Plant;
DOE program office: Office of Environmental Management;
Project purpose/objectives: The plant will separate high-level from low-
level radioactive waste currently stored in underground tanks,
processing and solidifying all high-level waste and a substantial
portion of the low-level waste, and will treat the remaining low-level
waste.
Source: DOE.
[End of table]
[End of section]
Appendix III: Independent Studies Reviewed:
National Ignition Facility:
Department of Energy, Office of Inspector General. Audit Report: Status
of the National Ignition Facility Project. DOE/IG-0598. Washington,
D.C.: April 28, 2003.
GAO. Department of Energy: Status of Contract and Project Management
Reforms. GAO-03-570T. Washington, D.C.: March 20, 2003.
GAO. Contract Reform: DOE Has Made Progress, but Actions Needed to
Ensure Initiatives Have Improved Results. GAO-02-798. Washington, D.C.:
September 13, 2002.
GAO. Department of Energy: Follow-up Review of DOE's National Ignition
Facility. GAO-01-677R. Washington, D.C.: June 1, 2001.
GAO. National Ignition Facility: Management and Oversight Failures
Caused Major Cost Overruns and Schedule Delays. GAO/RCED-00-141 and
GAO/RCED-00-271. Washington, D.C.: August 8, 2000.
The Mitre Corporation. NIF Ignition. JSR-05-340. McLean, VA: June 29,
2005.
Mixed Oxide Fuel Fabrication Facility:
Burns and Roe Enterprises, Inc. External Independent Review of the
Mixed Oxide Fuel Fabrication Facility (MFFF) Project Critical Decision
(CD) 2/3 Baseline: Performance Baseline (CD-2) and Start of
Construction (CD-3) Review. BREI-L-R-06-03. Oradell, NJ: July 7, 2006.
Burns and Roe Enterprises, Inc. External Independent Review of the
Basis of Design for the Aqueuous Polishing Process. BREI-SLP-R-06-01.
Oradell, NJ: March 27, 2006.
Civil Engineering Research Foundation. Independent Research Assessment
of Project Management Factors Affecting Department of Energy Project
Success. Washington, D.C.: July 12, 2004.
Department of Energy, Office of Inspector General. Audit Report: Status
of the Mixed Oxide Fuel Fabrication Facility. DOE/IG-0713. Washington,
D.C.: December 21, 2005.
Pit Disassembly and Conversion Facility:
Department of Energy, Office of Inspector General. Audit Report:
National Nuclear Security Administration's Pit Disassembly and
Conversion Facility. DOE/IG-0688. Washington, D.C.: May 3, 2005.
Los Alamos National Laboratory. Options for the Development and Testing
of the Pit Disassembly and Conversion Facility Government-Furnished
Design. LA-UR-03-3926. Los Alamos, NM: June 11, 2003.
Waste Treatment and Immobilization Plant:
Bechtel National, Inc. Hanford Tank Waste Treatment and Immobilization
Plant, May 2006 Estimate at Completion. Hanford Site, WA: May 31, 2006.
Bechtel National, Inc. Comprehensive Review of the Hanford Tank Waste
Treatment and Immobilization Plant Estimate at Completion. CCN 132848.
Hanford Site, WA: March 31, 2006.
Bechtel National, Inc. Comprehensive Review of the Hanford Waste
Treatment Plant Flowsheet and Throughput. CCN132846. Hanford Site, WA:
March 17, 2006.
Bechtel National, Inc. Hanford Tank Waste Treatment and Immobilization
Plant, December 2005 Estimate at Completion Executive Summary. Hanford
Site, WA: January 30, 2006.
Department of the Army Corp of Engineers. Complete Statement of Kim
Callan, to the Subcommittee on Energy and Water Development, Committee
on Appropriations, United States House of Representatives. Washington,
D.C.: April 6, 2006.
Department of Energy. External Independent Review, Independent Cost
Review, CD-3C Review of the Waste Treatment and Immobilization Plant
Project. Hanford Site, WA: September 2002.
Department of Energy. External Independent Review CD-3B Review of the
Waste Treatment and Immobilization Plant Project. Hanford Site, WA:
April 2002.
GAO. Hanford Waste Treatment Plant, Contractor and DOE Management
Problems Have Led to Higher Costs, Construction Delays, and Safety
Concerns. GAO-06-602T. Washington, D.C.: April 6, 2006.
GAO. Further Actions Are Needed to Strengthen Contract Management for
Major Projects. GAO-05-123. Washington, D.C.: March 18, 2005.
GAO. Nuclear Waste: Absence of Key Management Reforms on Hanford's
Cleanup Project Adds to Challenges of Achieving Cost and Schedule
Goals. GAO-04-611. Washington, D.C.: June 9, 2004.
GAO. Status of Contract and Project Management Reforms. GAO-03-57T.
Washington, D.C.: March 20, 2003.
GAO. Contract Reform: DOE Has Made Progress, but Actions Needed to
Ensure Initiatives Have Improved Results. GAO-02-798. Washington, D.C.:
September 13, 2002.
GAO. Nuclear Waste: Hanford Tank Waste Program Needs Cost, Schedule,
and Management Changes. GAO/RCED-93-99. Washington, D.C.: March 8,
1993.
LMI Government Consulting. Hanford Waste Treatment and Immobilization
Plant After-Action Fact-Finding Review. DE535T1. McLean, VA: January
2006.
LMI Government Consulting. External Independent Review, Follow-up
Review, Waste Treatment and Immobilization Plant (WTP) Out Briefing.
Washington, D.C.: March 14, 2003.
Spallation Neutron Source:
Civil Engineering Research Foundation. Independent Research Assessment
of Project Management Factors Affecting Department of Energy Project
Success. Washington, D.C.: July 12, 2004.
Department of Energy, Office of Inspector General. Audit Report:
Progress of the Spallation Neutron Source Project. DOE/IG-0532.
Washington, D.C.: November 19, 2001.
Department of Energy. Review Committee Report on the Baseline Review of
the Spallation Neutron Source (SNS) Project. Washington, D.C.: July 15,
1999.
Department of Energy. Technical, Cost, Schedule, and Management Review
of the Spallation Neutron Source Project. Washington, D.C.: January 28,
1999.
GAO. Department of Energy: Status of Contract and Project Management
Reforms. GAO-03-570T. Washington, D.C.: March 20, 2003.
GAO. Contract Reform: DOE Has Made Progress, but Actions Needed to
Ensure Initiatives Have Improved Results. GAO-02-798. Washington, D.C.:
September 13, 2002.
GAO. Department of Energy: Challenges Exist in Managing the Spallation
Neutron Source Project. GAO/T-RCED-99-103. Washington, D.C.: March 3,
1999.
Salt Waste Processing Facility:
Department of Energy, Office of Inspector General. Audit Report: Salt
Processing Project at the Savannah River Site. DOE/IG-0565. Washington,
D.C.: August 27, 2002.
Institute for Regulatory Science. Technical Peer Review Report of the
Review Panel on Salt Waste Processing Facility Technology Readiness.
CRTD-Vol. 75. Danvers, MA: October 31, 2003.
Tritium Extraction Facility:
Civil Engineering Research Foundation. Independent Research Assessment
of Project Management Factors Affecting Department of Energy Project
Success. Washington, D.C.: July 12, 2004.
Department of Energy, Office of Inspector General. Audit Report: The
Department of Energy's Tritium Extraction Facility. DOE/IG-0560.
Washington, D.C.: June 24, 2002.
GAO. Department of Energy: Further Actions Are Needed to Strengthen
Contract Management for Major Projects. GAO-05-123. Washington, D.C.:
March 18, 2005.
GAO. Department of Energy: Status of Contract and Project Management
Reforms. GAO-03-570T. Washington, D.C.: March 20, 2003.
GAO. Contract Reform: DOE Has Made Progress, but Actions Needed to
Ensure Initiatives Have Improved Results. GAO-02-798. Washington, D.C.:
September 13, 2002.
GAO. Nuclear Weapons: Design Reviews of DOE's Tritium Extraction
Facility. GAO/RCED-98-75. Washington, D.C.: March 31, 1998.
National Nuclear Security Administration. Program Review of the
Estimate to Complete Tritium Extraction Facility (TEF) at Savannah
River Site. Washington, D.C.: August 29, 2002.
Highly Enriched Uranium Materials Facility:
BWXT Y-12. Highly Enriched Uranium Materials Facility Project Causal
Analysis Report. Oak Ridge, TN: March 6, 2006.
Department of Energy. Limited External Independent Review for Baseline
Change Proposal Review. Oak Ridge, TN: August 31, 2004.
Department of Energy, Office of Inspector General. Audit Report, Design
of the Uranium Storage Facility at the Y-12 National Security Complex.
DOE/IG-0643. Washington, D.C.: March 19, 2004.
Department of Energy, Office of Inspector General. Audit Report,
Reestablishment of Enriched Uranium Operations at the Y-12 National
Security Complex. DOE/IG-0640. Washington, D.C.: February 24, 2004.
Department of Energy. External Independent Review - Performance
Baseline Review of the Highly Enriched Uranium Materials Facility
Project. Oak Ridge, TN: June 2003.
Depleted Uranium Hexafluoride 6 Conversion Facility:
Department of Energy. Report on the Independent Project Review of the
Depleted Uranium Hexafluoride Conversion Project. Washington, D.C.:
October 8, 2004.
Department of Energy, Office of Inspector General. Audit Report:
Depleted Uranium Hexafluoride Conversion. DOE/IG-0642. Washington,
D.C.: March 18, 2004.
GAO. Department of Energy: Status of Contract and Project Management
Reforms. GAO-03-570T. Washington, D.C.: March 20, 2003.
LMI Government Consulting. DUF6 Conversion Project CD-3 Corrective
Action Plan Review. DE538T1. McLean, VA: October 2005.
LMI Government Consulting. Construction Readiness EIR (for CD-3) of the
Depleted Uranium Hexafluoride Conversion Project. DE534T1. McLean, VA:
June 2005.
LMI Government Consulting. DUF6 Conversion Project CD-3C Construction
Readiness Review Preliminary Draft. Washington, D.C.: May 20, 2005.
LMI Government Consulting. DUF6 Limited Conversion Plan Project
External Independent Review for the Office of Engineering and
Construction Management. DE428T1. McLean, VA: June 2004.
Chemistry and Metallurgy Research Facility Replacement:
Jupiter Corporation. External Independent Review of the Chemistry and
Metallurgy Research Building Replacement Project. Approve Performance
Baseline and Approve Start of Construction. CD-2A/CD-3A. Wheaton, MD:
October 14, 2005.
[End of section]
Appendix IV: Survey Results for Primary Factors Affecting Cost and
Schedule on Nine Projects with Cost or Schedule Changes:
Table:
Absence of open communication, mutual trust, and close coordination.
Factor/Project: Depleted Uranium Hexafluoride 6 Conversion Facility;
Survey results for primary factors: To no extent: [Empty];
Survey results for primary factors: To a limited extent: X;
Survey results for primary factors: To a moderate extent: [Empty];
Survey results for primary factors: To a great extent: [Empty];
Survey results for primary factors: To a very great extent: [Empty];
Survey results for primary factors: No answer: [Empty].
Factor/Project: Highly Enriched Uranium Materials Facility; Survey
results for primary factors: To no extent: [Empty];
Survey results for primary factors: To a limited extent: X;
Survey results for primary factors: To a moderate extent: [Empty];
Survey results for primary factors: To a great extent: [Empty];
Survey results for primary factors: To a very great extent: [Empty];
Survey results for primary factors: No answer: [Empty].
Factor/Project: Mixed Oxide Fuel Fabrication Facility;
Survey results for primary factors: To no extent: [Empty];
Survey results for primary factors: To a limited extent: [Empty];
Survey results for primary factors: To a moderate extent: [Empty];
Survey results for primary factors: To a great extent: [Empty];
Survey results for primary factors: To a very great extent: [Empty];
Survey results for primary factors: No answer: X.
Factor/Project: National Ignition Facility;
Survey results for primary factors: To no extent: [Empty];
Survey results for primary factors: To a limited extent: [Empty];
Survey results for primary factors: To a moderate extent: X;
Survey results for primary factors: To a great extent: [Empty];
Survey results for primary factors: To a very great extent: [Empty];
Survey results for primary factors: No answer: [Empty].
Factor/Project: Pit Disassembly and Conversion Facility;
Survey results for primary factors: To no extent: [Empty];
Survey results for primary factors: To a limited extent: [Empty];
Survey results for primary factors: To a moderate extent: [Empty];
Survey results for primary factors: To a great extent: [Empty];
Survey results for primary factors: To a very great extent: [Empty];
Survey results for primary factors: No answer: X.
Factor/Project: Salt Waste Processing Facility;
Survey results for primary factors: To no extent: [Empty];
Survey results for primary factors: To a limited extent: [Empty];
Survey results for primary factors: To a moderate extent: X;
Survey results for primary factors: To a great extent: [Empty];
Survey results for primary factors: To a very great extent: [Empty];
Survey results for primary factors: No answer: [Empty].
Factor/Project: Spallation Neutron Source;
Survey results for primary factors: To no extent: [Empty];
Survey results for primary factors: To a limited extent: X;
Survey results for primary factors: To a moderate extent: [Empty];
Survey results for primary factors: To a great extent: [Empty];
Survey results for primary factors: To a very great extent: [Empty];
Survey results for primary factors: No answer: [Empty].
Factor/Project: Tritium Extraction Facility;
Survey results for primary factors: To no extent: [Empty];
Survey results for primary factors: To a limited extent: [Empty];
Survey results for primary factors: To a moderate extent: [Empty];
Survey results for primary factors: To a great extent: X;
Survey results for primary factors: To a very great extent: [Empty];
Survey results for primary factors: No answer: [Empty].
Factor/Project: Waste Treatment and Immobilization Plant;
Survey results for primary factors: To no extent: [Empty];
Survey results for primary factors: To a limited extent: X;
Survey results for primary factors: To a moderate extent: [Empty];
Survey results for primary factors: To a great extent: [Empty];
Survey results for primary factors: To a very great extent: [Empty];
Survey results for primary factors: No answer: [Empty].
Factor/Project: Total;
Survey results for primary factors: To no extent: 0;
Survey results for primary factors: To a limited extent: 4;
Survey results for primary factors: To a moderate extent: 2;
Survey results for primary factors: To a great extent: 1;
Survey results for primary factors: To a very great extent: 0;
Survey results for primary factors: No answer: 2.
Changes in "political will" during project execution (e.g., project
changes resulting from political decisions--includes politics internal
and external to the project ).
Factor/Project: Depleted Uranium Hexafluoride 6 Conversion Facility;
Survey results for primary factors: To no extent: [Empty];
Survey results for primary factors: To a limited extent: [Empty];
Survey results for primary factors: To a moderate extent: X;
Survey results for primary factors: To a great extent: [Empty];
Survey results for primary factors: To a very great extent: [Empty];
Survey results for primary factors: No answer: [Empty].
Factor/Project: Highly Enriched Uranium Materials Facility;
Survey results for primary factors: To no extent: [Empty];
Survey results for primary factors: To a limited extent: X;
Survey results for primary factors: To a moderate extent: [Empty];
Survey results for primary factors: To a great extent: [Empty];
Survey results for primary factors: To a very great extent: [Empty];
Survey results for primary factors: No answer: [Empty].
Factor/Project: Mixed Oxide Fuel Fabrication Facility;
Survey results for primary factors: To no extent: [Empty];
Survey results for primary factors: To a limited extent: [Empty];
Survey results for primary factors: To a moderate extent: [Empty];
Survey results for primary factors: To a great extent: [Empty];
Survey results for primary factors: To a very great extent: [Empty];
Survey results for primary factors: No answer: X.
Factor/Project: National Ignition Facility;
Survey results for primary factors: To no extent: X;
Survey results for primary factors: To a limited extent: [Empty];
Survey results for primary factors: To a moderate extent: [Empty];
Survey results for primary factors: To a great extent: [Empty];
Survey results for primary factors: To a very great extent: [Empty];
Survey results for primary factors: No answer: [Empty].
Factor/Project: Pit Disassembly and Conversion Facility;
Survey results for primary factors: To no extent: [Empty];
Survey results for primary factors: To a limited extent: [Empty];
Survey results for primary factors: To a moderate extent: [Empty];
Survey results for primary factors: To a great extent: [Empty];
Survey results for primary factors: To a very great extent: X;
Survey results for primary factors: No answer: [Empty].
Factor/Project: Salt Waste Processing Facility;
Survey results for primary factors: To no extent: [Empty];
Survey results for primary factors: To a limited extent: X;
Survey results for primary factors: To a moderate extent: [Empty];
Survey results for primary factors: To a great extent: [Empty];
Survey results for primary factors: To a very great extent: [Empty];
Survey results for primary factors: No answer: [Empty].
Factor/Project: Spallation Neutron Source;
Survey results for primary factors: To no extent: [Empty];
Survey results for primary factors: To a limited extent: [Empty];
Survey results for primary factors: To a moderate extent: [Empty];
Survey results for primary factors: To a great extent: [Empty];
Survey results for primary factors: To a very great extent: X;
Survey results for primary factors: No answer: [Empty].
Factor/Project: Tritium Extraction Facility;
Survey results for primary factors: To no extent: X;
Survey results for primary factors: To a limited extent: [Empty];
Survey results for primary factors: To a moderate extent: [Empty];
Survey results for primary factors: To a great extent: [Empty];
Survey results for primary factors: To a very great extent: [Empty];
Survey results for primary factors: No answer: [Empty].
Factor/Project: Waste Treatment and Immobilization Plant;
Survey results for primary factors: To no extent: [Empty];
Survey results for primary factors: To a limited extent: [Empty];
Survey results for primary factors: To a moderate extent: X;
Survey results for primary factors: To a great extent: [Empty];
Survey results for primary factors: To a very great extent: [Empty];
Survey results for primary factors: No answer: [Empty].
Factor/Project: Total;
Survey results for primary factors: To no extent: 2;
Survey results for primary factors: To a limited extent: 2;
Survey results for primary factors: To a moderate extent: 2;
Survey results for primary factors: To a great extent: 0;
Survey results for primary factors: To a very great extent: 2;
Survey results for primary factors: No answer: 1.
Interruptions in planning and committing budget funds.
Factor/Project: Depleted Uranium Hexafluoride 6 Conversion;
Survey results for primary factors: To no extent: X;
Survey results for primary factors: To a limited extent: [Empty];
Survey results for primary factors: To a moderate extent: [Empty];
Survey results for primary factors: To a great extent: [Empty];
Survey results for primary factors: To a very great extent: [Empty];
Survey results for primary factors: No answer: [Empty].
Factor/Project: Highly Enriched Uranium Materials Facility;
Survey results for primary factors: To no extent: [Empty];
Survey results for primary factors: To a limited extent: [Empty];
Survey results for primary factors: To a moderate extent: [Empty];
Survey results for primary factors: To a great extent: [Empty];
Survey results for primary factors: To a very great extent: X;
Survey results for primary factors: No answer: [Empty].
Factor/Project: Mixed Oxide Fuel Fabrication Facility;
Survey results for primary factors: To no extent: [Empty];
Survey results for primary factors: To a limited extent: [Empty];
Survey results for primary factors: To a moderate extent: [Empty];
Survey results for primary factors: To a great extent: X;
Survey results for primary factors: To a very great extent: [Empty];
Survey results for primary factors: No answer: [Empty].
Factor/Project: National Ignition Facility;
Survey results for primary factors: To no extent: X;
Survey results for primary factors: To a limited extent: [Empty];
Survey results for primary factors: To a moderate extent: [Empty];
Survey results for primary factors: To a great extent: [Empty];
Survey results for primary factors: To a very great extent: [Empty];
Survey results for primary factors: No answer: [Empty].
Factor/Project: Pit Disassembly and Conversion Facility;
Survey results for primary factors: To no extent: [Empty];
Survey results for primary factors: To a limited extent: [Empty];
Survey results for primary factors: To a moderate extent: [Empty];
Survey results for primary factors: To a great extent: [Empty];
Survey results for primary factors: To a very great extent: [Empty];
Survey results for primary factors: No answer: X.
Factor/Project: Salt Waste Processing Facility;
Survey results for primary factors: To no extent: X;
Survey results for primary factors: To a limited extent: [Empty];
Survey results for primary factors: To a moderate extent: [Empty];
Survey results for primary factors: To a great extent: [Empty];
Survey results for primary factors: To a very great extent: [Empty];
Survey results for primary factors: No answer: [Empty].
Factor/Project: Spallation Neutron Source;
Survey results for primary factors: To no extent: [Empty];
Survey results for primary factors: To a limited extent: [Empty];
Survey results for primary factors: To a moderate extent: [Empty];
Survey results for primary factors: To a great extent: [Empty];
Survey results for primary factors: To a very great extent: X;
Survey results for primary factors: No answer: [Empty].
Factor/Project: Tritium Extraction Facility;
Survey results for primary factors: To no extent: [Empty];
Survey results for primary factors: To a limited extent: [Empty];
Survey results for primary factors: To a moderate extent: X;
Survey results for primary factors: To a great extent: [Empty];
Survey results for primary factors: To a very great extent: [Empty];
Survey results for primary factors: No answer: [Empty].
Factor/Project: Waste Treatment and Immobilization Plant;
Survey results for primary factors: To no extent: [Empty];
Survey results for primary factors: To a limited extent: [Empty];
Survey results for primary factors: To a moderate extent: [Empty];
Survey results for primary factors: To a great extent: X;
Survey results for primary factors: To a very great extent: [Empty];
Survey results for primary factors: No answer: [Empty].
Factor/Project: Total;
Survey results for primary factors: To no extent: 3;
Survey results for primary factors: To a limited extent: 0;
Survey results for primary factors: To a moderate extent: 1;
Survey results for primary factors: To a great extent: 2;
Survey results for primary factors: To a very great extent: 2;
Survey results for primary factors: No answer: 1.
Project managers did not have adequate professional experience.
Factor/Project: Depleted Uranium Hexafluoride 6 Conversion;
Survey results for primary factors: To no extent: [Empty];
Survey results for primary factors: To a limited extent: [Empty];
Survey results for primary factors: To a moderate extent: X;
Survey results for primary factors: To a great extent: [Empty];
Survey results for primary factors: To a very great extent: [Empty];
Survey results for primary factors: No answer: [Empty].
Factor/Project: Highly Enriched Uranium Materials Facility;
Survey results for primary factors: To no extent: [Empty];
Survey results for primary factors: To a limited extent: [Empty];
Survey results for primary factors: To a moderate extent: X;
Survey results for primary factors: To a great extent: [Empty];
Survey results for primary factors: To a very great extent: [Empty];
Survey results for primary factors: No answer: [Empty].
Factor/Project: Mixed Oxide Fuel Fabrication Facility;
Survey results for primary factors: To no extent: [Empty];
Survey results for primary factors: To a limited extent: [Empty];
Survey results for primary factors: To a moderate extent: [Empty];
Survey results for primary factors: To a great extent: [Empty];
Survey results for primary factors: To a very great extent: [Empty];
Survey results for primary factors: No answer: X.
Factor/Project: National Ignition Facility;
Survey results for primary factors: To no extent: [Empty];
Survey results for primary factors: To a limited extent: [Empty];
Survey results for primary factors: To a moderate extent: X;
Survey results for primary factors: To a great extent: [Empty];
Survey results for primary factors: To a very great extent: [Empty];
Survey results for primary factors: No answer: [Empty].
Factor/Project: Salt Waste Processing Facility;
Survey results for primary factors: To no extent: X;
Survey results for primary factors: To a limited extent: [Empty];
Survey results for primary factors: To a moderate extent: [Empty];
Survey results for primary factors: To a great extent: [Empty];
Survey results for primary factors: To a very great extent: [Empty];
Survey results for primary factors: No answer: [Empty].
Factor/Project: Spallation Neutron Source;
Survey results for primary factors: To no extent: [Empty];
Survey results for primary factors: To a limited extent: [Empty];
Survey results for primary factors: To a moderate extent: X;
Survey results for primary factors: To a great extent: [Empty];
Survey results for primary factors: To a very great extent: [Empty];
Survey results for primary factors: No answer: [Empty].
Factor/Project: Tritium Extraction Facility;
Survey results for primary factors: To no extent: [Empty];
Survey results for primary factors: To a limited extent: X;
Survey results for primary factors: To a moderate extent: [Empty];
Survey results for primary factors: To a great extent: [Empty];
Survey results for primary factors: To a very great extent: [Empty];
Survey results for primary factors: No answer: [Empty].
Factor/Project: Pit Disassembly and Conversion Facility;
Survey results for primary factors: To no extent: [Empty];
Survey results for primary factors: To a limited extent: [Empty];
Survey results for primary factors: To a moderate extent: [Empty];
Survey results for primary factors: To a great extent: [Empty];
Survey results for primary factors: To a very great extent: [Empty];
Survey results for primary factors: No answer: X.
Factor/Project: Waste Treatment and Immobilization Plant;
Survey results for primary factors: To no extent: X;
Survey results for primary factors: To a limited extent: [Empty];
Survey results for primary factors: To a moderate extent: [Empty];
Survey results for primary factors: To a great extent: [Empty];
Survey results for primary factors: To a very great extent: [Empty];
Survey results for primary factors: No answer: [Empty].
Factor/Project: Total;
Survey results for primary factors: To no extent: 2;
Survey results for primary factors: To a limited extent: 1;
Survey results for primary factors: To a moderate extent: 4;
Survey results for primary factors: To a great extent: 0;
Survey results for primary factors: To a very great extent: 0;
Survey results for primary factors: No answer: 2.
Source: GAO.
[End of table]
[End of section]
Appendix V: Definitions of Technology Readiness Levels:
Table:
Technology readiness level(TRL): 1. Basic principles observed and
reported;
Level involved: Studies;
Basic objective of TRLs: Research to prove feasibility;
Components: None;
Integration: None;
Tests and Environment: Desktop "back of envelope" environment.
Technology readiness level(TRL): 2. Technology concept and/or
application formulated;
Level involved: Studies;
Basic objective of TRLs: Research to prove feasibility;
Components: None;
Integration: Paper studies indicate components ought to work together;
Tests and Environment: Academic environment. The emphasis here is still
on understanding the science but beginning to think about possible
applications of the scientific principles.
Technology readiness level(TRL): 3. Analytical and experimental
critical function and/or characteristic proof or concept;
Level involved: Pieces of components;
Basic objective of TRLs: Research to prove feasibility;
Components: No system components, just basic laboratory research
equipment to verify physical principles;
Integration: No attempt at integration; still trying to see whether
individual parts of the technology work. Lab experiments with available
components show they will work;
Tests and Environment: Uses of the observed properties are postulated
and experimentation with potential elements of subsystem begins. Lab
work to validate pieces of technology without trying to integrate.
Emphasis is on validating the predictions made during earlier
analytical studies to ensure that the technology has a firm scientific
underpinning.
Technology readiness level(TRL): 4. Component and/or breadboard
validation in lab environment;
Level involved: Low fidelity breadboard;
Basic objective of TRLs: Demonstrate technical feasibility and
functionality;
Components: Ad Hoc and available laboratory components are surrogates
for system components that may require special handling, calibration,
or alignment to get them to function. Not fully functional but
representative of technically feasible approach;
Integration: Available components assembled into subsystem breadboard.
Interfaces between components are realistic;
Tests and Environment: Tests in controlled laboratory environment. Lab
work at less than full subsystem integration, although starting to see
if components will work together.
Technology readiness level(TRL): 5. Component and/or breadboard
validation in relevant environment;
Level involved: High fidelity breadboard/brass-board (e.g., nonscale or
form components);
Basic objective of TRLs: Demonstrate technical feasibility and
functionality;
Components: Fidelity of components and interfaces are improved from TRL
4. Some special purpose components combined with available laboratory
components. Functionally equivalent but not of same material or size.
May include integration of several components with reasonably realistic
support elements to demonstrate functionality;
Integration: Fidelity of subsystem mock up improves (e.g., from
breadboard to brassboard). Integration issues become defined;
Tests and Environment: Laboratory environment modified to approximate
operational environment. Increases in accuracy of the controlled
environment in which it is tested.
Technology readiness level(TRL): 6. System/Subsystem model or prototype
demonstration in relevant environment;
Level involved: Subsystem closely configured for intended project
application. Demonstrated in relevant environment. (Shows will work in
desired configuration);
Basic objective of TRLs: Demonstrate applicability to intended project
and subsystem integration.
(Specific to intended application in project.);
Components: Subsystem is high fidelity functional prototype with (very
near same material and size of operational system). Probably includes
the integration of many new components and realistic supporting
elements/subsystems if needed to demonstrate full functionality.
Partially integrated with existing systems;
Integration: Components are functionally compatible (and very near same
material and size of operational system). Component integration into
system is demonstrated;
Tests and Environment: Relevant environment inside or outside the
laboratory, but not the eventual operating environment. The testing
environment does not reach the level of an operational environment,
although moving out of controlled laboratory environment into something
more closely approximating the realities of technology‘s intended use.
Technology readiness level(TRL): 7. Subsystem prototype demonstration
in an operational environment;
Level involved: Subsystem configured for intended project application.
Demonstrated in operational environment;
Basic objective of TRLs: Demonstrate applicability to intended project
and subsystem integration.
(Specific to intended application in project.);
Components: Prototype improves to preproduction quality. Components are
representative of project components (material, size, and function) and
integrated with other key supporting elements/subsystems to demonstrate
full functionality. Accurate enough representation to expect only minor
design changes;
Integration: Prototype not integrated into intended system but onto
surrogate system;
Tests and Environment: Operational environment, but not the eventual
environment. Operational testing of system in representational
environment. Prototype will be exposed to the true operational
environment on a surrogate platform, demonstrator, or test bed.
Technology readiness level(TRL): 8. Total system completed, tested, and
fully demonstrated;
Level involved: Full integration of subsystems to show total system
will meet requirements;
Basic objective of TRLs: Applied/Integrated into intended project
application;
Components: Components are right material, size, and function
compatible with operational system;
Integration: Subsystem performance meets intended application and is
fully integrated into total system;
Tests and Environment: Demonstration, test, and evaluation completed.
Demonstrates system meets procurement specifications. Demonstrated in
eventual environment.
Technology readiness level(TRL): 9. Total system used successfully in
project operations;
Level involved: System meeting intended operational requirements;
Basic objective of TRLs: Applied/Integrated into intended project
application;
Components: Components are successfully performing in the actual
environment”proper size, material, and function;
Integration: Subsystem has been installed and successfully deployed in
project systems;
Tests and Environment: Operational testing and evaluation completed.
Demonstrates that system is capable of meeting all mission
requirements.
Source: GAO analysis of DOD data.
[End of table]
[End of section]
Appendix VI: Comparison of DOD's Product Development Process with DOE's
Project Management Process:
[See PDF for Image]
Source: GAO analysis of DOD and DOE data.
[End of section]
Appendix VII: Comments from the Department of Energy:
Department of Energy:
Washington, DC 20585:
Mar 7 2007:
Mr. Gene Aloise:
Director, Natural Resources and Environment:
U.S. Government Accountability Office:
441 G Street NW:
Washington, DC 20548:
Dear Mr. Aloise:
The Department of Energy (DOE or Department) has reviewed the draft
Government Accountability Office (GAO) report entitled "Major
Construction Projects Need a Consistent Approach for Assessing
Technology Readiness to Help Avoid Cost Increases and Delays" (GAO-07-
336).
The report asserts broadly that the Department's management of projects
has led to growth in cost and schedules, but the report's
recommendations focus only on implementing a technology assessment
process. The Department agrees that appropriately assessing technology
readiness can be a significant factor in successfully completing its
projects. However, the Department has concerns with (1) the manner in
which GAO characterizes cost and schedule growth from a small sample of
our projects, and (2) the establishment of a standard for evaluating
(and approving) technology readiness for all DOE projects without
appropriate study.
The report inappropriately characterizes the cost growth associated
with DOE projects by using preliminary cost and schedule estimates
intended for internal DOE planning rather than validated and approved
baselines. These cost and schedule baselines are established at
Critical Decision-2 after having been validated by the Department's
Office of Engineering and Construction Management and approved by the
applicable Acquisition Executive. Tables 1 and 2, which compare
inconsistent combinations of preliminary and validated data, should be
deleted from the report or revised using only validated data for
initial and current baselines to allow for an accurate comparison. The
findings and conclusions supported by these tables should also be
corrected. While the report caveats the basis for cost and schedule
growth in the tables, the explanation provided is not sufficient to
prevent readers from reaching the wrong conclusions regarding cost and
schedule growth for the sample of projects in the study.
The report also attempts to identify the causes for the cost and
schedule increases in Table 3. The information contained in Table 3
should be revised to more clearly identify the correlation between cost
and schedule growth and technology maturity.
Additionally, GAO's data in Appendix IV, Survey Results, focuses on
communication, changes in "political will", funding uncertainties, and
project manager experience. It is unclear how these factors lead to the
recommendation to assess technology readiness.
Despite concerns with the study and report, the Department believes
GAO's recommendation regarding technology readiness analysis has merit.
The Department plans to pilot application of the technology readiness
assessment process on selected projects in order to better understand
the process and evaluate its potential use in its diverse portfolio of
projects. If warranted, DOE could implement a consistent technology
readiness assessment as part of its established project management
system. The assessment process would likely be applied to projects on a
case by case basis where critical technologies are being used.
The Department suggests that GAO's recommendation be restated as
follows:
As one tool to improve project planning and execution, we recommend
that the Secretary of Energy evaluate the use of a technology readiness
assessment for projects that involve critical technologies.
* The Department of Energy should consider lessons learned from NASA
and DoD on the use of Technology Readiness Levels (TRLs), as well as
the Department's own experience in determining technology readiness, as
it evaluates protocols for assessing and communicating the readiness of
critical project technologies.
* The Acquisition Executive should consider project risk imposed by
immature technology, and on a case by case basis, define an appropriate
level of technology readiness prior to establishing the project's
baseline.
Additional project-specific comments and corrections are attached. The
Department requests that this response letter be included in GAO's
final report.
Sincerely,
Signed by:
Ingrid Kolb:
Director:
Office of Management:
Attachment:
[End of section]
Appendix VIII: GAO Contact and Staff Acknowledgments:
GAO Contact:
Gene Aloise, (202) 512-3841:
Staff Acknowledgments:
In addition to the individual named above, Michaela Brown, Rudy
Chatlos, James Espinoza, Daniel Feehan (Assistant Director), Joseph
Keener, Thomas Kingham, Matthew Lea, Mehrzad Nadji, Omari Norman,
Christopher Pacheco, Thomas Perry, and Carol Herrnstadt Shulman made
key contributions to this report.
FOOTNOTES
[1] For this review, we lowered the threshold to $300 million out of
concern that some projects not considered major could later be defined
as major because of cost increases.
[2] This estimate includes design and construction costs, but does not
reflect the total life-cycle costs of the projects, such as operating
and maintenance costs.
[3] GAO, Hanford Waste Treatment Plant: Contractor and DOE Management
Problems Have Led to Higher Costs, Construction Delays, and Safety
Concerns, GAO-06-602T (Washington, D.C.: Apr. 6, 2006).
[4] A forthcoming GAO report will address actions taken by DOE to
improve overall project management.
[5] GAO, Best Practices: Better Management of Technology Development
Can Improve Weapon System Outcomes, GAO/NSIAD-99-162 (Washington, D.C.:
July 30, 1999); Defense Acquisitions: Assessments of Selected Major
Weapons Programs, GAO-06-391 (Washington, D.C.: Mar. 31, 2006); and
Defense Acquisitions: Space-Based Radar Effort Needs Additional
Knowledge before Starting Development, GAO-04-759 (Washington, D.C.:
July 23, 2004).
[6] GAO, Department of Energy: Further Actions Are Needed to Strengthen
Contract Management for Major Projects, GAO-05-123 (Washington, D.C.:
Mar. 18, 2005); and Civil Engineering Research Foundation, Independent
Research Assessment of Project Management Factors Affecting Department
of Energy Project Success (Washington, D.C: July 12, 2004).
[7] GAO, Oversight of DOE's Major Systems, GAO/RCED-97-146R
(Washington, D.C.: Apr. 30, 1997).
[8] The National Research Council was organized by the National Academy
of Sciences to advise the federal government on matters related to
science and technology.
[9] National Research Council, Improving Project Management in the
Department of Energy (Washington, D.C.: July 1999).
[10] Civil Engineering Research Foundation, Independent Research
Assessment of Project Management Factors Affecting Department of Energy
Project Success (Washington, D.C.: July 12, 2004).
[11] Department of Energy, Office of Inspector General, Audit Report:
Status of the Mixed Oxide Fuel Fabrication Facility, DOE/IG-0713
(Washington, D.C.: December 2005).
[12] GAO-06-602T.
[13] Department of Energy, Office of Inspector General, Audit Report:
The Department of Energy's Tritium Extraction Facility, DOE/IG-0560
(Washington, D.C.: June 2002).
[14] These 6 projects are the Mixed Oxide Fuel Fabrication Facility,
National Ignition Facility, Pit Disassembly and Conversion Facility,
Spallation Neutron Source, Tritium Extraction Facility, and Waste
Treatment and Immobilization Plant.
[15] These 5 projects are the Chemistry and Metallurgy Research
Facility Replacement, Depleted Uranium Hexafluoride 6 Conversion
Facility, Mixed Oxide Fuel Fabrication Facility, Pit Disassembly and
Conversion Facility, and Salt Waste Processing Facility.
[16] Los Alamos National Laboratory, Options for the Development and
Testing of the Pit Disassembly and Conversion Facility Government-
Furnished Design, LA-UR-03-3926 (Los Alamos, New Mexico: June 11,
2003).
[17] Pulse jet mixers, which do not have moving parts, use compressed
air to continuously mix tank waste so that it can be properly prepared
for further processing. While such devices have previously been used
successfully in other applications, they have never been used for
mixing wastes with high-solid content like those at the Waste Treatment
and Immobilization Plant.
[18] GAO/NSIAD-99-162 and GAO, Joint Strike Fighter Acquisition: Mature
Critical Technologies Needed to Reduce Risks, GAO-02-39 (Washington,
D.C.: Oct. 19, 2001).
[19] Improving Project Management.
[20] GAO, Department of Energy: Uncertainties and Management Problems
Have Hindered Cleanup at Two Nuclear Waste Sites, GAO/T-RCED-00-248
(Washington, D.C.: July 12, 2000).
[21] Burns and Roe Enterprises, Inc., External Independent Review of
the Basis of Design for the Aqueous Polishing Process for the Mixed
Oxide Fuel Fabrication Facility at The Savannah River Site for the U.S.
Department of Energy Office of Engineering and Construction Management
and National Energy Technology Laboratory Report, BREI-LSP-R-06-01
(Oradell, New Jersey: March 2006).
[22] GAO/NSIAD-99-162.
[23] Department of Energy, Tracking Technology Maturity in DOE's
Environmental Management Science and Technology Program; Revision 1
(Washington, D.C.: Jan. 1, 2001).
[24] We excluded the Yucca Mountain Repository project, with a total
estimated cost of $23 billion, from our review due to its uniqueness
and the fact that we have recently reported on the project and
currently have an ongoing review. Also, to review projects with
sufficient maturity, we included only the projects that were at least 1
year past completion of conceptual design.
[25] GAO, Department of Energy: Further Actions Are Needed to
Strengthen Contract Management for Major Projects, GAO-05-123
(Washington, D.C.: Mar. 18, 2005); and Civil Engineering Research
Foundation, Independent Research Assessment of Project Management
Factors Affecting Department of Energy Project Success (Washington,
D.C.: July 12, 2004).
[26] National Research Council, Improving Project Management in the
Department of Energy (Washington, D.C.: July 1999).
GAO's Mission:
The Government Accountability Office, the audit, evaluation and
investigative arm of Congress, exists to support Congress in meeting
its constitutional responsibilities and to help improve the performance
and accountability of the federal government for the American people.
GAO examines the use of public funds; evaluates federal programs and
policies; and provides analyses, recommendations, and other assistance
to help Congress make informed oversight, policy, and funding
decisions. GAO's commitment to good government is reflected in its core
values of accountability, integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through GAO's Web site (www.gao.gov). Each weekday, GAO posts
newly released reports, testimony, and correspondence on its Web site.
To have GAO e-mail you a list of newly posted products every afternoon,
go to www.gao.gov and select "Subscribe to Updates."
Order by Mail or Phone:
The first copy of each printed report is free. Additional copies are $2
each. A check or money order should be made out to the Superintendent
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or
more copies mailed to a single address are discounted 25 percent.
Orders should be sent to:
U.S. Government Accountability Office 441 G Street NW, Room LM
Washington, D.C. 20548:
To order by Phone: Voice: (202) 512-6000 TDD: (202) 512-2537 Fax: (202)
512-6061:
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Web site: www.gao.gov/fraudnet/fraudnet.htm E-mail: fraudnet@gao.gov
Automated answering system: (800) 424-5454 or (202) 512-7470:
Congressional Relations:
Gloria Jarmon, Managing Director, JarmonG@gao.gov (202) 512-4400 U.S.
Government Accountability Office, 441 G Street NW, Room 7125
Washington, D.C. 20548:
Public Affairs:
Paul Anderson, Managing Director, AndersonP1@gao.gov (202) 512-4800
U.S. Government Accountability Office, 441 G Street NW, Room 7149
Washington, D.C. 20548:
