Yucca Mountain
Quality Assurance at DOE's Planned Nuclear Waste Repository Needs Increased Management Attention Gao ID: GAO-06-313 March 17, 2006The Department of Energy (DOE) is working to obtain a license from the Nuclear Regulatory Commission (NRC) to construct a nuclear waste repository at Yucca Mountain in Nevada. The project, which began in the 1980s, has been beset by delays. In a 2004 report, GAO raised concerns that persistent quality assurance problems could further delay the project. Then, in 2005, DOE announced the discovery of employee e-mails suggesting quality assurance problems, including possible falsification of records. Quality assurance, which establishes requirements for work to be performed under controlled conditions that ensure quality, is critical to making sure the project meets standards for protecting public health and the environment. GAO was asked to examine (1) the history of the project's quality assurance problems, (2) DOE's tracking of these problems and efforts to address them since GAO's 2004 report, and (3) challenges facing DOE as it continues to address quality assurance issues within the project.
DOE has had a long history of quality assurance problems at the Yucca Mountain project. In the 1980s and 1990s, DOE had problems assuring NRC that it had developed adequate plans and procedures related to quality assurance. More recently, as it prepares to submit a license application for the repository to NRC, DOE has been relying on costly and time-consuming rework to resolve lingering quality assurance problems uncovered during audits and after-the-fact evaluations. DOE announced, in 2004, that it was making a commitment to continuous quality assurance improvement and that its efforts would be tracked by performance indicators that would enable it to assess progress and direct management attention as needed. However, GAO found that the project's performance indicators and other key management tools were not effective for this purpose. For example, the management tools did not target existing areas of concern and did not track progress in addressing them. The tools also had weaknesses in detecting and highlighting significant problems for management attention. DOE continues to face quality assurance and other challenges. First, DOE is engaged in extensive efforts to restore confidence in scientific documents because of the quality assurance problems suggested in the discovered e-mails between project employees, and it has about 14 million more project e-mails to review. Second, DOE faces quality assurance challenges in resolving design control problems associated with its requirements management process--the process for ensuring that high-level plans and regulatory requirements are incorporated into specific engineering details. Problems with the process led to the December 2005 suspension of certain project work. Third, DOE continues to be challenged to manage a complex program and organization. Significant personnel and project changes initiated in October 2005 create the potential for confusion over roles and responsibilities--a situation DOE found to contribute to quality assurance problems during an earlier transition.
RecommendationsOur recommendations from this work are listed below with a Contact for more information. Status will change from "In process" to "Open," "Closed - implemented," or "Closed - not implemented" based on our follow up work.
Director: Team: Phone:GAO-06-313, Yucca Mountain: Quality Assurance at DOE's Planned Nuclear Waste Repository Needs Increased Management Attention
This is the accessible text file for GAO report number GAO-06-313
entitled 'Yucca Mountain: Quality Assurance at DOE's Planned Nuclear
Waste Repository Needs Increased Management Attention' which was
released on March 23, 2006.
This text file was formatted by the U.S. Government Accountability
Office (GAO) to be accessible to users with visual impairments, as part
of a longer term project to improve GAO products' accessibility. Every
attempt has been made to maintain the structural and data integrity of
the original printed product. Accessibility features, such as text
descriptions of tables, consecutively numbered footnotes placed at the
end of the file, and the text of agency comment letters, are provided
but may not exactly duplicate the presentation or format of the printed
version. The portable document format (PDF) file is an exact electronic
replica of the printed version. We welcome your feedback. Please E-mail
your comments regarding the contents or accessibility features of this
document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
Report to the Chairman, Subcommittee on the Federal Workforce and
Agency Organization, Committee on Government Reform, House of
Representatives:
March 2006:
Yucca Mountain:
Quality Assurance at DOE's Planned Nuclear Waste Repository Needs
Increased Management Attention:
[Hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-06-313]:
GAO Highlights:
Highlights of GAO-06-313, a report to the Chairman, Subcommittee on the
Federal Workforce and Agency Organization, Committee on Government
Reform, House of Representatives:
Why GAO Did This Study:
The Department of Energy (DOE) is working to obtain a license from the
Nuclear Regulatory Commission (NRC) to construct a nuclear waste
repository at Yucca Mountain in Nevada. The project, which began in the
1980s, has been beset by delays. In a 2004 report, GAO raised concerns
that persistent quality assurance problems could further delay the
project. Then, in 2005, DOE announced the discovery of employee e-mails
suggesting quality assurance problems, including possible falsification
of records. Quality assurance, which establishes requirements for work
to be performed under controlled conditions that ensure quality, is
critical to making sure the project meets standards for protecting
public health and the environment.
GAO was asked to examine (1) the history of the project‘s quality
assurance problems, (2) DOE‘s tracking of these problems and efforts to
address them since GAO‘s 2004 report, and (3) challenges facing DOE as
it continues to address quality assurance issues within the project.
What GAO Found:
DOE has had a long history of quality assurance problems at the Yucca
Mountain project. In the 1980s and 1990s, DOE had problems assuring NRC
that it had developed adequate plans and procedures related to quality
assurance. More recently, as it prepares to submit a license
application for the repository to NRC, DOE has been relying on costly
and time-consuming rework to resolve lingering quality assurance
problems uncovered during audits and after-the-fact evaluations.
DOE announced, in 2004, that it was making a commitment to continuous
quality assurance improvement and that its efforts would be tracked by
performance indicators that would enable it to assess progress and
direct management attention as needed. However, GAO found that the
project‘s performance indicators and other key management tools were
not effective for this purpose. For example, the management tools did
not target existing areas of concern and did not track progress in
addressing them. The tools also had weaknesses in detecting and
highlighting significant problems for management attention.
DOE continues to face quality assurance and other challenges. First,
DOE is engaged in extensive efforts to restore confidence in scientific
documents because of the quality assurance problems suggested in the
discovered e-mails between project employees, and it has about 14
million more project e-mails to review. Second, DOE faces quality
assurance challenges in resolving design control problems associated
with its requirements management process”the process for ensuring that
high-level plans and regulatory requirements are incorporated into
specific engineering details. Problems with the process led to the
December 2005 suspension of certain project work. Third, DOE continues
to be challenged to manage a complex program and organization.
Significant personnel and project changes initiated in October 2005
create the potential for confusion over roles and responsibilities”a
situation DOE found to contribute to quality assurance problems during
an earlier transition.
View of Yucca Mountain and the Exploratory Tunnel for the Repository:
[See PDF for image]
[End of figure]
What GAO Recommends:
GAO recommends five actions DOE can take to improve the project‘s
management tools and identify and address quality assurance and other
problems.
In oral comments, DOE agreed with GAO‘s recommendations.
www.gao.gov/cgi-bin/getrpt?GAO-06-313.
To view the full product, including the scope and methodology, click on
the link above. For more information, contact Jim Wells at (202) 512-
3841 or wellsj@gao.gov.
[End of section]
Contents:
Letter:
Results in Brief:
Background:
DOE Has a Long History of Quality Assurance Problems at Yucca Mountain
and Is Relying on Costly and Time-Consuming Measures to Correct
Problems Before Submitting Its License Application for the Repository:
DOE Cannot Be Certain Its Efforts to Improve Quality Assurance Have
Been Effective Because of Weaknesses in Tracking Progress and
Identifying Problems:
DOE's 'New Path Forward' for Preparing to Submit Its License
Application Faces Substantial Quality Assurance and Other Challenges:
Conclusions:
Recommendations for Executive Action:
Agency Comments:
Appendixes:
Appendix I: Objectives, Scope, and Methodology:
Appendix II: Yucca Mountain Project Employee Concerns Programs:
Appendix III: GAO Contact and Staff Acknowledgments:
Tables:
Table 1: Visibility of Management Improvement Initiatives'
Effectiveness Indicators in Annunciator Panel When Last Prepared (Using
August 2005 Data):
Table 2: Key Indicators for Processes with Quality Elements, Their
Intended Focus, and Number of Times They Changed (March through August
2005):
Table 3: Employee Concerns Opened for Investigation under DOE's
Employee Concerns Program by Category of Concern, January through
November 2005:
Table 4: Employee Concerns Opened for Investigation under BSC's
Employee Concerns Program by Category of Concern, January through
November 2005:
Figures:
Figure 1: A Yucca Mountain Project Scientist Conducts Water
Infiltration Tests inside Yucca Mountain:
Figure 2: The 1.7-Mile Tunnel Built for Scientific Studies near the
Potential Repository Area:
Abbreviations:
AMR: Analysis and Model Report:
BSC: Bechtel/SAIC Company, LLC:
CAP: Corrective Action Program:
CR: condition report:
DOE: Department of Energy:
EPA: Environmental Protection Agency:
Initiatives: Management Improvement Initiatives:
INPO: Institute of Nuclear Power Operations:
LSN: Licensing Support Network:
NRC: Nuclear Regulatory Commission:
OCRWM: Office of Civilian Radioactive Waste Management:
RIT: Regulatory Integration Team:
USGS: U.S. Geological Survey:
Letter March 17, 2006:
The Honorable Jon C. Porter:
Chairman:
Subcommittee on the Federal Workforce and Agency Organization:
Committee on Government Reform:
House of Representatives:
Dear Mr. Chairman:
The nuclear waste created as a by-product of the nuclear power process
in reactors can remain highly radioactive for hundreds of thousands of
years, and will require proper disposal to protect public health and
the environment. Over 50,000 metric tons of this waste is currently
being stored at 72 sites around the country, principally at commercial
nuclear power plants. These wastes have been accumulating for several
decades in surface storage designed to be temporary. The Nuclear Waste
Policy Act of 1982 obligated the Department of Energy (DOE) to
construct an underground geological repository for permanent storage
and begin accepting these wastes by January 31, 1998. However, it was
not until 2002, after more than 15 years of scientific study, that
Yucca Mountain in Nevada was approved by Congress as a suitable
location for the repository. DOE is continuing to experience delays,
and it does not currently have a schedule for when construction of the
repository will begin. The project to build and operate a repository at
Yucca Mountain is highly complex. It is also highly controversial among
some of the public, in large part, because of their concern that the
repository may not be adequate, over the long term, to prevent the
release of radioactive material to the environment. DOE has established
quality assurance procedures to ensure that its work relative to the
project and the technical information it produces are of high quality
and defensible. However, persistent problems with implementing these
procedures and resulting questions about the quality of the work have
significantly contributed to project delays. Resolving these quality
issues is essential to proceeding with construction.
To construct a repository at the Yucca Mountain site, DOE must obtain a
license from the Nuclear Regulatory Commission (NRC). As part of the
licensing process, DOE must demonstrate to NRC that its plans for the
repository will meet standards for protecting public health and the
environment from harmful exposure to the radioactive waste. The
Environmental Protection Agency (EPA) set these standards in 2001, but
as a result of a 2004 court ruling, EPA is proposing to revise the
standards to extend the protection period from 10,000 years to 1
million years.[Footnote 1]
To demonstrate that it can meet these standards, DOE has been
conducting scientific and technical studies at the Yucca Mountain site
that will serve as supporting documentation for DOE's planned license
application. For example, it has developed mathematical models to
measure the probability that various combinations of natural and
engineered (human-made) features of a repository could safely contain
waste for the long term; the models take into account possible water
infiltration through the mountain (see fig. 1), earthquakes, volcanic
action, or other scenarios. Thus, one of DOE's most important tasks
during the licensing process will be to demonstrate the adequacy of its
data, software, and models. Accordingly, NRC requires nuclear
facilities to develop a quality assurance program that ensures that the
technical information submitted in support of a license application--
such as scientific data, models, and details on design and
construction--is well documented and defensible. More specifically,
data used to support conclusions about the safety and design of the
repository must meet transparency and traceability standards. That is,
the data must be clear in justifying and explaining any underlying
assumptions, calculations, and conclusions, and must be capable of
being traced back to original source materials.
Figure 1: A Yucca Mountain Project Scientist Conducts Water
Infiltration Tests inside Yucca Mountain:
[See PDF for image]
[End of figure]
To meet NRC's requirements, DOE established a quality assurance program
for the Yucca Mountain project. The program establishes requirements
that scientific, design, engineering, and other work, such as
procurement and record keeping, is to be performed under controlled
conditions that ensure quality and enable the work to be verified by
others. For example, the program establishes general requirements for
calibrating equipment before conducting tests, stipulating when and how
the equipment should be calibrated and how to document the results. The
project's line organizations, which are responsible for carrying out
various functions or aspects of the work, then create their own
policies and procedures to implement the requirements.
Project employees are required to follow such procedures to help ensure
the reliability of project information. Quality assurance auditors
periodically verify that the procedures have been followed. Project
employees, including quality assurance auditors, are required to
identify when procedures are not being followed or when they encounter
problems with the procedures. These problems can be identified in
"condition reports" under the project's Corrective Action Program,
which establishes procedures for the prompt identification and
correction of problems. Alternatively, project employees can submit
problems for resolution through the Employee Concerns Program, which
allows for submissions to be confidential or anonymous.
Because quality assurance plays a key role in ensuring that the
information DOE uses to support its license application is of high
quality and fully defensible, problems in this area raise concerns
about delays to DOE's submission and NRC's review of the license
application. In April 2004, for example, we reported that recurring
quality assurance problems at the Yucca Mountain project could delay
the licensing and operation of the repository.[Footnote 2] As we noted,
a 2004 NRC evaluation found quality assurance problems such as data
that could not be readily traced back to their sources. NRC indicated
that unless DOE rectified such problems before submitting the license
application, NRC could be in the position of requesting large volumes
of additional information, which could prevent it from making a
decision on the license within the time required by law. Then, in early
2005, DOE reported it had discovered a series of e-mail messages among
some U.S. Geological Survey (USGS) employees working on the Yucca
Mountain project under a contract with DOE that appeared to imply that
workers had falsified records for scientific work. Several of these
messages, written in the late 1990s, appeared to show disdain for the
project's quality assurance program and its requirements. As a result
of these e-mails, DOE is engaging in an extensive review of records to
restore confidence in scientific documents that will be used to support
its license application.
DOE's recent efforts to better manage quality assurance problems
include its Management Improvement Initiatives (Initiatives), which
began in 2002 and were reported completed in April 2004. The
Initiatives' purpose was to ensure that work and products consistently
met quality objectives and were fully defensible by establishing a
foundation for continuous improvement in areas of identified management
weaknesses. In our 2004 report, we concluded that, while DOE considered
the Initiatives to have been completed, it could not assess their
effectiveness in addressing the management weaknesses because its
performance goals lacked objective measures and time frames for
determining success.[Footnote 3] By the end of the Initiatives, DOE had
established two tools to alert management about quality-related and
other problems: (1) a one-page summary of performance indicators for
key project activities and processes (the summary, which DOE refers to
as a "panel," is prepared monthly for discussion and action by project
managers) and (2) quarterly trend evaluation reports analyzing patterns
and trends in problems identified through the Corrective Action
Program. Then, in October 2005, DOE initiated planning for an
aggressive series of changes to the facility design, organization, and
management of the Yucca Mountain project. This effort, known as the
"new path forward," is intended to address quality assurance and other
challenges prior to submission of a license application. According to
the project's Acting Director, DOE will be considering changes in
performance indicators and other management tools to better support the
new path forward.
In this context, you requested that we provide additional information
on the project's quality assurance problems and DOE's efforts to
correct them. As agreed with your office, this report discusses (1) the
history of the project's quality assurance problems since its start in
the 1980s, (2) DOE's tracking of quality problems and progress
implementing quality assurance requirements since our April 2004
report, and (3) challenges DOE faces as it continues to address quality
assurance issues at the project. In addition, you asked for information
about concerns raised in recent years through the project's Employee
Concerns Program, which is provided in appendix II.
To determine the history of quality assurance problems, we reviewed
previous GAO, DOE, and NRC documents, visited the project, and
interviewed officials from DOE, NRC, and Bechtel/SAIC Company, LLC
(BSC), which is DOE's management contractor for the Yucca Mountain
project. To assess DOE's tracking of quality-related problems and
progress in addressing them, we examined management tools and
associated documentation, such as monthly indicator panels and
quarterly trend reports, and interviewed BSC and DOE officials
regarding those tools. To identify current quality assurance and other
challenges, we attended quarterly NRC management meetings, interviewed
the Acting Director and other senior managers of the DOE project, and
gathered information on management turnover. Due to the criminal
investigation under way related to possible falsification of records
implied in USGS e-mail exchanges, we did not examine the investigated
issues beyond confirming that a concern about the e-mails had been
submitted to the Employee Concerns Program. However, to determine if
concerns about other instances of potential falsification of records
had been raised by project employees, we reviewed employee concerns
filed with the project's Employee Concerns Program from January 2004 to
December 2005. More information on our scope and methodology is
provided in appendix I. We conducted our work from July 2005 through
January 2006 in accordance with generally accepted government auditing
standards.
Results in Brief:
DOE has had a long history of quality assurance problems at the Yucca
Mountain project. In the late 1980s and early 1990s, DOE had problems
assuring NRC that it had developed adequate plans and procedures
related to quality assurance. For example, as GAO reported in 1988, NRC
had found that DOE's quality assurance procedures were inadequate and
its efforts to independently identify and resolve weaknesses in the
procedures were ineffective. By the late 1990s, DOE had largely
addressed NRC's concerns about its plans and procedures, but its own
audits identified quality assurance problems with the data, software,
and models used in the scientific work supporting its potential license
application. For example, in 1998, a team of project personnel
determined that 87 percent of the models used to simulate the site's
natural and environmental conditions, and to demonstrate the future
repository's performance over time, did not comply with requirements
for demonstrating their accuracy in predicting geologic events. More
recently, as it prepares to submit the license application for the
planned repository to NRC, DOE has been relying on costly and time-
consuming rework to resolve lingering quality assurance concerns. For
example, to address problems with the transparency and traceability of
scientific work in technical documents, DOE implemented, in the spring
of 2004, a roughly $20 million, 8-month project called the Regulatory
Integration Team. This effort involved about 150 full-time employees
from DOE, USGS, and multiple national laboratories, such as Sandia, Los
Alamos, and Lawrence Livermore, working to inspect technical documents
to identify and resolve quality problems.
DOE cannot be certain that its efforts to improve the implementation of
its quality assurance requirements have been effective because it
adopted management tools that did not target existing management
concerns and did not track progress with significant and recurring
problems. Although DOE announced, in 2004, that it was making a
commitment to continuous quality assurance improvement and that its
efforts would be tracked by performance indicators that would enable it
to assess progress and direct management attention as needed, its
adopted management tools have not been effective for this purpose.
Specifically, the one-page summary, or "panel," of selected performance
indicators that project managers used in monthly management meetings
was not an effective tool for assessing progress. The indicators
selected for the panel poorly represented the major management concerns
and changed frequently. For example, the panel did not include an
indicator to represent the management concern about unclear roles and
responsibilities--a problem that could undermine accountability within
the project. Use of the indicator panel was discontinued in late 2005,
and DOE is deciding on a tool to replace it. Moreover, a second
management tool--trend evaluation reports--also did not track relevant
concerns. The reports generally had technical weaknesses for
identifying recurrent and significant problems and inconsistently
tracked progress in resolving the problems. For example, lacking
reliable data and an appropriate performance benchmark for determining
the significance of human errors as a cause of quality problems, DOE's
trend reports offered no clear basis for tracking progress on such
problems. In addition, under the trend reports' rating categories, the
rating assigned to convey the significance of a problem was overly
influenced by a judgment that there were already ongoing management
actions to address the problem, rather than solely assessing the
problem's significance. For example, the trend report's rating of one
particular problem at the lowest level of significance did not
accurately describe the problem or sufficiently draw management's
attention to it.
Before DOE submits a license application, its aggressive "new path
forward" effort faces substantial quality assurance and other
challenges. First, the March 2005 announcement of the discovery of USGS
e-mails suggesting the possible falsification of quality assurance
records has resulted in extensive efforts to restore confidence in
scientific documents, and DOE is conducting a wide-ranging review of
approximately 14 million e-mails to determine whether they raise
additional quality assurance issues. Such a review creates a challenge
not just because of the sheer volume of e-mails to be reviewed, but
also because DOE will have to decipher their meaning and determine
their significance, sometimes without clarification from authors who
have left the project. Furthermore, if any of the e-mails raise quality
assurance concerns, further review, inspection, or rework may need to
be performed to resolve any newfound problems. Second, DOE faces
quality assurance challenges in resolving design control problems
associated with an inadequate requirements management process--the
process responsible for ensuring that broad plans and regulatory
requirements affecting the project are tracked and incorporated into
specific engineering details. In December 2005, DOE issued a stop-work
order on some design and engineering work until DOE can determine that
the requirements management process has been improved. Third, DOE
continues to be challenged by managing a changing and complex program
and organization. Significant project changes initiated in October 2005
under the new path forward create the potential for confusion over
accountability as roles and responsibilities change--a situation DOE
found to contribute to quality assurance problems during an earlier
transition period. For example, a proposed reorganization, establishing
a lead laboratory to assist the project, not only would have to be
effectively managed, but also would introduce a new player whose
accountability DOE would have to ensure. DOE has also experienced
turnover in 9 of 17 key management positions since 2001--including
positions related to quality assurance--that has created management
continuity challenges. For example, the director position for the
project has been occupied by three individuals since 1999 and is
currently occupied by an acting director. Since DOE is still
formulating its plans, it is too early to determine whether its new
effort will effectively resolve these challenges.
We are making recommendations to DOE aimed at improving the
effectiveness of its management tools for monitoring performance in key
areas, including quality assurance, by improving the tools' ability to
identify problems and track progress in addressing them. We provided
DOE and NRC with draft copies of this report for their review and
comment. In comments, DOE agreed with our recommendations. Both DOE and
NRC provided technical and editorial comments that we incorporated into
the report, as appropriate.
Background:
Congress enacted the Nuclear Waste Policy Act of 1982 to establish a
comprehensive policy and program for the safe, permanent disposal of
commercial spent fuel and other highly radioactive wastes in one or
more mined geologic repositories. The act charged DOE with (1)
establishing criteria for recommending sites for repositories; (2)
"characterizing" (investigating) three sites to determine each site's
suitability for a repository (1987 amendments to the act directed DOE
to investigate only the Yucca Mountain site); (3) recommending one
suitable site to the President, who, if he considered the site
qualified for a license application, would submit a recommendation to
Congress; and (4) seeking a license from NRC to construct and operate a
repository at the approved site. The act created the Office of Civilian
Radioactive Waste Management within DOE to manage its nuclear waste
program.
Since the 1980s, DOE has spent years conducting site characterization
studies at the Yucca Mountain site to determine whether it is suitable
for a high-level radioactive waste and spent nuclear fuel repository.
DOE, for example, has completed numerous scientific studies of the
mountain and its surrounding region for water flow and the potential
for rock movement, including volcanoes and earthquakes that might
adversely affect the performance of the repository. To allow scientists
and engineers greater access to the rock being studied, DOE excavated
two tunnels for studying the deep underground environment: (1) a five-
mile main tunnel that loops through the mountain, with several research
areas or alcoves connected to it; and (2) a 1.7-mile tunnel that
crosses the mountain (see fig. 2). This second tunnel allows scientists
to study properties of the rock and the behavior of water near the
potential repository area. In July 2002, Congress approved the
President's recommendation of the Yucca Mountain site for the
development of a repository.
Figure 2: The 1.7-Mile Tunnel Built for Scientific Studies near the
Potential Repository Area:
[See PDF for image]
[End of figure]
The Yucca Mountain project is currently focused on preparing an
application to obtain a license from NRC to construct a repository. The
required application information includes both repository design work
and scientific analyses. DOE is engaged in necessary tasks such as
compiling information and writing sections of the license application,
and is conducting technical exchanges with NRC staff and addressing key
technical issues identified by NRC to ensure that sufficient supporting
information is provided. It also plans to further develop the design of
the repository, including revised designs for the repository's surface
facilities and canisters to hold the waste. DOE is also identifying and
preparing potentially relevant documentary material that it is required
to make available on NRC's Web-based information system, known as the
Licensing Support Network. This is a critical step because DOE is
required to certify that the documentary material has been identified
and made electronically available no later than 6 months in advance of
submitting the license application.[Footnote 4]
In February 2005, DOE announced that it does not expect the repository
to open until 2012 at the earliest, which is more than 14 years later
than the 1998 goal specified by the Nuclear Waste Policy Act of 1982.
More recently, the conference report for DOE's fiscal year 2006
appropriations observed that further significant schedule slippages for
submitting a license application are likely. Further delays could arise
from factors such as the time needed for EPA to establish revised
radiation standards for Yucca Mountain and for DOE to revise its
technical documents in response. Such delays could be costly because
nuclear utilities, which pay for most of the disposal program through a
fee on nuclear power, have sued DOE, seeking damages for not starting
the removal of spent nuclear fuel from storage at commercial reactors
by the 1998 deadline. Estimates of the potential damages vary widely,
from DOE's estimate of about $5 billion to a nuclear industry's
estimate of about $50 billion, but the cost for the damages will likely
rise if there are further delays to opening the repository.
Given these schedule slippages, Congress has considered other options
for managing existing and future nuclear wastes, such as centralized
interim storage at one or more DOE sites. The conference report for
DOE's fiscal year 2006 appropriations directed DOE to develop a spent
nuclear fuel recycling plan to reuse the fuel. However, according to
the policy organization of the nuclear energy industry, no
technological option contemplated will eliminate the need to ultimately
dispose of nuclear waste in a geologic repository.
In October 2005, the project's Acting Director issued a memorandum
calling for the development of wide-ranging plans for the "new path
forward," DOE's effort to address quality assurance and other
challenges prior to applying for a license. To restore confidence in
scientific documents that will support the license application, some of
the plans will address the need to review and replace USGS work
products, a requirement for USGS to certify its scientific work
products, and establishing a lead national laboratory to assist the
project. Other plans are focused on a new simplified design for the
waste canisters and repository facilities, a design that is expected to
improve the safety and operation of the repository by eliminating the
need to directly handle and process the spent fuel at the repository.
Further, this aggressive effort called for management changes,
including a transition plan; more rigorous project management,
including a new baseline schedule; rescoping existing contracts and
developing new contracts; tracking project hiring actions; a financial
plan; and new reporting indicators.
After DOE submits the license application, NRC plans to take 90 days to
examine the application for completeness to determine whether DOE has
addressed all NRC requirements. One of the reviews for completeness
will include an examination of DOE's documentation of the quality
assurance program to assess whether it addresses all NRC criteria.
These criteria include, among other things, organization, design
control, document control, corrective actions, quality assurance
records, and quality audits. If it deems any part of the application is
incomplete, NRC may either reject the application or require that DOE
furnish the necessary documentation before proceeding with the detailed
technical review of the application. If it deems the application is
complete, NRC will docket the application, indicating its readiness for
a detailed technical review.[Footnote 5]
Once the application is accepted and placed on the docket, NRC will
conduct its 18-month technical review of the application to determine
if the application meets all NRC requirements, including the soundness
of scientific analyses and preliminary facility design, and NRC quality
assurance criteria. If NRC discovers problems with the technical
information used to support the application, it may conduct specific
reviews, including inspections, to determine the extent and effect of
the problem. Because the data, models, and software used in modeling
repository performance are integral parts of this technical review,
quality assurance plays a key role since it is the mechanism used to
verify the accuracy of the information DOE presents in the application.
NRC may conduct reviews, including inspections, of the quality
assurance program if technical problems are identified that are
attributable to quality problems. NRC will hold public hearings chaired
by its Atomic Safety and Licensing Board to examine specific topics.
After completing the proceedings, the board will forward its initial
decision to the NRC commissioners for their review. Finally, within 3
to 4 years from the date that NRC dockets the application, NRC will
make a decision to grant the construction authorization, reject the
application, or grant the construction authorization with
conditions.[Footnote 6] NRC will grant a construction authorization
only if it concludes from its reviews that the repository would meet
its reasonable expectation that the safety and health of workers and
the public would be protected.
DOE Has a Long History of Quality Assurance Problems at Yucca Mountain
and Is Relying on Costly and Time-Consuming Measures to Correct
Problems Before Submitting Its License Application for the Repository:
DOE has repeatedly experienced quality assurance problems with its work
on the Yucca Mountain project. In the late 1980s, DOE had been
challenged to fix and develop adequate plans and procedures related to
quality assurance. By the late 1990s, audits by GAO, DOE, and others
identified recurring quality assurance problems with several aspects of
key scientific data, models, and software. Currently, in preparing to
submit the license application to NRC, DOE is relying on costly and
time-consuming rework to resolve lingering quality assurance problems
with the transparency and traceability of data and in project design
and engineering documents uncovered during audits and after-the-fact
evaluations.
DOE Has Had Problems Implementing and Maintaining an Effective Quality
Assurance Program:
DOE has a long-standing history of attempting to address NRC concerns
about its quality assurance program. Although NRC will have
responsibility for regulating the construction, operation, and
decommissioning (closure) phases of the project, its regulatory and
oversight role does not begin until DOE submits a license application.
As a result, NRC's role in the project has been limited to providing
guidance to DOE to ensure an understanding of NRC regulations and that
the years of scientific and technical work will not later be found
inadequate for licensing purposes. Specifically, since 1984, NRC has
agreed to point out problems it identifies with the quality assurance
program so that DOE can take timely corrective action. Initially, this
NRC guidance was mainly focused on ensuring that DOE had the necessary
quality assurance organization, plans, and procedures.
As we reported in 1988, NRC had reviewed DOE's quality assurance plans
and procedures comprising the principal framework of its quality
assurance program, and concluded that they were inadequate and did not
meet NRC requirements.[Footnote 7] NRC also concluded that DOE's
efforts to independently identify and resolve weaknesses in the plans
and procedures were ineffective. After observing DOE quality assurance
audits, NRC determined that the audits were ineffective for measuring
whether quality assurance procedures were being effectively
implemented. Further, NRC identified additional concerns, during the
1980s, related to DOE management and organizational deficiencies
relating to the quality assurance program. Specifically, among other
things, NRC found the following:
* DOE had a small staff and relied heavily on contractors to provide
quality assurance oversight. Based on its experience in regulating
nuclear power plants, NRC found that these types of organizations
frequently developed major quality-related problems.
* DOE had indirect project control, with administrative and functional
control over the project split between different offices. NRC found
that such project control arrangements tend to have serious quality
assurance-related problems because conflicts can arise between quality
and other organizational goals, such as cost and schedule.
* During a 1984 NRC visit to Nevada, DOE project participants had
expressed the opinion that quality assurance is "unnecessary,
burdensome, and an imposition." Further, in 1986, DOE issued a stop-
work order to the USGS based on a determination that USGS staff did not
appreciate the importance of quality assurance and that USGS work would
not meet NRC expectations. NRC believed that organizational attitudes
can indicate whether a project is likely to experience problems
relating to quality assurance and found such examples troublesome.
Finally, based in part on the information obtained from its oversight
activities, NRC concluded, in 1989, that DOE and its key contractors
had yet to develop and implement an acceptable quality assurance
program.
However, by March 1992, NRC came to the conclusion that DOE had made
significant progress in improving its quality assurance program. NRC
noted that DOE had addressed many of its concerns, specifically that,
among other things, (1) all of the contractor organizations had
developed and were in the process of implementing quality assurance
programs that met NRC requirements, (2) quality assurance management
positions had been filled with full-time DOE personnel with appropriate
knowledge and experience, and (3) DOE had demonstrated that it is
capable of evaluating and correcting deficiencies in the overall
quality assurance program. Nevertheless, in October 1994, NRC found
problems with quality assurance, particularly with the site
contractor's ability to effectively implement corrective actions and
DOE's ability to oversee the site contractor's quality assurance
program.
Recurring Issues with Project Data, Models, and Software Illustrate
DOE's Difficulties Addressing Quality Assurance Problems:
As DOE's quality assurance program matured, it resolved NRC concerns
about its organization, plans, and procedures, and in the late 1990s
began successfully detecting new quality assurance problems in three
areas critical to the repository's successful performance: the adequacy
of the data sources, the validity of scientific models, and the
reliability of computer software developed at the site. These problems
surfaced in 1998 when DOE began to run the initial version of its
performance assessment model. Specifically, DOE was unable to ensure
that critical project data had been collected and tracked back to the
original sources. In addition, DOE did not have a standardized process
for developing scientific models used to simulate a variety of geologic
events or an effective process for ensuring that computer software used
to support the scientific models would work properly. As required by
DOE's quality assurance procedures, the department conducted a root
cause analysis and issued a corrective action plan in 1999. After
corrective actions were taken, DOE considered the issues resolved.
However, in 2001, similar deficiencies associated with models and
software resurfaced. DOE attributed the recurrence to ineffective
procedures and corrective actions, improper implementation of quality
procedures by line managers, and personnel who feared reprisal for
expressing quality concerns. Recognizing the need to correct these
recurring problems, DOE conducted a comprehensive root cause analysis
that included reviews of numerous past self-assessments and independent
program assessments, and identified weaknesses in management systems,
quality processes, and organization roles and responsibilities.
Following the analysis, in July 2002, DOE issued its Management
Improvement Initiatives (Initiatives) that addressed quality problems
with software and models. In addition, DOE added other corrective
actions to address management weaknesses that it found in areas such as
roles and responsibilities, quality assurance processes, written
procedures, corrective action plans, and work environment.
However, DOE continued to face difficulties in resolving quality
assurance problems concerning the data, software, and modeling to be
used in support of the licensing application:
* Data management. As part of NRC's quality assurance requirements,
data used to support conclusions about the safety and design of the
repository must be either collected under a quality assurance program
or subjected to prescribed testing procedures to ensure the data are
accurate for their intended use. In addition, the data supporting these
conclusions must also be traceable back to its original source. In
1998, DOE identified quality assurance problems with the quality and
traceability of data--specifically that some data had not been properly
collected or tested to ensure their accuracy and that data used to
support scientific analysis could not be properly traced back to their
source. DOE again found similar problems in April and September 2003,
when a DOE audit revealed that some data sets did not have the
documentation necessary to trace them back to their sources; the
processes for data control and management were unsatisfactory; and
faulty definitions were developed, which allowed unqualified data to be
used.
* Software management. DOE quality assurance procedures require that
software used to support analysis and conclusions about the performance
and safety of the repository be tested or created in such a way to
ensure that it is reliable. From 1998 to 2003, multiple DOE audits
found recurring quality assurance problems that could affect confidence
in the adequacy of software codes. For example, in 2003, DOE auditors
found problems related to software similar to those found previously in
areas such as technical reviews, software classification, planning,
design, and testing. Further, a team of industry professionals hired by
DOE to assess quality assurance problems with software reported in
February 2004 that these problems kept recurring because DOE did not
assess the effectiveness of its corrective actions and did not
adequately identify the root causes of the problems.
* Model validation. Models are used to simulate natural and
environmental conditions at Yucca Mountain, and to demonstrate the
performance of the future repository over time. However, before models
can be used to support the license application, DOE must demonstrate
through a process called validation that the models are able to
accurately predict geologic events. In 1998, a team of project
personnel evaluated the models and determined that 87 percent did not
comply with the validation requirements. In 2001, and again in 2003,
DOE audits found that project personnel were not properly following
procedures--specifically in the areas of model documentation, model
validation, and checking and review. Further, the 2003 audit concluded
that previous corrective actions designed to improve validation and
reduce errors in model reports were not fully implemented.
After many years of working to address these quality assurance problems
with data, software, and models, DOE had mostly resolved these problems
and closed the last of the associated condition reports by February
2005.
DOE Is Now Relying on Costly and Time-Consuming Rework to Resolve
Additional Problems:
As DOE prepares to submit the Yucca Mountain project license
application to NRC, it has relied on costly and time consuming rework
to ensure that the documents supporting the application are accurate
and complete. Specifically, DOE has relied on inspections and rework by
DOE personnel to resolve quality assurance problems with the
traceability and transparency of technical work products. These efforts
to deal with quality problems at the end, rather than effectively
ensuring that work organizations are producing quality products from
the beginning, add to the project's cost and could potentially delay
DOE's submission of the license application to NRC. In addition, DOE's
efforts indicate that some corrective actions have been ineffective in
resolving problems with the quality assurance process. Further, DOE is
now detecting quality assurance problems in design and engineering work
that are similar to the quality assurance problems it experienced with
its scientific work in the late 1990s.
Although DOE did not initiate its major effort to address these
problems until 2004, the department and NRC for years had known of
quality assurance problems with the traceability and transparency of
technical work products called Analysis and Model Reports (AMR). AMRs
are a key component of the license application, and contain the
scientific analysis and modeling data demonstrating the safety and
performance of the planned repository. Among other quality
requirements, AMRs must be traceable back to their original source
material and data, and must also be transparent in justifying and
explaining their underlying assumptions, calculations, and conclusions.
In 2003, based in part on these problems as well as DOE's long-standing
problems with data, software, and modeling, NRC conducted an
independent evaluation of three AMRs. The scope of the review was to
determine if the AMRs met NRC requirements for being traceable,
transparent, and technically appropriate for their use in the license
application. NRC found significant problems.[Footnote 8] First, in some
cases DOE was not transparent in explaining the basis on which it was
reaching conclusions. For example, in some circumstances, DOE selected
a single value from a range of data without sufficient justification.
Other times, DOE did not explain how a range of experimental conditions
were representative of repository conditions. Second, where DOE did
sufficiently explain the basis for a conclusion, it did not always
provide the necessary technical information, such as experimental data,
analysis, or expert judgment, to trace the support for that explanation
back to source materials. For example, DOE did not explain how
information on one type of material provided an appropriate comparison
for another material. Moreover, while DOE had identified similar
problems in the past, the actions taken to correct them did not
identify and resolve other deficiencies. NRC concluded that these
findings suggested that other AMRs possibly had similar problems, and
that if not resolved, such problems could delay NRC's review of the
license application as it would need to conduct special inspections to
resolve any issues it found with the quality of technical information.
To address problems of traceability and transparency, DOE in the spring
of 2004 initiated an effort called the Regulatory Integration Team
(RIT) to perform a comprehensive inspection and rework of the AMRs to
ensure they met NRC requirements and expectations.[Footnote 9]
According to DOE officials, the RIT involved roughly 150 full-time
personnel from DOE, USGS, and multiple national laboratories such as
Sandia, Los Alamos, and Lawrence Livermore. First, the RIT screened all
of the approximately 110 AMRs and prioritized its efforts on 89 that
needed additional rework. Ten AMRs were determined to be acceptable,
and 11 were canceled because they were no longer needed to support the
license application. According to DOE officials, approximately 8 months
later, the RIT project was completed at a cost of about $20 million,
with a total of over 3,700 problems and issues addressed or corrected.
In February 2005, in a letter to DOE, the site contractor stated that
the RIT effort was successful and that the AMRs had been revised to
improve traceability and transparency.
Subsequently, however, additional problems with traceability and
transparency have been identified, requiring further inspections and
rework. For example, after the March 2005 discovery of e-mails from
USGS employees written between May 1998 and March 2000 implying that
employees had falsified documentation of their work to avoid quality
assurance standards, DOE initiated a review of additional AMRs that
were not included in the scope of the 2004 RIT review. The additional
AMRs contained scientific work performed by the USGS employees and had
been assumed by the RIT to meet NRC requirements for traceability and
transparency. However, according to DOE officials, DOE's review
determined that these AMRs did not meet NRC's standards, and additional
rework was required. Further, similar problems were identified as the
focus of the project shifted to the design and engineering work
required for the license application. In February 2005, the site
contractor determined that in addition to problems with AMRs, similar
traceability and transparency problems existed in the design and
engineering documents that comprise the Safety Analysis Report--the
report necessary for demonstrating to NRC how the facilities and other
components of the repository site will meet the project's health,
safety, and environmental goals and objectives. In a root cause
analysis of this problem, the site contractor noted that additional
resources were needed to inspect and rework the documents to correct
the problems.
DOE Cannot Be Certain Its Efforts to Improve Quality Assurance Have
Been Effective Because of Weaknesses in Tracking Progress and
Identifying Problems:
DOE cannot be certain that it has met continuous improvement goals for
implementing its quality assurance requirements, a commitment DOE made
at the closure of its Management Improvement Initiatives (Initiatives)
in April 2004. At that time, DOE told us it expected that the progress
achieved with the initiatives would continue and that its performance
indicators would enable it to assess further progress and direct
management attention as needed. However, DOE's performance indicators,
as well as a second management tool--trend evaluation reports--have not
been effective for this purpose. More specifically, the indicators
panel did not highlight the areas of concern covered by the initiatives
and had weaknesses in assessing progress because the indicators kept
changing. The trend evaluation reports also did not focus on tracking
the concerns covered by the Initiatives, had technical weaknesses for
identifying significant and recurring problems, had inconsistently
tracked progress in addressing problems, and could not fully analyze
projectwide problems.[Footnote 10] In addition, the trend reports'
tracking of problems for which corrective actions were already being
taken was at times overly influenced by judgments about whether
additional management action was warranted rather than the problems'
significance.
The Panel's Focus and Frequent Changes Hindered the Tracking of
Progress with Management Concerns and Quality Problems:
By the time that the actions called for by the Initiatives had been
completed in April 2004, project management had already developed the
indicators panel, which DOE refers to as the annunciator panel, to use
at monthly management meetings to monitor project performance. The
panel was a single page composed of colored blocks representing
selected performance indicators and their rating or level of
performance. A manager viewing the panel would be able to quickly see
the color rating of each block or indicator. For example, red indicated
degraded or adverse performance warranting significant management
attention; yellow indicated performance warranting increased management
attention or acceptable performance that could change for the worse;
and green indicated good performance. The panel represented a hierarchy
of indicators in which the highest level, or primary, indicators were
shown; secondary indicators that determined the primary indicators'
ratings were shown for some primary indicators; but lower third-or
fourth-level indicators were not shown. Our review analyzed a subset of
these indicators that DOE designated as the indicators that best
predict performance in areas affecting quality. While we were
conducting our review, DOE suspended preparation of the panel after
August 2005 while it reconsiders its use of indicators to monitor
project performance. DOE had also suspended preparation of the panel
from late 2004 to early 2005 in order to make substantial revisions.
These revisions were made, in part, to emphasize fewer, more important
indicators for management attention.
The Initiatives raised concerns about five key areas of management
weakness as adversely affecting the implementation of quality assurance
requirements:
1. Roles and responsibilities were becoming confused as the project
transitioned from scientific studies to activities supporting
licensing. The confusion over roles and responsibilities was
undermining managers' accountability for results. The Initiatives'
objective was to realign DOE's project organization to give a single
point of responsibility for project functions, such as quality
assurance and the Corrective Action Program, and hold the project
contractor more accountable for performing the necessary work in
accordance with quality, schedule, and cost requirements.
2. Product quality was sometimes being achieved through inspections by
the project's Office of Quality Assurance rather than being routinely
implemented by the project's work organizations. As a result, the
Initiatives sought to increase work organizations' responsibility for
being the principle means for achieving quality.
3. Work procedures were typically too burdensome and inefficient, which
impeded work. The Initiatives sought to provide new user-friendly and
effective procedures, when necessary, to allow routine compliance with
safety and quality requirements.
4. Multiple corrective action programs existed, processes were
burdensome and did not yield useful management reports, and corrective
actions were not completed in a timely manner. The Initiatives sought
to implement a single program to ensure that problems were identified,
prioritized, and documented and that timely and effective corrective
actions were taken to preclude recurrence of problems.
5. The importance of a safety-conscious work environment that fosters
open communication about concerns was not understood by all managers
and staff, and they had not been held accountable when inappropriately
overemphasizing the work schedule, inadequately attending to work
quality, and acting inconsistently in practicing the desired openness
about concerns. Through issuing a work environment policy, providing
training on the policy, and improving the Employee Concerns Program,
the Initiatives sought to create an environment in which employees felt
free to raise concerns without fear of reprisal and with confidence
that issues would be addressed promptly and appropriately.
As shown in table 1, the Initiatives' effectiveness indicators for
tracking progress in addressing these management weaknesses did not
have equivalent performance indicators visible in the annunciator panel
when it was prepared for the last time, using August 2005 data.
Table 1: Visibility of Management Improvement Initiatives'
Effectiveness Indicators in Annunciator Panel When Last Prepared (Using
August 2005 Data):
Key area of management weakness identified in the Initiatives: Roles,
responsibilities, accountability, authority;
Effectiveness indicators from the Initiatives: An improving trend in
quality and work schedule performance;
DOE response on coverage of the management weakness in the panel's
performance indicators: No integrated analysis of trends in quality and
schedule performance;
GAO comments and observations: No indicator was visible in, or
underlies, the panel; Some indicators measured aspects of quality or
schedule, but provided no integrated analysis of these trends.
Effectiveness indicators from the Initiatives: A consistently
decreasing trend in quality problems related to roles and
responsibilities;
DOE response on coverage of the management weakness in the panel's
performance indicators: No aspect measured;
GAO comments and observations: No indicator was visible in, or
underlies, the panel.
Key area of management weakness identified in the Initiatives: Quality
assurance programs and processes;
Effectiveness indicators from the Initiatives: The numbers of high-
priority (significant) quality problems that are self-identified are at
least 80 percent of all significant quality problems;
DOE response on coverage of the management weakness in the panel's
performance indicators: One indicator looked at work organizations'
identification of problems, including less significant ones;
GAO comments and observations: No indicator, focused only on
significant problems, was visible in the panel; One fourth-level
indicator tracked work organizations' identification of significant
problems.
Effectiveness indicators from the Initiatives: A decreasing trend in
average time to resolve significant quality; problems and in number of
delinquent corrective actions for significant quality problems;
DOE response on coverage of the management weakness in the panel's
performance indicators: A new timeliness measure has been
developed.[A];
GAO comments and observations: No indicator was visible in panel;
Aspect of fourth-level indicator tracked average time of resolution.
Key area of management weakness identified in the Initiatives: Work
procedures;
Effectiveness indicators from the Initiatives: A decreasing number of
quality problems related to ineffective; Procedures;
DOE response on coverage of the management weakness in the panel's
performance indicators: No aspect measured;
GAO comments and observations: No indicator was visible in, or
underlies, the panel.
Effectiveness indicators from the Initiatives: A decreasing trend in
time needed to revise procedures;
DOE response on coverage of the management weakness in the panel's
performance indicators: No aspect measured;
GAO comments and observations: No indicator was visible in, or
underlies, the panel.
Effectiveness indicators from the Initiatives: A decreasing trend in
average time of interim procedure changes;
DOE response on coverage of the management weakness in the panel's
performance indicators: No aspect measured;
GAO comments and observations: No indicator was visible in, or
underlies, the panel.
Key area of management weakness identified in the Initiatives:
Corrective Action Program;
Effectiveness indicators from the Initiatives: A decreasing trend in
number of repetitive quality problems;
DOE response on coverage of the management weakness in the panel's
performance indicators: No aspect measured;
GAO comments and observations: No indicator was visible in, or
underlies, the panel.
Effectiveness indicators from the Initiatives: A decreasing trend in
average time to resolve significant quality; problems;
DOE response on coverage of the management weakness in the panel's
performance indicators: A new timeliness measure has been
developed.[A];
GAO comments and observations: No indicator was visible in panel;
Aspect of fourth-level indicator tracked average time of resolution.
Effectiveness indicators from the Initiatives: Less than 10 percent of
quality problems are resolved late;
DOE response on coverage of the management weakness in the panel's
performance indicators: A new timeliness measure has been
developed.[A];
GAO comments and observations: No indicator was visible in the panel; A
third-level indicator tracked percentage of problems with timely
resolution.
Key area of management weakness identified in the Initiatives: Work
environment;
Effectiveness indicators from the Initiatives: A decreasing number of
substantiated employee concerns for harassment, retaliation,
intimidation, and discrimination;
DOE response on coverage of the management weakness in the panel's
performance indicators: Aspects of this issue are measured by work
environment indicators;
GAO comments and observations: No indicator was visible in panel; A
third-level indicator measured this performance.
Effectiveness indicators from the Initiatives: Evaluation of routine
employee concerns in less than 30 days, or; 90 days for complex
employee concerns involving harassment or intimidation;
DOE response on coverage of the management weakness in the panel's
performance indicators: Goals have remained at 30 and 90 days;
GAO comments and observations: No indicator was visible in panel; Third-
level indicators measured the timely completion of routine and other
concerns.
Effectiveness indicators from the Initiatives: External evaluation of
work environment shows positive changes;
DOE response on coverage of the management weakness in the panel's
performance indicators: External evaluation is accomplished through
independent employee surveys, reflected in third-level indicators;
GAO comments and observations: No indicator was visible in panel; Four
third-level indicators were based on the employee surveys.
Source: GAO analysis of DOE data.
[A] New timeliness indicator was not implemented by the time of the
final panel using August 2005 data.
[End of table]
Two of the Initiatives' key areas of concern--(1) roles,
responsibilities, authority, and accountability; and (2) work
procedures--and their associated effectiveness indicators were not
represented in the panel's visible or underlying indicators. The
Initiatives' effectiveness indicator for tracking trends in recurring
problems also was not represented. In other cases, the Initiatives'
effectiveness indicators were represented in underlying lower-level
indicators that had very little impact on the rating of the visible
indicator. An example is the Initiatives' indicator for timely
completion of employee concerns. The panel's related visible indicator
was work environment, whose rating was based on 4 secondary and 23
tertiary indicators. Of the third-level indicators, two were for
timeliness of completion of employee concerns, and combined they
contributed 3 percent toward the rating of the work environment
indicator. As a result of the weighting of these many underlying
indicators, ratings for individual lower-level indicators could be
different from the visible indicator. For example, in August 2005, the
work environment indicator showed good performance. However, the
ratings of four underlying indicators from the project's employee
survey on the work environment--collectively accounting for 25 percent
of the work environment indicator's score--indicated the need for
increased management attention. Moreover, some of the Initiatives'
indicators, such as the work organizations' self-identification of
significant problems, had their impact on visible indicators diluted by
the inclusion of other indicators that were not focused solely on the
detection of significant problems.
Another shortcoming of the annunciator panel was that frequent changes
to the indicators hindered the ability to identify problems for
management attention and track progress in resolving them. The
indicators could change in many ways, such as changes in their
definition, calculation, or data sources used in calculations, or from
the deletion or addition of a subindicator. When such changes were made
to the indicators, progress became less clear because changes in
reported performance levels may have been the result of the indicator
changes rather than actual performance changes. Some of the indicators
for key project processes with quality elements changed from one to
five times during the 8-month period from April 2004 through November
2004. Even after the major revision of the panel in early 2005, most of
the performance indicators tracking quality issues continued to change
over the next 6 months--that is, from March 2005 through August 2005.
As shown in table 2, only one of the five relevant indicators did not
change during this period. One indicator was changed four times during
the 6-month period, resulting in it being different in more months than
it remained the same.
Table 2: Key Indicators for Processes with Quality Elements, Their
Intended Focus, and Number of Times They Changed (March through August
2005):
Indicators: Performance improvement;
Intended focus: Effectiveness of self-assessment of quality and other
issues, lessons learned, and Corrective Action Program;
Number of months changed: 4.
Indicators: Work management;
Intended focus: Quality of work products and documents;
Number of months changed: 1.
Indicators: Safety-conscious work environment;
Intended focus: Worker confidence in management support for raising
quality and other concerns without fear of retaliation; management
effectiveness in detecting and preventing retaliation for raising
concerns; effectiveness of normal and alternative problem resolution;
Number of months changed: 0.
Indicators: Human performance;
Intended focus: Preventing, detecting, and correcting human errors;
Number of months changed: 3.
Indicators: Quality performance;
Intended focus: Composite of quality indicators, in areas of
engineering products, self-assessment, Corrective Action Program, and
work products and documents;
Number of months changed: 1.
Source: GAO analysis of DOE data.
[End of table]
Moreover, the panel was not always available to identify problems and
track progress. The panel was not created for December 2004, January
2005, and February 2005 because it was undergoing a major revision. At
that time, DOE told NRC that the performance indicators for the panel
were revised to reflect the change in the work as the project moved
into the engineering, procurement, and construction phase. DOE also
reduced the total number of visible indicators from 60 to 30 to focus
on fewer, more critical aspects of project management. Panels with the
new indicators were then produced for 6 months, starting with March
2005 and ending after August 2005. This second interruption of the
panels resulted from another major revision to the indicators; this
time, indicators are being made congruent with project work as
designated by DOE's "new path forward," again to focus on fewer, more
important activities. In December 2005, a senior DOE official told us
that the project would begin to measure key activities, but without use
of the panel.
Trend Evaluation Reports Have Not Specifically Tracked the Initiatives'
Management Concerns and Have Had Weaknesses Tracking Significant and
Recurrent Problems for Management Attention:
According to DOE, some of the Initiatives' areas of concern and their
associated effectiveness indicators--for example, trends in quality
problems related to roles and responsibilities--were being captured, at
least partially, in the project's quarterly trend evaluation reports
rather than in the performance indicators. However, the trend reports
are a management tool designed more to identify emerging and
unanticipated problems than to monitor progress with already identified
problems, such as those addressed by the Initiatives. In developing
these reports, trend analysts seek to identify patterns and trends in
condition reports (CR), which document problematic conditions through
the project's Corrective Action Program. The trend reports analyze CRs
for more significant problems (Levels A and B) and minor problems
(Level C), but not at Level D (opportunities for improvement). The
trend analysis typically separates the reported problems into
categories such as organizational unit, type of problem, and cause.
These categories are intended to provide insights into the problems.
For example, analysis might reveal that most occurrences of a
particular type of problem are associated with a certain organization.
In practice, DOE missed opportunities to use trend reports to call
attention to progress in the Initiatives' areas of concern. For
example, the Initiatives sought to clarify roles and responsibilities
within and between DOE and BSC to ensure clear accountability for
project results during the project's transition from scientific studies
to the design and engineering activities necessary to license a
repository. Similar organizational transition problems were identified
in the November 2004 trend report. While that report attributed
increases in the number of causal factors associated with change
management, supervisory methods, and work organization to recent BSC
reorganizations and changes in the project from science-based to design
and engineering activities, it did not specifically mention issues of
roles and responsibilities or that roles and responsibilities was an
Initiatives' area of concern. However, an analysis of the cause of the
problems noted in various significant condition reports, which is
performed for certain condition reports and outside of the process of
developing trend reports, found evidence of weaknesses in the
organizational interfaces among BSC organizations, as well as between
BSC and DOE. According to this cause analysis, these organizational
interface weaknesses were associated with some manner of change and
represented weaknesses in the definition of roles and responsibilities.
Trend reports are generally based on condition reports, and problems
with roles and responsibilities seem to be identified in cause analyses
rather than in the condition reports themselves.
Similarly, DOE missed an opportunity to use trend reports to discuss
the Initiatives' goal that the project's line or work organizations
become more accountable for self-identifying significant problems. The
August 2005 trend report briefly cited an evaluation of a CR
highlighting the low rate of self-identification of significant
problems during the previous quarter and reported the evaluation's
conclusion that it was not a problem warranting management attention.
However, the trend report did not mention that about 35 percent of
significant problems were self-identified during the previous quarter,
while the Initiatives' goal was that 80 percent of significant problems
would be self-identified. Thus, the trend report missed an opportunity
to either raise a performance problem or pose the question of whether
the Initiatives' goal needed to be reassessed.
Beyond whether they effectively tracked the Initiatives' areas of
concern, trend reports face important obstacles, in general, to
adequately identify recurrent and significant problems:
* Recurring or similar conditions can be difficult to clearly identify
for management's attention and resolution. A trend report noted that
there will be few cases where recurrent conditions are obvious because
each condition slightly differs.
* Trend analysis tends to focus on the number of CRs issued, but the
number of CRs does not necessarily reflect the significance of a
problem. For example, the number of CRs involving requirements
management decreased by over half from the first quarter to the second
quarter of fiscal year 2005. However, this decrease was not a clear
sign of progress. Not only did the number rise again in the third
quarter, but the May 2005 trend report also noted that the number of
all condition reports had dropped during the second quarter. According
to the report, the volume of CRs in the first quarter had been high
because of reviews of various areas, including requirements management.
Another example is the records management problem. The November 2005
trend report stated that a records management problem identified in
various CRs, despite accounting for about 50 percent of all business
administration problems, reflected an underlying error rate of less
than 1 percent and thus was not a significant problem.
* The lack of an increasing trend in the number of reported problems
does not necessarily mean the lack of a significant problem for
management attention. Knowing the appropriate level of performance,
regardless of the trend, is difficult without having clearly
appropriate benchmarks from organizations engaged in activities similar
to the Yucca Mountain project. Such benchmarks would clarify, for
example, whether a project's percentages of human performance errors
compare favorably, regardless of whether the numbers are increasing.
Similarly, the trend in the number and types of CRs during any period
is not necessarily a sign of improvement or worsening conditions.
Trends can be attributed to various factors, including increases in the
number of audits or self-assessments, which can lead to more CRs being
issued.
* At the time of analysis, some trend data may not be sufficiently
reliable or complete to ensure sound findings for management's
attention. For example, although some actions were taken in December
2004 to ensure that cause and other codes were properly assigned, a BSC
audit in June 2005 again raised questions about the consistency of the
coding. With respect to completeness, the fourth quarter report for
2005 noted that 28 percent of the Level B CRs did not have a cause code
at the time of the trend analysis, and one finding was presented even
though two-thirds of the data was missing.
Due, in part, to these obstacles and changes to how the analysis is
done, trend reports have not consistently determined the significance
of problems or performed well in tracking progress in resolving
problems. For example, trend reports have questionably identified
significant human performance problems and ineffectively tracked
progress in resolving the problem because of no clearly appropriate or
precise benchmark for performance, inconsistent focus on the problem,
and unreliable data on cause codes.
The February 2004 trend report identified a human performance problem
based on Yucca Mountain project data showing the project's proportion
of skill-based errors to all human performance errors was two times
higher than benchmark data from the Institute of Nuclear Power
Operations (INPO).[Footnote 11] The report used this comparison to
suggest that the project needed to adopt successful commercial nuclear
practices for addressing skill-based errors. However, the report
cautioned that other comparisons with these INPO data may not be
appropriate because of differences in the nature, complexity, and scope
of work performed, but did not explain why the report's comparison of
INPO data for skill-based errors to the Yucca Mountain project should
be an exception to this caution. The May 2004 trend report repeated
this comparison to INPO, finding skill-based errors three times higher
than the benchmark data. However, this INPO benchmark has not been used
in subsequent reports.
The November 2004 trend report redefined the problem as the
predominance of human performance errors in general, rather than the
skill-based component of these errors--but later reports reinterpreted
this predominance as not a problem. The problem with skill-based errors
was unclear in the November 2004 report because these errors were
showing a decreasing trend, a finding that was attributed as likely the
result of unreliable assignment of cause codes. Instead, the report
cited an adverse trend based on the fact that the human performance
cause category accounted for over half of the total number of causes
for condition reports prepared during the quarter. Under the project's
trend analysis guidelines, this large predominance of human performance
causes--in contrast to management, communication or procedure, and
other cause categories--was designated an adverse trend. Nevertheless,
by February 2005, trend reports began interpreting this predominance as
generally appropriate, given the type of work done by the project. That
is, the project's work involves mainly human efforts and little
equipment, while work at nuclear power plants involves more
opportunities for errors caused by equipment. In our view, this
interpretation that a predominance of human performance errors would be
expected implies an imprecise benchmark for appropriate performance.
Although trend reports continued to draw conclusions about human
performance problems, the February 2005 report indicated that any
conclusions were hard to justify because of data reliability problems
with cause coding. For example, the majority of problems attributed to
human performance causes are minor, or Level C, problems that receive
less rigorous cause analysis, such as not completing a form. This less
rigorous analysis tends to reveal only individual human errors--that
is, human performance problems--whereas more rigorous analysis tends to
reveal less immediately obvious problems with management and
procedures.
Trend reports have also inconsistently tracked progress in resolving
the problem associated with the "flow-down" of requirements into the
project's procedures--that is, with ensuring that program, regulatory,
and statutory requirements are identified, allocated, and assigned to
the project organizations that are responsible for applicable
activities. Such requirements management problems can result in
inadequate control over design inputs and, possibly, inputs to
scientific models. Progress with this problem was less clear because of
inconsistent methods of categorizing requirements management problems
over time. Initially, based on reviews of annual trends in condition
reports, the September 2004 and November 2004 trend reports observed a
systemic and continuing problem in the flow-down of requirements from
BSC's Project Requirements Document and identified this as an adverse
trend. In subsequent reports, the requirements flow-down problem was
variously treated as an aspect of requirements management or records
management, or as a latent management weakness or weak change
management. When treated as an aspect of these broader problems, the
significance of the original flow-down problem and any progress in
resolving it became diluted and less clear. The primary focus
eventually became requirements management, which the February 2005
trend report designated as a potential trend, whereas the flow-down
problem had earlier been designated an adverse trend. Consequently, as
a result of this change, the flow-down of requirements got less direct
attention and analysis--for example, receiving only a footnote in the
August 2005 trend report stating that the April 2004 condition report
issued to address the adverse trend was still overseeing implementation
of corrective actions.
In addition, because trend reports examine only condition reports
issued to BSC, they do not always assess the projectwide significance
of problems such as requirements management.[Footnote 12] When
analyzing one category of issues associated with requirements
management, the November 2005 report stated that BSC and DOE shared the
process problems, which cannot be adequately addressed by just one of
the organizations. However, for a second category of these issues, the
report did not analyze most of the condition reports because 6 of the
10 relevant reports were assigned to DOE. For a third category of
issues, no analysis or recommendation was provided because all of the
reports were assigned to DOE and therefore did not fall within the
scope of the trend report.
DOE Has Not Adequately Tracked Problems Being Addressed by Ongoing
Management Actions:
The tracking of problems for which corrective actions are already being
taken appeared at times to be overly influenced by judgments, rather
than the problems' significance, about whether additional management
action is warranted. As a result, problems might be rated as less
significant, or not tracked further.
The situation of assigning a lower rating to a problem's significance
was apparently caused by the fact that ratings were simultaneously an
assessment of a problem's significance and of the need for management
action. In its current formulation, DOE's rating categories cannot
accurately represent both the assessment of a problem's significance
and a judgment that additional actions are not needed because the
designated rating category will distort one or the other. For instance,
the November 2005 trend report analyzed the four categories of
requirements management issues and designated one category that
included problems with requirements flow-down as a "monitoring trend"-
-defined as a small perturbation in numbers that does not warrant
action but needs to be monitored closely. Describing this trend as a
small perturbation, or a disturbance in numbers, did not accurately
describe the report's simultaneous recognition that significant process
problems spanned both BSC and DOE and the fact that the numbers and
types of problems were consistently identified over the previous three
quarters. A more understandable explanation for the low rating is that
designating the problem at any higher level of significance would have
triggered guidelines involving the issuance of a condition report,
which, according to the judgment expressed in the report, was not
needed. Specifically, the report indicated that existing condition
reports have already identified and were evaluating and resolving the
problem, thereby eliminating the need to issue a new condition report.
By rating the problem at the lowest level of significance and not
calling for additional actions, the trend report did not sufficiently
draw management's attention to the problem. The trend report's
assessment did not convey that other serious problems might have been
raised by the additional condition reports. At about the same time that
the trend report judged that no new condition reports were necessary,
an Employee Concerns Program's investigation of requirements management
resulted in 14 new condition reports--3 at the highest level of
significance and 8 at the second-highest level of
significance.[Footnote 13] For example, the Employee Concerns Program's
investigation resulted in condition reports calling for an analysis of
the collective significance of the numerous existing condition reports
and an assessment of whether the quality assurance requirement for
complete and prompt remedial action had been met.[Footnote 14] As a
result of the investigation and a concurrent DOE root cause analysis,
during the December 2005 Quarterly Management Meeting with NRC, DOE
stated that strong actions were required to address the problems with
its requirements management system and any resulting uncertainty about
the adequacy of its design products.[Footnote 15]
Trend reports identified significant problems in the February 2005
report but did not continue to track the problems after a separate
analysis identified ongoing improvement actions. According to the trend
report, Level B condition reports collectively indicated organizational
weaknesses associated with change management involving cross-
departmental interfaces. The trend report recommended that management
focus on these problems, and cited a condition report that would
further investigate them. The cause analysis for that condition report
and a related condition report found that the problems were well-known,
in part through a BSC review, and related to a variety of ongoing BSC
improvement actions. Since this was a broad category of problems with
many initiatives under way, the cause analysis recommended no new
actions other than for management to remain aware of the problems.
However, the trend reports that followed provided no further analyses
to focus management's awareness on these problems or to assess progress
in resolving them.
DOE's 'New Path Forward' for Preparing to Submit Its License
Application Faces Substantial Quality Assurance and Other Challenges:
In October 2005, DOE announced an aggressive series of proposed changes
to the design, organization, and management of the Yucca Mountain
project, but this effort--known as the "new path forward"--will face
substantial challenges. Some key challenges facing DOE are (1)
determining the extent of problems and restoring confidence in the
documents supporting the license application after the discovery of e-
mails raising the potential of falsified records, (2) settling design
issues and associated problems with requirements management, and (3)
replacing key personnel and managing the transition of new managers and
other organizational challenges. The current Acting Director of the
Office of Civilian Radioactive Waste Management (OCRWM) stated that DOE
will not announce a schedule for submitting a license application until
DOE addresses these important quality assurance and other challenges.
Since DOE is still formulating its plans, it is too early to determine
whether the new path will resolve these challenges.
Determining the Extent of Problems with Relevant Documents Will Delay
DOE's Submission of the License Application:
In March 2005, after announcing the discovery of USGS e-mails
suggesting the possible violation of quality assurance requirements,
including the falsification of records, DOE has taken steps to address
lingering concerns about the adequacy of the scientific work related to
the flow of water into the repository and whether similar quality
assurance problems are evident in other e-mails relevant to the
licensing application. Specifically, DOE is (1) conducting an extensive
review of approximately 14 million e-mails to determine whether these e-
mails raise additional quality assurance concerns and whether they
might be relevant to the licensing process, and (2) reworking the
technical documents created by USGS personnel to ensure that the
science underlying the conclusions on water infiltration are correct
and supportable in the license application. The Acting Director of
OCRWM has stated that DOE will not submit a license application until
these efforts are complete. Consequently, given the early planning
stage of these efforts, it is unknown how long this will delay the
submission of a license application.
As part of the licensing process, DOE is required to publicly disclose
all documents relevant to the licensing application, including e-mails,
by posting them on DOE's public Web site, which is accessible through
the NRC-sponsored, Internet-based Licensing Support Network (LSN). To
satisfy schedule requirements, DOE must certify that relevant documents
have been posted to the network and made available for public review 6
months before the submission of the license application. In preparation
for submitting the license application by December 2004, in June of
that year, DOE submitted almost 700,000 e-mails to the LSN that had
been reviewed by their original authors and determined to be relevant
to the licensing process. They were part of a group of approximately 6
million archived e-mails authored by individuals still associated with
the project. However, in August 2004, NRC's Atomic Safety and Licensing
Board ruled that DOE had not met its regulatory obligation to make all
relevant documentary material available. Specifically, DOE had not
reviewed a group of approximately 4 million archived e-mails authored
by individuals no longer affiliated with the project to determine
whether the e-mails were relevant to the licensing process. As part of
its effort to address the board's ruling, BSC began a review of e-mails
authored by employees who were not currently working at the project.
During this review, the contractor discovered and brought forward e-
mails between USGS scientists working on water infiltration models that
raised questions of the potential falsification of technical
information in order to sidestep quality assurance requirements.
Following the discovery of the e-mails, DOE conducted a search to
determine if there were similar e-mails in the approximately 1 million
e-mails previously determined relevant for licensing. However, the DOE
Inspector General reported in November 2005[Footnote 16] that there was
no evidence that the project requirements for identifying and
addressing conditions adverse to quality, such as those contained in
the USGS e-mails, were considered during the initial review of e-mails.
Further, among the approximately 10 million e-mails that had already
been reviewed for the licensing process, they found additional e-mails
that identified possible conditions adverse to quality that had not
been identified by project personnel as requiring further review. The
DOE Inspector General recommended, among other things, that DOE (1)
expand the review of archived e-mails to include both those deemed
relevant and those deemed not relevant to the licensing process, and
ensure that conditions adverse to quality are appropriately identified,
investigated, reported, and resolved; and (2) ensure that current and
future e-mails are reviewed for possible conditions adverse to quality
and that such conditions are appropriately addressed under the
Corrective Action Program (CAP) system. DOE accepted the Inspector
General's recommendations. Specifically, DOE agreed to develop a
corrective action plan to expand the review of archived e-mails to
ensure that conditions adverse to quality are appropriately identified
and processed under the CAP system. In addition to this review, the DOE
Inspector General opened a criminal investigation into the USGS e-mails
in March 2005. As of December 2005, the investigation was still in
progress.
According to NRC on-site representatives, completing these e-mail
reviews will be challenging because DOE now has to screen millions of e-
mails to ensure that records were not falsified. Further, many of these
e-mails were written by employees who no longer work at the project or
may be deceased, making it difficult to learn their true meaning and
context. Moreover, if additional e-mails are found that raise quality
assurance concerns, DOE may have to initiate further review,
inspections, or rework to address the newfound problems. NRC officials
stated that it takes the issue of potentially falsified documents by
USGS employees very seriously, wants a full understanding of the
situation regarding the USGS e-mails, and will conduct follow-up in
this area. Because NRC wants DOE to submit a high-quality license
application, it has encouraged DOE to take the time and actions
necessary to fully and adequately resolve these and other quality
assurance issues.
Immediately following the discovery of the USGS e-mails, DOE undertook
a scientific investigation into the technical documents created by USGS
personnel. In October 2005, DOE began developing an action plan for
reviewing, validating, augmenting, and replacing USGS work products
that had come under scrutiny. Although the plan is not yet complete,
the Acting Director told us that the license application would not be
submitted until the USGS work is replaced and there is confidence that
all requirements have been met. In an effort to ensure that the
scientific work underlying water infiltration modeling is accurate, DOE
is working to corroborate the original work by engaging multiple
agencies and organizations to rework the models. For example, DOE has
(1) had its lead project contractor work with the Idaho National
Laboratories to extensively review the software and data used in the
original science work, (2) engaged Sandia National Laboratories to
rework the model and calculations using different software than was
used originally, and (3) also asked USGS to rework the models.
Consequently, when this additional rework is completed, DOE will have
four sets of analysis (including the original scientific work) with
which they can evaluate, compare, and corroborate results. DOE will
then pick one set of scientific analysis for inclusion in the license
application, and work to explain and defend its choice.
Ongoing Design and Requirements Management Issues Could Delay DOE's
Submission of the License Application:
In October 2005, DOE announced significant changes to the design of the
Yucca Mountain repository to simplify the project and improve its
safety and operation. However, these changes will also require
additional design and engineering work that will add uncertainty about
the timing of the submission of a license application. DOE had been
considering a design where radioactive waste would be shipped to the
Yucca Mountain site, removed from its shipping container, placed and
sealed in a special disposal container, and finally moved into the
underground repository. As a result, DOE contemplated handling the
waste up to four separate times. In late 2003, DOE engineers began
identifying potential safety problems with this approach. First,
possible fissures or holes in the cladding surrounding the spent
nuclear fuel accidentally caused during the handling of the waste could
cause air to mix with the fuel and oxidize. Consequently, this
radioactive oxidized material could then leak and be dispersed into the
air. Second, DOE engineers determined that the original facility design
would not be able to adequately control the levels of radioactivity in
the buildings where the waste would be repackaged before being moved in
the repository. To address these problems, DOE researched a series of
options, including only accepting radioactive waste that had already
decayed to the point where oxidization would not be problematic, and
testing the waste shipments for oxidization and treating them at
another site before they arrived at the repository. In addition, DOE
also considered changing the design by filling the processing buildings
with inert gas to prevent oxidization and revising the electrical and
ventilation systems. According to a DOE official, these options were
impractical or added complexity to the design.
However, in October 2005, DOE proposed a new design that relies on
uniform canisters that would be filled and sealed before being shipped,
eliminating the need for direct handling of the waste prior to being
placed in the repository. As a result, DOE will not have to construct
several extremely large buildings costing millions of dollars for
handling radioactive waste. DOE believes this change will improve the
safety, operation, and long-term performance of the repository.
However, this change will also pose a challenge to the project because
of the widespread implications and the unknown time and effort required
to implement it. For example, to implement the new design, DOE will
need to, among other things,
* get approval from the Energy Systems Acquisition Advisory
Board[Footnote 17] for a new project plan, which, among other things,
includes details on the conceptual design, cost estimates, risk
management efforts, and acquisition strategies;
* plan, design, and produce standardized canisters for the
transportation of waste;
* coordinate this new approach with commercial nuclear power plants,
NRC, and government organizations that plan on shipping waste to the
project; and:
* revise procurement and contracting plans to support the new design.
Finally, DOE will need to perform the detailed design and engineering
work required to implement the new design, and create new technical
documents to support the license application. However, before it can
present its new plans and perform this design and engineering work, DOE
officials have stated that it will need to resolve long-standing
quality assurance problems involving requirements management.
Requirements management is the process that ensures the broad plans and
regulatory requirements affecting the project are tracked and
incorporated into specific engineering details. According to DOE's root
cause analyses, low-level documents were appropriately updated and
revised to reflect high-level design changes through fiscal year 1995.
However, from 1995 through 2002, many of these design documents were
not adequately maintained and updated to reflect current designs and
requirements. Further, a document that is a major component of the
project's requirements management process was revised in July 2002, but
has never been finalized or approved. Instead, the project envisioned a
transition to a new requirements management system after the planned
submission of the license application in December 2004. However, for
various reasons, the license application was not submitted at that
time, and the transition to a new requirements management system was
never implemented. As a result, the document refers to the out-of-date
NRC regulations contained in 10 CFR part 60, and not the regulations in
10 CFR part 63 that were finalized in October 2002.
The scope and cause of requirements management problems have been
identified in multiple DOE and NRC reviews.
* Multiple condition reports issued in 2004 and 2005 have identified
problems with requirements management. Due to these condition reports
and NRC concerns that repetitive deficiencies and the failure to
implement timely corrective actions could have direct implications on
the quality of the planned license application, NRC performed a review
of Corrective Action Program documents related to the requirements
management program in the late summer of 2005. NRC determined that
these reports identified approximately 35 deficiencies related to
requirements management. Because the requirements management documents
are not current and the new requirements management system has not been
implemented, NRC concluded that there does not appear to be a
requirements management mechanism in place. Further, based on the
number of reports and other issues identified by DOE audits, NRC
concluded that the project's Corrective Action Program was not
effective in, among other things, eliminating the repeated
identification of deficiencies relating to requirements management or
initiating the actions to identify and appropriately address the root
cause of these problems.
* In September 2005, DOE began reviewing the root causes associated
with CR-6278, a condition report identifying problems with requirements
management. As part of the review, DOE personnel analyzed 135 condition
reports and other events and allegations. Among other things, this
review found that DOE expectations for requirements management were
diluted and eventually neglected, that DOE reduced funding for
requirements management due to reductions in its annual budget, and
that these and other events caused the requirements management process
to become "completely dysfunctional" from July 2002 to the time of the
review in the fall of 2005. The analysis identified the root causes of
these conditions as DOE's failure to fund, maintain, and rigidly apply
a requirements management system.
* In November 2005, a team of DOE personnel concluded an investigation
into an employee's concerns regarding requirements management. The team
substantiated all of the concerns they investigated and found instances
of failures and breakdowns in the requirements management process. For
example, among other things, the team found that no procedure was
developed to describe how requirements management was to occur; some
existing requirements management procedures were not implemented; and
project management was aware of these conditions but corrective actions
were deferred because the planned requirements management system was
expected to address the problem. As a contributing factor, the team
also observed that the project's lead contractor had not implemented a
"traditional systems engineering approach" as it did not have, among
other things, typical engineering management plans or a separate
systems engineering organization responsible for requirements
management. As a result of the investigation, the team initiated 14
condition reports, 13 of which identified quality-related problems.
To address these problems, on December 19, 2005, DOE issued a stop-work
order on design and engineering for the surface facility and certain
other technical work. DOE stated that the root cause analysis for CR
6278 and the investigation into employee concerns revealed that the
project has not maintained or properly implemented its requirements
management system, resulting in inadequacies in the design control
process. This stop-work order will be in effect until, among other
things, the project's lead contractor improves the requirements
management system; validates that processes exist and are being
followed; and requirements are appropriately traced to implementing
mechanisms and products. Further, DOE will establish a team to take
other actions necessary to prevent inadequacies in requirements
management and other management systems from recurring.
An example of the potential risks of a breakdown with requirements
management was noted during a BSC audit on the design process in March
2005. NRC on-site representatives observing this audit reported that
the audit team noted problems with inconsistencies between the design
documents of the planned fuel-handing facility that would be receiving,
preparing, and packaging the waste before it is placed in the
repository. The original set of requirements specified that no water
from a fire protection system was to be used in the fuel-handling areas
of the facility because under certain scenarios, water used for fire
suppression could facilitate an accidental nuclear reaction, a
condition known as criticality. Later, as the project began to review
the design of the fuel-handling facility, the design was changed to
allow the use of water sprinklers in the fuel-handling areas of the
facility to suppress possible fires. NRC noted that personnel working
on the design knew of the inconsistencies between older and newer
design documents, but no formal tracking mechanism had been provided to
ensure that those issues were rectified. According to an NRC on-site
representative in December 2005, this was an example of a concern with
requirements management, and that repetitive and uncorrected issues
associated with the requirements management process could have direct
implications on the quality of the license application.
While the project may be able to resolve these inconsistencies through
an informal process, the lack of a formal design control and
requirements management process increases the risk that not all such
problems will be addressed. These requirements management problems are
potentially significant because if the high-level engineering needs of
the project are not accurately or completely reflected in the detailed
design, then the quality of the license application may be compromised
and cause delays in the license application review process. For
example, according to a 1989 speech prepared by NRC's Office of General
Counsel stressing the importance of quality assurance, a West Coast
nuclear power plant experienced similar quality assurance problems with
requirements management. After a license was issued by NRC, power plant
personnel discovered that the wrong diagrams were used to develop
design requirements. As a result of this and other quality assurance
weaknesses identified by NRC, the license was suspended and the power
plant was required to initiate an independent program to verify the
correctness of the design. Further, NRC reopened hearings on the issue
of the adequacy of the power plant's quality assurance program related
to the plant's design.
DOE Faces Challenges in Managing the Transition, Complexity, and
Continuity of Its 'New Path Forward':
In October 2005, DOE announced a "new path forward" that would create a
new project schedule and financial plan to address the completion of
scientific and engineering work in support of a license application.
However, DOE faces challenges to successfully implementing the new
path, in terms of managing the transition, program and organizational
complexities, and the continuity of management. According to DOE
managers involved with planning the new path forward, the
organizational transition could take several months to complete. It is
too early to determine whether DOE's new effort will resolve quality
assurance issues and move the project forward to the submission of a
license application.
Accountability for quality and results, which was identified as a
significant transition issue in the Initiatives, will likely pose a
challenge for managing the transition to the new path forward. The
Initiatives sought to clarify roles and responsibilities within and
between DOE and contractor organizations to ensure clear accountability
for results and quality during the transition from OCRWM's
organization, processes, procedures, and skills supporting scientific
studies to those supporting the activities necessary to license a
repository. As the project realigns organizations, processes,
procedures, and skills to support the new path forward, it will also be
faced with the challenge of ensuring that accountability is not
undermined during the transition. For instance, according one DOE
manager, transitioning project work to a lead laboratory under a direct
contract with DOE could pose a significant challenge for quality
assurance because the laboratories are currently working under BSC
quality assurance procedures and will now have to develop their own
procedures.
Implicitly recognizing the importance of accountability issues,
elements of the new path forward seek to address issues that can
negatively affect quality assurance and project management in general.
For instance, the new path includes plans for developing and
transmitting requirements to USGS for the certification of scientific
work. In addition, a senior project official told us that the lead
laboratory would provide a single point of accountability that will
enhance the quality of the science work. The Acting Director indicated
that OCRWM's management structure may have to be reorganized to have a
single manager clearly accountable for each of the new path's major
tasks in science, engineering, and licensing. Moreover, the project is
developing new performance indicators to allow the project to assess
important activities under the new path forward. Outside of the new
path, as the result of a September 2005 DOE Inspector General
report[Footnote 18] on accountability problems with managing contract
incentives, OCRWM agreed to develop a comprehensive corrective action
plan to provide clearer and more objective performance standards in the
BSC contract.
Program complexity and other project characteristics are also likely to
pose challenges to managing quality assurance. Based on its experience
with licensing and regulating nuclear power plants, NRC observed in the
mid-1980s that the Yucca Mountain project's characteristics, such as a
large and complicated program, increased the likelihood of major
quality-related problems. Although the new path is intended to simplify
design, licensing, and construction, the project remains a complicated
program that seeks to both restore confidence in its scientific studies
and pursue new design and engineering activities. As a result, the
project has to manage quality assurance issues simultaneously in both
areas. Moreover, the project involves a complicated organizational
structure. The project will continue contracting work with BSC, USGS,
and the Sandia National Laboratory, which involves working with
organizations in various locations. In our 1988 report, we noted that
the geographic distance between the various organizations may hamper
OCRWM's quality assurance communication and oversight objectives.
The project also faces management challenges related to ensuring
management continuity at the project, since DOE has experienced
turnover in 9 of 17 key management positions since 2001. To ensure the
right managers move the project forward to licensing, the project has a
recruitment effort for replacing key departing managers. In the past
year, the project has lost key managers through the departures of the
director of Project Management and Engineering, the director of the
License Application and Strategy, the director of Quality Assurance,
and the contractor's general manager. According to NRC on-site
representatives in August and October 2005, management turnover is a
concern for NRC because it would like to see continuity of qualified
managers rather than a series of acting managers. Recruiting
replacement managers can impact project continuity, and newly acting
managers may not take full rein of project tasks. However, the Acting
Director told us that the recruitment process is an opportunity to
improve project managers and staff, but recruiting the right people is
challenging for various reasons--for example, government salaries are
less than those in industry, and employment clauses restrict subsequent
employment in related industries.
Finally, since new directors sometimes give new direction to the
project, a critical issue for sustaining the current new path forward
is continuity with OCRWM's director. This position was occupied by
three individuals between late 1999 and early 2005. The last OCRWM
director assumed the position in April 2002, started the Management
Improvement Initiatives in 2002, and left the position in February
2005. The current Acting Director began functioning in his position in
the summer of 2005, and initiated the new path forward in October 2005.
DOE is currently awaiting congressional confirmation of a nominee to
take the director position. However, the Acting Director told us he
expects that the new path forward will be sustained because it has been
endorsed by the Secretary of Energy.
Conclusions:
DOE's Yucca Mountain project has been wrestling with quality assurance
problems for a long time. Now, after more than 20 years of project
work, DOE is again faced with substantial quality assurance and other
challenges to submit a fully defensible license application to NRC.
Unless these challenges are effectively addressed, further delays on
the project are likely. Furthermore, even as DOE faces new quality
assurance challenges, it cannot be certain it has resolved past
problems, largely because the department has not been well served by
management tools--specifically, its performance indicators and trend
evaluation reports--that have not effectively identified and tracked
progress on significant and recurring problems. First, the management
tools have provided limited coverage of the areas of concern identified
in the Management Improvement Initiatives and thus have not enabled DOE
managers to effectively monitor progress in these important areas.
Second, the tools have often not reflected the full extent or
significance of problems because their scope has been limited and not
based on projectwide analysis. Third, the trend evaluation reports
have, at times, not accurately characterized problems because reliable
and complete data and appropriate performance benchmarks were not
available at the time of analysis. Fourth, frequent changes in
performance indicators and the way analysis is done have made it
difficult to accurately identify trends over time. Fifth, the tools'
rating categories have sometimes been misleading as to the significance
of problems because the ratings tend to be skewed by the fact that
corrective actions were already being taken, without considering their
effectiveness or considering the significance of the problem on its own
terms. These shortcomings with the tools limit project managers'
ability to direct and oversee such a large and complex undertaking as
constructing an underground repository for nuclear wastes. Further
complicating DOE's ability to manage the project are the vacancies in
key managerial positions for the quality assurance program and
elsewhere on the project. The tools become even more important for new
managers who need to quickly understand project management issues.
Recommendations for Executive Action:
To improve the effectiveness of DOE's efforts to monitor performance in
key areas at the Yucca Mountain project, including quality assurance,
we recommend that the Secretary of Energy direct the Director, Office
of Civilian Radioactive Waste Management, to take the following five
actions to strengthen the project's management tools:
* Reassess the coverage that the management tools provide for the areas
of concern identified in the Management Improvement Initiatives and
ensure that performance in these important areas is effectively
monitored, especially in light of the more recent condition reports and
associated cause analyses, trend reports, and other reviews indicating
continuing problems.
* Base future management tools, such as the trend evaluation reports,
on projectwide analysis of problems, unless there are compelling
reasons for a lesser scope.
* Establish quality guidelines for trend evaluation reports to ensure
sound analysis when reporting problems for management's attention. Such
guidelines should address, among other things, having reliable and
complete data and appropriate benchmarks.
* To the extent practicable, make analyses and indicators of
performance consistent over time so that trends or progress can be
accurately identified and, where changes to analyses or indicators are
made for compelling reasons, provide a clear history of the changes and
their impact on measuring progress.
* Focus the management tools' rating categories on the significance of
the monitored condition, not on a judgment of the need for management
action.
Agency Comments:
We provided DOE and NRC with draft copies of this report for their
review and comment. In oral comments, DOE agreed with our
recommendations and provided technical and editorial comments that we
have incorporated in the report, as appropriate. We also incorporated,
as appropriate, NRC's oral editorial comments, which primarily served
to clarify its role.
As agreed with your office, unless you publicly announce the contents
of this report earlier, we plan no further distribution until 30 days
from the report date. At that time, we will send copies to interested
congressional committees and Members of Congress, the Secretary of
Energy, and the Chairman of the Nuclear Regulatory Commission. We will
also make copies available to others upon request. In addition, the
report will be available at no charge on the GAO Web site at
[Hyperlink, http://www.gao.gov].
If you or your staff have any questions about this report, please
contact me at (202) 512-3841 or [Hyperlink, wellsj@gao.gov]. Contact
points for our Offices of Congressional Relations and Public Affairs
may be found on the last page of this report. GAO staff who made major
contributions to this report are listed in appendix III.
Sincerely yours,
Signed by:
Jim Wells:
Director, Natural Resources and Environment:
[End of section]
Appendixes:
Appendix I: Objectives, Scope, and Methodology:
The objectives of this review were to determine (1) the history of the
Yucca Mountain project's quality assurance problems since the project's
start in the 1980s, (2) the Department of Energy's (DOE) tracking of
quality problems and progress implementing quality assurance
requirements since our April 2004 report, and (3) challenges that DOE
faces as it continues to address quality assurance issues within the
project. In addition, we were asked to provide information about
implementation of the project's Employee Concerns Program and the types
of concerns raised in recent years through the program.
To determine the history of the project's quality assurance problems,
we reviewed our prior reports and those of DOE's Office of the
Inspector General concerning the Yucca Mountain project. We also
reviewed internal DOE evaluations and audit reports written about the
quality assurance program and Nuclear Regulatory Commission (NRC)
reports and NRC-prepared summaries of NRC and DOE quarterly management
meetings, technical exchange meetings, and quality assurance meetings
dating to early 2004. In addition, we reviewed letters and
communications between DOE and NRC regarding quality assurance from the
NRC Web archives from the late 1980s. Furthermore, we reviewed plans
for the Regulatory Integration Team (RIT) and subsequent correspondence
between Bechtel/SAIC Company, LLC (BSC), DOE's management contractor
for the Yucca Mountain project, and DOE. Moreover, we discussed quality
assurance issues with officials of DOE's Office of Civilian Radioactive
Waste Management (OCRWM), including the Acting Director and Deputy
Director, at DOE headquarters in Washington, D.C., and at its field
office in Las Vegas. In addition, we interviewed representatives of
Navarro Quality Services, a DOE subcontractor, as well as BSC, and NRC
officials in the agency's field office in Las Vegas, Nevada, and at its
headquarters in Rockville, Maryland.
To determine DOE's tracking of quality problems and progress
implementing quality assurance requirements since our April 2004
report, we interviewed OCRWM, BSC, and NRC officials about the status
of these efforts since the issuance of our prior report. We also
reviewed DOE's Management Improvement Initiatives (2002), DOE's
Management Improvement Initiatives Transition Approach (2003), and our
2004 report to understand the history of the improvement efforts. To
understand DOE's management tools to monitor problems and progress, we
reviewed the available performance indicators panels from April 2004
through August 2005, when it was last produced; the documentation on
the individual indicators applied to August 2005 data; and the
quarterly trend reports from the fourth quarter of fiscal year 2003
through the fourth quarter of fiscal year 2005. We also reviewed
information from condition reports and examined documentation on DOE's
Quality Assurance Requirements and Description (issued in August 2004),
BSC's Trend Evaluation and Reporting, and DOE's Procedure: Condition
Reporting and Resolution (issued in November 2005).
To determine challenges that DOE faces as it continues to address
quality assurance issues within the project, we reviewed information
from condition reports, NRC on-site representative reports, DOE
Inspector General reports, and an OCRWM Office of Concerns Program's
investigative report on past quality assurance problems and DOE's
efforts to address them. We obtained information on turnover in key
management positions at DOE and BSC since 2000. In addition, we
discussed with DOE and NRC officials DOE's difficulties in addressing
recurring quality assurance problems and the quality assurance
implications of the Yucca Mountain project moving from the site
characterization phase to design and licensing. Also, to better
understand issues and challenges, we attended quarterly meetings held
between DOE and NRC in Rockville in September 2005 and Las Vegas in
December 2005.
To identify recent employee concerns related to quality assurance, such
as falsification of records and a safety-conscious work environment, as
well as to identify the actions taken to address those concerns, we
reviewed all concerns received by the OCRWM and BSC Employee Concerns
Programs from January through November 2005. For the OCRWM program, we
reviewed all employee concerns files to identify concerns related to
quality assurance. For the BSC program, we first read summary
descriptions of each concerns file, and reviewed the concerns files for
only those we identified as related to quality assurance. We then
conducted a content analysis of all concerns files that we reviewed.
Next, our three team members reached consensus about the correct
classification of a concern as a quality assurance problem, such as
potential falsification of records. Finally, through a second review of
concerns files, we verified our recorded information for those concerns
that seemed to be important illustrations of problems. In addition, we
also spot-checked a sample of OCRWM and BSC concerns received in 2005
to verify the accuracy of their placement in various concerns
categories. We found that the concerns were generally categorized
accurately.
We performed our work from July 2005 through January 2006 in accordance
with generally accepted government auditing standards.
[End of section]
Appendix II: Yucca Mountain Project Employee Concerns Programs:
NRC expects licensees to establish a safety conscious work environment-
-that is, one in which (1) employees are encouraged to raise concerns
either to their own management or to NRC without fear of retaliation
and (2) employees' concerns are resolved in a timely and appropriate
manner according to their importance. NRC encourages but does not
require licensees to establish employee concerns programs to help
achieve such a work environment, and both DOE and BSC have established
such programs.[Footnote 19] DOE's Employee Concerns Program is
currently operated under the requirements of DOE Order 442.1A, but the
department, in anticipation of becoming a licensee, is in the process
of establishing the program to meet NRC expectations.
DOE and contractor employees at the Yucca Mountain project may raise
concerns about quality, safety, or other work environment issues--such
as harassment, intimidation, retaliation, and discrimination--through
various means. Employees are encouraged to resolve concerns at the
lowest possible level in the organization, in the following order:
* Use normal supervisory channels, such as by raising an issue to a
manager for resolution.
* Initiate a condition report through the Corrective Action Program--a
process in which any employee can formally identify a problem on the
project, such as with policies, procedures, or the work environment,
and have the issue investigated and, if necessary, fixed through
corrective actions.
* Submit a concern via e-mail, telephone, or in person to one of the
project's two Employee Concerns Programs--a BSC program for BSC
employees and other subcontractors and another run by DOE for either
DOE or BSC employees.
* Contact NRC directly.
The DOE and BSC concerns programs are intended to supplement rather
than replace the resolution of problems through managers or the
Corrective Action Program.
DOE and BSC Employee Concerns Programs have each established a
communication network to allow employees to register concerns. These
networks include brochures and regular newsletters on the programs and
numerous links to the program on the project's intranet, where
employees can obtain concerns forms. Both the DOE and BSC concerns
programs of the Yucca Mountain project have four main steps:
1. Employees notify the concerns program staff about issues that they
feel should be corrected, such as safety or health issues; harassment,
intimidation, retaliation, or discrimination; concerns raised through
the Corrective Action Program; and quality assurance problems.
2. The concerns program staff document and handle the concern in
accordance with the requirements of DOE Order 442.1A.
3. The concerns program notifies the employees of the results of the
investigation and notifies management of any deficiencies.
4. Project management develops corrective actions for deficiencies, and
the program validates that the concerns have been effectively addressed
by the actions.
Under DOE Order 442.1A, concerns may be addressed through an
investigation by the concerns program staff, an independent
investigation, a referral, a transfer, or a dismissal of the concern.
Employees can request or waive confidentiality. If a concern is
submitted anonymously, interpreting the main issues and problems is
left up to the concerns program staff, and action on the concern may be
limited if the submitted information does not clearly or sufficiently
define the concern.
The concerns program may conduct its own investigation of the concern.
Alternatively, it may refer the concern to another project organization
for investigation or resolution. After the results of the investigation
or resolution are reported to the concerns program within a specified
period, the concerns program accepts the results or requires additional
actions. In other cases, concerns may be transferred to another
organization with the appropriate subject matter responsibility or
expertise, such as the Office of Human Relations, Office of General
Counsel, or Office of the Inspector General.
After investigating a concern, the concerns programs determines whether
the concern is substantiated, partially substantiated, unsubstantiated,
or indeterminate. If a concern is substantiated or partially
substantiated, the investigation results are presented to the
responsible senior managers. A concern is considered indeterminate when
evidence is insufficient to substantiate a concern or allow for a
conclusion to be drawn. Some concerns can be resolved through a
noninvestigative resolution, a method to address concerns promptly when
minimal effort is required for resolution. Some resolutions involve the
development of management corrective action plans that are tracked
until they are closed. In addition, for deficiencies that identify
systemic problems, the concerns programs may file a condition report
through the Corrective Action Program. Moreover, DOE and contractor
employees are required to report certain conditions or alleged
conditions to DOE's Office of the Inspector General under DOE Order
221.1, which covers waste, fraud, and abuse. The concerns program
handles some employee concerns in this way.
From January through November 2005, DOE's concerns program opened 139
employee concerns for investigation, and the BSC concerns program
opened 112 concerns for investigation.[Footnote 20] DOE's concerns
program places concerns into 14 categories, while the BSC program uses
20 categories.[Footnote 21] For both DOE and BSC, the category
receiving by far the most concerns for calendar year 2005 was
management: "management/mismanagement" for DOE and "management
practices" for BSC. According to DOE, management concerns generally
involved conditions related to management behavior, policy practice,
budget allocation, or use of resources. According to the manager of
BSC's program, about half of the concerns in the management practices
category involve hiring and human relations issues and the other half
involve organizational policies and other issues. The "quality"
category accounts for a relatively small portion of total concerns--18
percent of concerns for the DOE program and 4 percent for the BSC
program. Tables 3 and 4 show the concerns received by the DOE and BSC
programs for January through November 2005.
Table 3: Employee Concerns Opened for Investigation under DOE's
Employee Concerns Program by Category of Concern, January through
November 2005:
Concern category: Management/mismanagement;
Percentage of total concerns: 42.
Concern category: Workplace violence;
Percentage of total concerns: 0.
Concern category: Harassment, intimidation, retaliation, and
discrimination;
Percentage of total concerns: 6.
Concern category: Reprisal;
Percentage of total concerns: 0.
Concern category: Chilling effect;
Percentage of total concerns: 5.
Concern category: Security;
Percentage of total concerns: 0.
Concern category: Health;
Percentage of total concerns: 0.
Concern category: Safety;
Percentage of total concerns: 4.
Concern category: Environment;
Percentage of total concerns: 0.
Concern category: Fraud, waste and abuse;
Percentage of total concerns: 4.
Concern category: Human resources;
Percentage of total concerns: 12.
Concern category: Equal Employment Opportunity;
Percentage of total concerns: 2.
Concern category: Quality;
Percentage of total concerns: 18.
Concern category: Other;
Percentage of total concerns: 8.
Concern category: Total;
Percentage of total concerns: 100.
Source: DOE.
Note: Percentages may not add to 100 because of rounding.
[End of table]
Table 4: Employee Concerns Opened for Investigation under BSC's
Employee Concerns Program by Category of Concern, January through
November 2005:
Concern category: Management practices;
Percentage of total concerns: 48.
Concern category: Industrial;
Percentage of total concerns: 1.
Concern category: Health;
Percentage of total concerns: 4.
Concern category: Fraud;
Percentage of total concerns: 3.
Concern category: Fitness for duty;
Percentage of total concerns: 1.
Concern category: Ethics;
Percentage of total concerns: 5.
Concern category: Cyber;
Percentage of total concerns: 0.
Concern category: Access authorization;
Percentage of total concerns: 0.
Concern category: Environmental;
Percentage of total concerns: 1.
Concern category: Employee relations;
Percentage of total concerns: 5.
Concern category: Intimidation;
Percentage of total concerns: 1.
Concern category: Harassment;
Percentage of total concerns: 0.
Concern category: Discrimination;
Percentage of total concerns: 4.
Concern category: Chilling effect;
Percentage of total concerns: 4.
Concern category: Abuse;
Percentage of total concerns: 4.
Concern category: Training;
Percentage of total concerns: 1.
Concern category: Safety-conscious work environment;
Percentage of total concerns: 3.
Concern category: Retaliation;
Percentage of total concerns: 4.
Concern category: Quality;
Percentage of total concerns: 4.
Concern category: Other;
Percentage of total concerns: 6.
Concern category: Total;
Percentage of total concerns: 100.
Source: BSC.
Note: Percentages may not add to 100 because of rounding.
[End of table]
The Employee Concerns Programs, which are designed to provide an
alternative to raising issues through the Corrective Action Program and
issuing condition reports, have been playing an active and sometimes
key role in identifying and addressing quality assurance problems, as
can be seen in the following examples:
* As part of an effort to identify e-mails relevant to the licensing
process and that therefore should be included in the Licensing Support
Network, BSC employees in late 2004 discovered e-mails suggesting
potential falsification of technical records. The e-mails were
submitted to the Employee Concerns Program in March 2005 and were
eventually reported to the DOE Inspector General for investigation. The
quality assurance issues raised by the e-mails have resulted in a
substantial effort by DOE to restore confidence in the quality of
technical documents that will support its license application to
construct the repository.
* In mid-2005, the DOE concerns program referred to the project's
senior management an employee's allegation that the project's schedule
was taking priority over quality in the review of technical documents.
In this instance, the Office of Concerns Program Manager negotiated
with senior management to address the time and resource needs for
ensuring quality assurance, rather than simply communicating to the
organization that quality should take priority over the schedule.
* As the result of an employee's concerns referred to DOE by NRC in mid-
2005, the Employee Concerns Program initiated an extensive
investigation of issues related to requirements management. That
investigation substantiated the employee's concerns and led to the
issuance of 14 condition reports for problem resolution. Signifying the
importance of this issue, DOE discussed problems with requirements
management with NRC at their quarterly meeting in December 2005.
The Employee Concerns Programs' role in identifying and addressing
quality assurance and other issues is dependent upon employees'
willingness to submit concerns, but the employees' willingness has
sometimes been in doubt. A late 2004 DOE survey of project employees
indicated, for example, that less than two-thirds of employees were
confident that submitted concerns would be thoroughly investigated and
appropriately resolved. DOE recognizes the need to improve employee
trust and willingness to use the concerns program, and both the DOE and
BSC program are engaged in outreach efforts. However, employees'
willingness to submit concerns may be affected by factors outside the
programs' control. According to a DOE manager, the project's recent and
pending workforce reductions may account for a decreasing number of
concerns submitted to the DOE program in late 2005. Based on OCRWM
Employee Concerns Program data, the program averaged about 13 concerns
a month from January through November 2005. However, the number of
monthly concerns dropped to 5 in October and 3 in November 2005.
During our review of concerns opened for investigation from January
2004 through November 2005, we did not identify any concerns alleging
problems similar to the falsification of technical records suggested by
the USGS e-mails. Although we found records of an early 2004 concern
about an instance of inappropriate management of a technical document,
this instance was resolved and did not appear to be an intentional or
systematic effort to falsify records. The manager of the BSC program
told us of a concern raised about another set of e-mails, but this
concern was not about record falsification. The manager of the DOE
program told us that she had not seen any reportable allegations of
falsification of technical records since she took her position in July
2004.
[End of section]
Appendix III: GAO Contact and Staff Acknowledgments:
GAO Contact:
Jim Wells, (202) 512-3841 or [Hyperlink, wellj@gao.gov].
Staff Acknowledgments:
In addition to the contact named above, Raymond Smith (Assistant
Director), Casey Brown, John Delicath, James Espinoza, and Terry
Hanford made key contributions to this report.
(360611):
FOOTNOTES
[1] Once EPA finalizes its new standards, NRC will revise its licensing
regulations to make them consistent with the standards.
[2] GAO, Yucca Mountain: Persistent Quality Assurance Problems Could
Delay Repository Licensing and Operation, GAO-04-460 (Washington, D.C.:
Apr. 30, 2004).
[3] GAO-04-460, 5.
[4] In addition, DOE must update this certification at the time of
license application submittal, as required by NRC regulations.
[5] Docketing is the formal acceptance of the license application by
NRC after it determines that it contains adequate information for a
formal review.
[6] NRC is required to issue or deny a construction authorization not
later than 3 years after receiving a license application, unless it
extends this period by not more than 1 year and reporting the reasons
for doing so to the Secretary of Energy and Congress.
[7] GAO, Nuclear Waste: Repository Work Should Not Proceed Until
Quality Assurance Is Adequate, GAO/RCED-88-159 (Washington, D.C.: Sept.
29, 1988).
[8] U.S. Nuclear Regulatory Commission, U.S. Nuclear Regulatory
Commission Staff Evaluation of U.S. Department of Energy Analysis Model
Reports, Process Controls, and Corrective Actions (Washington, D.C.,
Apr. 7, 2004).
[9] In addition, the RIT edited the AMRs to assure consistency and ease
of technical and regulatory reviews.
[10] Similar to our findings, a January 2006 NRC observation audit
report noted that a DOE audit had found the trend program was
unsatisfactory. Specifically, the DOE audit found that the trend
program was not handled as a priority for management attention, and has
historically identified broad causal issues with no adverse trends.
[11] Skill-based errors are defined in trend reports as unintentional
errors resulting from people not paying attention to the task at hand.
[12] DOE prepares a separate trend report for CRs issued to DOE, but
that report's findings are not integrated with those of BSC to provide
a projectwide analysis.
[13] The difference in number of CRs issued also reflects the fact that
the scope of the investigations was broader than the trending report's
scope, which focused only on CRs assigned to BSC. The investigation
resulted in CRs assigned to DOE as well as BSC.
[14] Appendix II offers a detailed description of the employee concerns
program and the variety of employee submitted concerns, which are often
not related to quality assurance, and our examination of concerns
submitted since 2004 that did not find concerns similar to those raised
about the potential falsification of records by USGS employees.
[15] A root cause analysis in such reports involves methods for
determining the root cause of a problem, which is the underlying cause
that must change in order to prevent the problem from reoccurring. A
root cause analysis is required for the most significant CRs--those
determined to be at Level A in the Corrective Action Program.
[16] U.S. Department of Energy, Office of Inspector General, Office of
Inspections and Special Inquiries, Quality Assurance Weaknesses in the
Review of Yucca Mountain Electronic Mail for Relevancy to the Licensing
Process, DOE/IG-0708 (Washington, D.C., November 2005).
[17] The Energy Systems Acquisition Advisory Board is a special
organization within DOE that advises the Secretary in approving and
revising plans for major acquisition projects.
[18] U.S. Department of Energy, Office of Inspector General, Office of
Audit Services, DOE/IG-0702, Use of Performance Based Incentives by the
Office of Civilian Waste Management (Washington, D.C., September 2005).
[19] The discussion in this report involves the DOE OCRWM office in Las
Vegas, Nevada and not the separately operated office in OCRWM
headquarters in Washington, D.C. The DOE employee concerns program
oversees BSC's program.
[20] The number of concerns opened for investigation by BSC includes
referrals from DOE. Both organizations track disposition of these
concerns and include them in their totals. According to the manager of
the BSC Employee Concerns Program, about one-fourth of BSC's concerns
are DOE referrals.
[21] We reviewed all of the concerns submitted to DOE and BSC from
January 2004 through November 2005 for their relevance to quality
assurance issues. Although we did not verify the accuracy of DOE's and
BSC's placement of all concerns into the above categories, we spot-
checked a sample of DOE and BSC concerns and found that they were
generally categorized accurately.
GAO's Mission:
The Government Accountability Office, the investigative arm of
Congress, exists to support Congress in meeting its constitutional
responsibilities and to help improve the performance and accountability
of the federal government for the American people. GAO examines the use
of public funds; evaluates federal programs and policies; and provides
analyses, recommendations, and other assistance to help Congress make
informed oversight, policy, and funding decisions. GAO's commitment to
good government is reflected in its core values of accountability,
integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through the Internet. GAO's Web site ( www.gao.gov ) contains
abstracts and full-text files of current reports and testimony and an
expanding archive of older products. The Web site features a search
engine to help you locate documents using key words and phrases. You
can print these documents in their entirety, including charts and other
graphics.
Each day, GAO issues a list of newly released reports, testimony, and
correspondence. GAO posts this list, known as "Today's Reports," on its
Web site daily. The list contains links to the full-text document
files. To have GAO e-mail this list to you every afternoon, go to
www.gao.gov and select "Subscribe to e-mail alerts" under the "Order
GAO Products" heading.
Order by Mail or Phone:
The first copy of each printed report is free. Additional copies are $2
each. A check or money order should be made out to the Superintendent
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or
more copies mailed to a single address are discounted 25 percent.
Orders should be sent to:
U.S. Government Accountability Office
441 G Street NW, Room LM
Washington, D.C. 20548:
To order by Phone:
Voice: (202) 512-6000:
TDD: (202) 512-2537:
Fax: (202) 512-6061:
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Web site: www.gao.gov/fraudnet/fraudnet.htm
E-mail: fraudnet@gao.gov
Automated answering system: (800) 424-5454 or (202) 512-7470:
Public Affairs:
Jeff Nelligan, managing director,
NelliganJ@gao.gov
(202) 512-4800
U.S. Government Accountability Office,
441 G Street NW, Room 7149
Washington, D.C. 20548:
