Information Security
National Nuclear Security Administration Needs to Improve Contingency Planning for Its Classified Supercomputing Operations Gao ID: GAO-11-67 December 9, 2010In the absence of underground nuclear weapons testing, the National Nuclear Security Administration (NNSA) relies on its supercomputing operations at its three weapons laboratories to simulate the effects of changes to current weapons systems, calculate the confidence of future untested systems, and ensure military requirements are met. GAO was requested to assess the extent to which (1) NNSA has implemented contingency and disaster recovery planning and testing for its classified supercomputing systems, (2) the laboratories are able to share supercomputing capacity for recovery operations, and (3) NNSA tracks the costs for contingency and disaster recovery planning for supercomputing assets. To do this work, GAO examined contingency and disaster recovery planning policies and activities, and analyzed classified supercomputing capabilities at the weapons laboratories, and NNSA budgetary data.
All three NNSA weapons laboratories--Los Alamos, Sandia, and Lawrence Livermore--have implemented some components of a contingency planning and disaster recovery program. NNSA, however, has not provided effective oversight to ensure that the laboratories have comprehensive and effective contingency and disaster recovery planning and testing. Further, due to lack of planning and analysis by NNSA and the laboratories, the impact of a system outage is unclear. Only one of the three laboratories--Los Alamos--had conducted a business impact analysis to assess the criticality of resources and acceptable outage time frames; yet, NNSA and all three laboratories consider the consequence associated with the loss of system availability to be low impact and do not consider the classified supercomputers to be mission critical. Nonetheless, NNSA classified supercomputing capabilities serve as a computational surrogate to nuclear weapons testing and are used to address other areas of national security. Despite the absence of business impact analyses, all laboratories had key components of a contingency planning program in place. However, shortcomings existed. For example, all laboratories had backup processes in place and had developed contingency plans, but the plans were not comprehensive. Specifically, one plan did not address the supercomputing operations, and none of the plans had been tested at the time of GAO's review. In addition, the laboratories addressed disaster recovery to a limited extent, but not specifically for the supercomputers. These shortcomings existed, at least in part, because NNSA's component organizations, including the Office of the Chief Information Officer, were unclear about their roles and responsibilities for providing oversight in the laboratories' implementation of contingency and disaster recovery planning. Until the agency fully implements a contingency and disaster recovery planning program for its weapons laboratories, it has limited assurance that vital information can be recovered and made available to meet national security priorities and requirements. Although the laboratories have the technological capability to share supercomputing capacity across all three weapons laboratories, barriers exist that could impede recovery operations. For example, the laboratories do not know the minimum supercomputing capacity needed to meet program requirements, such as simulating the effects of changes to weapons systems, should a disruption occur. In addition, the laboratories have not tested the technological capability to share the capacity on an on-demand basis for recovery operations. Without having an understanding of capacity needs and subsequent testing, the laboratories have little assurance that they could effectively share capacity if needed. Although NNSA obligated approximately $1.7 billion to help implement its classified supercomputing program from fiscal years 2007 through 2009, the agency has not tracked costs for contingency and disaster recovery planning and is uncertain of actual funds that were spent toward these efforts. GAO recommends, among other things, that NNSA clearly define roles and responsibilities for its component organizations in providing oversight for contingency and disaster recovery planning for the classified supercomputing environment. NNSA agreed with most of GAO's recommendations, but did not concur with recommendations relating to capacity planning and cost tracking.
RecommendationsOur recommendations from this work are listed below with a Contact for more information. Status will change from "In process" to "Open," "Closed - implemented," or "Closed - not implemented" based on our follow up work.
Director: Gregory C. Wilshusen Team: Government Accountability Office: Information Technology Phone: (202) 512-6244GAO-11-67, Information Security: National Nuclear Security Administration Needs to Improve Contingency Planning for Its Classified Supercomputing Operations
This is the accessible text file for GAO report number GAO-11-67
entitled 'Information Security: National Nuclear Security
Administration Needs to Improve Contingency Planning for Its
Classified Supercomputing Operations' which was released on December
9, 2010.
This text file was formatted by the U.S. Government Accountability
Office (GAO) to be accessible to users with visual impairments, as
part of a longer term project to improve GAO products' accessibility.
Every attempt has been made to maintain the structural and data
integrity of the original printed product. Accessibility features,
such as text descriptions of tables, consecutively numbered footnotes
placed at the end of the file, and the text of agency comment letters,
are provided but may not exactly duplicate the presentation or format
of the printed version. The portable document format (PDF) file is an
exact electronic replica of the printed version. We welcome your
feedback. Please E-mail your comments regarding the contents or
accessibility features of this document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
United States Government Accountability Office:
GAO:
Report to Congressional Requesters:
December 2010:
Information Security:
National Nuclear Security Administration Needs to Improve Contingency
Planning for Its Classified Supercomputing Operations:
GAO-11-67:
GAO Highlights:
Highlights of GAO-11-67, a report to congressional requesters.
Why GAO Did This Study:
In the absence of underground nuclear weapons testing, the National
Nuclear Security Administration (NNSA) relies on its supercomputing
operations at its three weapons laboratories to simulate the effects
of changes to current weapons systems, calculate the confidence of
future untested systems, and ensure military requirements are met.
GAO was requested to assess the extent to which (1) NNSA has
implemented contingency and disaster recovery planning and testing for
its classified supercomputing systems, (2) the laboratories are able
to share supercomputing capacity for recovery operations, and (3) NNSA
tracks the costs for contingency and disaster recovery planning for
supercomputing assets. To do this work, GAO examined contingency and
disaster recovery planning policies and activities, and analyzed
classified supercomputing capabilities at the weapons laboratories,
and NNSA budgetary data.
What GAO Found:
All three NNSA weapons laboratories”-Los Alamos, Sandia, and Lawrence
Livermore”-have implemented some components of a contingency planning
and disaster recovery program. NNSA, however, has not provided
effective oversight to ensure that the laboratories have comprehensive
and effective contingency and disaster recovery planning and testing.
Further, due to lack of planning and analysis by NNSA and the
laboratories, the impact of a system outage is unclear. Only one of
the three laboratories-”Los Alamos-”had conducted a business impact
analysis to assess the criticality of resources and acceptable outage
time frames; yet, NNSA and all three laboratories consider the
consequence associated with the loss of system availability to be low
impact and do not consider the classified supercomputers to be mission
critical. Nonetheless, NNSA classified supercomputing capabilities
serve as a computational surrogate to nuclear weapons testing and are
used to address other areas of national security. Despite the absence
of business impact analyses, all laboratories had key components of a
contingency planning program in place. However, shortcomings existed.
For example, all laboratories had backup processes in place and had
developed contingency plans, but the plans were not comprehensive.
Specifically, one plan did not address the supercomputing operations,
and none of the plans had been tested at the time of GAO‘s review. In
addition, the laboratories addressed disaster recovery to a limited
extent, but not specifically for the supercomputers. These
shortcomings existed, at least in part, because NNSA‘s component
organizations, including the Office of the Chief Information Officer,
were unclear about their roles and responsibilities for providing
oversight in the laboratories‘ implementation of contingency and
disaster recovery planning. Until the agency fully implements a
contingency and disaster recovery planning program for its weapons
laboratories, it has limited assurance that vital information can be
recovered and made available to meet national security priorities and
requirements.
Although the laboratories have the technological capability to share
supercomputing capacity across all three weapons laboratories,
barriers exist that could impede recovery operations. For example, the
laboratories do not know the minimum supercomputing capacity needed to
meet program requirements, such as simulating the effects of changes
to weapons systems, should a disruption occur. In addition, the
laboratories have not tested the technological capability to share the
capacity on an on-demand basis for recovery operations. Without having
an understanding of capacity needs and subsequent testing, the
laboratories have little assurance that they could effectively share
capacity if needed.
Although NNSA obligated approximately $1.7 billion to help implement
its classified supercomputing program from fiscal years 2007 through
2009, the agency has not tracked costs for contingency and disaster
recovery planning and is uncertain of actual funds that were spent
toward these efforts.
What GAO Recommends:
GAO recommends, among other things, that NNSA clearly define roles and
responsibilities for its component organizations in providing
oversight for contingency and disaster recovery planning for the
classified supercomputing environment. NNSA agreed with most of GAO‘s
recommendations, but did not concur with recommendations relating to
capacity planning and cost tracking.
View [hyperlink, http://www.gao.gov/products/GAO-11-67] or key
components. For more information, contact Gregory C. Wilshusen at
(202) 512-6244 or wilshuseng@gao.gov, Gene Aloise at (202) 512-3841 or
aloisee@gao.gov, or Naba Barkakati at (202) 512-6415 or
barkakatin@gao.gov.
[End of section]
Contents:
Letter:
Background:
NNSA Has Not Fully Implemented Contingency and Disaster Recovery
Planning and Testing for Its Classified Supercomputing Assets:
The Laboratories Have the Ability to Share Supercomputing Capacity,
but Barriers Exist:
NNSA Does Not Track the Costs for Ensuring Contingency and Disaster
Recovery Planning for Its Supercomputing Assets:
Conclusions:
Recommendations for Executive Action:
Agency Comments and Our Evaluation:
Appendix I: Objectives, Scope, and Methodology:
Appendix II: NNSA Annual Obligations for Its Advanced Simulation and
Computing Program, Fiscal Years 2007 through 2009:
Appendix III: Comments from the National Nuclear Security
Administration:
Appendix IV: GAO Contacts and Staff Acknowledgments:
Table:
Table 1: Inventory of NNSA-Deployed Classified Supercomputing Systems
(as of October 2010):
Figures:
Figure 1: Common Hardware Components of a Supercomputing System:
Figure 2: NNSA's Classified Supercomputing Network Infrastructure:
Figure 3: Total Usable Supercomputing Capacity at Each Weapons
Laboratory, 2010 and 2011:
Figure 4: Annual Obligations for NNSA's Advanced Simulation and
Computing Program, Fiscal Years 2007 through 2009:
Abbreviations:
ASC: Advanced Simulation and Computing:
BIA: business impact analysis:
CNSS: Committee on National Security Systems:
DISCOM: Distance Computing:
DOD: Department of Defense:
DOE: Department of Energy:
FISMA: Federal Information Security Management Act of 2002:
FLOPS: floating-point operations per second:
Livermore: Lawrence Livermore National Laboratory:
Los Alamos: Los Alamos National Laboratory:
NIST: National Institute of Standards and Technology:
NNSA: National Nuclear Security Administration:
Sandia: Sandia National Laboratories:
[End of section]
United States Government Accountability Office:
Washington, DC 20548:
December 9, 2010:
The Honorable Henry Waxman:
Chairman:
Committee on Energy and Commerce:
House of Representatives:
The Honorable Edward J. Markey:
Chairman:
Subcommittee on Energy and the Environment:
Committee on Energy and Commerce:
House of Representatives:
The Honorable Bart Stupak:
Chairman:
Subcommittee on Oversight and Investigations:
Committee on Energy and Commerce:
House of Representatives:
The National Nuclear Security Administration[Footnote 1] (NNSA)
provides classified supercomputing capabilities for assessing the
performance of nuclear weapons. In the absence of nuclear weapons
testing--which ceased in 1992--the simulation capabilities of NNSA's
supercomputers are a necessary means to determine the effects of
changes to current weapons systems and to determine a level of
confidence in the performance of future untested systems.[Footnote 2]
These simulation capabilities also contribute to the enhancement of
NNSA's ability to predict the performance of weapons systems to ensure
the systems meet all military requirements established by the
Department of Defense (DOD).
NNSA's three nuclear weapons laboratories--Los Alamos National
Laboratory (Los Alamos) in New Mexico, Lawrence Livermore National
Laboratory (Livermore) in California, and the Sandia National
Laboratories (Sandia) with locations in New Mexico and California--use
these supercomputing simulation capabilities to obtain a comprehensive
understanding of the entire nuclear weapons life cycle, from design to
safe processes for dismantlement. These classified supercomputing
capabilities are a considerable investment and serve as a cornerstone
for NNSA's Stockpile Stewardship Program.[Footnote 3] In addition,
classified supercomputing capabilities are essential for informing
critical decisions related to the nuclear stockpile, including all
stockpile modernization and warhead studies. NNSA classified
supercomputing capabilities are also used to address other areas of
national security, including intelligence analyses, nuclear forensics,
and emergency response. Because of the importance of these classified
supercomputing capabilities to issues central to national security,
contingency and disaster recovery planning[Footnote 4] are key to
ensuring that, when unexpected events occur, NNSA can recover and
reconstitute its classified supercomputing systems, data, and
operations.
Our objectives were to assess the extent to which (1) NNSA has
implemented contingency and disaster recovery planning and testing for
its classified supercomputing assets, (2) the three laboratories are
able to share classified supercomputing capacity for recovery
operations, should service disruptions occur, and (3) NNSA tracks the
costs for ensuring contingency and disaster recovery planning for its
classified supercomputing assets. To accomplish these objectives, we
examined contingency and disaster recovery planning controls for the
systems within the classified supercomputing environment that are a
necessary means for NNSA's achievement of its nuclear weapons mission.
In addition, we performed technical assessments of classified
supercomputing capabilities at each weapons laboratory, including each
laboratory's ability to share supercomputing capacity. Further, we
obtained information from NNSA and laboratory officials to determine
how expenditures were tracked for contingency and disaster recovery
planning of the classified supercomputing systems at each of the
laboratories, as well as projected future cost estimates for ensuring
the recovery of these assets.
We conducted this performance audit from December 2009 through
December 2010 in accordance with generally accepted government
auditing standards. Those standards require that we plan and perform
the audit to obtain sufficient, appropriate evidence to provide a
reasonable basis for our findings and conclusions based on our audit
objectives. We believe that the evidence obtained provides a
reasonable basis for our findings and conclusions based on our audit
objectives. A more detailed description of our objectives, scope, and
methodology is contained in appendix I.
Background:
NNSA relies on its Stockpile Stewardship Program to ensure the safety,
security, and effectiveness of the nuclear weapons stockpile. The
Stockpile Stewardship Program is comprised of various elements,
including, but not limited to: (1) the Advanced Simulation and
Computing (ASC) Campaign, which provides the computational science and
simulation tools to understand the behaviors and effects of nuclear
weapons; (2) Directed Stockpile Work, which provides evidence of the
health of the nuclear weapons stockpile and involves day-to-day
maintenance of these weapons, including life extension efforts; (3)
the Science Campaign, which provides tools and capabilities geared
toward advancing the general understanding of all nuclear weapons
systems; and (4) the Engineering Campaign, which provides a sustained
basis for stockpile certification and assessments throughout the life
cycle of each weapon. The coordination among the Stockpile Stewardship
elements is instrumental to increasing NNSA's confidence in the
performance of nuclear weapons.
To help accomplish its Stockpile Stewardship mission, NNSA relies on
the three weapons laboratories--Los Alamos, Livermore, and Sandia. Los
Alamos and Livermore are the two design laboratories that are
responsible for designing the nuclear weapons' explosive package and
conducting research to better understand nuclear weapons phenomena.
Sandia is an engineering laboratory and has principal responsibility
for the research, design, and development of nonnuclear warhead
components; integration of these components with Los Alamos and
Livermore; and overall warhead systems integration with DOD. In
accordance with NNSA, management and operations contractors, who are
responsible for day-to-day operations of the laboratories, are
required to adhere to agency policies.[Footnote 5]
At the time of our review, NNSA's classified supercomputing resources
consisted of 12 classified supercomputing systems. Figure 1 shows the
hardware configuration of a supercomputing system.
Figure 1: Common Hardware Components of a Supercomputing System:
[Refer to PDF for image: illustration]
Compute chip:
Compute card:
Node card:
Cabinet:
System:
Source: GAO, data provided by Los Alamos, Livermore, and Sandia.
[End of figure]
NNSA classified supercomputing systems employ a large number of
interdependent processors, which are the core unit of a computer that
gathers instructions and data. These processors are mounted onto a
compute chip, which is the portion of the system that carries out the
instructions of a computer program. These compute chips are inserted
onto a compute card, which also holds memory for the compute chips to
use. A number of compute cards are attached to a node card, which have
one or more processors with a common memory and are connected by high-
speed interconnection networks. Each node card is inserted into a
single cabinet, and that configuration is repeated many times to build
a single supercomputing system. Each supercomputing system has a peak
performance, which is the maximum rate of floating-point operations
per second (FLOPS) that the system can sustain.[Footnote 6] Currently,
almost all NNSA classified supercomputer systems operate at the
teraFLOP level, which represents a trillion FLOPS.
According to NNSA, the laboratories have three types of classified
supercomputing systems:
Capacity: Small systems that execute parallel problems with more
modest computational requirements. These systems serve as the
workhorse for the ASC program and are responsible for processing the
day-to-day supercomputing workload.
Capability: This type of supercomputer is used to solve the largest
and most demanding problems that other computing systems cannot manage.
Advanced architecture: Research and development systems that assist
the ASC program in preparing to rapidly deploy and exploit the next
generation of supercomputing technology. These systems have a targeted
workload and serve as the foundation for the next generation of NNSA
supercomputers.
Table 1 shows the classified supercomputing systems currently in use
at the three weapons laboratories.
Table 1: Inventory of NNSA-Deployed Classified Supercomputing Systems
(as of October 2010):
Site: Los Alamos:
System name: Roadrunner Base;
System type: Capacity;
Delivery date: 10/2006;
Total processors: 18,432;
Peak performance (TeraFLOPS): 76.0.
System name: Roadrunner Phase-3;
System type: Advanced architecture;
Delivery date: 9/2008;
Total processors: 24,480;
Peak performance (TeraFLOPS): 1,280.0.
System name: Hurricane;
System type: Capacity;
Delivery date: 9/2008;
Total processors: 5,760;
Peak performance (TeraFLOPS): 51.2.
Site: Livermore:
System name: BlueGene/L;
System type: Advanced architecture;
Delivery date: 11/2004;
Total processors: 131,072;
Peak performance (TeraFLOPS): 367.0.
System name: Purple[A];
System type: Capability;
Delivery date: 6/2005;
Total processors: 12,288;
Peak performance (TeraFLOPS): 93.4.
System name: Rhea;
System type: Capacity;
Delivery date: 9/2006;
Total processors: 4,608;
Peak performance (TeraFLOPS): 22.1.
System name: Minos;
System type: Capacity;
Delivery date: 6/2007;
Total processors: 6,912;
Peak performance (TeraFLOPS): 33.2.
System name: Juno;
System type: Capacity;
Delivery date: 5/2008;
Total processors: 18,432;
Peak performance (TeraFLOPS): 162.2.
System name: Dawn;
System type: Advanced architecture;
Delivery date: 1/2009;
Total processors: 147,456;
Peak performance (TeraFLOPS): 501.4.
Site: Sandia-NM:
System name: Red Storm;
System type: Advanced architecture;
Delivery date: 3/2005;
Total processors: 31,680;
Peak performance (TeraFLOPS): 284.0.
System name: Unity;
System type: Capacity;
Delivery date: 3/2009;
Total processors: 4,352;
Peak performance (TeraFLOPS): 38.0.
Site: Sandia-CA;
System name: Whitney;
System type: Capacity;
Delivery date: 3/2009;
Total processors: 4,352;
Peak performance (TeraFLOPS): 38.0.
Source: GAO summary of data from Los Alamos National Laboratory,
Lawrence Livermore National Laboratory, and Sandia National
Laboratories.
[A] Although Purple was the capability system in use at the time of
our site visits, Livermore retired the system in November 2010.
[End of table]
NNSA's classified supercomputing capabilities consist of supporting
resources, including (1) parallel files systems, which store
transitory data; (2) network file systems, which store user and
project data for a calculation; (3) archival storage systems, which
serve as storage for data; and (4) visualization systems, which enable
users to better comprehend the results of their computations.
NNSA's classified supercomputing systems are connected via its
Enterprise Secure Network and the Distance Computing (DISCOM) network,
which function as supporting resources for the classified
supercomputing environment. The Enterprise Secure Network provides
classified communications across the nuclear weapons complex,
including security services and other activities that ensure the flow
of NNSA's data sharing and business missions. DISCOM provides secure,
high-speed remote access for intra-and inter-site file transfers and
enables users, across the three weapons laboratories, to operate on
remote computing resources as if they were local. DISCOM and the
Enterprise Secure Network serve as the backup networks to each other.
Figure 2 shows the composition of NNSA's classified supercomputing
network infrastructure.
Figure 2: NNSA's Classified Supercomputing Network Infrastructure:
[Refer to PDF for image: illustration]
The illustration depicts the following connections:
Connected to DISCOM Network (10 Gigabits per second):
and to NNSA Enterprise Secure Network (1 Gigabit per second):
Lawrence Livermore National Laboratory:
* BlueGene/L;
* Purple;
* Rhea;
* Minos;
* Juno;
* Dawn.
Los Alamos National Laboratory:
* Road Runner;
* Road Runner Phase 3;
* Hurricane.
Sandia National Laboratories, California:
* Whitney.
Sandia National Laboratories, New Mexico:
* Red Storm;
* Unity.
DISCOM Network and NNSA Enterprise Secure Network are interconnected.
Source: GAO, data provided by Los Alamos, Livermore, and Sandia.
[End of figure]
NNSA reported obligating approximately $1.7 billion from fiscal years
2007 through 2009 to support ASC program activities at the three
weapons laboratories.[Footnote 7] The $1.7 billion was predominantly
associated with three efforts:
Weapons codes and models. This effort is intended to develop and
improve weapons simulation models and codes for predicting the
behavior of weapons systems and devices in the nuclear stockpile.
Computational systems and software environment. This effort is
intended to provide ASC users a stable, seamless computing environment
for ASC-deployed platforms. It is responsible for procuring,
delivering, and deploying ASC computational systems and user
environments via technology development and integration across the
three weapons laboratories.
Facility operations and user support. This effort is intended to
provide both the necessary physical facility and operational support
for reliable supercomputing and storage environments, as well as a
suite of user services for effective use of the three weapons
laboratories' computing resources. Facility operations cover physical
space, power and other utility infrastructure, and local-and wide-area
networking, as well as system administration, cyber security, and
operations services for ongoing support. The user support function
includes planning, development, integration and deployment, continuing
product support, and quality and reliability activity collaborations.
To strengthen the security of information and information systems
across the federal government, including those at NNSA's weapons
laboratories, the Federal Information Security Management Act of 2002
(FISMA) requires each agency to develop, document, and implement an
agencywide information security program that supports the operations
and assets of the agency, including those provided or managed by
another agency or contractor on its behalf.[Footnote 8] This security
program is to include plans and procedures to ensure the continuity of
operations for information systems that support the agency's
operations.[Footnote 9] Pursuant to its FISMA responsibilities, the
National Institute of Standards and Technology (NIST) has issued
federal standards and guidelines on information security, such as a
contingency planning guide for federal information systems, and
recommended security controls, which address contingency and disaster
recovery planning and testing.[Footnote 10] To further ensure the
security of national security systems, the Committee on National
Security Systems (CNSS)[Footnote 11] requires federal agencies with
national security systems to implement a comprehensive set of security
controls and enhancements for these systems.[Footnote 12] CNSS
requires that each agency implement a contingency and disaster
recovery planning capability that ensures the integrity and
availability of its national security information and information
systems.[Footnote 13]
FISMA, NIST guidelines,[Footnote 14] and CNSS policies all call for
contingency and disaster recovery planning--also referred to as
continuity of operations for information systems--for critical
components of information protection. DOE and NNSA policies also
regard contingency and disaster recovery plans as being necessary for
information protection. If normal operations are interrupted,
contingency and disaster recovery plans allow senior agency officials
to detect, mitigate, and recover operations. Examples of the key
components that make up contingency and disaster recovery planning
programs include (1) assessing the criticality and sensitivity of
computerized operations and identification of supporting resources
such as developing business impact analyses (BIA), (2) taking steps to
prevent and minimize potential damage and interruption such as
establishing data backup processes, (3) developing comprehensive
contingency and disaster recovery plans,[Footnote 15] and (4)
conducting periodic testing of contingency and disaster plans.
The extent to which controls--such as contingency and disaster
recovery planning--are implemented depends on a level of risk assigned
to the system or information maintained on the system. NIST standards
and guidelines, CNSS instructions, and NNSA policy allow consideration
of risk in determining the level of protection of systems and data.
These standards and policies require that organizations consider the
impact or consequences of loss as it relates to the confidentiality,
integrity, and availability of the information, and assign a value of
low, moderate, or high impact levels. For contingency and disaster
recovery planning, consideration of "availability" is the key element.
NNSA policy defines the values for the consequences of loss associated
with availability as follows:
High: Loss of life might result from loss of availability; information
must always be available on request, with no tolerance for delay; loss
of availability will have an adverse effect on national-level
interests; federal requirement (i.e., requirement for material control
and accountability inventory); or loss of availability will have an
adverse effect on confidentiality.
Moderate: Information must be readily available with minimum tolerance
for delay; bodily injury might result from loss of availability; or
loss of availability will have an adverse effect on organizational-
level interests.
Low: Information must be available with flexible tolerance for delay.
NNSA Has Not Fully Implemented Contingency and Disaster Recovery
Planning and Testing for Its Classified Supercomputing Assets:
Contingency and disaster recovery planning and testing for NNSA's
classified supercomputing systems have not been fully implemented at
each of the three weapons laboratories--Los Alamos, Sandia, and
Livermore. Specifically, NNSA did not ensure that the laboratories (1)
developed BIAs to determine the impact of potential service
disruptions, (2) fully tested data backup processes, and (3) developed
and tested contingency and disaster recovery plans. These shortcomings
existed, at least in part, because NNSA's component organizations were
unclear of their roles and responsibilities for providing oversight in
the laboratories' implementation of contingency and disaster recovery
planning. Until the agency fully implements a contingency and disaster
recovery planning program for its classified supercomputing assets at
the weapons laboratories, it has limited assurance that vital
information can be recovered and made available to meet national
security priorities and requirements.
Not All of the Laboratories Assessed the Criticality and Sensitivity
of Supercomputer Operations and Resources, or Potential Outage Impact:
To assess the criticality and sensitivity of computerized operations
and identification of supporting resources, NIST guidelines state that
agencies should determine their recovery strategies by performing
business impact analyses of their systems. A BIA is an analysis of
information technology system requirements, processes, and
interdependencies used to characterize system contingency requirements
and priorities in the event of a significant disruption. NIST
guidelines state that agencies conduct a BIA to identify critical
information systems to fully characterize the system's requirements,
processes, and interdependencies to determine contingency requirements
and priorities. In addition, according to NIST guidelines, the BIA
process should follow three main steps: (1) identify critical data and
information technology resources, (2) identify outage impacts and
allowable outage times,[Footnote 16] and (3) develop recovery
priorities and strategies. NNSA policy also requires a BIA to identify
systems that provide critical services to site operations and
prioritize these systems and their components.
One of the laboratories--Los Alamos--had conducted a BIA that
addressed its classified supercomputing systems, generally following
the three steps of a BIA. However, the BIA was not always specific.
For example, the laboratory identified critical information technology
resources for each of its classified supercomputing systems, but did
not specifically identify the critical data. Instead, Los Alamos noted
that the systems are not considered mission critical nor mission
essential to the business needs of the laboratory,[Footnote 17] and
that the consequence of loss for system availability is low.
Additionally, it defined a specific number of days for the allowable
time frames for fully and partially disabled systems, but did not
provide specifics on allowable outage impacts. Further, the analyses
indicated high-level recovery priorities, but did not provide
specifics regarding the recovery process or strategies that would be
used for recovery efforts.
The other two laboratories did not conduct BIAs specifically for
classified supercomputing systems, but plan to do so. Livermore has a
BIA in place for its logical assets--the applications and services
that provide basic operational support to the Livermore computing
environment, but the BIA did not address any of the classified
supercomputing systems. However, at the time of our site visit,
Livermore officials stated they were beginning the process of
developing a BIA that would address their information technology needs
for their classified supercomputing systems, but the process was still
in the planning stage. Similarly, according to Sandia officials, the
laboratory has BIAs that address its unclassified information
technology systems, but does not currently have one specifically for
its classified supercomputing systems. However, Sandia officials
indicated that they plan to conduct a BIA for classified
supercomputing systems in 2011.
Although the two laboratories have not conducted any BIAs--in line
with the BIA conducted by Los Alamos--they have considered the risk of
consequence of loss from availability as low impact. NNSA also
considers the consequence of loss as low impact. In addition, NNSA and
the three laboratories do not consider the classified supercomputers
to be "mission critical." One laboratory categorized the systems as
"mission essential," while another referred to them as "mission
support elements, not mission essential elements." However, NNSA's
mission includes maintaining the safety, security, and effectiveness
of the nuclear deterrent without nuclear testing. The supercomputers
provide a necessary means to determine the effects of changes to
current weapons systems and to determine a level of confidence in the
performance of future untested systems. The classified supercomputing
capabilities serve as the computational surrogate to nuclear weapons
testing and are central to national security.
Regarding recovery priorities and strategies, each of the laboratories
indicated that it would likely rely on a process that is currently
being used for the capability system shared among the laboratories.
The laboratories generally rely on the Capability Computing Campaign
to prioritize the workload and develop priorities for jobs that need
to be run on the capability system.[Footnote 18] In the event of a
service disruption or emergency, laboratory officials told us that
they would likely rely on the same process for all of their systems.
However, this process has not been documented as a means for
establishing overall recovery priorities across the laboratories.
Until all of the laboratories have a BIA in place for their classified
supercomputing systems that (1) identifies and categorizes critical
data, (2) identifies acceptable allowable outage impacts and time
frames, and (3) establishes emergency processing priorities and
strategies, the potential impact of a system outage will remain
unclear.
The Laboratories Have Backup Processes in Place, but One Storage Site
May Be Susceptible to Damage:
Data backup processes offer a means of taking steps to prevent and
minimize potential damage and interruption to computerized services.
NIST guidelines, as well as CNSS instructions and NNSA policies, call
for agencies to conduct backups of user-level information, system-
level information, and information system documentation. In addition,
NIST, CNSS, and NNSA all provide that agencies establish an alternate
storage site that is separated from the primary storage site so that
both are not susceptible to the same hazards. To ensure the
availability of data stored in the alternate storage site, NIST and
CNSS require that agencies test the backup information to verify the
integrity of the data.
All of the laboratories had backup processes in place. Each of the
laboratories follows similar data backup processing--both manual and
automated procedures--to back up user-level information, system-level
information, and information system documentation. For example, this
information can include global directories, user home directories,
project directories, desktop systems, and critical systems
documentation. Backups occur in increments: daily incremental backups
to disk, weekly full backups to tape, and monthly full-system backups
to tape (with a 6-month on-site storage retention policy). The
laboratories also have vendor-provided software that takes periodic
snapshots of user directories for storage retention purposes. The
snapshot process can be performed manually or can be set up for
automatic processing. Users are encouraged to maintain their data in a
shared environment on the network and are allowed to make their own
determinations regarding what data should be backed up from the
classified supercomputing systems.
Not all of the laboratories have an alternate storage site
sufficiently separated from the primary site to not be susceptible to
the same hazards. Two of the three laboratories have alternate storage
sites a considerable distance from their primary storage site.
Livermore sends its system backups electronically to Los Alamos every
6 months. Sandia sends its backup data to its alternate site locations
(e.g, the California site sends its data to the New Mexico site and
the New Mexico site sends its data to the California site). However,
Los Alamos maintains its alternate storage facility on-site in a
building located less than 1 mile away from the primary local backup
storage facility. Consequently, both sites could be susceptible to the
same hazards, such as a wildfire.
The laboratories had processes in place to verify the integrity of the
backed up data. However, tests of their backup procedures rely
predominantly on ad hoc recovery, rather than periodically planned
tests. Los Alamos officials indicated that thousands of file
recoveries have been performed over the years by end users as part of
their testing. Livermore officials stated that the laboratory tests
its local backup procedures through actual system usage on almost a
daily basis, and tests their remote backup procedures at least once
annually. Further, Sandia officials told us they had successfully
tested a sample of data at their offsite facility.
Not All Laboratories Had Developed and Tested Contingency and Disaster
Recovery Plans:
NIST guidelines and CNSS policies call for the development and testing
of contingency plans and the development of disaster recovery plans
for each information system to ensure that, in the event of a service
disruption, the work and supporting functions of the agency can
continue to be performed. According to NIST guidelines, at a minimum,
the contingency plan should address the identification and
notification of key personnel, plan activation, system recovery, and
system reconstitution to meet the needs of the agency's critical
supporting operations. The guidelines also state that the plan should
be tested periodically; CNSS specifies that the frequency of testing
should be annually. NIST also notes that the disaster recovery plan
should be designed to restore operability of the targeted system,
application, or computer facility at an alternate site after a major
service disruption. DOE and NNSA policies also require the development
of contingency and disaster recovery plans and the testing of these
plans in line with NIST and CNSS.
Each of the laboratories had developed contingency plans for their
classified supercomputing systems; however, the plans were not always
comprehensive, and at the time of our site visits, these plans had not
been tested. The laboratories addressed disaster recovery planning to
a limited extent; none specifically addressed the supercomputing
environment. For example,
* Two laboratories--Los Alamos and Sandia--had contingency plans that
addressed the classified supercomputing systems. Although Livermore
had an information technology contingency plan and a master security
plan, neither specifically addressed the supercomputers. In addition,
the plans for both Los Alamos and Sandia included key components such
as the identification and notification of key personnel, plan
activation, system recovery, and system reconstitution procedures;
however, the sufficiency of the level of detail varied. For instance,
one plan provided specific details regarding system recovery processes
and the notification and identification of key personnel, but provided
limited details regarding plan activation and system reconstitution
procedures.
* At the time of our site visits, none of the laboratories had tested
their contingency plans, which were less than a year old. One of the
laboratories--Los Alamos--had created testing guides but had not yet
conducted formal testing. Subsequent to our site visit, Los Alamos
officials indicated that the first test of their plan took place in
September 2010 and noted that the results would be finalized in
December. Additionally, although Sandia had a contingency plan in
place, the plan states that testing is not required because, in the
event of a service disruption, the laboratory would either wait until
the equipment was fully operational or simply acquire new equipment.
This is contrary to NIST guidelines, CNSS instructions, and DOE and
NNSA policies.
* Each of the laboratories had addressed disaster recovery planning to
a limited extent. For example,
- Los Alamos included disaster recovery planning as a section within
their classified supercomputing system's contingency plans. Although
it provided high-level instructions such as directing individuals to
call 911 for all emergencies, it did not include information regarding
the specifics for restoring operability of the classified
supercomputing system at an alternate site after a major service
disruption.
- None of the plans submitted by Livermore specifically addressed the
supercomputing environment, although in the disaster recovery section
of the master security plan, the laboratory noted that it had no
mission-essential systems in the computing environment, and systems
may be offline for an extended period for system upgrades.
- Sandia also had disaster recovery plans in place, designed for
emergency preparedness and disease response planning needs for the
laboratory. These plans focused on emergencies involving the
facilities, operations, and activities for the laboratory, and
provided individuals with emergency information should pandemics
plague the laboratory. However, these plans did not include any
information regarding the classified supercomputing systems.
Unless each of the laboratories develops and sufficiently tests
comprehensive contingency and disaster recovery plans in accordance
with applicable policies and guidance for their classified
supercomputing systems, they face a risk of not being able to
successfully recover their supercomputing assets and operations after
a service disruption.
NNSA Component Organizations Were Unclear of Their Roles and
Responsibilities for Providing Oversight:
The aforementioned shortcomings existed, at least in part, because
NNSA's component organizations were unclear of their roles and
responsibilities for providing oversight in the laboratories'
implementation of contingency and disaster recovery planning. FISMA
requires that the chief information officer, in coordination with
other senior agency officials, manage the development and
implementation of an agencywide information security program that
includes plans and procedures to ensure continuity of operations for
information systems that support the operations and assets of the
agency. NIST guidelines and DOE policies call for individuals with
information system or security management and oversight
responsibilities to take responsibility for the development,
implementation, assessment, monitoring, reviewing, and updating of
security planning policies and procedures, which includes contingency
and disaster recovery plans. Further, the NNSA Safety Management
Function and Responsibilities and Authorities Manual states that the
chief information officer is responsible for information technology
programs and initiatives and for ensuring the security of the agency's
information and systems.
Although roles and responsibilities are defined at a high level in
FISMA, NIST guidelines, as well as DOE and NNSA policies, NNSA
component organizations were confused about their roles in providing
oversight of the laboratories' implementation of contingency and
disaster recovery planning for the supercomputing systems. For
example, at the beginning of our review, ASC officials told us that,
although they were responsible for administering and managing the
program that uses the classified supercomputing systems, they were not
responsible for contingency and disaster recovery planning. Instead,
they directed us to the Office of the Chief Information Officer
(OCIO), where officials told us that they were not responsible for
contingency and disaster recovery planning for these systems, and
noted that they would only provide guidance if requested by ASC.
Further, OCIO officials told us that ASC has not requested any
assistance. ASC officials subsequently acknowledged that they had
responsibility for contingency and disaster recovery planning;
however, this organizational responsibility is contrary to NIST
guidelines and DOE policies, as well as NNSA's own manual, which gives
this responsibility to the OCIO. In the absence of effective
oversight, the laboratories did not consistently comply with, or fully
implement, federal requirements and guidance related to contingency
planning and disaster recovery. Until NNSA clearly establishes and
carries out defined roles and responsibilities for OCIO and ASC
pertaining to contingency and disaster recovery planning for the
classified supercomputing environment, it will not be able to
effectively manage and oversee the recovery of its supercomputing
operations should service disruptions occur.
The Laboratories Have the Ability to Share Supercomputing Capacity,
but Barriers Exist:
Technologically, the weapons laboratories have demonstrated the
ability to share classified supercomputing capacity using their
capacity and capability systems under normal operating conditions.
Although these supercomputers process unique workloads and operate
independently, they are designed with a similar operating system,
resource manager, and job scheduler, which is built on a LINUX
foundation. These supercomputers also include application codes that
are portable across supercomputing systems and a data network, which
allows authorized users local and remote access to the systems.
Although the weapons laboratories have the ability to share
supercomputing capacity, barriers exist. One barrier to sharing
supercomputing capacity is that the weapons laboratories do not know
the minimum supercomputing capacity needed to achieve processing
priorities in the event of a service disruption. NIST guidelines
recommend, and NNSA policy requires, that capacity planning be
conducted so that there is adequate capacity for information
processing and supporting resources during contingency operations.
Although the weapons laboratories have identified the supercomputing
processing needed for normal business operations, they have not
identified the minimum supercomputing capacity needed to achieve
processing priorities in the event of a service disruption.
Another barrier to sharing supercomputing processing is the disparity
in usable supercomputing processing across the laboratories. Figure 3
depicts this disparity by identifying the amount of total usable
supercomputing capacity, in teraFLOPS, for each of the three weapons
laboratories for 2010 and 2011.[Footnote 19]
Figure 3: Total Usable Supercomputing Capacity at Each Weapons
Laboratory, 2010 and 2011:
[Refer to PDF for image: vertical bar graph]
TeraFLOPS: Los Alamos;
Total usable capacity: 2010: 127;
Total usable capacity: 2011: 1,427.
TeraFLOPS: Livermore;
Total usable capacity: 2010: 311;
Total usable capacity: 2011: 218.
TeraFLOPS: Sandia;
Total usable capacity: 2010: 76;
Total usable capacity: 2011: 76.
Source: GAO analysis of supercomputing capacity data provided by Los
Alamos, Livermore, and Sandia.
[End of figure]
For example, in 2010, total usable capacity at Livermore has been 311
teraFLOPS, whereas Los Alamos and Sandia have had 127 teraFLOPS and 76
teraFLOPS, respectively. Should Livermore experience service
disruptions for a sustained amount of time, neither Los Alamos nor
Sandia possesses the necessary usable supercomputing capacity to
accommodate the additional workload and NNSA will have to reprioritize
the computational workloads across the other two laboratories. As
previously noted, officials at the laboratories told us that, should
disruptions occur, they would use the Capability Computing Campaign
model for re-prioritizing the workload. However, this process has not
been documented for recovery activities.
Further limiting the ability of the weapons laboratories to recover
from a service disruption, in 2011, there will be a significant
disparity in projected usable supercomputing capacity. For example, in
2011, Los Alamos' usable capacity is projected to be 1,427 teraFLOPS,
whereas usable capacity at Livermore and Sandia is to be 218 teraFLOPS
and 76 teraFLOPS, respectively. Should Los Alamos' supercomputing
systems become unavailable for an extended period of time, neither
Livermore nor Sandia possesses sufficient usable supercomputing
capacity to achieve its workload and accommodate the additional
potential computational workload from Los Alamos. According to
laboratory officials, an additional supercomputer will be deployed at
Los Alamos in 2011. This supercomputer is a replacement for the single
capability supercomputer currently at Livermore that was retired in
2010. Therefore, a significant amount of usable supercomputing
capacity will be centralized at Los Alamos. Because the weapons
laboratories have not determined the minimum supercomputing capacity
requirements for their emergency processing priorities, they may not
be able to meet the minimum computational workload required to meet
Stockpile Stewardship milestones.
Another barrier to sharing supercomputing capacity across the weapons
laboratories is that the capability to share usable capacity on an "on-
demand"[Footnote 20] basis has not been fully tested in a recovery
scenario. According to officials at the laboratories, during normal
operating conditions, simulation programs have run on other
supercomputing systems. However, consideration has not been given to
include and test these abilities in a disaster recovery scenario
should a service disruption occur. As a result, NNSA has limited
assurance that its disaster recovery approach would work effectively
should a service disruption occur.
NNSA Does Not Track the Costs for Ensuring Contingency and Disaster
Recovery Planning for Its Supercomputing Assets:
Although NNSA reported obligating approximately $1.7 billion from
fiscal 2007 through 2009 to implement its ASC program activities at
the three weapons laboratories, the costs for ensuring the recovery of
its classified supercomputing operations are unknown. Under GAO's
Standards for Internal Control in the Federal Government, financial
information should be recorded and communicated to program managers
who need this information to make operational decisions and to
effectively allocate resources for program activities.
NNSA officials reported obligating approximately $390 million for
facility operations and user support activities, which include the
funds associated with contingency and disaster recovery planning
activities, but they were unable to provide detailed financial
information for contingency and disaster recovery planning activities.
According to NNSA officials, costs for contingency and disaster
recovery planning for classified supercomputing systems are unknown
because ASC program expenditures are part of the NNSA ASC operational
budget, whose costs are tracked at an aggregate level. As a result,
neither NNSA nor the three weapons laboratories can track what has
been spent since fiscal year 2007 for ensuring the recovery of
classified supercomputing operations and, consequently, they do not
know whether funding levels for these activities have been adequate.
Although certain components of contingency and disaster recovery
planning are in place at the three weapons laboratories, NNSA is
uncertain as to what funds were spent on these information protection
activities.
Furthermore, NNSA has not developed contingency planning and disaster
recovery cost estimates for its classified supercomputing assets. For
fiscal years 2011 through 2014, NNSA projects that it needs about $2.2
billion to implement its ASC activities, which support its Stockpile
Stewardship program--$984 million for weapons codes and models, $604
million for computational systems and software environment, and $588
million for facility operations and user support. Although NNSA has
developed its out-year funding needs over the next 4 years, it has not
developed estimates regarding the future costs for ensuring the
recovery of its classified supercomputing assets in the event of
service disruptions. Until NNSA develops a means for tracking current
contingency and disaster recovery costs and for developing estimates
of future costs, the agency will not have the information needed to
determine whether it is meeting its goals for effective stewardship of
public resources.
Conclusions:
All three NNSA weapons laboratories have implemented some components
of a contingency planning and disaster recovery program. NNSA,
however, has not provided effective oversight to ensure that the
laboratories have comprehensive and effective contingency and disaster
recovery planning and testing. For example, BIAs that identify
critical resources and outage impacts have not been developed for all
classified supercomputing systems and existing contingency plans at
the laboratories have not been thoroughly tested. Although one
laboratory's analysis is not comprehensive and the other two
laboratories have not completed a BIA, NNSA and the laboratories
consider the consequence of loss of availability of the classified
supercomputers as a low-risk impact, and do not consider them to be
mission critical. However, it is unclear how NNSA made this
determination given that (1) the analyses have not been completed; (2)
NNSA's mission includes maintaining the safety, security, and
effectiveness of the nuclear deterrent without nuclear testing; (3)
the classified supercomputing capabilities serve as the computational
surrogate to underground nuclear weapons testing and are central to
our national security; and (4) NNSA has obligated about $1.7 billion
over 3 fiscal years to support the Advanced Simulation and Computing
program, which includes classified supercomputing activities.
Beyond the activities undertaken by the laboratories, NNSA has not
developed a means for identifying, tracking, or re-prioritizing the
classified supercomputing workload across the operating environment.
In addition, the laboratories have not tested offsite recovery
capabilities and the agency has not tested the laboratories' ability
to share "on-demand" capacity if needed or determined the minimum
capacity needed to meet Stockpile Stewardship Program requirements,
particularly in the event that it may need to establish emergency
processing priorities. Further, although over a billion dollars have
been obligated to support the classified supercomputing capabilities
within the last 3 years, NNSA has not tracked the costs for ensuring
the recovery of the classified supercomputing systems, data, and
supporting resources should a service disruption occur. The classified
supercomputing program represents a significant investment, and
accountability for these systems is essential. Until NNSA clearly
defines its component organizations' roles and responsibilities and
fully implements an effective contingency and disaster planning
program, it has limited assurance that, in the event of a service
disruption, vital information could be recovered and made available to
meet national security priorities.
Recommendations for Executive Action:
To improve the effectiveness of contingency and disaster recovery
planning for NNSA's classified supercomputing capabilities, we
recommend that the Administrator of NNSA direct the weapons
laboratories to take the following four actions, where not already
implemented:
* Develop business impact analyses that, among other things, (1)
identify and prioritize critical systems, data, and supporting
resources; (2) identify allowable outage times and impacts for
classified supercomputing capabilities; and (3) identify recovery
priorities and strategies.
* Develop and implement comprehensive contingency and disaster
recovery plans for all classified supercomputing systems that identify
how each weapons laboratory's classified supercomputing capabilities
will be recovered following service disruptions.
* Conduct contingency and disaster recovery plan testing.
* Test the three weapons laboratories' ability to share "on-demand"
classified supercomputing capacity to ensure this capability will work
in the event of unexpected service disruptions.
In addition, we recommend that the Administrator of NNSA take the
following five actions:
* Document an agencywide means for reprioritizing the workload across
NNSA's classified supercomputing systems should a disruption occur.
* Clearly define the oversight responsibilities of the NNSA ASC
program office and the NNSA Office of the Chief Information Officer,
as they relate to contingency and disaster recovery planning for
NNSA's classified supercomputing operations.
* Identify, assess, and communicate the minimum classified
supercomputing capacity needed to meet Stockpile Stewardship
requirements in the event of a service disruption.
* Develop, document, and implement a process that identifies and
tracks expenditures for contingency and disaster recovery planning for
NNSA's classified supercomputing assets.
* Develop and document the total anticipated costs for contingency and
disaster recovery planning of NNSA's classified supercomputing assets,
which includes the replacement costs for these assets.
Agency Comments and Our Evaluation:
In providing written comments (reprinted in appendix III) on a draft
of this report, NNSA's Associate Administrator for Management and
Administration agreed that improvements can be made in contingency and
disaster recovery planning for supercomputing operations. He indicated
that NNSA agreed with six of our nine recommendations and outlined the
agency's intent to conduct business impact analyses, develop and test
appropriate contingency and disaster recovery plans, document workload
prioritization, and clearly define roles and responsibilities.
However, NNSA did not agree with our recommendation related to
identifying the minimum capacity needed to meet Stockpile Stewardship
requirements in the event of a service disruption. The Associate
Administrator stated that this recommendation did not take into
account that the different types of supercomputers--capacity and
capability--serve different functions and are procured and managed
differently. In our report, we recognize that different types of
supercomputers exist and that they are used for different purposes,
they process unique workloads and operate independently. However, as
we point out in the report, although the weapons laboratories have
identified supercomputing processing needed for normal business
operations, they have not identified the minimum capacity needed to
achieve processing priorities in the event of a service disruption. We
believe that the recommendation appropriately focuses on meeting
NNSA's Stockpile Stewardship mission and that capacity planning is
essential to ensure that information processing and supporting
resources exist during contingency operations, regardless of the type
of system used. Although NNSA did not agree with the recommendation,
the Associate Administrator stated that the agency will conduct a BIA
and build appropriate contingency strategies for both types of
supercomputers, as well as enhance capacity sizing actions to account
for contingency and disaster recovery operations.
Further, NNSA did not agree with two recommendations related to
identifying and tracking expenditures for contingency and disaster
recovery planning and documenting anticipated recovery planning costs,
including replacement costs of the assets. The Associate Administrator
asserted that this information would not add significant value to
managing contingency and disaster recovery planning. However, we
believe such actions reflect good government practices and would add
value by providing NNSA program managers with useful expenditure and
cost information to aid decision making with regards to contingency
and disaster recovery planning. As our report points out, GAO's
Standards for Internal Control in the Federal Government states that
financial information should be recorded and communicated to program
managers to help them make operational decisions and effectively
allocate resources for program activities. Strong financial and
internal controls are a major part of managing any organization
because they help government program managers achieve desired results
through effective stewardship of public resources. Accordingly, we
believe our recommendations have merit.
We are sending copies of this report to the Secretary of Energy; the
Administrator of NNSA; and the Directors of Los Alamos, Livermore, and
Sandia laboratories. Copies of the report will also be available to
others at no charge on the GAO Web site at [hyperlink,
http://www.gao.gov].
If you or your staffs have any questions about this report, please
contact Gene Aloise at (202) 512-6870, or aloisee@gao.gov; Nabajyoti
Barkakati at (202) 512-6415 or barkakatin@gao.gov; or Gregory C.
Wilshusen at (202) 512-6244 or wilshuseng@gao.gov. Contact points for
our Offices of Congressional Relations and Public Affairs may be found
on the last page of this report. GAO staff who made major
contributions to this report are included in appendix IV.
Signed by:
Gene Aloise:
Director, Natural Resources and Environment:
Signed by:
Nabajyoti Barkakoti:
Director, Center for Technology and Engineering:
Signed by:
Gregory C. Wilshusen:
Director, Information Security Issues:
[End of section]
Appendix I: Objectives, Scope, and Methodology:
The objectives of our review were to assess the extent to which (1)
the National Nuclear Security Administration (NNSA) has implemented
contingency and disaster recovery planning and testing for its
classified supercomputing assets, (2) the three laboratories are able
to share classified supercomputing capacity for recovery operations,
should service disruptions occur, and (3) NNSA tracks the costs for
ensuring contingency and disaster recovery planning for its classified
supercomputing assets. To address these objectives, we focused on
contingency and disaster recovery planning activities at NNSA
headquarters, as well as the operating environment for the 12
classified supercomputing systems at the three weapons laboratories--
Los Alamos National Laboratory, Livermore National Laboratory, and
Sandia National Laboratories.
To assess the extent to which NNSA has implemented contingency and
disaster recovery planning and testing for its classified
supercomputing assets, we examined contingency and disaster recovery
planning controls for the systems within the classified supercomputing
environment that are critical to NNSA's achievement of its nuclear
weapons mission. We collected and reviewed policies, procedures, and
guidelines from the National Institute of Standards and Technology,
the Committee on National Security Systems, the Department of Energy,
and NNSA. We also reviewed contingency plans and business impact
analyses provided by the weapons laboratories and compared them to
federal guidelines. We interviewed NNSA and laboratory officials to
determine whether they had documented critical system, data, and
supporting resources and whether contingency plans had been tested.
Further, we interviewed NNSA officials to determine to what extent
they have provided specific guidance and oversight for the
laboratories to ensure that contingency and disaster recovery planning
requirements are being met.
To determine the extent to which the three weapons laboratories have
the ability to share supercomputing capacity for backup and recovery
operations, we visited each weapons laboratory and gained an
understanding of the overall classified supercomputing infrastructure
and identified interconnectivity and control points. We performed
technical assessments of supercomputing capabilities at each weapons
laboratory, including each laboratory's ability to share
supercomputing capacity under normal operating conditions. We reviewed
the weapons laboratories' efforts to determine the minimal
supercomputing capacity needed to meet NNSA Stockpile Stewardship
Program requirements along with the ability of the weapons
laboratories to share supercomputing capacity on an "on-demand" basis,
including the use of advanced architecture systems. In addition, we
obtained documents describing the supercomputing system environment as
well as capacity information, along with the views of officials from
NNSA and the three weapons laboratories.
To assess the extent to which NNSA tracks costs for ensuring
contingency and disaster recovery planning for classified
supercomputing assets, we interviewed NNSA and weapons laboratory
officials to determine how expenditures were tracked for contingency
and disaster recovery planning of the classified supercomputing
systems at each of the laboratories. We also requested the amount of
funds NNSA obligated to the three weapons laboratories, and the amount
of funds the laboratories spent, in implementing NNSA's classified
supercomputing capabilities from fiscal years 2007 through 2009.
Further, we interviewed NNSA and laboratory officials to determine how
they projected future cost estimates for ensuring the recovery of
these assets for fiscal years 2011 through 2014. To assess the
reliability of data provided, we reviewed (1) the fiscal year 2009
financial statement audit for the system and (2) responses NNSA
provided to questions about processes and procedures for ensuring the
accuracy and completeness of data. Based on this information, we
determined the data are sufficiently reliable for the purposes of this
report.
We conducted this performance audit from December 2009 through
December 2010 in accordance with generally accepted government
auditing standards. Those standards require that we plan and perform
the audit to obtain sufficient, appropriate evidence to provide a
reasonable basis for our findings and conclusions based on our audit
objectives. We believe that the evidence obtained provides a
reasonable basis for our findings and conclusions based on our audit
objectives.
[End of section]
Appendix II: NNSA Annual Obligations for Its Advanced Simulation and
Computing Program, Fiscal Years 2007 through 2009:
NNSA reported obligating approximately $1.7 billion from fiscal years
2007 through 2009 to support Advanced Simulation and Computing (ASC)
program activities at the three weapons laboratories. The $1.7 billion
was used mainly for three efforts:
Weapons codes and models. This effort is intended to develop and
improve weapons simulation codes and models for predicting the
behavior of weapons systems and devices in the nuclear stockpile.
Computational systems and software environment. This effort is
intended to provide ASC users a stable, seamless computing environment
for ASC-deployed platforms. It is responsible for procuring,
delivering, and deploying ASC computational systems and user
environments via technology development and integration across the
three weapons laboratories.
Facility operations and user support. This effort is intended to
provide both the necessary physical facility and operational support
for reliable supercomputing and storage environments, as well as a
suite of user services for effective use of the three weapons
laboratories' computing resources. Facility operations cover physical
space, power and other utility infrastructure, and local-and wide-area
networking, as well as system administration, cyber security, and
operations services for ongoing support. The user support function
includes planning, development, integration and deployment, continuing
product support, and quality and reliability activity collaborations.
Figure 4 depicts NNSA's annual obligations for each of the three
efforts from fiscal years 2007 through 2009.
Figure 4: Annual Obligations for NNSA's Advanced Simulation and
Computing Program, Fiscal Years 2007 through 2009:
[Refer to PDF for image: stacked vertical bar graph]
Fiscal year: 2007;
Weapons codes and models: $280 million;
Computational systems and software environment: $183 million;
Facility operations and user support: $128 million;
Total: $591 million.
Fiscal year: 2008;
Weapons codes and models: $249 million;
Computational systems and software environment: $183 million;
Facility operations and user support: $114 million;
Total: $546 million.
Fiscal year: 2009;
Weapons codes and models: $221 million;
Computational systems and software environment: $161 million;
Facility operations and user support: $147 million;
Total: $529 million.
Source: GAO analysis of data provided by NNSA.
[End of figure]
As shown in figure 4, NNSA annual obligations for its classified
supercomputing operations decreased from about $591 million to $529
million between fiscal years 2007 and 2009. The largest obligation for
the classified supercomputing program was for weapons codes and
models, which accounted for approximately $750 million (or 45 percent)
of total obligations.
Obligations for computational systems and software environment
accounted for approximately $527 million (or 32 percent) of total
obligations. For the period, obligations for this effort decreased
from $183 million to $161 million.
The facility operations and user support activities, which includes,
among other things, expenditures for contingency and disaster recovery
planning, accounted for $390 million (or 23 percent) of total
obligations over the period. These obligations ranged from $114
million to $147 million for the 3 fiscal years.
[End of section]
Appendix III: Comments from the National Nuclear Security
Administration:
Department of Energy:
National Nuclear Security Administration:
Washington, DC 20585:
November 23, 2010:
Mr. Gregory C. Wilshusen:
Director:
Information Security Issues:
U.S. Government Accountability Office:
441 G Street, NW:
Washington, DC 20548:
Dear Mr. Wilshusen:
The National Nuclear Security Administration (NNSA) appreciates the
opportunity to review the Government Accountability Office's (GAO)
draft report, GAO-11-67, Information Security: National Nuclear
Security Administration Needs to Improve Contingency Planning for Its
Classified Supercomputing Operations. I understand the House Committee
on Energy and Commerce requested GAO to assess various aspects of
NNSA's Continuity of Operations Program to ensure that, in case of
service disruptions, the three weapons laboratories can maintain the
computer simulation capabilities needed to meet nuclear weapons
assessment and certification requirements. Specifically, GAO
identified, (1) whether NNSA has Continuity of Operations planning and
testing procedures in place across the classified supercomputing
environment of the three weapons laboratories; (2) whether the weapons
laboratories are able to share capacity for backup and recovery
operations; and (3) the past, present and future resources needed to
maintain supercomputing capabilities.
We are pleased that GAO recognizes the importance of the simulation
capabilities of NNSA's supercomputers to address stockpile stewardship
and other national security matters. While the draft report implies,
without explicitly stating, that the timeframe for reconstitution of
supercomputing assets should be similar to that required for
Continuity of Operations for national command and control, major
financial systems, and health and emergency services, that time
urgency is not consistent with existing policies for the recovery of
research and development capabilities, nor should it be. Nevertheless,
we agree that improvements can be made in contingency and disaster
recovery planning for supercomputing operations.
After careful review, additional time is required to establish further
plans and schedule solutions for addressing the GAO's recommendations.
NNSA will provide a more detailed response to the recommendations when
the final report is issued. However, we are providing a summary of
responses to the recommendations presented in the draft report.
Recommendation 1: Develop business impact analyses that, among other
things, (I) identify and prioritize critical systems, data, and
supporting resources, (2) identiI5, allowable outage times and impacts
for classified supercomputing capabilities, and (3) identify recovery
priorities and strategies.
Concur: NNSA will leverage current Business Impact Analysis (BIA)
activities underway at Lawrence Livermore National Laboratory, Los
Alamos National Laboratory, and Sandia National Laboratories and
perform a national level BIA to provide a consistent assessment across
the laboratories for classified supercomputing.
Recommendation 2: Develop and implement comprehensive contingency and
disaster recovery plans for all classified supercomputing systems that
identify how each weapons laboratory's classified supercomputing
capabilities will be recovered following service disruptions.
Concur: NNSA will develop appropriate plans based on the assessment
results of the BIA performed per the GAO's first recommendation.
Recommendation 3: Conduct contingency plan testing.
Concur: NNSA will conduct contingency plan testing according to
contingency and disaster recovery plans to be implemented based on the
assessment results of the BIA performed per the GAO's first
recommendation.
Recommendation 4: Classified supercomputing capacity to ensure this
capability will work in the event of unexpected service disruptions.
Concur: NNSA will test the three weapons laboratories' ability to
share classified capacity supercomputers according to contingency and
disaster recovery plans to be implemented based on the assessment
results of the BIA.
Recommendation 5: Document an agency-wide means for reprioritizing the
workload across NNSA's classified supercomputing systems should a
disruption occur.
Concur: The NNSA will adapt, apply and exercise procedures that are
routinely being used for prioritizing workload in capability computing
campaigns for use in contingencies and disasters.
Recommendation 6: Clearly define the oversight responsibilities of the
NNSA ASC program office and the NNSA Office of the Chief Information
Officer, as they relate to contingency and disaster recovery planning
for NNSA's classified supercomputing operations.
Concur: In general, the NNSA Office of the Chief Information Officer
provides policy and guidance and the Office of Advanced Simulation and
Computing (ASC) has responsibilities for execution. Oversight
responsibilities will be clearly defined through the BIA and
development and implementation of the contingency and disaster
recovery plans.
Recommendation 7: Identify, assess, and communicate the minimum
classified supercomputing capacity needed to meet Stockpile
Stewardship requirements in the event of a service disruption.
Nonconcur: This recommendation does not take into account that
capacity and capability systems serve different functions under
different cost of ownership models, and consequently are procured and
managed differently. The ASC program has deployed its supercomputing
assets to mitigate single site failures. For the future, the program
will enhance capacity sizing actions to account for contingency and
disaster recovery operations when planning host sites for capacity
computing capabilities. We will conduct a BIA assessment recognizing
the differences and build appropriate contingency strategies for both
classes of computing.
Recommendation 8: Develop, document, and implement a process that
identifies and tracks expenditures for contingency and disaster
recovery planning for NNSA 's classified supercomputing assets.
Nonconcur: Almost all classified supercomputing contingency and
disaster recovery planning leverages computing resources and
activities funded as part of a production simulation environment for
weapons designers and engineers. These expenses are integral to ASC's
Facility Operations and User Support Program element and tracking them
separately would not add significant value to managing contingency and
disaster recovery.
Recommendation 9: Develop and document the total anticipated costs for
contingency and disaster recovery planning of NNSA 's classified
supercomputing assets, which includes the replacement costs for these
assets.
Nonconcur: As stated in the response to Recommendation 8, these
expenses are integral to ASC's Facility Operations and User Support
program element and tracking them separately would not add significant
value to managing contingency and disaster recovery.
If you have any questions related to this response, please contact
JoAnne Parker, Director, Office of Internal Controls, at 202-586-1913.
Sincerely,
Signed by:
Gerald L. Talbot, Jr.
Associate Administrator for Management and Administration:
cc: Acting Chief Information Officer:
Deputy Administrator for Defense Programs:
[End of section]
Appendix IV: GAO Contacts and Staff Acknowledgments:
GAO Contacts:
Gene Aloise (202) 512-3841 or aloisee@gao.gov Nabajyoti Barkakati
(202) 512-6415 or barkakatin@gao.gov Gregory C. Wilshusen (202) 512-
6244 or wilshuseng@gao.gov:
Staff Acknowledgments:
In addition to the individuals named above, Glen Levis, Edward M.
Glagola, Jr., and Jeffrey Knott (Assistant Directors); and Preston S.
Heard, Jennifer R. Franks, Kevin Metcalfe, and Zsaroq Powe were key
contributors to this report. Neil Doherty, Nancy Glover, Franklin
Jackson, and Jonathan Kucskar also made key contributions to this
report.
[End of section]
Footnotes:
[1] NNSA was established in 2000 as a separately organized agency
within the Department of Energy (DOE) and is responsible for the
nation's nuclear weapons, nonproliferation, and naval reactors
programs.
[2] For nearly half a century, the United States' nuclear program was
spearheaded by underground nuclear testing and never had to rely on
weapon systems that had exceeded their design life times. The United
States last produced a nuclear weapon in 1991 and performed its last
underground nuclear test in 1992.
[3] The National Defense Authorization Act for Fiscal Year 1994, Pub.
L. No. 103-160, § 3138 (1993), directed DOE to establish the Stockpile
Stewardship Program. In the absence of underground nuclear testing,
the program encompasses a broad range of activities to increase
understanding of the basic phenomena associated with nuclear weapons,
provide better predictive understanding of the safety and reliability
of weapons, and ensure a strong scientific and technical basis for
future nuclear weapons policy objectives. The Stockpile Stewardship
Program is carried out through the nuclear weapons complex, which
includes three nuclear weapons laboratories.
[4] Continuity of operations focuses on restoring an organization's
mission-essential functions at an alternate site and performing those
functions for a short period of time before returning to normal
operations. Contingency and disaster recovery planning include a broad
scope of activities designed to sustain and recover critical
information and information system services for a range of potential
service disruptions. Contingency and disaster recovery planning
components may include the relocation of information systems and
operations to an alternate site, recovery of information system
functions using alternate equipment, or the performance of information
system functions using alternative methods. For the purposes of this
report, the term contingency and disaster recovery planning refer to
the interim measures NNSA should use to recover information system
services after an unexpected service disruption.
[5] Los Alamos is managed and operated by Los Alamos National
Security, LLC, which is a consortium of contractors that includes
Bechtel National, the University of California, the Babcock and Wilcox
Company, and the Washington Division of URS. Livermore is managed and
operated by Lawrence Livermore National Security, LLC, which is
comprised of a corporate management team that includes Bechtel
National, the University of California, the Babcock and Wilcox
Company, and the Washington Division of URS. Sandia is managed and
operated by Sandia Corporation, a wholly owned subsidiary of Lockheed
Martin Corporation.
[6] FLOPS are a measure of a supercomputing system's performance.
Floating-point performance is the rate at which a computer executes
floating-point operations.
[7] For additional information regarding budgetary information for the
classified supercomputing program from fiscal years 2007 through 2009,
see appendix II.
[8] 44 U.S.C. § 3544(b); FISMA was enacted as title III, E-Government
Act of 2002, Pub. L. No. 107-347, 116 Stat. 2899, 2946 (Dec. 17, 2002).
[9] For the purposes of this report, we will refer to "continuity of
operations procedures for information systems" as contingency and
disaster recovery planning.
[10] NIST Special Publication 800-34, Contingency Planning Guide for
Information Technology Systems (Washington, D.C.: June 2002) and NIST
Special Publication 800-53 Revision 3, Recommended Security Controls
for Federal Information Systems and Organizations (Gaithersburg, Md.:
August 2009).
[11] Formerly known as the National Security Telecommunications and
Information Systems Security Committee, CNSS provides a forum for the
discussion of policy issues, sets national policy, and provides
direction, operational procedures, and guidance for the security of
national security systems. DOD chairs the committee under the
authorities established by National Security Directive 42, National
Policy for the Security of National Security Telecommunications and
Information Systems, issued in July 1990. This directive designates
the Secretary of Defense and the Director of the National Security
Agency as the Executive Agent and National Manager, respectively. The
committee has 21 voting representatives from various departments and
agencies, including the Department of Energy.
[12] National security systems include any information system used or
operated by an agency, or by a contractor of an agency, that
processes, stores, or transmits national security information. They do
not include those systems used for routine administrative and business
applications.
[13] CNSS Instruction 1253 provides federal government departments,
agencies, bureaus, and offices with a process for security
categorization of national security systems that collect, generate,
process, store, display, transmit, or receive national security
information. In addition, this instruction serves as a companion
document to NIST Special Publication 800-53, Revision 3.
[14] Although NIST guidelines note they shall not apply to national
security systems without the express approval of appropriate federal
officials exercising policy authority over such systems, CNSS
instructions, as well as DOE and NNSA policies for national security
systems, refer to the NIST guidelines as being applicable.
[15] A contingency plan is designed to maintain or restore business
operations, including computer operations, possibly at an alternate
location in the event of emergencies, system failures, or disaster. A
disaster recovery plan is a written plan for processing critical
applications in the event of a major hardware or software failure or
destruction of facilities.
[16] Outage impacts and allowable outage times enable the organization
to develop and prioritize recovery strategies that personnel will
implement during contingency plan activation. The effects of the
outage may be tracked over time, which will enable the organization to
identify the maximum allowable time that a resource may be unavailable
before it inhibits the performance of an essential function. The
effects of the outage can also be tracked across related resources,
identifying any cascading effects that may occur as an effect of a
service disruption.
[17] The Department of Energy defines "mission critical" as an
information system that supports an organization's core missions and
goals, and "mission-essential (or business essential)" as an
information system whose failure would not preclude organizations from
accomplishing core business functions in the long term.
[18] The Capability Computing Campaign includes a committee made up of
staff from the NNSA ASC program office, as well as ASC executives
located at the laboratories at Los Alamos, Livermore, and Sandia.
[19] Total usable supercomputing capacity includes the supercomputers
that have the ability to run all weapons program codes and could be
used in the event of a service disruption, and includes capacity and
capability systems.
[20] The term "on demand" is defined as the ability to move an
application (simulation program/code) from one supercomputer to a
different supercomputer at a different physical facility and use the
existing computational resources without the need for major
modifications.
[End of section]
GAO's Mission:
The Government Accountability Office, the audit, evaluation and
investigative arm of Congress, exists to support Congress in meeting
its constitutional responsibilities and to help improve the performance
and accountability of the federal government for the American people.
GAO examines the use of public funds; evaluates federal programs and
policies; and provides analyses, recommendations, and other assistance
to help Congress make informed oversight, policy, and funding
decisions. GAO's commitment to good government is reflected in its core
values of accountability, integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each
weekday, GAO posts newly released reports, testimony, and
correspondence on its Web site. To have GAO e-mail you a list of newly
posted products every afternoon, go to [hyperlink, http://www.gao.gov]
and select "E-mail Updates."
Order by Phone:
The price of each GAO publication reflects GAO‘s actual cost of
production and distribution and depends on the number of pages in the
publication and whether the publication is printed in color or black and
white. Pricing and ordering information is posted on GAO‘s Web site,
[hyperlink, http://www.gao.gov/ordering.htm].
Place orders by calling (202) 512-6000, toll free (866) 801-7077, or
TDD (202) 512-2537.
Orders may be paid for using American Express, Discover Card,
MasterCard, Visa, check, or money order. Call for additional
information.
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]:
E-mail: fraudnet@gao.gov:
Automated answering system: (800) 424-5454 or (202) 512-7470:
Congressional Relations:
Ralph Dawn, Managing Director, dawnr@gao.gov:
(202) 512-4400:
U.S. Government Accountability Office:
441 G Street NW, Room 7125:
Washington, D.C. 20548:
Public Affairs:
Chuck Young, Managing Director, youngc1@gao.gov:
(202) 512-4800:
U.S. Government Accountability Office:
441 G Street NW, Room 7149:
Washington, D.C. 20548:
