NASA's Space Vision
Business Case for Prometheus 1 Needed to Ensure Requirements Match Available Resources
Gao ID: GAO-05-242 February 28, 2005
In 2003, the National Aeronautics and Space Administration (NASA) initiated the Prometheus 1 project to explore the outer reaches of the Solar System. The Prometheus 1 spacecraft is being designed to harness nuclear energy that will increase available electrical power from about 1,000 watts to over 100,000 watts and enable the use of electric propulsion thrusters. Historically, NASA has had difficulty implementing some initiatives. NASA's failure to adequately define requirements and quantify the resources needed to meet those requirements has resulted in some projects costing more, taking longer, and achieving less than originally planned. Prometheus 1 will need to compete for NASA resources with other space missions--including efforts to return the shuttle safely to flight and complete the International Space Station. GAO was asked to determine (1) whether NASA is establishing initial justification for its investment in the Prometheus 1 project and (2) how the agency plans to ensure that critical technologies will be sufficiently mature at key milestones.
NASA is in the process of establishing initial justification for its investment in the Prometheus 1 project but faces challenges establishing preliminary requirements and developing accurate cost estimates. Decision makers will not get their first comprehensive picture of the project's requirements and the resources needed to meet those requirements until the preliminary mission and systems review, scheduled for summer 2005. Defining the project's requirements and developing life-cycle cost estimates by then could be challenging, given the short time frames. The fidelity of this information should improve by the preliminary design review scheduled for 2008. At that time, NASA has the opportunity to use these more refined requirements and cost estimates to establish a sound business case for its investment in the Prometheus 1 project. According to Prometheus 1 project management, a flat funding profile is inadequate to ramp up for the planned 2015 launch of Prometheus 1, the project's first spacecraft to its original destination of Jupiter's Icy Moons. By matching requirements to resources a sound business case would allow NASA to determine whether trade-offs in the design of the spacecraft or the agency's expectations are needed to avoid outstripping available resources. Significant program cost and schedule increases in past programs can be traced to not matching requirements with resources at preliminary design review. While development of the Prometheus 1 technologies is under way, each will require extensive advancement before they are mature enough to support reliable cost estimates. NASA is preparing technology development plans that include measurable criteria to ensure the Prometheus 1 technologies are on track for meeting NASA's maturity requirements through the end of the preliminary design phase. GAO's best practices work has shown, however, that establishing a formal business case based on a knowledge-based approach that includes matching requirements and available resources--which include technical and engineering knowledge, time, and funding--and controls to ensure that sufficient knowledge has been attained at critical junctures within the product development process is an essential part of any product development justification. NASA's current policy does not require projects to develop knowledge-based business cases that match requirements to available resources and include controls to ensure that sufficient knowledge has been attained. Therefore, the agency had not planned to develop such a business case for Prometheus 1. Since GAO provided our draft report to NASA for comment, the agency released its fiscal year 2006 budget request that includes changes to Prometheus 1. If properly implemented, these changes could be positive steps in addressing the findings and recommendations in this report.
Recommendations
Our recommendations from this work are listed below with a Contact for more information. Status will change from "In process" to "Open," "Closed - implemented," or "Closed - not implemented" based on our follow up work.
Director:
Team:
Phone:
GAO-05-242, NASA's Space Vision: Business Case for Prometheus 1 Needed to Ensure Requirements Match Available Resources
This is the accessible text file for GAO report number GAO-05-242
entitled 'NASA's Space Vision: Business Case for Prometheus 1 Needed to
Ensure Requirements Match Available Resources' which was released on
March 23, 2005.
This text file was formatted by the U.S. Government Accountability
Office (GAO) to be accessible to users with visual impairments, as part
of a longer term project to improve GAO products' accessibility. Every
attempt has been made to maintain the structural and data integrity of
the original printed product. Accessibility features, such as text
descriptions of tables, consecutively numbered footnotes placed at the
end of the file, and the text of agency comment letters, are provided
but may not exactly duplicate the presentation or format of the printed
version. The portable document format (PDF) file is an exact electronic
replica of the printed version. We welcome your feedback. Please E-mail
your comments regarding the contents or accessibility features of this
document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
Report to Congressional Requesters:
United States Government Accountability Office:
GAO:
February 2005:
NASA's Space Vision:
Business Case for Prometheus 1 Needed to Ensure Requirements Match
Available Resources:
GAO-05-242:
GAO Highlights:
Highlights of GAO-05-242, a report to congressional requesters
Why GAO Did This Study:
In 2003, the National Aeronautics and Space Administration (NASA)
initiated the Prometheus 1 project to explore the outer reaches of the
Solar System. The Prometheus 1 spacecraft is being designed to harness
nuclear energy that will increase available electrical power from about
1,000 watts to over 100,000 watts and enable the use of electric
propulsion thrusters.
Historically, NASA has had difficulty implementing some initiatives.
NASA‘s failure to adequately define requirements and quantify the
resources needed to meet those requirements has resulted in some
projects costing more, taking longer, and achieving less than
originally planned. Prometheus 1 will need to compete for NASA
resources with other space missions”including efforts to return the
shuttle safely to flight and complete the International Space Station.
GAO was asked to determine (1) whether NASA is establishing initial
justification for its investment in the Prometheus 1 project and (2)
how the agency plans to ensure that critical technologies will be
sufficiently mature at key milestones.
What GAO Found:
NASA is in the process of establishing initial justification for its
investment in the Prometheus 1 project but faces challenges
establishing preliminary requirements and developing accurate cost
estimates. Decision makers will not get their first comprehensive
picture of the project‘s requirements and the resources needed to meet
those requirements until the preliminary mission and systems review,
scheduled for summer 2005. Defining the project‘s requirements and
developing life-cycle cost estimates by then could be challenging,
given the short time frames. The fidelity of this information should
improve by the preliminary design review scheduled for 2008. At that
time, NASA has the opportunity to use these more refined requirements
and cost estimates to establish a sound business case for its
investment in the Prometheus 1 project. According to Prometheus 1
project management, a flat funding profile is inadequate to ramp up for
the planned 2015 launch of Prometheus 1, the project‘s first spacecraft
to its original destination of Jupiter‘s Icy Moons. By matching
requirements to resources a sound business case would allow NASA to
determine whether trade-offs in the design of the spacecraft or the
agency‘s expectations are needed to avoid outstripping available
resources. Significant program cost and schedule increases in past
programs can be traced to not matching requirements with resources at
preliminary design review.
While development of the Prometheus 1 technologies is under way, each
will require extensive advancement before they are mature enough to
support reliable cost estimates. NASA is preparing technology
development plans that include measurable criteria to ensure the
Prometheus 1 technologies are on track for meeting NASA‘s maturity
requirements through the end of the preliminary design phase.
GAO‘s best practices work has shown, however, that establishing a
formal business case based on a knowledge-based approach that includes
matching requirements and available resources”which include technical
and engineering knowledge, time, and funding”and controls to ensure
that sufficient knowledge has been attained at critical junctures
within the product development process is an essential part of any
product development justification. NASA‘s current policy does not
require projects to develop knowledge-based business cases that match
requirements to available resources and include controls to ensure that
sufficient knowledge has been attained. Therefore, the agency had not
planned to develop such a business case for Prometheus 1.
Since GAO provided our draft report to NASA for comment, the agency
released its fiscal year 2006 budget request that includes changes to
Prometheus 1. If properly implemented, these changes could be positive
steps in addressing the findings and recommendations in this report.
What GAO Recommends:
GAO is making recommendations aimed at ensuring that NASA prepares a
sound business case for Prometheus 1. NASA concurred with our
recommendations.
www.gao.gov/cgi-bin/getrpt?GAO-05-242.
To view the full product, including the scope and methodology, click on
the link above. For more information, contact Allen Li at (202) 512-
3600 or Lia@gao.gov.
[End of section]
Contents:
Letter:
Results in Brief:
Background:
Initial Justification for Prometheus 1 Project Could Form the Basis of
a Sound Business Case:
NASA's Plans to Ensure Mature Technologies Rely on Prototype
Demonstrations:
Conclusions:
Recommendations for Executive Action:
Agency Comments:
Scope and Methodology:
Appendix I: Technology Readiness Levels Definitions:
Appendix II: Prometheus 1 Critical Technologies:
Nuclear Reactor:
Power Conversion:
Heat Rejection:
Nuclear Electric Propulsion:
Radiation Hardening:
High Power Communications:
Autonomous Rendezvous and Docking:
Appendix III: Comments from the National Aeronautics and Space
Administration:
Appendix IV: GAO Contact and Staff Acknowledgments:
GAO Contact:
Acknowledgments:
GAO Related Products:
Figures:
Figure 1: Prometheus 1 Spacecraft:
Figure 2: NASA Prometheus 1 Fiscal Year 2005 Budget Request Profile:
Figure 3: Prometheus 1 Milestone Reviews:
Figure 4: Technology Maturity Levels for Product Development:
Abbreviations:
AR&D: autonomous rendezvous and docking:
JIMO: Jupiter Icy Moons Orbiter:
JPL: Jet Propulsion Laboratory:
MCT: maturity criteria tables:
NASA: National Aeronautics and Space Administration:
PDR: preliminary design review:
PMSR: preliminary mission and systems review:
SLI: Space Launch Initiative:
TRL: technology readiness levels:
United States Government Accountability Office:
Washington, DC 20548:
February 28, 2005:
The Honorable Daniel K. Inouye:
Co-Chairman:
Committee on Commerce, Science, and Transportation:
United States Senate:
The Honorable John McCain:
United States Senate:
In 2003, the National Aeronautics and Space Administration (NASA)
initiated the Prometheus 1 project to explore the outer reaches of the
Solar System in the hopes of finding answers to some of humankind's
most profound questions about life and its origins. The Prometheus 1
spacecraft is being designed to harness nuclear energy that will
efficiently increase available electrical power from about 1,000 watts
to over 100,000 watts and enable the use of electric propulsion
thrusters. The availability of nuclear power at this magnitude will
fundamentally change NASA's capability to explore deep space.
While NASA has been successful in missions such as Mars Pathfinder and
Exploration rovers, the agency has had difficulty implementing a number
of other costly initiatives because it was overly optimistic in what
could be achieved within available resources. NASA's failure to
adequately define requirements and quantify the resources needed to
meet those requirements has resulted in some projects costing more,
taking longer, and achieving less than originally planned. Prometheus 1
will compete for NASA resources with other space missions--including,
in the near term, efforts to return the shuttle safely to flight and
completing the International Space Station.
Cognizant of the outlook for a constrained federal budget for the
foreseeable future and NASA's difficulty in implementing some major
programs within projected resources, you asked us to review the
Prometheus 1 project to determine (1) whether NASA is establishing
initial justification for its investment in the Prometheus 1 project
and (2) how the agency plans to ensure that critical technologies will
be sufficiently mature at key milestones.
To address our objectives, we obtained and reviewed Prometheus 1 plans,
schedules, risk assessments, budget documentation, technology maturity
assessments, and technology development plans. We conducted further
qualitative and quantitative analyses of these documents and compared
them to criteria established in NASA and Jet Propulsion Laboratory
(JPL) policies governing development programs and in GAO's best
practices body of work (see GAO Related Products). Our work was
conducted between April 2004 and January 2005 in accordance with
generally accepted government auditing standards.
Since our draft report was provided to NASA for comment, a significant
change has been made to the Prometheus 1 project. NASA's fiscal year
2006 budget request includes changes to the Prometheus 1 project that
directly address the findings and recommendations of this report. These
changes are briefly outlined in the Agency Comments section of this
report.
Results in Brief:
NASA is in the process of establishing initial justification for its
investment in the Prometheus 1 project but faces challenges preparing
preliminary requirements and cost estimates. NASA wants to have a
defined set of preliminary system requirements and an initial estimate
of the life-cycle cost for Prometheus 1 by summer 2005--when the
project enters into the preliminary design phase. While the project
office has drafted cost guidelines and a technical baseline for use in
developing an initial life-cycle cost estimate, due to the early nature
of the project, the development of this estimate is just under way.
Before Prometheus 1 can move forward into the preliminary design phase,
NASA must make funding decisions. At this time, however, the level of
funding NASA needs to execute the project is not fully defined. NASA
requested $438 million for fiscal year 2005, but according to project
officials, the "flat line" estimate for the next few years would need
to be increased to reflect project needs of a mission to Jupiter's Icy
Moons. A business case providing an understanding of the potential
return on such investment would be helpful to decision makers for
determining whether continued investment--currently estimated by the
Congressional Budget Office at about $10 billio[Footnote 1]n--is
warranted. Developing a sound business case that includes matching
requirements with expected resources, would enable NASA to make early
trade-offs either in the preliminary design of the spacecraft or in the
agency's expectations to avoid outstripping available resources. While
NASA will have information available to match preliminary requirements
to expected resources, the fidelity of this information is not expected
to be completely defined until the preliminary design review (PDR),
currently scheduled for 2008.
NASA plans to ensure critical technologies are mature by demonstrating
subsystem prototypes before the end of the preliminary design phase.
There are several technologies, however, whose maturity will influence
NASA's ability to develop initial requirements and reliable resource
estimates. The Prometheus 1 design conceived for the mission to
Jupiter's Icy Moons relied on advancements in several breakthrough
technologies, including nuclear electric power and propulsion, high
power communications, radiation-hardened electronics, and autonomous
rendezvous and docking (AR&D). NASA is preparing technology development
plans that include measurable criteria to ensure each technology is on
track for meeting the maturity requirements through the end of the
preliminary design phase. While development of these technologies is
under way, each will require extensive advancement before they are
mature enough to provide the revolutionary capabilities of the
Prometheus 1 spacecraft.
Our past work on the best practices of product developers in government
and industry has found that development of a sound business case based
on matching requirements to resources is a key factor in successfully
addressing such challenges. Despite the fact that the Prometheus 1
project is in the very early stages of development, the use of a sound
business case that includes well-defined requirements, realistic cost
estimates, and mature technology is an essential part of any product
development investment justification. NASA's current policy does not
require projects to develop formal knowledge-based business cases that
match requirements to available resources and include controls to
ensure that sufficient knowledge has been attained. NASA had not
planned to develop such a business case. Subsequent to our draft being
provided to NASA for comment, the agency announced in its fiscal year
2006 budget request that it was conducting an analysis of alternatives
to identify a new mission with reduced technical, schedule, and
operational risk. As NASA will be establishing a new justification for
the Prometheus 1 project, we are recommending that the NASA
Administrator establish a sound business case for the Prometheus 1
project wherein resources are matched to requirements and controls are
in place to ensure that sufficient knowledge has been attained at
critical phases of the product development process.
Background:
The Prometheus 1 project is part of NASA's Prometheus Nuclear Systems
and Technology program[Footnote 2] to develop nuclear power
technologies capable of providing power and propulsion for a new
generation of missions. The Prometheus 1 spacecraft is being designed
to use nuclear power and electric propulsion technologies to explore
the outer reaches of the solar system. The Jupiter Icy Moons Orbiter
(JIMO) mission--a 4 to 6-year study of three of Jupiter's moons:
Callisto, Europa, and Ganymede--was the original destination identified
by NASA.[Footnote 3] The JIMO mission's overarching science objectives
were to (1) investigate the origin and evolution of the three moons;
(2) scout their potential for sustaining life; and (3) determine the
current rate of movement of surface ice and the rates at which the
moons are weathered. With an unprecedented level of power, Prometheus
1, the first in a potential series of spacecraft, is expected to
support the use of high capability science instruments and high power
communications systems to provide scientists with an a unprecedented
amount of scientific information. Figure 1 depicts the notional
Prometheus 1 spacecraft.
Figure 1: Prometheus 1 Spacecraft:
[See PDF for image]
[End of figure]
NASA contracted with the Jet Propulsion Laboratory (JPL) to manage the
Prometheus 1 project and to manage development of the science mission
payload. In turn, JPL awarded a $400-million contract for the initial
development of the Prometheus 1 spacecraft to Northrop Grumman Space
Technology in September 2004. NASA is collaborating with the Department
of Energy's Office of Naval Reactors to develop and handle all issues
related to the spacecraft's nuclear reactor.
The Prometheus 1 project will have to compete for funding with other
NASA programs. In January 2004, the President charged NASA with
implementing a new strategy for space exploration--which includes the
Prometheus 1 project--while simultaneously returning the shuttle to
flight status and completing the International Space Station. NASA laid
out its plan for implementing the strategy in its fiscal year 2005
budget request. In essence, NASA's implementation plan holds
aeronautics, science, and other activities at near constant levels and
transitions funding levels currently dedicated to the Space Station and
shuttle programs to the space exploration strategy as the Space Station
and shuttle programs phase out. This plan was predicated upon NASA's
annual funding level receiving increases to about $18 billion a year by
fiscal year 2008 and then remaining near that level, except for
inflation, through at least 2020.
Best Practices Reveal Elements of a Sound Business Case for Product
Development:
In the last several years, we have undertaken a best practices body of
work on how leading developers in industry and government use a
knowledge-based approach to develop products that reduces risks and
increases the likelihood of successful outcomes. Development of a sound
business case based on this best practices model enables decision
makers to be reasonably certain about their products at critical
junctures during development and helps them make informed investment
decisions.
Our best practice work has shown that developing a sound business case
based on matching requirements to resources is essential to
implementing a knowledge-based approach. A sound business case includes
the following elements:
* well-defined requirements,
* preliminary design,
* realistic cost estimates, and:
* mature technology.
A knowledge-based business case also involves the use of controls or
exit criteria to ensure that the required knowledge has been attained
at each critical juncture. It ensures that managers will (1) conduct
activities to capture relevant product development knowledge, (2)
provide evidence that knowledge was captured, and (3) hold decision
reviews to determine that appropriate knowledge was captured to allow a
move to the next phase. If the knowledge attained at each juncture does
not confirm the business case on which the effort was originally
justified, the program does not go forward.
Use of this approach has enabled leading organizations to deliver high
quality products on time and within budget. Product development efforts
that have not followed a knowledge-based business case approach can be
frequently characterized by poor cost, schedule, and performance
outcomes.
Although NASA does not require projects to develop a formal business
case based on matching requirements to resources, JPL project
implementation policy,[Footnote 4] which establishes JPL's
institutional structure for implementation and management of JPL flight
projects in accordance with NASA policies, does require projects to
develop documentation that includes elements essential to a sound
business case. For example, before entering the preliminary design
phase, JPL projects are required to develop preliminary requirements, a
conceptual design, realistic cost estimates, and technology development
plans. JPL projects are required to update and improve the fidelity of
information in these documents by PDR. The information in these
documents could provide NASA decision makers with the information
necessary to support sound business case decisions based on matching
requirements to resources at preliminary mission and systems review
(PMSR) and PDR.
In September 2004, the Congressional Budget Office reported that if
NASA's costs for implementing the strategy were similar to prior
analogous NASA programs--such as Apollo, Viking, and Mars Exploration
Rover--NASA's funding needs could increase by 15 to 23 percent--or $40
billion to $61 billion--over the 16-year estimate. The Congressional
Budget Office concluded that if funding were held constant, NASA would
likely have to either eliminate mission content or delay schedules.
Initial Justification for Prometheus 1 Project Could Form the Basis of
a Sound Business Case:
NASA is still in the process of preparing initial justification for the
Prometheus 1 project to enter the preliminary design phase.
Consequently, at this time the level of funding NASA needs to execute
the project is not fully defined. According to project officials,
however, funding levels would need to be increased to support the
planned launch of Prometheus 1 to Jupiter's Icy Moons. While NASA plans
to have defined preliminary system requirements and an initial estimate
of the life-cycle cost for Prometheus 1 by summer 2005--when the
project enters the preliminary design phase--the agency faces
significant challenges in doing so.
Project Management Believes Current Funding Is Insufficient to Support
Launch of Prometheus 1:
According to Prometheus 1 project management, current funding is
inadequate to support a 2015 launch of Prometheus 1 as initially
planned. Following small funding increases from fiscal years 2005
through 2007, the budget profile[Footnote 5] becomes relatively flat
through fiscal year 2009 (see fig. 2). Project officials believe that
the current profile would need to be increased beginning in fiscal year
2007 to reflect project needs of a Jupiter Icy Moons mission. Decision
makers will not get their first comprehensive picture of the project's
requirements and the resources needed to meet those requirements--the
first basis for funding decisions--until PMSR scheduled for summer
2005. While the fiscal year 2006 request includes an updated Prometheus
1 funding profile, a funding profile based on life-cycle cost
estimates--which NASA plans to have when it enters the preliminary
design phase--will not be included until NASA's fiscal year 2007
request.
Figure 2: NASA Prometheus 1 Fiscal Year 2005 Budget Request Profile:
[See PDF for image]
[End of figure]
Short Time Frames and History Foretell Difficulty in Defining
Requirements and Developing Life-Cycle Cost Estimates:
The Prometheus 1 project office is required to develop preliminary
requirements by PMSR. Defining the project's requirements and
developing life-cycle cost estimates by then could be challenging,
given the short time frames and NASA's past difficulties developing
requirements and estimates. While it is not unusual for a project at
this stage in acquisition to still be defining requirements, several
factors could make it difficult for NASA to develop preliminary
requirements by PMSR. The contractor, Northrop Grumman, was only
recently selected, and according to project officials, input from both
the contractor and Office of Naval Reactors is needed to finalize the
preliminary ground, space, and launch systems requirements mandatory
for PMSR. In addition, NASA continues to refine its requirements. For
example, Prometheus 1 project management increased requirements for
reactor lifetime, reactor power, and propellant tank capacity to ensure
that the Prometheus 1 spacecraft and reactor designs could be used to
support follow-on missions. Currently, project managers are working
with broad NASA requirements for deep space exploration and more
refined project requirements specific to the Prometheus 1 ground,
space, and launch systems.
NASA is also required to have an initial life-cycle cost estimate for
Prometheus 1 at PMSR.[Footnote 6] However, because the estimate is
based on a conceptual design, preliminary system requirements, and
detailed technology development plans that are not yet complete, it
will be difficult for NASA to develop an estimate in the short time
available by PMSR.
The project office is working with Northrop Grumman to merge and
finalize the conceptual design. Once the conceptual design is
finalized, the project office will update the work breakdown
structure[Footnote 7] and develop a "grass roots" estimate of the
spacecraft cost. However, project officials do not expect to receive
cost estimates from the Office of Naval Reactors and Northrop Grumman,
which are also needed to develop the estimate, until the end of
February 2005. The JPL Costing Office will prepare a separate cost
estimate based on its experiences with prior programs, and both JPL and
NASA will contract for additional independent cost estimates.
Adding to these complexities, NASA has historically had difficulty
establishing life-cycle cost estimates. In May 2004, we reported that
NASA's basic cost-estimating processes--an important tool for managing
programs--lack the discipline needed to ensure that program estimates
are reasonable.[Footnote 8] Specifically, we found that 10 NASA
programs that we reviewed in detail did not meet all of our cost-
estimating criteria--based on criteria developed by Carnegie Mellon
University's Software Engineering Institute. Moreover, none of the 10
programs fully met certain key criteria--including clearly defining the
program's life cycle to establish program commitment and manage program
costs, as required by NASA. In addition, only three programs provided a
breakdown of the work to be performed. Without this knowledge, we
reported that the programs' estimated costs may be understated and
thereby subject to underfunding and cost overruns, putting programs at
risk of being reduced in scope or requiring additional funding to meet
their objectives. In this report we recommended that NASA take a number
of actions to improve its cost -estimating practices. NASA concurred
noting that our recommendations validated and reinforced the importance
of activities underway at NASA.
Business Case Allows Match of Needs to Resources and Facilitates
Informed Decision Making:
By PDR--which occurs at end of the preliminary design phase and is
scheduled for 2008--the fidelity of the information is expected to
improve and could allow NASA to develop a business case that would
match requirements with resources and provide decision makers with the
information needed to determine whether continued investment in the
project is warranted. However, in the past NASA has had difficulties
developing the realistic requirements and cost estimates needed to
develop a sound business case.
To help ensure program requirements do not outstrip resources, leading
commercial firms obtain the right knowledge about a new product's
technology, design, and production at the right time. We have issued a
series of reports[Footnote 9] on the success these firms have had in
estimating the time and money to develop new and more sophisticated
products--the kinds of results that NASA seeks. Our best practice work
has shown that developing business cases based on matching requirements
to resources before program start leads to more predictable program
outcomes--that is, programs are more likely to be successfully
completed within cost and schedule estimates and deliver anticipated
system performance.
Figure 3: Prometheus 1 Milestone Reviews:
[See PDF for image]
[End of figure]
A sound business case includes the following elements--well-defined
requirements, a preliminary design, realistic cost estimates, and
mature technology. While NASA does not require projects to develop a
formal business case based on matching requirements to resources, JPL
policy, which implements NASA policy, does require projects to develop
documentation that could support formulation of a sound business case.
Before a JPL project enters the preliminary design phase, JPL project
implementation policy requires that the project develop preliminary
requirements, a conceptual design, realistic cost estimates, and
technology development plans. This policy also requires that the
fidelity of information in these documents improve by PDR.
The requirements and resource estimates NASA is developing for PMSR
could form the basis for an initial business case based on matching
Prometheus 1 requirements to available resources. However, Prometheus 1
project management plans to continue directing requirements changes to
accommodate follow-on missions. While our work shows that the
preliminary design phase is the appropriate place to conduct systems
engineering to support requirement/cost trade-off decisions, NASA needs
to remain cognizant that adding requirements could increase cost and
risk. In addition, NASA has had past difficulty developing the
realistic requirements and cost estimates needed to develop a sound
business case. These difficulties have resulted in the termination of
several major efforts after significant investment of resources. For
example, in 2002 NASA terminated the Space Launch Initiative (SLI)
program--a $4.8 billion, 5-year program to build a new generation of
space vehicles to replace its aging space shuttle. SLI was a complex
and challenging endeavor for NASA, both technically and from a business
standpoint. The SLI program faced some of the same challenges that
Prometheus 1 is struggling with today, such as the need to develop and
advance new airframe and propulsion technologies. SLI did not achieve
its goals, in part, because NASA did not develop realistic requirements
and cost estimates.
NASA's Plans to Ensure Mature Technologies Rely on Prototype
Demonstrations:
Leading firms make an important distinction between technology
development and product development. Technologies that are not mature
continue to be developed in the technology base--they are not included
in a product development. Our best practices work has also shown that
there is a direct relationship between the maturity of technologies and
the accuracy of cost and schedule estimates. NASA's Prometheus 1
technologies are currently immature. The Prometheus 1 project office is
preparing technology development plans to guide the development of each
key technology during the preliminary design phase.
Mature Technologies Are Key to Minimizing Risk:
Maturing technologies during the preliminary design phase is a key
element of matching needs to resources before entering the product
development phase. Our best practices work has shown that technology
readiness levels[Footnote 10] (TRL)--a concept developed by NASA--can
be used to gauge the maturity of individual technologies (see fig.
4).[Footnote 11] (See app. I for detailed definition of TRLs.)
Specifically, TRL 6--demonstrating a technology as a fully integrated
prototype in a realistic environment--is the level of maturity needed
to minimize risks for space systems entering product development.
Figure 4: Technology Maturity Levels for Product Development:
[See PDF for image]
[End of figure]
While development of Prometheus 1 critical technologies is under way,
the technologies will require extensive advancement before they are
mature enough to provide the revolutionary capabilities of the
Prometheus 1 spacecraft. The overall technology objective for
Prometheus 1 is to safely develop and operate a spacecraft with a
nuclear-reactor-powered electric propulsion system. To achieve this
objective, the spacecraft will require advancement in several
technology areas, including, nuclear electric power, power conversion
and heat rejection systems, nuclear electric propulsion, high power
communications, radiation-hardened electronics, and AR&D. (See app. II
for a more detailed explanation of these technologies.) NASA's fiscal
year 2005 budget request indicates that these technologies are either
at TRL 3 (individual technologies have been demonstrated in a
laboratory environment) or TRL 4 (system components have been
demonstrated in a laboratory environment). Before NASA conducts the PDR
in 2008, it will need to mature the technologies--each of which comes
with a unique set of engineering challenges.
To gauge the maturation of the Prometheus 1 technologies, the
Prometheus 1 project office is preparing technology development plans,
which rely on the use of maturity criteria tables (MCT), a concept
similar to TRLs.[Footnote 12] The specific maturation criteria for each
technology vary greatly, but all technologies are to be matured by PDR
to the point that:
* developmental models are complete,
* all major risks to each technology are retired,
* all major manufacturing issues are resolved, and:
* plans for obtaining life data that will provide confidence that the
hardware will meet the mission lifetime requirements are in hand.
Prometheus 1 project officials believe these criteria roughly
correspond to a TRL 5 (component and/or breadboard validation in a
relevant environment) or a TRL 6 (system/subsystem model or prototype
demonstration in a relevant environment). The program office's position
is that using MCTs that are equivalent to TRL 5 and TRL 6 at PDR is
appropriate because the program office is both the technology developer
and product developer and, as such, has a thorough understanding of how
mature the technologies need to be at certain points in time as the
program progresses. Nevertheless, the dual role of project office as
both technology and product developer is not unique, and our best
practices body of work shows that a TRL 6 is the level of maturity
needed to minimize risks for space systems entering product
development.
Conclusions:
NASA is quickly approaching one of the most critical phases in its
acquisition of Prometheus 1--the preliminary design phase. While the
impetus for the changes made to the program--subsequent to our
providing a draft of this report to NASA for comment--recognize the
technical, schedule, and operational risk of this program, there is
still much work to be done. Based on the information presented at PMSR,
now scheduled for summer 2005, NASA will need to decide at what level
to fund the project. However, NASA will be challenged to develop the
information required at PMSR, given the compressed time frames.
Although PDR is still several years out, NASA will face significant
challenges in meeting this milestone, given the immaturity of the
revolutionary technologies that NASA anticipates will be needed to
successfully launch Prometheus 1. While NASA is developing well-defined
criteria tables for maturing Prometheus 1 technologies, the many
inherent unknowns in developing technologies frequently results in
unanticipated difficulties and delays. NASA's current policy does not
require projects to develop knowledge-based business cases that match
requirements to available resources and include controls to ensure that
sufficient knowledge has been attained and therefore the agency had not
planned to develop such a business case for Prometheus 1. We have
found, however, that establishing a formal business case based on a
knowledge-based approach that includes matching requirements and
available resources--which include technical and engineering knowledge,
time and funding--and controls to ensure that sufficient knowledge has
been attained at critical junctures within the product development
process is an essential part of any product development justification.
The risk associated with failing to meet these challenges is
considerable. If NASA decides to move forward without adequate
information at PMSR--that matches requirements and available resources
and provides NASA decision makers with a clear understanding of
Prometheus 1's potential return on investment--Prometheus 1 may be
unable to compete for funding within NASA. Ultimately, NASA could find,
as it has in the past, that the program must be cancelled after having
invested millions of dollars.
Recommendations for Executive Action:
We recommend that the NASA Administrator take the following two
actions:
* identify at PMSR the level of resources the agency is committing to
the project and direct project officials to develop project
requirements based on this resource constraint and:
* ensure that prior to proceeding beyond PDR (currently planned for
2008) a sound business case is established which includes confirmation
that (1) critical technologies have been successfully demonstrated as
mature, (2) systems engineering has been conducted to support
requirements/cost trade-off decisions, (3) requirements and resource
estimates have been updated based on the results of the preliminary
design phase, (4) knowledge based criteria are established at each
critical juncture to ensure that relevant product development knowledge
is captured, and (5) decision reviews are held to determine that
appropriate knowledge was captured to allow a move to the next phase.
Agency Comments:
In written comments on a draft of this report, NASA's Deputy
Administrator stated that the agency concurs with the recommendations,
adding that the recommendations validate and reinforce the importance
of activities underway at NASA to improve NASA's management of complex
technical programs.
Subsequent to our draft report being provided to NASA for comment,
significant changes were made to the Prometheus 1 project. NASA's
fiscal year 2006 budget request includes changes to the Prometheus 1
project that directly address the recommendations in this report.
According to NASA's budget justification, the agency is planning a less
complex mission than the original JIMO mission. According to program
officials who we consulted with following the release of the budget,
eliminating the long reactor lifetime, stringent radiation hardening,
multiple launches, and AR&D required for the JIMO mission will allow
NASA "to walk before it runs" and significantly reduce cost and
technical risks. As a result, NASA has delayed PMSR until summer 2005
and is conducting an analysis of alternatives to identify a relevant
mission with reduced technical, schedule, and operational risk. The
fiscal year 2006 budget request also reshapes the Prometheus 1 funding
profile to provide an orderly increase in developmental activities.
Notwithstanding agreement with our recommendations, the Deputy
Administrator stated that NPR 7120.5B requires projects to develop a
business case. As we noted in this draft report, we recognize that NASA
policy requires the development of elements that could support the
formulation of a knowledge-based business case. However, we found no
explicit requirement within NPR 7120.5B for NASA projects to develop a
business case of any kind. More importantly, while NPR 7120.5B does
require that projects establish controls to monitor performance against
cost, schedule, and performance baselines and to conduct reviews
throughout the project's lifecycle, it does not establish specific
knowledge-based controls to ensure that the knowledge necessary to
match resources to requirements is in hand before moving forward. For
example, whereas NPR 7120.5B requires projects to conduct a preliminary
design review before entering NASA's implementation phase, i.e.,
product development, it does not establish knowledge-based criteria to
ensure that technologies needed to meet essential product requirements
have been demonstrated to work in a realistic environment. Likewise,
NASA policy requires a critical design review during a project's
implementation phase but does not include knowledge-based criteria to
ensure the design is stable. We have found that such knowledge-based
criteria, when tied to major events on a program's schedule, can
disclose whether gaps or shortfalls exist in demonstrated knowledge,
which can presage future cost, schedule and performance problems.
In his comments, the Deputy Administrator also noted that the
Exploration Systems Mission Directorate is in the process of initiating
a number of reforms to its project management policies and specified
formulation dates in the coming months. He outlined these reforms and
explained how they will allow NASA to address the recommendations in
our report. We are encouraged by these planned changes. If properly
implemented, they could be positive steps toward implementing a
knowledge-based approach to project management.
The Deputy Administrator also requested that the relationship between
JPL and NASA project management requirements be explicitly stated in
the report. We moved the information from a footnote into the body of
the report to clarify that relationship. We also addressed NASA's
technical comments as appropriate throughout the report.
Scope and Methodology:
To determine whether NASA is establishing justification for the project
and ensuring critical technologies are mature, we conducted interviews
with NASA Exploration Systems Mission Directorate and Prometheus 1
project officials at NASA Headquarters, Washington, D.C; Marshall Space
Flight Center, Huntsville, Ala; and the Jet Propulsion Laboratory,
Pasadena, Calif. We obtained and reviewed pertinent documents from the
agency. We conducted quantitative and qualitative analyses of project
schedules, risk assessments, budget documentation, technology maturity
assessments and technology development plans. We compared these
documents to criteria established in JPL and NASA policies governing
developmental programs and to criteria for a knowledge based approach
to acquisition described in GAO's best practices body of work. We
discussed key project challenges with Prometheus 1 project officials,
and conducted GAO team meetings to discuss analyses and developing
issues. Our audit work was completed between April 2004 and January
2005.
As agreed with your office, unless you announce its contents earlier,
we will not distribute this report further until 30 days from its
issuance date. At that time, we will send copies to the NASA
Administrator and interested congressional committees. We will make
copies available to others upon request. In addition, the report will
be available at no charge on the GAO web site at http://www.gao.gov.
If you or your staff have any questions concerning this report, please
contact me at (202) 512-4841 or lia@gao.gov. Key contributors to this
report are acknowledged in appendix IV.
Signed by:
Allen Li:
Director:
Acquisition and Sourcing Management:
[End of section]
Appendix I: Technology Readiness Levels Definitions:
Technology readiness level: 1. Basic principles observed and reported;
Description: Lowest level of technology readiness. Scientific research
begins to be translated into applied research and development. Examples
might include paper studies of a technology's basic properties;
Hardware and software: None (Paper studies and analysis);
Demonstration environment: None.
Technology readiness level: 2. Technology concept and/or application
formulated;
Description: Invention begins. Once basic principles are observed,
practical applications can be invented. The application is speculative
and there is no proof or detailed analysis to support the assumption.
Examples are still limited to paper studies;
Hardware and software: None (Paper studies and analysis);
Demonstration environment: None.
Technology readiness level: 3. Analytical and experimental critical
function and/or characteristic proof of concept;
Description: Active research and development is initiated. This
includes analytical studies and laboratory studies to physically
validate analytical predictions of separate elements of the technology.
Examples include components that are not yet integrated or
representative;
Hardware and software: Analytical studies and demonstration of nonscale
individual components (pieces of subsystem);
Demonstration environment: Lab.
Technology readiness level: 4. Component and/or breadboard. Validation
in laboratory environment;
Description: Basic technological components are integrated to establish
that the pieces will work together. This is relatively "low fidelity"
compared to the eventual system. Examples include integration of "ad
hoc" hardware in a laboratory;
Hardware and software: Low fidelity breadboard. Integration of nonscale
components to show pieces will work together. Not fully functional or
form or fit but representative of technically feasible approach
suitable for flight articles;
Demonstration environment: Lab.
Technology readiness level: 5. Component and/or breadboard validation
in relevant environment;
Description: Fidelity of breadboard technology increases significantly.
The basic technological components are integrated with reasonably
realistic supporting elements so that the technology can be tested in a
simulated environment. Examples include "high fidelity" laboratory
integration of components;
Hardware and software: High fidelity breadboard. Functionally
equivalent but not necessarily form and/or fit (size weight, materials,
etc.). Should be approaching appropriate scale. May include integration
of several components with reasonably realistic support elements/
subsystems to demonstrate functionality;
Demonstration environment: Lab demonstrating functionality but not form
and fit. May include flight demonstrating breadboard in surrogate
aircraft. Technology ready for detailed design studies.
Technology readiness level: 6. System/subsystem model or prototype
demonstration in a relevant environment;
Description: Representative model or prototype system, which is well
beyond the breadboard tested for TRL 5, is tested in a relevant
environment. Represents a major step up in a technology's demonstrated
readiness. Examples include testing a prototype in a high fidelity
laboratory environment or in simulated operational environment;
Hardware and software: Prototype--Should be very close to form, fit and
function. Probably includes the integration of many new components and
realistic supporting elements/subsystems if needed to demonstrate full
functionality of the subsystem;
Demonstration environment: High-fidelity lab demonstration or limited/
restricted flight demonstration for a relevant environment. Integration
of technology is well defined.
Technology readiness level: 7. System prototype demonstration in an
operational environment;
Description: Prototype near or at planned operational system.
Represents a major step up from TRL 6, requiring the demonstration of
an actual system prototype in an operational environment, such as in an
aircraft, vehicle or space. Examples include testing the prototype in a
test bed aircraft;
Hardware and software: Prototype. Should be form, fit and function
integrated with other key supporting elements/subsystems to demonstrate
full functionality of subsystem;
Demonstration environment: Flight demonstration in representative
operational environment such as flying test bed or demonstrator
aircraft. Technology is well substantiated with test data.
Technology readiness level: 8. Actual system completed and "flight
qualified" through test and demonstration;
Description: Technology has been proven to work in its final form and
under expected conditions. In almost all cases, this TRL represents the
end of true system development. Examples include developmental test and
evaluation of the system in its intended weapon system to determine if
it meets design specifications;
Hardware and software: Flight qualified hardware;
Demonstration environment: DT&E in the actual system application.
Technology readiness level: 9. Actual system "flight proven" through
successful mission operations;
Description: Actual application of the technology in its final form and
under mission conditions, such as those encountered in operational test
and evaluation. In almost all cases, this is the end of the last "bug
fixing" aspects of true system development. Examples include using the
system under operational mission conditions;
Hardware and software: Actual system in final form;
Demonstration environment: OT&E in operational mission conditions.
Source: GAO.
[End of table]
[End of section]
Appendix II: Prometheus 1 Critical Technologies:
Nuclear Reactor:
The nuclear reactor is the key element of the Prometheus 1 spacecraft.
Without the power levels supplied by the reactor, the proposed
propulsion, science, and communication systems are not feasible.
Designing, constructing, and utilizing highly reliable, safe, portable
nuclear reactors is not new--nuclear reactors have been used in
submarines for almost 50 years. However, the United States has very
little experience operating nuclear reactors in a space environment and
tackling space unique nuclear application issues. The Office of Naval
Reactors, the organizational unit in the Department of Energy
responsible for developing nuclear reactors for the Navy, will be
responsible for all portions of the Prometheus 1 reactor development
effort.
The space environment places significant weight constraints on the
reactor design and requires semi-autonomous control. Unlike submarines
and aircraft carriers, all spacecraft have serious weight constraints
driven by the cost of launching payloads into orbit. Consequently,
spacecraft designers put great effort into eliminating weight. Further,
where conventional reactors have hands on operators, the Prometheus 1
reactor must be remotely controlled. NASA estimates that control
communications will take about 40 minutes to travel one way between
Earth and the Jovian system.
Power Conversion:
A power conversion system accepts the thermal energy from the reactor
and converts it to useful electrical power for the spacecraft. Power
conversion is an integral part of any power generation system taking
the form of steam turbine generators in terrestrial utility plants and
nuclear submarines. NASA is considering two types of power conversion
systems--dynamic and static. According to NASA, the dynamic systems
under consideration offer the benefits of increased efficiency, reduced
weight and mass, and decreased nuclear fuel requirements. The static
systems, however, have a technology heritage in prior spacecraft and
could offer increased reliability because they have no moving parts.
Heat Rejection:
Since the conversion process in a fission reactor is never 100 percent
efficient, heat rejection is required to dissipate waste energy. This
is usually accomplished with large pumped-water cooling systems on
earth. Space based power conversion would require a large radiator
system to dissipate the waste heat in the vacuum of space. The
requirement to fold the large radiator system into the launch vehicle
fairing and deploy it after launch complicates the radiator system
design. (See fig. 1.)
Nuclear Electric Propulsion:
Operating electric propulsion systems in space applications, including
deep space, is not new. There is extensive experience with electric
propulsion systems on satellites. In addition, NASA's Deep Space 1
spacecraft was propelled using an electric propulsion ion thruster,
similar in nature to the concept being developed for Prometheus 1. The
thruster power levels required by Prometheus 1 have been demonstrated
in a laboratory environment. The lifetime required by Prometheus 1,
however, has not been demonstrated. Furthermore, lifetime testing of
existing ion thrusters has demonstrated that these thrusters were
approaching "wear out failure"[Footnote 13] after 30,352 hours. The
Prometheus 1 thrusters will need to be qualified for operational
durations approaching 120,000 hours. NASA recognizes that they will
have to develop models and accelerated aging techniques to demonstrate
the lifetime requirement.
Radiation Hardening:
All electronic components of the Prometheus 1 spacecraft must be
radiation hardened or shielded from radiation produced by the onboard
nuclear reactor and the harsh radiation environment of the Jovian
system. Shielding electronics and the science packages, requires a
heavy metal or some other material, which may not yet be developed, and
increases the weight and mass of the spacecraft. The additional weight
used to shield the spacecraft lessens the science payload package that
can be taken onboard for scientific data collection and research and
increases the size and power of the launch vehicle required to launch
Prometheus 1 into earth orbit:
High Power Communications:
The nuclear reactor will provide increased electrical power for
communications. This translates to increased bandwidth and data rates.
The high power communications system onboard the Prometheus 1
spacecraft, will provide tens of compact disks full of data back to
earth. Analogous missions such as Cassini provide only a couple of
floppy disks full of data. (A floppy disk typically holds about 1.44 MB
of data. A compact disk typically holds about 700 MB of data.)
According to project officials, the higher power communications system
on the Prometheus 1 spacecraft will require upgrades to the Deep Space
Network, which are out of the purview of the Prometheus 1 project.
Autonomous Rendezvous and Docking:
There is no launch vehicle in the present or proposed U.S. inventory
capable of launching the Prometheus 1 spacecraft, conceived for a
mission to Jupiter's Icy Moons, into orbit in one piece. The conceptual
design currently shows the Prometheus 1 spacecraft to weigh between 29
and 36 metric tons and be about 58 meters in length. The current
concept is to use multiple launches, 2 to 5, to place the spacecraft
components in orbit and to use AR&D technology to assemble the
spacecraft in orbit. Prometheus 1 is relying on NASA's Demonstration
Autonomous Rendezvous Technology and Hubble Robotic Servicing Mission,
and the Defense Advanced Research Projects Agency's Orbital Express
programs for AR&D technology. These programs use different sensors and
approaches to AR&D thereby providing Prometheus 1 with various options
for consideration.
[End of section]
Appendix III: Comments from the National Aeronautics and Space
Administration:
National Aeronautics and Space Administration:
Office of the Administrator:
Washington, DC 20546-0001:
February 15, 2005:
Mr. Allen Li:
Director, Acquisition and Sourcing Management Team:
United States General Accounting Office:
Washington, DC 20548:
Dear Mr. Li:
The National Aeronautics and Space Administration (NASA) appreciates
the opportunity to review and comment on your draft report General
Accounting Office (GAO)-05-242, "NASA's Space Vision: Business Case for
Prometheus I Needed to Ensure Requirements Match Available Resources."
We concur with your recommendations and note that they validate and
reinforce the importance of activities currently underway at NASA to
strengthen our management of complex technical programs, including the
development of Prometheus 1 and its attendant supporting infrastructure
and technologies. However, we would like to clarify the status of our
program management practices and policies.
The draft report states that the Jet Propulsion Laboratory's (JPL)
project implementation policy requires documentation that includes
elements essential to developing a sound business case. However, GAO
states that NASA does not require projects to develop knowledge-based
business cases that match requirements to available resources. The
report also suggests that NASA does not employ controls to ensure that
sufficient knowledge has been attained during project development.
As NASA's Federally Funded Research and Development Center (FFRDC), JPL
programs and practices are required to conform to NASA requirements.
The NASA requirements for program/project management are documented in
NASA's "Program and Project Management Processes and Requirements"
document (NASA Procedural Requirements (NPR) 7120.513, dated November
2002). As stated in NPR 7120.513, NASA requires development of a
business case for all of its projects, and JPL has instituted polices
and procedures to conform to the NASA requirements. The draft report
text does not explicitly acknowledge the fact that JPL's "Project
Implementation Policy" is written to conform to these NASA
requirements. NASA believes that for the sake of clarity, the
relationship between JPL and NASA program/project management
requirements should be explicitly stated in the report. We would be
happy to discuss this issue further with you or provide you with a copy
of the NPR. The NPR is also available online at
http://www.hq.nasa.gov/office/codeq/doctree/71205.htm.
As you noted, NASA is in the process of conducting conceptual design
(Phase A) for the Prometheus 1 project in anticipation of entering
preliminary design (Phase B). Concurrent with these activities, NASA's
Exploration Systems Mission Directorate (ESMD) is developing detailed
program management requirements on why, when, and how to implement NASA
programs and projects. These requirements consist of a three document
suite based on the Department of Defense (DoD) and NASA best practices
cited in your draft report, i.e., well-defined requirements, a
conceptual design, realistic cost estimates, and independently reviewed
technology development plans. This suite of documents, which also
considers lessons learned at NASA, will succinctly define the policies
and requirements that all ESMD program/project managers must use in
overseeing NASA's exploration activities. ESMD's document suite
consists of the Single Acquisition Management Plan (SAMP), the Systems
Engineering Management Plan (SEMP), and the Program Management Handbook
(PMH).
The SAMP documents the ESMD overarching acquisition strategy.
Recognizing that each program has unique challenges, the SAMP discusses
application of the ESMD acquisition strategy for each major program
under the ESMD's purview. The SAMP explicitly addresses the use of
spiral development and implementation of the "Strategy to Task to
Technology" (STT) approach for program management. The STT uses an
integrated team of stakeholders and an Operational Advisory Group (OAG)
to identify and assess user-defined operational needs and priorities.
This process is designed to implement the best practices identified in
GAO-04-386SP by creating a credible, auditable trail of the ESMD
decision-making process.
The SEMP identifies the sequence of systems engineering processes and
activities for the technical aspects of ESMD systems engineering and
integration management. The intent of the SEMP is to combine DoD and
NASA best practices in a way which supports NASA's unique mission and
which defines a rigorous process for determining whether proposed
solutions meet ESMD requirements.
The PMH defines the organizational roles, responsibilities, and
processes required for program management. Processes are defined in
terms of program life cycles and are tailored to the characteristics of
basic and applied research, technology development, and mission system
development. Program life. cycles are punctuated by Key Decision Points
for transitioning from one program phase to the next. Throughout the
PMH, program managers are encouraged to be candid and forthcoming about
program status, explicitly including identifying risks and problems
without fear of personal consequences.
The ESMD management is committed to providing the best resources and
tools available to its staff and consistently stresses the need for
credibility, affordability, and sustainability of its programs. The
NASA Associate Administrator for Exploration Systems, Craig E. Steidle,
has personally visited each NASA Center and JPL to explain the upcoming
program management practices. He emphasizes that NASA must be prepared
to demonstrate a positive return on our resource investments. ESMD
goals are to:
1. Convene a diverse group of stakeholders, including designers,
operations staff, and technicians to evaluate and identify performance
trade-offs early in the program.
2. Apply available technologies to reduce system life-cycle costs,
specifically infrastructure development.
3. Mature a technology before employing it.
4. Partner with industry as appropriate.
As a civilian Government agency, NASA's mission is to expand human
knowledge of the Earth and phenomena in the atmosphere and space.
Consequently, many of NASA's exploration projects require first-of-a-
kind technology development. Nevertheless, NASA must ensure that
available resources, both human and otherwise, are effectively and
efficiently applied in our technology development activities.
The following paragraphs provide the current status and planned
approach for addressing each of the GAO recommendations:
Recommendation 1: Identify at Preliminary Mission and Systems Review
(PMSR) the level of resources the agency is committing to the project
and direct project officials to develop project requirements based on
this resource constraint.
NASA concurs with this recommendation and is documenting the ESMD
process for transitioning from Phase A to Phase B for all programs
under ESMD's purview. JPL's PMSR is an in-depth review of the project
conducted shortly before the NASA program review. The project must pass
both reviews in order for the project to move forward. The ESMD's PMH
requires that program managers define the mission and system concepts,
parameters, constraints, and requirements that will enable development
of the system and mission on a given schedule to meet established goals
within a reasonable cost estimate. Cost estimates are commensurate with
the available information at that point in project development. As the
project matures and the design progresses, project cost estimates
become more refined. Program managers will periodically review program
activities against resource constraints, using earned value management
techniques, and report actual or anticipated deviations from program
costs and/or schedule to senior management for resolution.
Stakeholders, including operations, science, engineering, and the OAG
staff will review mission objectives, technology requirements, and
resource constraints throughout the project life cycle to ensure that
requirements can be met within defined resource and technology
constraints. Before transitioning to Phase B, ESMD also requires an
independent review by an Independent Program Assessment Team to
determine whether or not the program is ready to proceed.
The ESMD's PMH will be issued in February 2005, and the SAMP and SEMP
will follow thereafter in the summer of 2005.
Recommendation 2: Ensure that prior to proceeding beyond Preliminary
Design Review (PDR) (currently planned for 2008) a sound business case
is established which includes confirmation that (1) critical
technologies have been successfully demonstrated as mature, (2) systems
engineering has been conducted to support requirement/cost trade-off
decisions, (3) requirements and resource estimates have been updated
based on the results of the preliminary design phase, (4) knowledge
based criteria are established at each critical juncture to ensure that
relevant product development knowledge is captured, and (5) decision
reviews are held to determine that appropriate knowledge was captured
to allow a move to the next phase.
The ESMD will be issuing the PMH in February 2005. The PMH will
explicitly define the requirements for transitioning between each phase
of a program/project. The PMH defines specific entrance and exit
criteria for each program phase and requires periodic reassessment of
technology maturity, systems engineering interfaces and requirements,
independent cost and technical analysis, value engineering, updating
requirements and resource estimates, and analysis of alternatives to
determine whether a given program/project should proceed. Earned value
management techniques will be used. The PMH also requires documentation
of decision bases to ensure that the STT process is credible and
auditable. The three-document suite will be revised to incorporate
lessons learned as NASA gains experience in implementing this new
program management acquisition strategy to ensure that we continue to
improve our processes.
We also note that both the report summary page and the report itself
cite the Congressional Budget Office's (CBO) project cost estimate as
$10 billion dollars. The CBO estimate was not coordinated with NASA.
NASA is currently developing a disciplined process to develop NASA's
initial project cost estimate. We are also providing technical comments
under separate cover.
Thank you for the opportunity to comment on the draft report. We
appreciate your recommendations and will continue to work on improving
how we do business.
Cordially,
Signed by:
Frederick D. Gregory:
Deputy Administrator:
Enclosure:
[End of section]
Appendix IV: GAO Contact and Staff Acknowledgments:
GAO Contact:
Allen Li, (202) 512-4841:
Acknowledgments:
In addition to the contact named above, James Morrison, Jerry Herley,
John Warren, Tom Gordon, Ruthie Williamson, Karen Sloan and Sylvia
Schatz made key contributions to this report.
[End of section]
GAO Related Products:
Best Practices: Using a Knowledge-Based Approach to Improve Weapon
Acquisition, GAO-04-386SP. Washington, D.C.: January 2004.
Best Practices: Setting Requirements Differently Could Reduce Weapon
Systems' Total Ownership Costs. GAO-03-57. Washington, D.C.: February
11, 2003.
Defense Acquisitions: DOD's Revised Policy Emphasizes Best Practices,
but More Controls Are Needed. GAO-04-53. Washington, D.C.: November 10,
2003.
Best Practices: Capturing Design and Manufacturing Knowledge Early
Improves Acquisition Outcomes. GAO-02-701. Washington, D.C.: July 15,
2002.
Defense Acquisitions: DOD Faces Challenges in Implementing Best
Practices. GAO-02-469T. Washington, D.C.: February 27, 2002.
Best Practices: Better Matching of Needs and Resources Will Lead to
Better Weapon System Outcomes. GAO-01-288. Washington, D.C.: March 8,
2001.
Best Practices: A More Constructive Test Approach Is Key to Better
Weapon System Outcomes. GAO/NSIAD-00-199. Washington, D.C.: July 31,
2000.
Defense Acquisition: Employing Best Practices Can Shape Better Weapon
System Decisions. GAO/T-NSIAD-00-137. Washington, D.C.: April 26, 2000.
Best Practices: DOD Training Can Do More to Help Weapon System Program
Implement Best Practices. GAO/NSIAD-99-206. Washington, D.C.: August
16, 1999.
Best Practices: Better Management of Technology Development Can Improve
Weapon System Outcomes. GAO/NSIAD-99-162. Washington, D.C.: July 30,
1999.
Defense Acquisitions: Best Commercial Practices Can Improve Program
Outcomes. GAO/T-NSIAD-99-116. Washington, D.C.: March 17, 1999.
Defense Acquisition: Improved Program Outcomes Are Possible. GAO/T-
NSIAD-98-123. Washington, D.C.: March 18, 1998.
Best Practices: DOD Can Help Suppliers Contribute More to Weapon System
Programs. GAO/NSIAD-98-87. Washington, D.C.: March 17, 1998.
Best Practices: Successful Application to Weapon Acquisition Requires
Changes in DOD's Environment. GAO/NSIAD-98-56. Washington, D.C.:
February 24, 1998.
Major Acquisitions: Significant Changes Underway in DOD's Earned Value
Management Process. GAO/NSIAD-97-108. Washington, D.C.: May 5, 1997.
Best Practices: Commercial Quality Assurance Practices Offer
Improvements for DOD. GAO/NSIAD-96-162. Washington, D.C.: August 26,
1996.
[End of section]
(120346):
FOOTNOTES
[1] This estimate is based on conducting the Jupiter Icy Moons Orbiter
Mission. The Congressional Budget Office did not consult with NASA to
develop this estimate.
[2] The Prometheus Nuclear Systems and Technology program stems from
the former Nuclear Systems Initiative to develop nuclear power for deep
space exploration.
[3] Subsequent to being provided a draft of this report for comment,
NASA announced that it was conducting an analysis of alternatives to
identify a new mission.
[4] JPL's Project Implementation Policy establishes JPL's institutional
structure for implementation and management of JPL flight projects in
accordance with NASA Procedural Requirements 7120.5B, NASA Program and
Project Management Processes and Requirements, which governs all NASA
development programs and projects.
[5] Between 10 and 15% of the budget shown is allocated to Advanced
Systems and Technology Development activities supporting current and
future missions.
[6] NASA guidance requires that life-cycle costs be estimated,
assessed, and controlled throughout a program's life cycle. The
estimates are to be prepared to support major program reviews and the
development of budget submissions.
[7] A work breakdown structure is a product oriented division of
hardware, software, services and project unique tasks that organizes
and defines the product to be developed and serves as the basis for
estimating both cost and schedule.
[8] GAO, NASA: Lack Of Disciplined Cost Estimating Processes Hinders
Effective Program Management, GAO-04-642 (Washington, D.C.: May 28,
2004).
[9] GAO, Best Practices: Better Matching of Needs and Resources Will
Lead to Better Weapon System Outcomes, GAO-01-288 (Washington, D.C.:
Mar. 8, 2001).
[10] Technology readiness levels characterize the readiness of
technologies for handoff to project implementers. Nine levels are
defined representing concepts from fundamental research level through
technologies fully qualified and demonstrated in flight.
[11] GAO, Best Practices: Using a Knowledge-Based Approach to Improve
Weapon Acquisition, GAO-04-386SP (Washington D.C.: January 2004).
[12] According to the Prometheus project office, the MCT [system],
[also developed by NASA], is an improvement over the TRL [system]
because MCT definitions are more detailed and quantifiable and,
therefore, provide more clarity to technology developers, managers, and
independent reviewers. Project office officials also note that the
criteria in the tables have been reviewed through an independent peer
assessment to validate that they provide the comprehensive set of
measurements that need to be made to verify that a particular maturity
has been met in a specific technology.
[13] Wear out failure is the point at which a system stops operating
because its mechanical or physical parts are worn to the point they
will no longer function.
GAO's Mission:
The Government Accountability Office, the investigative arm of
Congress, exists to support Congress in meeting its constitutional
responsibilities and to help improve the performance and accountability
of the federal government for the American people. GAO examines the use
of public funds; evaluates federal programs and policies; and provides
analyses, recommendations, and other assistance to help Congress make
informed oversight, policy, and funding decisions. GAO's commitment to
good government is reflected in its core values of accountability,
integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through the Internet. GAO's Web site ( www.gao.gov ) contains
abstracts and full-text files of current reports and testimony and an
expanding archive of older products. The Web site features a search
engine to help you locate documents using key words and phrases. You
can print these documents in their entirety, including charts and other
graphics.
Each day, GAO issues a list of newly released reports, testimony, and
correspondence. GAO posts this list, known as "Today's Reports," on its
Web site daily. The list contains links to the full-text document
files. To have GAO e-mail this list to you every afternoon, go to
www.gao.gov and select "Subscribe to e-mail alerts" under the "Order
GAO Products" heading.
Order by Mail or Phone:
The first copy of each printed report is free. Additional copies are $2
each. A check or money order should be made out to the Superintendent
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or
more copies mailed to a single address are discounted 25 percent.
Orders should be sent to:
U.S. Government Accountability Office
441 G Street NW, Room LM
Washington, D.C. 20548:
To order by Phone:
Voice: (202) 512-6000:
TDD: (202) 512-2537:
Fax: (202) 512-6061:
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Web site: www.gao.gov/fraudnet/fraudnet.htm
E-mail: fraudnet@gao.gov
Automated answering system: (800) 424-5454 or (202) 512-7470:
Public Affairs:
Jeff Nelligan, managing director,
NelliganJ@gao.gov
(202) 512-4800
U.S. Government Accountability Office,
441 G Street NW, Room 7149
Washington, D.C. 20548: