NASA
Implementing a Knowledge-Based Acquisition Framework Could Lead to Better Investment Decisions and Project Outcomes Gao ID: GAO-06-218 December 21, 2005The National Aeronautics and Space Administration (NASA) plans to spend over $100 billion on capabilities and technologies to achieve the initial goals of the President's 2004 Vision for Space Exploration. In the past, NASA has had difficulty meeting cost, schedule, and performance objectives for some of its projects because it failed to adequately define project requirements and quantify resources. NASA will be further challenged by a constrained federal budget and a shrinking experienced NASA workforce. To help face these challenges and manage projects with greater efficiency and accountability, NASA recently updated its program and project management policy and is developing an agencywide systems engineering policy. GAO has issued a series of reports on the importance of obtaining critical information and knowledge at key junctures in major system acquisitions to help meet cost and schedule objectives. This report (1) evaluates whether NASA's policy supports a knowledge-based acquisition approach and (2) describes how NASA centers are implementing the agency's acquisition policies and guidance.
While NASA's revised policy for developing flight systems and ground support projects incorporates some of the best practices used by successful developers, it lacks certain key criteria and major decision reviews that support a knowledge-based acquisition framework. For example, NASA's policy requires projects to conduct a major decision review before moving from formulation to implementation. Further, before moving from formulation to implementation, projects must validate requirements and develop realistic cost and schedule estimates, human capital plans, a preliminary design, and a technology plan--key elements for matching needs to resources. However, NASA's policies do not require projects to demonstrate technologies at high levels of maturity before program start. By not establishing a minimum threshold for technology maturity, NASA increases the risk that design changes will be required later in development, when such changes are typically more costly to make. In addition, although NASA's policy does require project managers to establish a continuum of technical and management reviews, it does not specify what these reviews should be, nor does it require major decision reviews at other key points in a product's development. Acquiring knowledge at key junctures will become increasingly important as NASA proceeds to implement elements of the Vision. Without a major decision review at key milestones to ensure that the appropriate level of knowledge has been achieved to proceed to the next phase, the risk of cost and schedule overruns, as well as performance shortfalls, increases. NASA centers have varying approaches for implementing the agency's policies and guidance. Some centers have established product development criteria that are similar to the criteria used in a knowledge-based acquisition, while other centers have not. As a result, each center reports a different level and type of knowledge about a project at key decision points. Centers also rely on project managers and systems engineers to employ good project management and systems engineering practices. However, given the loss of experienced project managers and the decline of in-house systems engineering and technical capabilities, that reliance could be problematic. These situations make it difficult for decision makers to evaluate projects on the same basis and make sound investment decisions and tradeoffs based on those evaluations. A standardized, knowledge-based approach would prepare NASA to face competing budgetary priorities and better position the agency to make difficult decisions regarding the investment in and termination of projects.
RecommendationsOur recommendations from this work are listed below with a Contact for more information. Status will change from "In process" to "Open," "Closed - implemented," or "Closed - not implemented" based on our follow up work.
Director: Team: Phone:GAO-06-218, NASA: Implementing a Knowledge-Based Acquisition Framework Could Lead to Better Investment Decisions and Project Outcomes
This is the accessible text file for GAO report number GAO-06-218
entitled 'NASA: Implementing a Knowledge-Based Acquisition Framework
Could Lead to Better Investment Decisions and Project Outcomes' which
was released on January 23, 2006.
This text file was formatted by the U.S. Government Accountability
Office (GAO) to be accessible to users with visual impairments, as part
of a longer term project to improve GAO products' accessibility. Every
attempt has been made to maintain the structural and data integrity of
the original printed product. Accessibility features, such as text
descriptions of tables, consecutively numbered footnotes placed at the
end of the file, and the text of agency comment letters, are provided
but may not exactly duplicate the presentation or format of the printed
version. The portable document format (PDF) file is an exact electronic
replica of the printed version. We welcome your feedback. Please E-mail
your comments regarding the contents or accessibility features of this
document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
Report to Congressional Requesters:
United States Government Accountability Office:
GAO:
December 2005:
NASA:
Implementing a Knowledge-Based Acquisition Framework Could Lead to
Better Investment Decisions and Project Outcomes:
GAO-06-218:
GAO Highlights:
Highlights of GAO-06-218, a report to congressional requesters:
Why GAO Did This Study:
The National Aeronautics and Space Administration (NASA) plans to spend
over $100 billion on capabilities and technologies to achieve the
initial goals of the President‘s 2004 Vision for Space Exploration. In
the past, NASA has had difficulty meeting cost, schedule, and
performance objectives for some of its projects because it failed to
adequately define project requirements and quantify resources. NASA
will be further challenged by a constrained federal budget and a
shrinking experienced NASA workforce. To help face these challenges and
manage projects with greater efficiency and accountability, NASA
recently updated its program and project management policy and is
developing an agencywide systems engineering policy.
GAO has issued a series of reports on the importance of obtaining
critical information and knowledge at key junctures in major system
acquisitions to help meet cost and schedule objectives. This report (1)
evaluates whether NASA's policy supports a knowledge-based acquisition
approach and (2) describes how NASA centers are implementing the
agency‘s acquisition policies and guidance.
What GAO Found:
While NASA‘s revised policy for developing flight systems and ground
support projects incorporates some of the best practices used by
successful developers, it lacks certain key criteria and major decision
reviews that support a knowledge-based acquisition framework. For
example, NASA‘s policy requires projects to conduct a major decision
review before moving from formulation to implementation. Further,
before moving from formulation to implementation, projects must
validate requirements and develop realistic cost and schedule
estimates, human capital plans, a preliminary design, and a technology
plan”key elements for matching needs to resources. However, NASA‘s
policies do not require projects to demonstrate technologies at high
levels of maturity before program start. By not establishing a minimum
threshold for technology maturity, NASA increases the risk that design
changes will be required later in development, when such changes are
typically more costly to make. In addition, although NASA‘s policy does
require project managers to establish a continuum of technical and
management reviews, it does not specify what these reviews should be,
nor does it require major decision reviews at other key points in a
product‘s development. Acquiring knowledge at key junctures will become
increasingly important as NASA proceeds to implement elements of the
Vision. Without a major decision review at key milestones to ensure
that the appropriate level of knowledge has been achieved to proceed to
the next phase, the risk of cost and schedule overruns, as well as
performance shortfalls, increases.
NASA centers have varying approaches for implementing the agency‘s
policies and guidance. Some centers have established product
development criteria that are similar to the criteria used in a
knowledge-based acquisition, while other centers have not. As a result,
each center reports a different level and type of knowledge about a
project at key decision points. Centers also rely on project managers
and systems engineers to employ good project management and systems
engineering practices. However, given the loss of experienced project
managers and the decline of in-house systems engineering and technical
capabilities, that reliance could be problematic. These situations make
it difficult for decision makers to evaluate projects on the same basis
and make sound investment decisions and tradeoffs based on those
evaluations. A standardized, knowledge-based approach would prepare
NASA to face competing budgetary priorities and better position the
agency to make difficult decisions regarding the investment in and
termination of projects.
What GAO Recommends:
GAO is making several recommendations to help ensure NASA uses a
knowledge-based acquisition approach in making informed investment
decisions. NASA concurred with GAO‘s recommendations.
www.gao.gov/cgi-bin/getrpt?GAO-06-218.
To view the full product, including the scope and methodology, click on
the link above. For more information, contact Allen Li at (202) 512-
4841 or lia@gao.gov.
[End of section]
Contents:
Letter:
Results in Brief:
Background:
NASA's Revised Policy Does Not Fully Support a Knowledge-Based Approach
to Acquisitions:
Lack of NASA-Wide Project Management Criteria May Result in Investment
Decisions Based on Inconsistent Information:
Conclusions:
Recommendations for Executive Action:
Agency Comments and Our Evaluation:
Appendix I: Scope and Methodology:
Appendix II: Activities That Enable the Capture of Design and
Manufacturing Knowledge:
Appendix III: Comments from the National Aeronautics and Space
Administration:
Appendix IV: GAO Contact and Staff Acknowledgments:
Related GAO Products:
Tables:
Table 1: Examples of Problems With NASA Programs/Projects:
Table 2: KP1 Criteria Compared to NASA Policy Criteria:
Table 3: Activities to Capture Design Knowledge and Make Decisions:
Table 4: Activities to Capture Manufacturing Knowledge and Make
Decisions:
Figures:
Figure 1: Conceptual Drawing of NASA Crew Vehicle Docked with Lunar
Lander and Departure Stage in Earth Orbit:
Figure 2: Project Categorization Scheme and Oversight Authorities from
NPR 7120.5C:
Figure 3: Knowledge-Based Acquisition Life Cycle:
Figure 4: Life Cycle Phases for NASA's Flight Systems and Ground
Support Projects:
Figure 5: Technology Maturity Levels for Product Development:
Figure 6: Comparison of NASA's Life Cycle with a Knowledge-Based
Acquisition Life Cycle:
Abbreviations:
CADRe: Cost analysis data requirement:
CDR: Critical design review:
EVM: Earned value management:
GPMC: Governing Program Management Committee:
GSFC: Goddard Space Flight Center:
ITA: Independent Technical Authority:
JPL: Jet Propulsion Laboratory:
JSC: Johnson Space Center:
KP1: knowledge point 1:
KP2: knowledge point 2:
KP3: knowledge point 3:
MSFC: Marshall Space Flight Center:
NAR: Non-Advocate Review:
NASA: National Aeronautics and Space Administration:
NPD: NASA Policy Directive:
NPR: NASA Procedural Requirement:
PDR: Preliminary design review:
Pre-NAR: Preliminary Non-Advocate Review:
TRL: Technology readiness level:
United States Government Accountability Office:
Washington, DC 20548:
December 21, 2005:
The Honorable Bart Gordon:
Ranking Minority Member:
Committee on Science:
House of Representatives:
The Honorable Mark Udall:
Ranking Minority Member:
Subcommittee on Space and Aeronautics:
Committee on Science:
House of Representatives:
The National Aeronautics and Space Administration (NASA) plans to spend
over $100 billion to develop new capabilities and technologies critical
to supporting the initial goals outlined in the President's 2004 Vision
for Space Exploration (see fig. 1 for a conceptual drawing of NASA's
proposed crew vehicle). The Vision, which NASA has characterized as
bold, includes plans to explore the moon, Mars, and beyond.[Footnote 1]
Despite many successes, such as landing the Pathfinder and Exploration
Rovers on Mars, NASA has had difficulty carrying out a number of other
missions, such as the X-33, a technology demonstrator for future
reusable launch vehicles, because the agency was overly optimistic in
what could be achieved within available resources. NASA's failure to
define requirements adequately and quantify the resources needed to
meet those requirements resulted in some projects costing more, taking
longer, and achieving less than originally planned. In addition to
these project management challenges, a constrained federal budget and a
shrinking experienced project manager and technical workforce, well-
versed in systems engineering and architecture design, will present
further challenges to NASA in the years ahead.
Figure 1: Conceptual Drawing of NASA Crew Vehicle Docked with Lunar
Lander and Departure Stage in Earth Orbit:
[See PDF for image]
[End of figure]
To meet these challenges, NASA recently updated its program and project
management policy[Footnote 2] and is developing an agencywide systems
engineering policy. NASA expects that these new polices will help the
agency manage its projects with greater efficiency, responsibility, and
accountability. We have issued a series of reports on the importance of
obtaining critical information and knowledge at key junctures in major
system acquisitions before additional investments are made. This work
has shown that following a knowledge-based approach helps developers
meet cost and schedule objectives in developing new and more
sophisticated products--the kinds of results that NASA seeks.
Given the major endeavor that NASA is about to undertake, the agency's
history of cost and schedule overruns and technical problems, and the
significant challenges the agency currently faces and will likely face,
you asked us to (1) evaluate whether NASA's policies support an
acquisition approach consistent with best practices identified in GAO's
work on system acquisitions and (2) describe how NASA-wide acquisition
policies are implemented at the various NASA centers.[Footnote 3]
To conduct our work, we reviewed and analyzed NASA-wide program and
project management policies as they relate to flight systems and ground
support projects and systems engineering guidance and compared the
policies and guidance with criteria contained in GAO's best practices
work on systems acquisition, including space systems. We also
interviewed NASA headquarters officials from the Office of the Chief
Engineer responsible for the policy and guidance. In addition, we
reviewed NASA center-specific program and project management policies
and systems engineering policies, as they relate to flight systems and
ground support projects, and interviewed officials responsible for
implementing those policies. We focused primarily on Goddard Space
Flight Center (GSFC), the Jet Propulsion Laboratory (JPL), Johnson
Space Center (JSC), and Marshall Space Flight Center (MSFC), which
manage the majority of NASA's flight systems and ground support
projects. Complete details of our scope and methodology can be found in
appendix I. We performed our work from May 2005 to November 2005 in
accordance with generally accepted government auditing standards.
Results in Brief:
NASA's revised policy for developing flight systems and ground support
projects incorporates some of the best practices used by successful
developers, but the policy lacks certain key criteria and decision
reviews necessary to fully support the knowledge-based acquisition
framework. For example, NASA's policy requires projects to conduct a
major decision review before moving from formulation to implementation.
Further, before moving from formulation to implementation, projects
must validate requirements and develop realistic cost and schedule
estimates, human capital plans, a preliminary design, and a technology
plan--all key elements for matching needs to resources before a
commitment to major investment is made at project start. However,
NASA's policy does not require that projects demonstrate technologies
at high levels of maturity at that point. By not establishing a minimum
threshold for technology maturity, NASA increases the risk that
requirements will not be met and design changes will be required later
in development, when such changes are typically more costly to make. In
addition, although NASA's policy does require project managers to
establish a continuum of technical and management reviews, the policy
does not specify what these reviews should be, nor does it require
major decision reviews at other key points in a product's development.
Acquiring knowledge at key junctures will become increasingly important
as NASA proceeds to implement various elements of the Vision. Without a
major decision review at key milestones to ensure that the appropriate
level of knowledge has been achieved to proceed to the next phase, NASA
increases the risk of cost and schedule overruns as well as performance
shortfalls.
NASA centers have varying approaches to implementing agencywide project
management policy and systems engineering guidance for flight systems
and ground support projects. Some centers use criteria at key decision
points that are similar to the criteria required to ensure a knowledge-
based approach is followed, while others lack such criteria. As a
result, each center reports a different level and type of knowledge
about a project at key decision points. Centers also rely on project
managers and systems engineers to employ good project management and
systems engineering practices. However, given the loss of experienced
project managers and the decline of in-house systems engineering and
technical capabilities agencywide, that reliance could be problematic.
These situations make it difficult for NASA decision makers to evaluate
center projects on a common foundation of knowledge and make sound
investment decisions and tradeoffs based on those evaluations. A
standardized knowledge-based approach would prepare NASA to face
competing budgetary priorities and better position the agency to make
difficult decisions regarding the investment in and termination of
projects.
To ensure NASA uses a knowledge-based acquisition approach in making
informed investment decisions, we are making several recommendations to
improve the agency's policies. We are recommending that NASA (1)
require the capture of specific knowledge to be used as criteria for
allowing projects to enter implementation and proceed through
development and to support informed investment decisions and (2)
institute additional reviews for flight system and ground support
projects during project implementation, which result in recommendations
to the appropriate decision authority. In written comments on a draft
of this report, NASA agreed with our recommendations. NASA's comments
are included in their entirety in appendix III.
Background:
Over the past decade, NASA has experienced significant problems with
several of its projects, which GAO and others have reported on (see
table 1 for some examples of such problems). In addition, in 2002 we
reported on several sources of failures in NASA programs, including
underestimating complexity and technology maturity, and inadequate
review and systems engineering processes. Further, we reported that the
sources of these problems were not new and that NASA failed to
consistently apply lessons previously learned.[Footnote 4] The failures
identified in our 2002 report were in part the result of the "faster,
better, cheaper" approach to managing its major acquisitions, which
NASA adopted in the 1990s.[Footnote 5]
Table 1: Examples of Problems With NASA Programs/Projects:
X-33:
Program/Project description: The X-33 was to be an unmanned technology
demonstrator, which tested a range of technologies needed for future
reusable launch vehicles;
Reported problems: In 1999, GAO reported that technical problems on the
X-33 project led to cost overruns of $75 million dollars and over a
year's delay in schedule. These problems resulted from NASA developing
unrealistic cost estimates in the early stages of the X-33 project and
not following its own policies with regard to project and risk
management. In February 2001, NASA announced it would provide no
additional funding for the X-33 program.
Space Launch Initiative (SLI):
Program/Project description: Approved in February 2001, SLI was
designed as a $4.8 billion, 6-year effort to fly a half-scale flight
demonstrator intended to validate advanced technologies that would
dramatically reduce the cost of putting a pound of payload into space-
-from $10,000 to $1,000;
Reported problems: In 2002, both the NASA Inspector General and GAO
released reports assessing SLI with particular emphasis on requirement
definition and implementation of management controls; The NASA
Inspector General found NASA did not verify and validate the basic
requirements for its second-generation space transportation system. GAO
found NASA could not implement key management controls until it defined
the SLI's basic requirements. In October 2002, in response to the GAO
report, NASA indefinitely postponed the System Readiness Review for
SLI.
Comet Nucleus Tour (COUNTOUR):
Program/Project description: NASA's CONTOUR project was intended to
visit at least two comets to provide the first detailed look at the
differences between primitive building blocks of the solar system and
answer questions about how comets act and evolve; Reported problems: A
May 31, 2003 report released by a NASA Mishap Investigation Board (MIB)
found that inadequate systems engineering processes and an inadequate
review function were root causes of the December 2002 loss of the $159
million NASA COUNTOUR spacecraft. Further, based on the findings
related to the CONTOUR project, the MIB board recommended that NASA
establish clear standards for conducting and documenting engineering
work and associated peer and independent reviews and reevaluate its
oversight and review requirements.
Prometheus 1:
Program/Project description: The Prometheus 1 project is part of NASA's
Prometheus Nuclear Systems and Technology project to develop nuclear
power technologies capable of providing power and propulsion for a new
generation of missions. It is being designed to use nuclear power and
electric propulsion technologies to explore the outer reaches of the
solar system. The Jupiter Icy Moons Orbiter mission was to be a 4-to 6-
year study of three of Jupiter's moons;
Reported problems: In February 2005, GAO issued a report on NASA's
Prometheus I project which found that the agency faced challenges
preparing preliminary requirements and cost estimates and that the
critical technologies supporting the project would require extensive
advancement before they could satisfy requirements. According to
Prometheus 1 project management, the approved funding profile for the
project was inadequate to support the planned mission, a 2015 launch to
Jupiter's Icy Moons. GAO recommended that NASA identify the level of
resources the agency is committing to the project and direct project
officials to develop requirements based on this resource constraint.
NASA has since deferred the Jupiter Icy Moons Orbiter mission, citing
concerns over cost and technical complexity.
Source: GAO and NASA.
[End of table]
These problems, along with others, highlighted the need for the agency
to reevaluate its approach to cost and schedule estimating, risk
assessments, technology development, project reviews, and systems
engineering. In 1998, NASA adopted a new program and project management
policy, which was revised in 2002. The policy provided significant
flexibility, including allowing tailoring and projects to opt out of
requirements at the discretion of the project manager. In March 2005--
following a series of internal and external assessments of NASA that
showed that the agency faced significant problems with project
management--NASA again revised its program and project management
policy, which includes policy requirements for the development of
flight systems and ground support projects.[Footnote 6] According to
NASA officials, further changes to the policy are anticipated in light
of new agency leadership.[Footnote 7]
The March 2005 policy document, NASA Procedural Requirement (NPR)
7120.5C, differs from previous versions of the policy in that it
delineates requirements based on four NASA investment areas: Basic and
Applied Research, Advanced Technology Development, Flight Systems and
Ground Support, and Institutional Infrastructure. The policy also
reinstitutes a life cycle phased approach to product development and
institutes a project categorization scheme, based on project cost and
priority, which denotes the oversight authorities[Footnote 8] and the
level of detail that is needed to support project planning documents.
See figure 2 below.
Figure 2: Project Categorization Scheme and Oversight Authorities from
NPR 7120.5C:
[See PDF for image]
[End of figure]
NASA has also attempted to address some of its cost-estimating
weaknesses by instituting a cost analysis data requirement (CADRe) and
establishing thresholds for the use of earned value management
(EVM).[Footnote 9] Another major change to the policy is the
establishment of the Independent Technical Authority (ITA). According
to the policy, the purpose of ITA is to establish sound technical
requirements and decisions for safe and reliable system operations
separate from the project management and reporting chain. Finally,
agency officials told us that while the requirements in previous
versions of the policy were easy to tailor, projects now must document
compliance with the requirements in a "compliance matrix" and must
request and have approved any deviations and/or waivers to
requirements.
Although NPR 7120.5C contains some mandatory requirements with regard
to systems engineering, according to agency officials, NASA has never
had an agencywide systems engineering policy to inform the development
of flight systems and ground support projects. Since 1995, in the
absence of a policy on systems engineering, project managers and
systems engineers have relied on information contained in NASA's
Systems Engineering Handbook to guide their systems engineering
approach on projects.[Footnote 10] Project managers and systems
engineers, however, are not required to follow the handbook.
Recognizing the need for a more structured and rigorous approach to
systems engineering agencywide, NASA's Office of the Chief Engineer is
currently leading the effort to develop a systems engineering policy.
The policy is in draft form.
Knowledge-Based Acquisition Framework:
Over the last several years, we have undertaken a body of work on how
leading developers in industry and government use a knowledge-based
approach to deliver high quality products on time and within
budget.[Footnote 11] A knowledge-based approach to product development
efforts enables developers to be reasonably certain, at critical
junctures or "knowledge points" in the acquisition life cycle, that
their products are more likely to meet established cost, schedule, and
performance baselines and, therefore provides them with information
needed to make sound investment decisions. See figure 3 for a depiction
of a knowledge-based acquisition life cycle.
Figure 3: Knowledge-Based Acquisition Life Cycle:
[See PDF for image]
[End of figure]
* Knowledge point 1 (KP1): Resources and needs match. Knowledge point 1
occurs when a sound business case is made for the product--that is, a
match is made between the customer's requirements and the product
developer's available resources in terms of knowledge, time, workforce,
and money. To determine available resources, successful developers rely
on current and valid information from predecessor projects, new
technologies that have demonstrated a high level of maturity, system
engineering data, and experienced people. Successful developers also
communicate extensively with customers to match their wants and needs
with available resources and with the developers ability to manufacture
an appropriate product.
* Knowledge point 2 (KP2): Product design is stable. Knowledge point 2
occurs when a developer determines that a product's design is stable--
that is, it will meet customer requirements and cost and schedule
targets. A best practice is to achieve design stability at the
product's critical design review (CDR), usually held midway through
development. Completion of at least 90 percent of engineering drawings
at the CDR provides tangible evidence to decision makers that the
design is stable.
* Knowledge point 3 (KP3): Production processes are mature. This level
of knowledge is achieved when it has been demonstrated that the product
can be manufactured within cost, schedule, and quality targets. A best
practice is to ensure that all key manufacturing processes are in
statistical control--that is, they are repeatable, sustainable, and
capable of consistently producing parts within the product's quality
tolerances and standards--at the start of production. It is important
that the product's reliability be demonstrated before production
begins, as investments can increase significantly if defective parts
need to be repaired or reworked.
If the knowledge attained at each juncture does not confirm the
business case on which the initial investment was originally justified,
the project should not go forward and additional resources should not
be committed. Product development efforts that do not follow a
knowledge-based approach can be frequently characterized by poor cost,
schedule, and performance outcomes.
NASA's Revised Policy Does Not Fully Support a Knowledge-Based Approach
to Acquisitions:
NASA's revised acquisition policy for developing flight and ground
support systems incorporates some of the elements of a knowledge-based
acquisition approach. However, it lacks specific key criteria and
decision reviews necessary to fully support such an approach. NASA's
policy defines a phased life cycle approach and requires a major
decision review to move from project formulation to implementation. The
policy requirements for this review address many of the key elements
necessary to match needs to resources, such as requirements to
establish project baselines. However, the policy does not require that
projects demonstrate technologies at high levels of maturity before
launching a project and investing a large amount of resources. In
addition, NASA's policy does not require any further major decision
reviews following the formulation phase of the project. Major decision
reviews in the implementation phase based on specific evaluation
criteria at the final design and fabrication, assembly, and testing
milestones--critical decision points in any product development--could
help NASA ensure that sufficient knowledge has been gained to warrant
moving forward in the development process.
NASA's Policy Establishes a Phased Life Cycle:
The life cycle for all NASA projects is divided into two major phases-
-formulation and implementation. Because flight systems and ground
support projects are particularly complex and have long life cycles,
NASA has further divided the formulation and implementation phases for
these projects to allow managers to assess management and engineering
progress (see fig. 4).
Flight systems and ground support projects must successfully complete
two major decision reviews: a Preliminary-Non Advocate Review (Pre-NAR)
between Phases A and B and a Non-Advocate Review (NAR) between Phases B
and C. At these reviews, the Governing Program Management Committee
(GPMC) evaluates the cost, schedule, safety, and technical content of
the project to ensure that the project is meeting commitments specified
in key management documents.[Footnote 12] Following each of these
reviews, the GPMC recommends to the appropriate decision
authority[Footnote 13] whether the project should be authorized to
proceed.[Footnote 14]
Figure 4: Life Cycle Phases for NASA's Flight Systems and Ground
Support Projects:
[See PDF for image]
[End of figure]
As with KP1 in a knowledge-based acquisition life cycle, the NAR marks
the official project approval point in the life cycle. After approval
at the NAR, projects are included as part of implementation reviews of
their parent program. These implementation reviews are conducted
biennially and are not tied to any design or production milestones.
After entering the implementation phase, the GPMC is notified if cost
or schedule performance exceeds the baselines established at the NAR by
10 percent or when key performance criteria are not met. Exceeding the
NAR baselines can result in the GPMC conducting a termination review to
determine whether or not to continue the project.
NASA's Policy Lacks Requirements for Maturing Technologies:
To help ensure project requirements do not outstrip resources, leading
developers obtain the right knowledge about a new product's technology,
design, and production at the right time. NASA's policy emphasizes many
elements needed at the NAR (or KP1) to match needs to resources, such
as validating requirements, developing realistic cost and schedule
estimates and human capital plans, and establishing a preliminary
design. The policy does not, however, require projects to demonstrate
technologies at high levels of maturity before launching a project.
Table 2 compares KP1 criteria and NASA's policy criteria.
Table 2: KP1 Criteria Compared to NASA Policy Criteria:
KP1 Criteria (resources and needs match): High level of technology
maturity; NPR 7120.5C NAR requirements: A technology plan describing
the technology needed for the project, including plans for technology
maturation, validation, and insertion, and quantifiable milestones,
decision gates, and resources required.
KP1 Criteria (resources and needs match): Informed requirements;
NPR 7120.5C NAR requirements: A set of requirements that are well
formed (clear and unambiguous), complete (agrees with customer and
stakeholder expectations), and consistent (conflict free); and each
requirement is verifiable and traceable to higher level requirements.
KP1 Criteria (resources and needs match): Realistic cost and schedule
estimates;
NPR 7120.5C NAR requirements: A cost analysis data requirement
(CADRe)[A] documents the programmatic, technical, and life cycle cost
information. The CADRe includes a life cycle cost estimate, integrated
master project schedule, and a work breakdown structure (WBS).[B]
KP1 Criteria (resources and needs match): Human capital in place;
NPR 7120.5C NAR requirements: Plans to staff the project with personnel
with the appropriate skills, abilities, and experience, and provide
integrated team training to successfully execute the project. The
project manager is required to negotiate to obtain the needed resources
identified in the plan.
KP1 Criteria (resources and needs match): Preliminary system design;
NPR 7120.5C NAR requirements: A preliminary system design is required
and is reviewed at the NAR.
KP1 Criteria (resources and needs match): Conduct a decision review
before project launch;
NPR 7120.5C NAR requirements: The GPMC determines the project's
readiness to proceed to implementation and recommends a course of
action to the appropriate decision authority.
Source: NASA and GAO.
[A] A CADRe is required for Category I and Category II flight system
and ground support projects.
[B] A WBS is a product-oriented division of hardware, software,
services, and project unique tasks that organizes and defines the
product to be developed and serves as the basis for estimating both
cost and schedule.
[End of table]
While NASA requires projects to develop plans that describe how
technologies will be matured and to provide alternative development
strategies for technologies that do not mature as expected, it does not
establish a minimum threshold for technology maturity. Consequently,
projects can enter the implementation phase with immature technologies
and embark on a risky path of having to build technology, design, and
production knowledge concurrently. Our best practices work has shown
that maturing technologies during the preliminary design phase and
before entering product development is a key element of matching needs
to resources and that there is a direct relationship between the
maturity of technologies and the risk of cost and schedule
growth.[Footnote 15] Allowing technology development to carry over into
the product development phase increases the risk that significant
problems will be discovered late in development. Addressing such
problems at this stage may require extensive retrofitting and redesign
as well as retesting, which can jeopardize performance and result in
more time and money to fix. This approach also makes it more difficult
for projects to demonstrate the same level of design stability in later
phases of implementation since technology and design activities will be
done concurrently.
Technology readiness levels (TRL)--a concept developed by NASA--can be
used to gauge the maturity of individual technologies. The higher the
TRL, the more the technology has been proven and the lower the risk of
performance problems and cost and schedule overruns (see fig. 5). TRL
7--demonstrating a technology as a fully integrated prototype in an
operational environment--is the level of maturity preferred by product
developers to minimize risks when entering product development.
Figure 5: Technology Maturity Levels for Product Development:
[See PDF for image]
[End of figure]
Successful developers will not commit to undertaking product
development, and more importantly investing resources, unless they have
high confidence that they have achieved a match between what the
customer wants and what the project can deliver. Technologies that are
not mature continue to be developed in an environment that is focused
solely on technology development. Once matured, these technologies can
be transitioned to projects. This puts developers in a better position
to succeed because they can focus on integrating the technologies and
testing and proving the product design.
NASA's Policy Does Not Support Informed Design and Production
Decisions:
Our prior work has shown that successful developers establish specific
criteria to ensure that requisite knowledge has been attained before
moving forward from final design into the latter stages of
development.[Footnote 16] Before making significant increases in
investments to fabricate, assemble, and test the product, these
developers conduct a decision review to determine if the design is
stable and performs as expected and the project is ready to enter the
next phase. To make this determination and reach KP2, successful
developers use specific, knowledge-based standards and criteria. (See
app. II for more information on the specific knowledge-based standards
and criteria successful developers use to judge readiness to proceed
beyond detailed design activities.)
Successful developers also demand proof that manufacturing processes
are in control and product reliability goals are attained before
committing to production. To determine whether they have achieved this
knowledge point, KP3, successful developers conduct another mandatory
decision review in which they use specific, knowledge-based standards
and criteria to determine if the product can be produced within cost,
schedule, and quality targets. (See app. II for more information on
specific knowledge-based standards and criteria used by successful
developers to judge readiness to enter into production.)
Contrary to these best practices, the NAR at the end of the preliminary
design phase--KP1--is the last major decision review in the NASA
project life cycle. (See fig. 6.) Although NPR 7120.5C requires that
projects document in the project plan a continuum of technical and
management reviews, such as a PDR and CDR, it does not require any
specific reviews.[Footnote 17] In addition, NASA's March 2005 policy
does not require a NAR-type decision review to ensure a project has
obtained the knowledge needed to proceed beyond the final design phase
into the fabrication, assembly, and test phase, which serves as both
the demonstration and production phase of the NASA life cycle.[Footnote
18] According to NASA officials, projects conduct a CDR at the end of
the final design phase to ensure adequate information is available
about product design and producibility before entering the fabrication,
assembly, and test phase. The CDR, however, is a technical review--not
a major decision review like the NAR. Furthermore, the policy does not
establish criteria as to what constitutes successful completion of a
CDR.
Figure 6: Comparison of NASA's Life Cycle with a Knowledge-Based
Acquisition Life Cycle:
[See PDF for image]
[End of figure]
NASA's policy also does not require a major decision review before
beginning manufacturing. (See fig. 6 above.) Therefore, the transition
from final design to fabrication, assembly, and test often marks a de
facto production decision. According to NASA officials, the agency
rarely enters a formal production phase due to the small quantities of
space systems that they build. However, due to the high cost of failure
associated with NASA projects and the costs and risks involved in
repairing a system in-orbit, a major decision review at production that
assesses product reliability is essential even for these limited
production systems. In addition, although NASA's production quantities
are typically low, in some instances NASA does produce larger
quantities of a system or subsystem, such as the external tanks for the
Space Shuttle. Furthermore, NASA's plans indicate that the agency may
be increasing production for elements of their future systems. For
example, NASA's Exploration Systems Architecture Study indicates that
NASA plans to build several new crew exploration vehicles, with
disposable elements, such as the lunar lander, solid rocket boosters,
and space shuttle main engines that will require higher numbers of
production runs.
Rather than establish specific criteria by which all projects are
judged, NASA's policy requires that projects manage to baselines and
plans established in key management documents and approved at the
project NAR. The baselines and plans serve as their primary tools for
measuring project progress and as the primary basis for judgment at
project reviews. While the plans may include some information that
addresses knowledge-based criteria for design and production, the
instructions for preparing them leaves the establishment of thresholds
and success criteria to the discretion of the project manager. For
example, NASA policy requires that projects include, as part of the
project plan, a Verification and Validation Sub Plan that describes the
project's approach to verifying and validating hardware and software as
part of the project plan. The policy, however, includes no instruction
as to what constitutes a sufficient approach to testing. In other
words, there are no requirements concerning the fidelity of test
articles or the realism of the test environment. Similarly, NASA's
policy requires that projects include, as part of the project plan, a
Systems Engineering Sub Plan that describes the project's approach to
systems engineering and the technical standards that are applicable,
including metrics that verify the processes. The policy, however, does
not identify the types of metrics appropriate to verify the process or
establish any threshold criteria.
The absence of major decision reviews, along with specific criteria in
the fabrication, assembly, and test phase, could result in concurrent
design and manufacturing activities, a practice our past work has found
increases risk in acquisition programs.[Footnote 19] Furthermore,
lacking a major decision review to ensure that projects have gained the
appropriate levels of knowledge at KP2 and KP3, NASA decision makers
cannot be provided a high level of certainty that the project will meet
cost, schedule, and performance requirements and have no assurance that
the provisions of the key management documents required by NPR 7120.5C
are being executed after the NAR.
Lack of NASA-Wide Project Management Criteria May Result in Investment
Decisions Based on Inconsistent Information:
NASA centers have varying approaches to implementing project management
policies and systems engineering guidance for flight systems and ground
support projects. Some centers use criteria at key decision points that
are similar to the criteria required to ensure a knowledge-based
approach is followed, while others lack such criteria. As a result,
each center reports a different level and type of knowledge about a
project at key decision points. Centers also rely on project managers
and systems engineers to employ good project management and systems
engineering practices. However, given the loss of experienced project
managers and the decline of in-house systems engineering and technical
capabilities agencywide, that reliance could be problematic. These
situations make it difficult for NASA decision makers to evaluate
center projects on a common foundation of knowledge and make sound
investment decisions and tradeoffs based on those evaluations. A
standardized, knowledge-based approach would prepare NASA to face
competing budgetary priorities and make difficult decisions regarding
the investment in and termination of projects.
Individual Centers Tailor Implementation of Agencywide Project
Management Policies and Systems Engineering Guidance:
While NASA centers are given discretion about how they implement
agencywide policies, they are expected to have procedures and
guidelines in place for implementing those policies. Some centers have
developed center-specific policies and criteria for implementing NASA's
project management policies and system engineering guidance, while
others have not. Centers also rely on project managers and systems
engineers to implement the requirements of NPR 7120.5C and to use
NASA's Systems Engineering Handbook as guidance for good systems
engineering practices.[Footnote 20] Some NASA centers have also
developed criteria in their policies that are similar to the criteria
used to ensure a knowledge-based approach is followed; other centers
lack such criteria.
Because of their varying policies and criteria, each center requires a
different level of knowledge at the same point in a project's
development cycle. For example, GSFC requires its projects to mature
technologies to TRL 6 by the preliminary design review--before entering
the implementation phase. On the other hand, JPL, MSFC, and JSC
policies do not require projects to mature technology to a particular
level before entering implementation, leaving the determination of
needed technology maturity up to the project manager.
Requirements for assessing design maturity also vary across the
centers. While most of the center policies require a CDR to enter
"Phase D--Fabrication, Assembly, and Test"--the criteria used to assess
projects at this point vary. For example, both GSFC and MSFC require
projects to have completed a percentage of design drawings at this
review. MSFC requires that 90 percent of design drawings to be complete
by CDR, which is consistent with best practices. While GSFC establishes
a minimum threshold of drawings to be complete by CDR--greater than 80
percent--neither JSC nor JPL establish a minimum percentage drawing
requirement. Instead, JSC requires the design be complete and drawings
ready to begin production, and JPL requires that the design be mature
and provide confidence in the integrity of the flight system design.
Almost none of the center policies include a requirement to assess the
maturity of production processes. According to NASA officials, due to
the low quantities of systems generally produced by the agency, most of
the center policies do not require a review before beginning
manufacturing. Only JSC requires a Production Readiness Review to
ensure that production plans, facilities, and personnel are in place
and ready to begin production. However, the criteria do not specify
quantifiable thresholds to measure production readiness at the review.
JPL, MSFC, and GSFC do not have policies that outline requirements for
such a review.
In addition to individual policy requirements, centers may rely on
project managers and systems engineers to employ good project
management and systems engineering practices. For example, an
experienced project manager at JPL told us that although JPL policy
does not require a particular TRL at PDR to enter implementation, he
required that all technologies for his project be around a TRL 6 in
order to separate technology development from systems development.
Reliance on project managers to implement good practices, however,
could be problematic given the diminishing number of experienced
project managers available to lead projects and the decline of in-house
systems engineering and technical capabilities agencywide caused by
increasing retirements and outsourcing.
Informed Investment Decisions Require Consistent Knowledge:
In a knowledge-based process, the achievement of each successive
knowledge point builds on the preceding one, giving decision makers the
information they need, when they need it, to make decisions about
whether to invest significant additional funds to move forward with
product development. Our work has shown that successful product
development efforts are marked by adherence to a disciplined process
that establishes and uses common and consistent criteria for decision
making at these key points. With varying project management and systems
engineering criteria, NASA centers' technical reviews, such as PDR,
provide different levels of knowledge to support NASA's major decision
reviews, such as the NAR, which NASA uses to support its major
investment decisions for flight systems and ground support projects.
In the near future, NASA will need to determine the resources necessary
to develop the systems and supporting technologies to achieve the
President's Vision for Space Exploration and structure its investment
strategy accordingly. Initial implementation of the Vision as explained
in NASA's Exploration Systems Architecture Study calls for completing
the International Space Station, developing a new crew exploration
vehicle, and returning to the moon no later than 2020. NASA estimates
that it will cost approximately $104 billion over the next 13 years to
accomplish these initial goals. These priorities, along with NASA's
other missions, will be competing within NASA for funding. It will
likely be difficult for NASA managers to agree on which projects to
invest in and which projects to terminate. The NASA Administrator has
acknowledged that NASA faces difficult choices about its missions in
the future--for example, between human space flight, science, and
aeronautics missions.
Using consistent criteria to evaluate all NASA projects would help
ensure that the same level and type of knowledge is available about
individual projects at key decision points. Analogous information about
all flight systems and ground support projects would allow decision
makers to make apples-to-apples comparisons across projects and make
investment decisions, and trade-offs, based upon these comparisons.
Further, policies with consistent criteria can provide inexperienced
project managers and systems engineers with the necessary guidance to
implement good project management and systems engineering practices and
ensure that the right knowledge is available for decision makers.
NASA officials within the Chief Engineer's Office acknowledged the need
for more consistency in criteria across the various NASA centers in
order to successfully achieve NASA's Vision for Space Exploration. Some
NASA centers, however, are resistant to standardized criteria because
they feel it could be overly prescriptive. These centers indicated that
because of the unique nature of the work done at each of the 10 NASA
centers, it would be unrealistic to hold every project to the same
criteria. Nonetheless, our work has shown that the process used by
successful product developers to develop leading-edge technology and
products does not differ based upon the type of product being
developed. Further, these developers adhere to a disciplined process
that establishes and uses common and consistent criteria for decision
making--regardless of the type of product or technology being
developed. Using consistent criteria can allow NASA decision makers to
assess the likely return on competing investment priorities and to
reevaluate alternatives and make investment decisions across projects
to increase the likelihood of attaining the strategic goals of the
agency.
Conclusions:
Several of NASA's major acquisitions have been marked by cost,
schedule, and performance problems. Yet the challenges NASA faces in
the future are likely to far exceed those it has faced in the past. The
complex technical requirements associated with fulfilling the
President's Vision, the fiscal constraints under which NASA will be
required to operate, the diminished number of experienced project
managers and systems engineers, and the potential for increased
production make following a knowledge-based approach for flight systems
and ground support projects all the more critical. While NASA has made
improvements to its policies governing project management, the lack of
major decision reviews beyond the initial project approval gate leaves
decision makers with little knowledge about the progress of the
agency's projects. Further, without a standard set of criteria to
measure projects at crucial phases in the development life cycle, NASA
cannot be assured that its decisions will result in the best possible
return on its investments. Since NASA is currently in the process of
revising its program and project management policies and developing an
agencywide systems engineering policy, we believe this presents a
unique opportunity for the agency to correct some of the problems
identified during our review.
Recommendations for Executive Action:
In order to close the gaps between NASA's current acquisition
environment and best practices on knowledge-based acquisition, we
recommend that NASA take steps to ensure NASA projects follow a
knowledge-based approach for product development. Specifically, we
recommend that the NASA Administrator direct the Office of the Chief
Engineer to take the following two actions:
* In drafting its systems engineering policy, incorporate requirements
in the policy for flight systems and ground support projects to capture
specific product knowledge by key junctures in project development. The
demonstration of this knowledge should be used as exit criteria for
decision making at the following key junctures:
* Before projects are approved to transition from formulation to
implementation, the policy should require that projects demonstrate
that key technologies have reached a high maturity level.
* Before projects are approved to transition from final design to
fabrication, assembly, and test, the policy should require that
projects demonstrate that the design is stable.
* Before projects are approved to transition into production, the
policy should require projects to demonstrate that the design can be
manufactured within cost, schedule, and quality targets.
* Revise NPR 7120.5C to institute additional major decision reviews
following the NAR for flight systems and ground support projects, which
result in recommendations to the appropriate decision authority. These
reviews should be tied to the key junctures during project development
mentioned above in order to increase the likelihood that cost,
schedule, and performance requirements of the project will be met.
Agency Comments and Our Evaluation:
We provided a draft of this report to NASA for review and comment. In
written comments, NASA indicated that it agreed with our
recommendations and outlined specific actions that the agency plans to
take to address them.
The actions that NASA plans to take to address our recommendations are
a positive step toward achieving successful project outcomes and
ensuring that decision makers are appropriately investing the agency's
resources. We are pleased to hear that many knowledge-based practices
identified in our report are currently being practiced agencywide in
the management and development NASA systems. The addition of such
practices to NASA's policies will only strengthen their use agencywide
and ensure that these practices continue to be utilized by less
experienced project managers and systems engineers as the more
experienced workforce retires. The effectiveness of such practices,
however, will be limited if project officials are not held accountable
for demonstrating a high level of knowledge, consistent with the
success criteria that NASA plans to require in its policies, at key
junctures in development. It is critical that project officials not
only have a high level of knowledge about a project at key junctures,
but also that this information is used by decision makers to make
decisions on whether to invest additional resources and allow a project
to proceed through the development life cycle.
NASA's comments are reprinted in appendix III. NASA also provided
technical comments, which we addressed throughout the report as
appropriate.
As agreed with your offices, unless you announce its contents earlier,
we will not distribute this report further until 30 days from its date.
At that time, we will send copies to NASA's Administrator and
interested congressional committees. We will make copies available to
others upon request. In addition, the report will be available at no
charge on the GAO Web site at http://www.gao.gov.
If your or your staff have any questions concerning this report, please
contact me at (202) 512-4841 or lia@gao.gov. Contact points for our
Offices of Congressional Relations and Public Affairs may be found on
the last page of this report. Key contributors to this report are
acknowledged in appendix IV.
Signed by:
Allen Li:
Director:
Acquisition and Sourcing Management:
[End of section]
Appendix I: Scope and Methodology:
To determine the extent to which NASA's current policies support an
acquisition approach consistent with best practices identified in GAO's
work on system acquisitions, we reviewed and analyzed NASA-wide program
and project management policies and systems engineering guidance. Our
review and analysis of the policy focused on requirements for flight
systems and ground support projects. We interviewed NASA Headquarters
officials from the Office of the Chief Engineer who are responsible for
the policy and guidance. We compared NASA's policy on program and
project management with criteria contained in GAO best practices work
on systems acquisition and space system acquisitions. We concentrated
on whether the policy provides a framework for a knowledge-based
process and the criteria necessary to carry out this intent.
To determine how NASA-wide acquisition policies are implemented across
the various NASA centers, we reviewed NASA center-specific program and
project management policies and systems engineering policies and
interviewed officials responsible for implementing those policies.
While we interviewed officials from all centers, we focused on the
centers that manage the majority of NASA's flight systems and ground
support projects--GSFC, JPL, JSC, and MSFC. This approach included site
visits with GSFC, JPL, JSC, and MSFC and teleconferences with the
remaining centers. We compared examples of the centers' implementation
of the policies and specific criteria included in these policies with
our best practices work on systems acquisition.
We performed our work from May 2005 to November 2005 in accordance with
generally accepted government auditing standards.
[End of section]
Appendix II: Activities That Enable the Capture of Design and
Manufacturing Knowledge:
Table 3: Activities to Capture Design Knowledge and Make Decisions:
Knowledge: Design is stable and performs as expected (knowledge point
2);
Decision: Product is ready for initial manufacturing and system
demonstration phase;
Key indicator: 90 percent of engineering drawings completed.
Activities to achieve stable design knowledge;
* Limit design challenge--The initial design challenge is limited to a
product that can be developed and delivered quickly and provide the
user with an improved capability. A time-phased plan is used to develop
improved products--future generations--in increments as technologies
and other resources become available;
* Demonstrate design meets requirements-- The product's design is
demonstrated to meet the user's requirements. For a new product that is
not based on an existing product, prototypes are built and tested. If
the product is a variant of an existing product, companies often used
modeling and simulation or prototypes at the component or subsystem
level to demonstrate the new product's design;
* Complete critical design reviews--Critical design reviews are used to
assess whether a product's design meets requirements and is ready to
start initial manufacturing. Reviews are conducted for the system,
subsystems, and components to assess design maturity and technical
risk;
* Stakeholders agree drawings complete and producible-
-The agreement by stakeholders (engineers, manufacturers, and other
organizations) is used to signify confidence that the design will work
and the product can be built;
* Executive level review to begin initial manufacturing--Corporate
stakeholders meet and review relevant product knowledge, including
design stability, to determine whether a product is ready to initiate
manufacturing of production representative prototypes used during
system demonstrations. The decision is tied to the capture of
knowledge.
[End of table]
Source: GAO.
Table 4: Activities to Capture Manufacturing Knowledge and Make
Decisions:
Knowledge: Product can be produced within cost, schedule, and quality
targets (knowledge point 3);
Decision: Product is ready for production and will be reliable;
Key indicator: Critical processes in statistical control and product
reliability demonstrated.
Activities to Achieve Manufacturing Knowledge;
* Identify key system characteristics and critical manufacturing
processes--Key product characteristics and critical manufacturing
processes are identified. Because there can be thousands of
manufacturing processes required to build a product, companies focus on
the critical processes--those that build parts that influence the
product's key characteristics such as performance, service life, or
manufacturability;
* Determine processes in control and capable--Statistical process
control is used to determine if the processes are consistently
producing parts. Once control is established, an assessment is made to
measure the process's ability to build a part within specification
limits as well as how close the part is to that specification. A
process is considered capable when it has a defect rate of less than 1
out of every 15,152 parts produced;
* Conduct failure modes and effects analysis--Bottom- up analysis is
done to identify potential failures for product reliability. It begins
at the lowest level of the product design and continues to each higher
tier of the product until the entire product has been analyzed. It
allows early design changes to correct potential problems before
fabricating hardware;
* Set reliability growth plan and goals--A product's reliability is its
ability to perform over an expected period of time without failure,
degradation, or need of repair. A growth plan is developed to mature
the product's reliability over time through reliability growth testing
so that it has been demonstrated by the time production begins;
* Conduct reliability growth testing--Reliability growth is the result
of an iterative design, build, test, analyze, and fix process for a
product's design with the aim of improving the product's reliability
over time. Design flaws are uncovered and the design of the product is
matured;
* Conduct executive level review to begin production--Corporate
stakeholders meet and review relevant product knowledge, including
manufacturing and reliability knowledge, to determine whether a product
is ready to begin production. The decision is tied to the capture of
knowledge.
Source: GAO.
[End of table]
[End of section]
Appendix III: Comments from the National Aeronautics and Space
Administration:
National Aeronautics and Space Administration:
Office of the Administrator:
Washington, DC 20546-0001:
December 15, 2005:
Mr. Allen Li:
Director, Acquisition and Sourcing Management:
United States Government Accountability Office:
Washington, DC 20548:
Dear Mr. Li:
NASA appreciates the opportunity to comment on your draft report
(General Accountability Office (GAO) 06-218) entitled "Implementing a
Knowledge-Based Acquisition Framework Could Lead to Better Investment
Decisions and Project Outcomes."
Per the National Aeronautics and Space Act (Pub. L. No 85-568), NASA's
charter is "lo provide for research into problems of flight within and
outside the Earth's atmosphere, and for other purposes." NASA is a
research and development organization, and its projects face greater
uncertainty further into the life cycle than traditional high-volume
acquisitions. Most NASA missions fabricate one-of-a-kind spacecraft
using a proto-flight or very small quantity approach.
Projects are primarily governed by two NASA Procedural Requirements
(NPRs). NPR 7120.5C, NASA Program and Project Management Processes and
Requirements, establishes the management system which governs the
formulation, approval, implementation, and evaluation of NASA programs
and projects. Requirements for performing, supporting, and evaluating a
systems approach applied to all elements and levels of a system over
the complete project life cycle are established for the implementing
organizations in NPR 7123, NASA Systems Engineering Processes and
Requirements. NPR 7123 is in the final approval cycle.
Many of the best practices you recommend are currently being practiced
Agencywide but may not be apparent in our current policy. Issuance of
NPR 7123 will establish a consistent minimum set of project reviews
that are already practiced within NASA (i.e., Systems Requirement
Review (SRR), Preliminary Design Review (PDR), Critical Design Review
(CDR)). While NPR 7120.5C does not depict decision reviews at key
junctures beyond the Non-Advocate Review (NAR), NASA gains continuous
knowledge throughout the implementation phase. Monthly or quarterly
status reviews are typically held utilizing the hierarchy of governing
program management councils. This process provides for decision points
throughout the life cycle.
The following paragraphs provide the GAO recommendations and the NASA
response addressing these recommendations:
First GAO Recommendation:
In order to close the gaps between NASA's current acquisition
environment and best practices on knowledge-based acquisition, we
recommend that NASA take steps to ensure NASA projects follow a
knowledge-based approach for product development. Specifically, we
recommend that the NASA Administrator direct the Office of the Chief
Engineer to take the following two actions:
* In drafting its systems engineering policy, incorporate requirements
in the policy for flight systems and ground support projects to capture
specific product knowledge by key junctures in project development. The
demonstration of this knowledge should be used as exit criteria for
decision making at the following key junctures:
- Before projects are approved to transition from formulation to
implementation, the policy should require that projects demonstrate
that key technologies have reached a high maturity level.
- Before projects are approved to transition from final design to
fabrication, assembly, and test, the policy should require that
projects demonstrate that the design is stable.
- Before projects are approved to transition into production, the
policy should require projects to demonstrate that the design can be
manufactured within cost, schedule and quality targets.
NASA agrees, with the following comments:
NASA takes a risk-based approach to technology readiness. NASA will
define the applicable technology readiness level for each phase of the
life cycle and include these definitions in NPR 7123. NASA will also
strengthen the current wording in NPR 7120.5C to ensure that a design
solution incorporating sufficiently mature technology will be available
at PDR This does not preclude the program/project manager from pursuing
a parallel design approach utilizing a higher performance but less
mature solution as long as the mature technology remains available and
appropriate risk mitigation processes are employed. However, exceptions
should be made in the cases where the primary mission objective is to
mature and demonstrate technologies to reduce risk on future missions
(i.e., New Millennium Program).
NASA implements a review process that includes peer reviews both within
NASA and at the supplier. Per NPR 7123, CDRs are one of a minimum set
of flight systems and ground support technical reviews held for the
purpose of demonstrating "that the maturity of the design is
appropriate to support proceeding with full scale fabrication,
assembly, integration, and test, and that the technical effort is on
track to complete the flight and ground system development and mission
operations in order to meet mission performance requirements within the
identified cost and schedule constraints." NASA will establish
requirements for the CDR success criteria in NPR 7123 to ensure that
the design is stable and that the system can be fabricated within cost,
schedule, and performance specifications. In addition, NASA will
establish requirements for the success criteria in NPR 7123 for other
required technical reviews in the life cycle.
Very few NASA projects enter into a traditional production process.
Most NASA missions fabricate one-of-a-kind spacecraft. For this class
of mission, the CDR will thoroughly review the readiness to proceed to
fabrication of the flight system. CDR success criteria will be reviewed
and updated to include any additional fabrication information required
to enable a knowledge-based decision (see above paragraph). In the
event that systems or elements are intended for large scale production,
including major components of unique systems, NASA will conduct a
Production Readiness Review (PRR) that will meet required success
criteria. This review and its success criteria will be added to NPR
7123.
Second GAO Recommendation:
Revise NPR 7120.5C to institute additional major decision reviews
following the NAR for flight systems and ground support projects, which
result in recommendations to the appropriate decision authority. These
reviews should be tied to the key junctures during project development
mentioned above in order to increase the likelihood that cost,
schedule, and performance requirements of the project will be met.
NASA agrees, with the following comments:
The CDR results will provide the information needed for the decision
authority. NASA will revise NPR 7120.5C to require the results of the
CDR to be out-briefed to the appropriate decision authority in a timely
manner for a decision to proceed. An additional decision point will be
added, as applicable, to out-brief the results of a PRR in the case of
large scale production lots to the appropriate decision authority for a
decision to proceed.
Thank you for the opportunity to comment on this draft report.
Sincerely,
Signed by:
Shana Dale:
Deputy Administrator:
[End of section]
Appendix IV: GAO Contact and Staff Acknowledgments:
GAO Contact:
Allen Li (202) 512-4841:
Staff Acknowledgments:
In addition to the individual named above, James L. Morrison, Assistant
Director; Valerie Colaiaco, Tom Gordon, Alison Heafitz, Shelby S.
Oakley, Ron Schwenn, Karen Sloan, and John S. Warren, Jr., made key
contributions to this report.
[End of section]
Related GAO Products:
Space Reports:
Space Acquisitions: Stronger Development Practices and Investment
Planning Need to Address Continuing Problems. GAO-05-891T. Washington,
D.C.: July 12, 2005.
Defense Acquisitions: Incentives and Pressures That Drive Problems
Affecting Satellite and Related Acquisitions. GAO-05-570R. Washington,
D.C.: June 23, 2005.
Defense Acquisitions: Space-Based Radar Effort Needs Additional
Knowledge before Starting Development. GAO-04-759. Washington, D.C.:
July 23, 2004.
Defense Acquisitions: Risks Posed by DOD's New Space Systems
Acquisition Policy. GAO-04-379R. Washington, D.C.: January 29, 2004.
Space Acquisitions: Committing Prematurely to the Transformational
Satellite Program Elevates Risks for Poor Cost, Schedule, and
Performance Outcomes. GAO-04-71R. Washington, D.C.: December 4, 2003.
Defense Acquisitions: Improvements Needed in Space Systems Acquisition
Policy to Optimize Growing Investment in Space. GAO-04-253T.
Washington, D.C.: November 18, 2003.
Defense Acquisitions: Despite Restructuring, SBIRS High Program Remains
at Risk of Cost and Schedule Overruns. GAO-04-48. Washington, D.C.:
October 31, 2003.
Defense Acquisitions: Improvements Needed in Space Systems Acquisition
Management Policy. GAO-03-1073. Washington, D.C.: September 15, 2003.
Military Space Operations: Common Problems and Their Effects on
Satellite and Related Acquisitions. GAO-03-825R. Washington, D.C.: June
2, 2003.
Military Space Operations: Planning, Funding, and Acquisition
Challenges Facing Efforts to Strengthen Space Control. GAO-02-738.
Washington, D.C.: September 23, 2002.
Polar-Orbiting Environmental Satellites: Status, Plans, and Future Data
Management Challenges. GAO-02-684T. Washington, D.C.: July 24, 2002.
Defense Acquisitions: Space-Based Infrared System-Low at Risk of
Missing Initial Deployment Date. GAO-01-6. Washington, D.C.: February
28, 2001.
Best Practices Reports:
Defense Acquisitions: Assessments of Selected Major Weapon Programs.
GAO-05-301. Washington, D.C.: March 31, 2005.
Defense Acquisitions: Stronger Management Practices Are Needed to
Improve DOD's Software-Intensive Weapon Acquisitions. GAO-04-393.
Washington, D.C.: March 1, 2004.
Defense Acquisitions: Assessments of Selected Major Weapon Programs.
GAO-04-248. Washington, D.C.: March 31, 2004.
Defense Acquisitions: DOD's Revised Policy Emphasizes Best Practices,
but More Controls Are Needed. GAO-04-53. Washington, D.C.: November 10,
2003.
Defense Acquisitions: Assessments of Selected Major Weapon Programs.
GAO-03-476. Washington, D.C.: May 15, 2003.
Best Practices: Setting Requirements Differently Could Reduce Weapon
Systems' Total Ownership Costs. GAO-03-57. Washington, D.C.: February
11, 2003.
Best Practices: Capturing Design and Manufacturing Knowledge Early
Improves Acquisition Outcomes. GAO-02-701. Washington, D.C.: July 15,
2002.
Defense Acquisitions: DOD Faces Challenges in Implementing Best
Practices. GAO-02-469T. Washington, D.C.: February 27, 2002.
Best Practices: Better Matching of Needs and Resources Will Lead to
Better Weapon System Outcomes. GAO-01-288. Washington, D.C.: March 8,
2001.
Best Practices: A More Constructive Test Approach Is Key to Better
Weapon System Outcomes. GAO/NSIAD-00-199. Washington, D.C.: July 31,
2000.
Defense Acquisition: Employing Best Practices Can Shape Better Weapon
System Decisions. GAO/T-NSIAD-00-137. Washington, D.C.: April 26, 2000.
Best Practices: DOD Training Can Do More to Help Weapon System Program
Implement Best Practices. GAO/NSIAD-99-206. Washington, D.C.: August
16, 1999.
Best Practices: Better Management of Technology Development Can Improve
Weapon System Outcomes. GAO/NSIAD-99-162. Washington, D.C.: July 30,
1999.
Defense Acquisitions: Best Commercial Practices Can Improve Program
Outcomes. GAO/T-NSIAD-99-116. Washington, D.C.: March 17, 1999.
Defense Acquisition: Improved Program Outcomes Are Possible. GAO/T-
NSIAD-98-123. Washington, D.C.: March 18, 1998.
Best Practices: Successful Application to Weapon Acquisition Requires
Changes in DOD's Environment. GAO/NSIAD-98-56. Washington, D.C.:
February 24, 1998.
Major Acquisitions: Significant Changes Underway in DOD's Earned Value
Management Process. GAO/NSIAD-97-108. Washington, D.C.: May 5, 1997.
Best Practices: Commercial Quality Assurance Practices Offer
Improvements for DOD. GAO/NSIAD-96-162. Washington, D.C.: August 26,
1996.
FOOTNOTES
[1] The Vision includes a return to the moon that is intended
ultimately to enable future exploration of Mars and other destinations.
To accomplish this, NASA initially plans to (1) complete its work on
the International Space Station by 2010, fulfilling its commitment to
15 international partner countries; (2) begin developing a new manned
exploration vehicle to replace the space shuttle; and (3) return to the
moon as early as 2015 and no later than 2020 in preparation for future,
more ambitious missions.
[2] NASA Policy Directive (NPD) 7120.4C describes the management system
by which NASA formulates, approves, implements, and evaluates all
programs and projects. NASA Procedural Requirement (NPR) 7120.5C
establishes the management system requirements for implementing NPD
7120.4C. NPR 7120.5C governs the formulation, approval, implementation,
and evaluation of all agency programs and projects. For purposes of
this report, we refer to NPR 7120.5C as NASA's program and project
management policy.
[3] NASA consists of NASA headquarters, nine centers, the Jet
Propulsion Laboratory (operated under contract to NASA by the
California Institute of Technology), and several ancillary
installations and offices in the United States and abroad. The
implementation of NASA programs and aeronautical and space/earth
science research occurs primarily at the centers.
[4] GAO, NASA: Better Mechanisms Needed for Sharing Lessons Learned,
GAO-02-195 (Washington, D.C.: Jan. 30, 2002).
[5] The approach was intended to help NASA reduce costs, become more
efficient, and increase scientific results by conducting more and
smaller missions in less time.
[6] NPR 7120.5C contains specific requirements for both programs and
projects. The policy defines a program as a strategic investment by a
Mission Directorate or Mission Support Office that has defined goals,
objectives, architecture, funding level, and a management structure
that supports one or more projects. A project is defined as a specific
investment identified in a Program Plan having defined goals,
objectives, requirements, life cycle cost, a beginning, and an end. For
purposes of this report, we have focused our analysis of the policy as
it relates to the traditional development approach to flight systems
and ground support projects.
[7] According to the NASA Administrator, evolutionary acquisition
processes will not be practiced at NASA. Therefore, NASA officials
indicated that evolutionary acquisition for flight systems and ground
support projects will likely be deleted in the next version of NPR
7120.5.
[8] One oversight authority is called a Project Management Committee
(PMC). The PMC is defined as one of the hierarchy of forums, composed
of senior management, that assesses project planning and
implementation, and provides oversight and direction as appropriate.
These are established at the agency, mission directorate, center, and
lower levels. The Independent Program Assessment Office (IPAO) is the
NASA organization responsible for scheduling, organizing, and
conducting the independent reviews at the Preliminary Non-Advocate
Review (Pre-NAR) and the Non-Advocate Review (NAR) for programs and
projects reporting to the agency PMC. The Systems Management Office
(SMO) is the center organization responsible for independent review and
assessment of projects at the Pre-NAR and the NAR, whose findings are
reported to the center PMC.
[9] As defined by NPR 7120.5C, a CADRe is a formal document to
understand the cost and cost risk of space flight projects. The policy
defines EVM as a tool for measuring and assessing project performance
through the integration of technical scope with schedule and cost
objectives during the execution of the project. As defined in NPR
7120.5C, all projects over $20 million must apply EVM.
[10] National Aeronautics and Space Administration, NASA Systems
Engineering Handbook, SP-6105, (Washington, D.C.: June 1995).
[11] Our best practice reviews are identified in the "Related GAO
Products" section at the end of this report.
[12] The key management documents for flight system and ground support
projects are the Project Formulation Authorization Document (FAD), or
equivalent, and the Project Plan. The FAD authorizes a Project Manager
to initiate the formulation phase of a new project. Because a new
project represents a major commitment of resources, the FAD requires
approval of the project decision authority before the project enters
Phase A of the life cycle. The Project Plan is an agreement between the
Mission Directorate Associate Administrator, the Center Director,
Program Manager, and Project Manager, as applicable. It defines, at a
high level, the scope of the project, the implementation approach, the
environment within which the project operates, and the commitments of
the project. The Project Plan also establishes the cost, schedule, and
technical baselines for the implementation phase and is used by the
GPMC in the review process to determine if the project is fulfilling
its agreements. A preliminary Project Plan is due at the Pre-NAR, and
an updated or final version is due at the NAR.
[13] The decision authority for Category I projects is the Deputy
Administrator; for Category II projects, the Mission Directorate
Associate Administrator; and for Category III projects, the Center
Director (of the executing center).
[14] A positive recommendation may be unconditional, or conditional on
the Project Manager completing assigned action items. A negative
recommendation could result in the decision authority either directing
the Project Manager to address the deficiencies, or in the
authorization of a termination review.
[15] GAO's 2005 assessment of 54 systems within the Department of
Defense showed that development costs for programs that started
development with mature technology increased by an average of 9 percent
over the first full estimate, whereas the development costs for the
programs that started development with immature technologies increased
an average of 41 percent over the first full estimate. Likewise,
program acquisition unit costs for the programs with mature technology
increased by less than 1 percent, whereas the programs that started
development with immature technologies experienced an average program
acquisition unit cost increase of nearly 21 percent over the first full
estimate. Programs with mature technology experienced an average
schedule delay of 7 months--a 9 percent increase--whereas the schedule
for the programs that started development with immature technology
increased an average of 13 months--a 13 percent increase. GAO, Defense
Acquisitions: Assessments of Selected Major Weapon Programs, GAO-05-301
(Washington, D.C.: Mar. 31, 2005).
[16] In NASA this is "Phase D--Fabrication, Assembly and Test."
[17] Projects may also be required to complete specific technical and
management reviews per individual center policy.
[18] According to NASA officials, some NASA centers require projects to
provide status updates to the center PMC after the NAR, for example
through monthly or quarterly status reports.
[19] GAO, Better Management of Technology Development Can Improve
Weapon System Outcomes, GAO/NSIAD-99-162 (Washington, D.C., July 30,
1999) and GAO, Best Practices: Capturing Design and Manufacturing
Knowledge Early Improves Acquisition Outcomes, GAO-02-701 (Washington,
D.C., July 15, 2002).
[20] NASA's Systems Engineering Handbook provides systems engineers and
project managers with a generic description of NASA systems engineering
and a common language and perspective of the systems engineering
process. See National Aeronautics and Space Administration, NASA
Systems Engineering Handbook, SP-6105. (Washington, D.C.: June 1995).
GAO's Mission:
The Government Accountability Office, the investigative arm of
Congress, exists to support Congress in meeting its constitutional
responsibilities and to help improve the performance and accountability
of the federal government for the American people. GAO examines the use
of public funds; evaluates federal programs and policies; and provides
analyses, recommendations, and other assistance to help Congress make
informed oversight, policy, and funding decisions. GAO's commitment to
good government is reflected in its core values of accountability,
integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through the Internet. GAO's Web site (www.gao.gov ) contains
abstracts and full-text files of current reports and testimony and an
expanding archive of older products. The Web site features a search
engine to help you locate documents using key words and phrases. You
can print these documents in their entirety, including charts and other
graphics.
Each day, GAO issues a list of newly released reports, testimony, and
correspondence. GAO posts this list, known as "Today's Reports," on its
Web site daily. The list contains links to the full-text document
files. To have GAO e-mail this list to you every afternoon, go to
www.gao.gov and select "Subscribe to e-mail alerts" under the "Order
GAO Products" heading.
Order by Mail or Phone:
The first copy of each printed report is free. Additional copies are $2
each. A check or money order should be made out to the Superintendent
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or
more copies mailed to a single address are discounted 25 percent.
Orders should be sent to:
U.S. Government Accountability Office
441 G Street NW, Room LM
Washington, D.C. 20548:
To order by Phone:
Voice: (202) 512-6000:
TDD: (202) 512-2537:
Fax: (202) 512-6061:
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Web site: www.gao.gov/fraudnet/fraudnet.htm
E-mail: fraudnet@gao.gov
Automated answering system: (800) 424-5454 or (202) 512-7470:
Public Affairs:
Jeff Nelligan, managing director,
NelliganJ@gao.gov
(202) 512-4800
U.S. Government Accountability Office,
441 G Street NW, Room 7149
Washington, D.C. 20548:
